STAR: Sparse Transformer-based Action Recognition
STAR：基于稀疏变换器的动作识别

The Figure 2 shows the comparison between our data format and the format used by previous works. In the data format adopted by previous works, longer videos are cut off to the predefined length and shorter videos are padded with repeated frames. Furthermore, the frames with a single person are all zero-padded to match the fixed number of persons. The upper data format from Figure2 illustrates the NTU RGB+D data format used by previous works. In each fixed-length video $V^{(i)}$, $P_{1}^{(i)}$ and $P_{2}^{(i)}$ represent two persons. In NTU RGB+D 120 dataset, only 26 out of 120 actions are mutual actions, which means that the second person’s skeleton is just zeros ($P_{2}^{(i)}=0$ in Figure 2) in most data samples. In contrast to the previous data format, the proposed format maintains the original length of each video clip. Additionally, when a video clip contains two persons, we concatenate them along the frame dimension. Instead of keeping an individual dimension for a batch of video clips, we further concatenate the video clips in a batch along the frame dimension, and the auxiliary vector stores the batch indices to indicate to which video clip a frame belongs, as shown in the bottom data format of Figure 2. Moreover, given the new dimensions ($N$, $V$, $C$) as shown in Figure 2, where $N$ is the total number of frames after concatenating the video clips along the temporal dimension and $V$ is the number of skeleton’s joints, we regard dimension $N$ as the logical batch size for spatial attention and dimension $V$ as the logical batch size for temporal attention.
图 2 显示了我们的数据格式与先前工作中使用的格式的比较。在先前工作中采用的数据格式中，较长的视频被截断到预定义的长度，较短的视频则通过重复帧进行填充。此外，单人帧全部用零填充以匹配固定的人数数量。图 2 上方的数据格式展示了先前工作中使用的 NTU RGB+D 数据格式。在每个固定长度的视频 $V^{(i)}$ 、 $P_{1}^{(i)}$ 和 $P_{2}^{(i)}$ 代表两个人。在 NTU RGB+D 120 数据集中，只有 120 个动作中的 26 个是相互动作，这意味着在大多数数据样本中，第二个人的骨骼只是零（图 2 中的 $P_{2}^{(i)}=0$ ）。与先前数据格式相比，所提出的格式保持了每个视频片段的原始长度。此外，当视频片段包含两个人时，我们在帧维度上拼接它们。我们不是为一批视频片段保留一个单独的维度，而是进一步在帧维度上拼接一批视频片段，辅助向量存储批索引以指示帧属于哪个视频片段，如图 2 底部的数据格式所示。 此外，根据图 2 所示的新维度（ $N$ ， $V$ ， $C$ ），其中 $N$ 是沿时间维度拼接视频片段后的总帧数， $V$ 是骨骼关节的数量，我们将维度 $N$ 视为空间注意力的逻辑批量大小，维度 $V$ 视为时间注意力的逻辑批量大小。

With the new data format introduced in the previous section, we propose a novel linear multi-head attention tailored for this data format. We call it a Segmented Linear Attention. As stated in the previous sections, Transformers are originally designed for sequential data. In the human skeleton sequence, each joint’s dynamic movement across the frames can be regarded as a time series. Therefore, the 3D coordinates, i.e., $(x,y,z)$, of every joint can be processed individually through the trajectory along the time dimension, and the application of attention extracts the interaction among time steps represented by frames.
随着上一节引入的新数据格式，我们提出了一种针对此数据格式的创新线性多头注意力机制。我们称之为分段线性注意力。如前几节所述，Transformer 最初是为序列数据设计的。在人体骨骼序列中，每个关节在帧之间的动态运动可以被视为一个时间序列。因此，每个关节的 3D 坐标，即 $(x,y,z)$ ，可以通过沿着时间维度的轨迹单独处理，应用注意力机制提取由帧表示的时间步之间的交互。

Linear Attention. Standard dot product attention mechanism (Vaswani et al. 2017) (Equation 4) with the global receptive field of $N$ inputs are prohibitively slow due to the quadratic time and memory complexity $\mathcal{O}(N^{2})$. The quadratic complexity also makes Transformers hard to train and limits the context. Recent research toward the linearized attention mechanism derives the approximation of the Softmax-based attention. The most appealing ones are linear Transformers (Katharopoulos et al. 2020; Choromanski et al. 2021; Shen et al. 2021) based on kernel functions approximating the Softmax. The linearized Transformers can improve inference speeds up to three orders of magnitude without much loss in predictive performance (Tay et al. 2020). Given the projected embeddings $Q$, $K$, and $V$ for input tensors of queries, keys, and values, respectively, according to the observation from the accumulated value $V_{i}^{\prime}\in\mathbf{R}^{d}$ for the query $Q_{i}\in\mathbf{R}^{d}$ in position $i$, $d$ is the channel dimension, the linearized attention can be transformed from Equation 4 to Equation 5, the computational complexity is reduced to $\mathcal{O}(Nd)$, when $d$ is much smaller than $N$, the computational complexity is approaching linear $\mathcal{O}(N)$:
线性注意力。标准点积注意力机制（Vaswani 等人，2017）（公式 4）由于输入的全局感受野的二次时间和内存复杂度，导致速度过慢。二次复杂度也使得 Transformer 难以训练并限制了上下文。针对线性化注意力机制的最新研究推导出基于 Softmax 的注意力近似。其中最吸引人的是线性 Transformer（Katharopoulos 等人，2020；Choromanski 等人，2021；Shen 等人，2021），它们基于近似 Softmax 的核函数。线性化 Transformer 可以提高推理速度高达三个数量级，同时预测性能损失不大（Tay 等人，2020）。考虑到查询、键和值输入张量的投影嵌入 $Q$ 、 $K$ 和 $V$ ，分别根据查询 $Q_{i}\in\mathbf{R}^{d}$ 在位置 $i$ 的累积值 $V_{i}^{\prime}\in\mathbf{R}^{d}$ 的观察， $d$ 是通道维度，线性化注意力可以从方程 4 转换为方程 5，当 $d$ 远小于 $N$ 时，计算复杂度降低到 $\mathcal{O}(Nd)$ ，接近线性 $\mathcal{O}(N)$ ：

where $\phi(\cdot)$ is the kernel function. In work of (Katharopoulos et al. 2020), kernel function is simply simulated with ELU, $\phi(x)=elu(x)+1$; while (Choromanski et al. 2021) introduces the Fast Attention via Orthogonal Random Feature (FAVOR) maps as the kernel function, $\phi(x)=\frac{c}{\sqrt{M}}f(Wx+b)^{T}$, where $c>0$ is a constant, and $W\in\mathbf{R}^{M\times d}$ is a Gaussian random feature matrix, and $M$ is the dimensionality of this matrix that controls the number of random features.
; $\phi(\cdot)$ 是核函数。在 (Katharopoulos 等人 2020 年) 的研究中，核函数简单地通过 ELU 进行模拟， $\phi(x)=elu(x)+1$ ；而 (Choromanski 等人 2021 年) 引入了通过正交随机特征 (FAVOR) 映射的快速注意力作为核函数， $\phi(x)=\frac{c}{\sqrt{M}}f(Wx+b)^{T}$ ，其中 $c>0$ 是一个常数， $W\in\mathbf{R}^{M\times d}$ 是高斯随机特征矩阵， $M$ 是该矩阵的维度，它控制随机特征的数量。

Segmented Linear Attention. Since we concatenate the various length of video clips within a single batch along the time dimension, directly applying linear attention will cause the cross clip attention, leading to irrelevant information taken into account from one video clip to another, as shown in Figure 4. Therefore, we consider the frames of a video clip arranged as a segment, and then we design the segmented linear attention by reformulating Equation 5 with segment index. Therefore, for each $V_{i}$ in segment $\mathcal{S}_{m}$, we summarize
分段线性注意力。由于我们在时间维度上将不同长度的视频片段拼接在一个批次中，直接应用线性注意力将导致跨片段注意力，导致从一个视频片段到另一个视频片段考虑了不相关信息，如图 4 所示。因此，我们将视频片段的帧排列为一个段，然后通过使用段索引重新表述方程 5 来设计分段线性注意力。因此，对于每个分段 $\mathcal{S}_{m}$ 中的 $V_{i}$ ，我们总结

where $\mathcal{S}_{m}$ is the $m$-th segment, and the reduction operation $\sum_{j\in\mathcal{S}_{m}}f(x)$ can be easily implemented through the indexation to segments; and with help of the gathering and scattering operations (Fey 2021a), the segmented linear attention maintains the highly-paralleled computation. Figure 3 illustrates the comparison of different attention operations.
; $\mathcal{S}_{m}$ 是第 $m$ 个段，缩减操作 $\sum_{j\in\mathcal{S}_{m}}f(x)$ 可以通过段索引轻松实现；借助聚集和散射操作（Fey 2021a），分段线性注意力保持高度并行计算。图 3 说明了不同注意力操作的比较。

In previous works (Yan, Xiong, and Lin 2018; Shi et al. 2019b), before connecting to the final fully-connected layer for classification, summarizing the video clip embedding along the temporal dimension is implemented by global average pooling. Alternatively, we utilize a probabilistic approach through context-aware attention, which is extended from the work of (Bai et al. 2019), to enhance this step’s robustness, as demonstrated in Figure 6. Denote an input tensor embedding of video clip $S_{m}$ as $V\in R^{F\times N\times D}$, for $F$ is the number of frames in video clip $S_{m}$, $N$ is the number of joints of skeleton, and each joint possessing $D$ features, where $v_{i}\in R^{N\times D}$ is the embedding of frame $i$ of $V$. First, a global context $c\in R^{N\times D}$ is computed, which is a simple average of embedding of frames followed by a nonlinear transformation: $c=tanh\left(\frac{1}{F}W\sum^{F}_{i\in S_{m}}v_{i}\right)$, where $W\in R^{D\times D}$ is a learnable weight matrix. The context $c$ provides the global structural and feature information of the video clip that is adaptive to the similarity between frames in video clip $S_{m}$, via learning the weight matrix. Based on $c$, we can compute one attention weight for each frame. For frame $i$, to make its attention an aware of the global context, we take the inner product between $c$ and its embedding. The intuition is that, frames similar to the global context should receive higher attention weights. A sigmoid function $\sigma(x)=\frac{1}{1+exp(-x)}$ is applied to the result to ensure the attention weights is in the range $(0,1)$. Finally, the video clip embedding $v^{\prime}\in R^{N\times D}$ is the weighted sum of video clip embeddings, $v^{\prime}=\sum^{F}_{i\in S_{m}}a_{i}v_{i}$. The following equations summarize the proposed context-aware attentive mechanism:
在先前的工作（Yan, Xiong, 和 Lin 2018；Shi 等人 2019b）中，在连接到最终的完全连接层进行分类之前，通过全局平均池化实现了沿时间维度的视频片段嵌入的总结。或者，我们利用一种基于上下文感知注意力的概率方法，该方法扩展自（Bai 等人 2019）的工作，以提高这一步骤的鲁棒性，如图 6 所示。将视频片段 $S_{m}$ 的输入张量嵌入表示为 $V\in R^{F\times N\times D}$ ，对于 $F$ 是视频片段 $S_{m}$ 中的帧数， $N$ 是骨骼关节数，每个关节具有 $D$ 个特征，其中 $v_{i}\in R^{N\times D}$ 是 $V$ 中帧 $i$ 的嵌入。首先，计算一个全局上下文 $c\in R^{N\times D}$ ，这是一个简单平均的帧嵌入，随后进行非线性变换： $c=tanh\left(\frac{1}{F}W\sum^{F}_{i\in S_{m}}v_{i}\right)$ ，其中 $W\in R^{D\times D}$ 是一个可学习的权重矩阵。上下文 $c$ 通过学习权重矩阵，提供了视频片段的全局结构和特征信息，该信息适应于视频片段 $S_{m}$ 中帧之间的相似性。基于 $c$ ，我们可以为每个帧计算一个注意力权重。 对于帧 $i$ ，为了使其关注具有全局上下文意识，我们计算 $c$ 与其嵌入的内积。直觉是，与全局上下文相似的帧应获得更高的关注权重。对结果应用 sigmoid 函数 $\sigma(x)=\frac{1}{1+exp(-x)}$ ，以确保关注权重在 $(0,1)$ 范围内。最后，视频片段嵌入 $v^{\prime}\in R^{N\times D}$ 是视频片段嵌入 $v^{\prime}=\sum^{F}_{i\in S_{m}}a_{i}v_{i}$ 的加权求和。以下方程式总结了所提出的上下文感知注意力机制：

As the attention mechanism is order-agnostic to the permutation in the input sequence (Vaswani et al. 2017; Tsai et al. 2019) and treats the input as an unordered bag of element. Therefore, an extra positional embedding is necessary to maintain the data order, i.e., time-series data are in the inherently sequential ordering. Then these positional embedding are participating the evaluation of the attention weight and value between token $i$ and $j$ in the input sequence.
由于注意力机制对输入序列的排列（Vaswani 等人，2017；Tsai 等人，2019）是无序的，并将输入视为无序的元素集合。因此，需要额外的位置嵌入来维持数据顺序，即时间序列数据具有固有的顺序性。然后这些位置嵌入参与评估输入序列中标记 $i$ 和 $j$ 之间的注意力权重和值。

Segmented Sequential Positional Encoding However, as we arrange the variable-length video clips into a batch along the temporal dimension, it is not feasible to directly apply positional encoding to the whole batch. Therefore, we introduce the segmented positional encoding where each video clip gets its positional encoding according to batch indices. An example of such encoding is shown in Figure 7.
分段顺序位置编码然而，当我们沿着时间维度将可变长度的视频剪辑批量排列时，无法直接对整个批量应用位置编码。因此，我们引入了分段位置编码，其中每个视频剪辑根据批量索引获得其位置编码。这种编码的示例如图 7 所示。

Structural Positional Encoding. we also attempt to apply the structural positional encoding, e.g., tree-based positional encoding (Shiv and Quirk 2019; Omote, Tamura, and Ninomiya 2019), to the spatial dimension, i.e., the tree topology of skeleton. Experiments show that the current approach which we used cannot improve our model’s performance significantly. Hence, to reduce our model’s complexity, we decide not to apply the structural positional encoding for this work and leave it for future research.
结构位置编码。我们还尝试将结构位置编码应用于空间维度，即骨骼的树拓扑结构，例如基于树的编码（Shiv 和 Quirk 2019；Omote，Tamura 和 Ninomiya 2019）。实验表明，我们目前使用的方法并不能显著提高我们模型的表现。因此，为了降低我们模型的复杂性，我们决定不将结构位置编码应用于这项工作，并将其留待未来的研究。

This dataset contains 56,880 video clips involving 60 human action classes. The samples are performed by 40 volunteers and captured by three Microsoft Kinect v2 cameras (Shahroudy et al. 2016). It contains four modalities, including RGB videos, depth sequences, infrared frames, and 3D skeleton data. Our experiments are only conducted with the 3D skeleton data. The length of the action samples vary from 32 frames to 300 frames. In each frame, there are at most 2 subjects and each subject contains 25 3D joint coordinates. The dataset follows two evaluation criteria, which are Cross-Subject and Cross-View. In the Cross-View evaluation (X-View), there are 37,920 training samples captured from camera 2 and 3 and 18,960 test samples captured from camera 1. In the Cross-Subject evaluation (X-Sub), there are 40,320 training samples from 20 subjects and 26,560 test samples from the rest. We follow the original two benchmarks and report the Top-1 accuracy as well as the profiling metrics.
此数据集包含 56,880 个视频片段，涉及 60 个人动作类别。样本由 40 名志愿者执行并由三台 Microsoft Kinect v2 相机（Shahroudy 等人，2016 年）捕捉。它包含四种模态，包括 RGB 视频、深度序列、红外帧和 3D 骨骼数据。我们的实验仅使用 3D 骨骼数据。动作样本的长度从 32 帧到 300 帧不等。在每一帧中，最多有 2 个主体，每个主体包含 25 个 3D 关节坐标。该数据集遵循两个评估标准，即跨主体和跨视图。在跨视图评估（X-View）中，有 37,920 个训练样本来自相机 2 和 3，以及 18,960 个测试样本来自相机 1。在跨主体评估（X-Sub）中，有 40,320 个训练样本来自 20 个主体，以及 26,560 个测试样本来自其他主体。我们遵循原始的两个基准，并报告 Top-1 准确率以及配置文件指标。

The dataset (Liu et al. 2019) extends from NTU RGB+D 60 and is currently the largest dataset with 3D joint annotations. It contains 57,600 new skeleton sequences representing 60 new actions, a total of 114,480 videos involving 120 classes of 106 subjects captured from 32 different camera setups. The dataset follows two criteria, which are Cross-Subject and Cross-Setup. In the Cross-Subject evaluation, similar to the previous dataset, splits subjects in half to training and testing dataset. In the Cross-Setup evaluation, the samples are divided by the 32 camera setup IDs, where the even setup IDs are for training and the odd setup IDs for testing. Similar to the previous dataset, there is no preprocessing to set the uniform video length for all the samples. We follow the two criteria and report the Top-1 accuracy and the profiling metrics.
数据集（刘等人，2019 年）从 NTU RGB+D 60 扩展而来，是目前拥有 3D 联合标注的最大数据集。它包含 57,600 个新的骨架序列，代表 60 个新的动作，共计 114,480 个视频，涉及 106 个受试者的 120 个类别，由 32 个不同的摄像机设置捕获。该数据集遵循两个标准，即跨受试者和跨设置。在跨受试者评估中，与之前的数据集类似，将受试者分为训练集和测试集。在跨设置评估中，样本根据 32 个摄像机设置 ID 进行划分，偶数设置 ID 用于训练，奇数设置 ID 用于测试。与之前的数据集类似，没有预处理来设置所有样本的统一视频长度。我们遵循这两个标准，并报告 Top-1 准确率和性能指标。

Unlike GCN-based models, where the length of all the samples and the number of subjects need to be fixed (e.g. 300 frames and 2 subjects), our model can process varying length of input samples and of the number of subjects. So no further preprocessing with padding is done on the samples.
与基于 GCN 的模型不同，其中所有样本的长度和主题数量需要固定（例如，300 帧和 2 个主题），我们的模型可以处理不同长度的输入样本和主题数量。因此，对样本不再进行填充等进一步预处理。

We first evaluate the effectiveness of our Transformer encoder based model compared to ST-TR and ST-GCN models. Each model’s accuracy is evaluated with the NTU RGB+D 60 and 120 testing datasets. As shown in the Table 1, our model outperforms ST-GCN in both cross-view (cross-set) and cross-subject benchmarks of the two dataset. Our model achieves 3.6 $\sim$ 7.7 percent lower accuracy compared to ST-TR, which heavily relies on convolution-based key components inherited from ST-GCN and utilizes them in both spatial and temporal pipelines. Our model yields modest performance compared to the state-of-the-art models in NTU RGB+D 60 and 120 when trained from scratch. The Figure 11 shows that there exists a performance gap between the training and testing. Transformer architectures’ lack of inductive biases, especially translation equivariance and locality that are essential to convolution networks, could result in weak generalization. In NLP, Transformers are usually pretrained on a large corpus of text and fine-tuned on a smaller task-specific dataset to boost the performance. We would like to conduct extensive experiments on pre-training and fine-tuning our model on a larger dataset in the future to improve the accuracy comparable to those of the state-of-the-art models. For our future study, we want to address effective generalization methods for Transformer models, which resolves overfitting issues and improve the overall performance.
首先评估我们的 Transformer 编码器模型相对于 ST-TR 和 ST-GCN 模型的有效性。每个模型的准确性使用 NTU RGB+D 60 和 120 测试数据集进行评估。如表 1 所示，我们的模型在两个数据集的交叉视图（交叉集）和交叉主题基准测试中均优于 ST-GCN。与严重依赖从 ST-GCN 继承的基于卷积的关键组件并在空间和时间管道中利用它们的 ST-TR 相比，我们的模型实现了 3.6 $\sim$ 7.7%的更低准确性。当从头开始训练时，我们的模型在 NTU RGB+D 60 和 120 的现有模型中表现出适度的性能。图 11 显示，训练和测试之间存在性能差距。Transformer 架构缺乏归纳偏差，特别是对卷积网络至关重要的平移等变性和局部性，可能导致泛化能力较弱。在 NLP 中，Transformers 通常在大型文本语料库上预训练，并在较小的特定任务数据集上进行微调以提升性能。 我们希望在未来的某个时间点，在更大的数据集上对我们模型的预训练和微调进行广泛的实验，以提高其准确性，使其与最先进的模型相当。对于我们的未来研究，我们希望解决 Transformer 模型的有效泛化方法，这可以解决过拟合问题并提高整体性能。

In this section, we evaluate the efficiency of the different models. As shown in Table 2, our model (STAR) is significantly efficient in terms of model size, the number of multiply-accumulate operations (GMACs) and the latency. Each metric for the different models is evaluated by running inference with sample dataset. Our model is fed with the original skeleton sequence of varying length. The other two models are fed with fix-sized skeleton sequence padded to 300 frames and 2 persons. We use the official profiler of PyTorch (v1.8.1) (Paszke et al. 2019), and Flops-Profiler of DeepSpeed (Rasley et al. 2020) to measure the benchmarks. The results are summarized with the following metrics:
本节中，我们评估了不同模型的效率。如表 2 所示，我们的模型（STAR）在模型大小、乘累加操作数（GMACs）和延迟方面效率显著。通过使用样本数据集进行推理，评估了不同模型的每个指标。我们的模型使用的是不同长度的原始骨架序列。其他两个模型使用的是填充到 300 帧和 2 个人的固定大小骨架序列。我们使用 PyTorch（v1.8.1）的官方分析器（Paszke 等人，2019 年）和 DeepSpeed 的 Flops-Profiler（Rasley 等人，2020 年）来测量基准。结果以下列指标总结：

MACs. The number of Multiply-Accumulate (MAC) operations is used to determine the efficiency of deep learning models. Each MAC operation is counted as two floating point operations. With the same hardware configuration, more efficient models require fewer MACs than other models to fulfill the same task. As shown in Table 2, both of our model with different channel sizes execute only $\frac{1}{3}\sim\frac{1}{17}$ amount of GMACs (i.e., Giga MACs) compared to ST-GCN and ST-TR models, respectively.
MACs. 乘累加（MAC）操作的次数用于确定深度学习模型的效率。每个 MAC 操作计为两个浮点操作。在相同的硬件配置下，更高效的模型比其他模型完成相同任务所需的 MACs 更少。如表 2 所示，我们具有不同通道大小的模型与 ST-GCN 和 ST-TR 模型相比，分别仅执行 $\frac{1}{3}\sim\frac{1}{17}$ 数量的 GMACs（即十亿 MACs）。

Model size. Model size is another metric to measure the efficiency of a machine learning model. Given the same task, smaller model delivering the same or very close performance is preferable. Smaller model is not only beneficial for the higher speedup and less memory accesses but also gives better energy consumption, especially for embedded systems and edge devices with scarce computational resources and small storage volume. The column of the number of parameters in Table 2 depicts the size of the models, these parameters are trainable weights in the model. Among all the model, STAR possesses the smallest model size, 0.42M and 1.26M for STAT-64 and STAR-128, respectively.
模型大小。模型大小是衡量机器学习模型效率的另一个指标。在相同任务下，较小的模型能够提供相同或非常接近的性能，这是更可取的。较小的模型不仅有利于更高的加速和更少的内存访问，还能提供更好的能耗，尤其是在计算资源和存储空间有限的嵌入式系统和边缘设备中。表 2 中参数数量的列描述了模型的大小，这些参数是模型中的可训练权重。在所有模型中，STAR 具有最小的模型大小，分别为 STAT-64 的 0.42M 和 STAR-128 的 1.26M。

Breakdown analysis. The breakdown analysis is used to identify potential bottlenecks within different models (STAR-64, ST-TR, and ST-GCN). Table 3 provides the detailed profiling results for the top-3 computation modules that are dominant in each models. According to Figure 8, 9 and 10, the convolution operations cost significant number of MAC operations and lead to computation bound. ST-GCN and ST-TR mainly consist of the convolution operations followed by batch normalization, which requires relatively large computational resources. Our Transformer model is based on sparse and linear attention mechanisms. It only produces relatively small attention weights from sparse attention; and performs low-rank matrix multiplication for linear attention ($\mathcal{O}(n)$). This replaces huge dynamic weights of attention coefficients from the standard attention mechanism, which has a quadratic time and space complexity ($\mathcal{O}(n^{2})$).
分解分析。分解分析用于识别不同模型（STAR-64、ST-TR 和 ST-GCN）中的潜在瓶颈。表 3 提供了每个模型中占主导地位的 Top-3 计算模块的详细分析结果。根据图 8、9 和 10，卷积操作消耗了大量的 MAC 操作，导致计算受限。ST-GCN 和 ST-TR 主要由卷积操作组成，随后是批量归一化，这需要相对较大的计算资源。我们的 Transformer 模型基于稀疏和线性注意力机制。它仅从稀疏注意力中产生相对较小的注意力权重；并对线性注意力执行低秩矩阵乘法（ $\mathcal{O}(n)$ ）。这取代了标准注意力机制中巨大的动态权重，该机制具有二次时间和空间复杂度（ $\mathcal{O}(n^{2})$ ）。