Mixture-of-Experts with Expert Choice Routing
专家选择路由的专家混合模型

MoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-$k$ experts for each token [31, 21, 10]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.
虽然 MoE 在计算上可能比密集模型更具优势，但必须使用一种路由策略将每个令牌分配给最合适的专家。传统的 MoE 模型采用令牌选择路由，它为每个令牌独立选择前 $k$ 个专家 [31, 21, 10]。我们认为这种策略有一些缺陷，导致训练效果不佳。

Load Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.
负载不平衡：令牌选择路由经常导致不同专家之间的负载不平衡。即一些专家可能会被大多数令牌训练，而其他专家则未被充分利用。专家可能会因为未充分利用的专家中的大量模型容量被浪费而变得不够专门化。另一方面，由于过度使用的专家在每一步只能处理一定数量的令牌以避免内存不足，一些令牌将无法被处理。负载不平衡还会影响步骤延迟，从而影响推理时间，因为步骤延迟可以由负载最重的专家决定。先前的方法在负载平衡上添加了辅助损失以缓解问题。然而，这种辅助损失不能保证负载平衡，尤其是在训练的重要早期阶段。实际上，我们在实验中观察到，在令牌选择路由中，某些专家的超容量比可以达到 20%–40%，这表明分配给这些专家的令牌中有相当一部分会被丢弃。

Under Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity. Ideally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.
在专业化不足的情况下：每个 MoE 层使用一个门控网络来学习 token 到专家的亲和性。理想情况下，学习到的门控网络应生成这样的亲和性，使得相似或相关的 token 被路由到相同的专家。一个次优的策略可能会导致冗余的专家和/或不够专业化的专家。专业化不足可能是由于设置了过大的辅助损失，以偏向于更平衡负载但效率较低的路由。在确保负载平衡和专业化的辅助损失之间找到合适的平衡对于 token 选择路由来说是具有挑战性的。

Same Compute for Every Token: Finally, in a token-choice strategy each token receives exactly $k$ experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.
每个 Token 相同的计算：最终，在 token 选择策略中，每个 token 正好接收 $k$ 个专家，因此占用相同数量的计算资源。我们假设这既不是必需也不是期望的。相反，MoE 模型应该根据输入的复杂性灵活分配其计算资源。基于上述观察，接下来我们将描述一种简单但有效的方法，它基于专家选择产生负载均衡的分配。

Different from conventional routing, an expert choice method independently selects top-$k$ tokens for each expert, where $k$ is a fixed expert capacity (i.e. the number of tokens each expert can take). Despite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.
与传统路由不同，专家选择方法独立为每个专家选择前 $k$ 个 tokens，其中 $k$ 是固定的专家容量（即每个专家可以接受的 tokens 数量）。尽管其简单，但设计上专家选择实现了完美的负载平衡。它还允许更灵活的模型计算分配，因为 tokens 可以由可变数量的专家接收。

In our experiments, we set $k$ as
在我们的实验中，我们将 $k$ 设置为

where $n$ is the total number of tokens in the input batch (such as batch size $\times$ sequence length), $c$ is the capacity factor, and $e$ is the number of experts. The capacity factor $c$ denotes on average how many experts are utilized by a token. Given input token representations $X\in\mathbb{R}^{n\times d}$ where $d$ is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices $I$, $G$ and $P$. The matrix $I$ is an index matrix where $I[i,j]$ specifies $j$-th selected token of the $i$-th expert. The gating matrix $G\in\mathbb{R}^{e\times k}$ denotes the weight of expert for the selected token, and $P\in\mathbb{R}^{e\times k\times n}$ refers to an one-hot version of $I$ that will be used to gather tokens for each expert. These matrices are computed using a gating function,
其中 $n$ 是输入批次中的 token 总数（例如批次大小 $\times$ 序列长度）， $c$ 是容量因子， $e$ 是专家数量。容量因子 $c$ 表示平均每个 token 使用多少个专家。给定输入 token 表征 $X\in\mathbb{R}^{n\times d}$ ，其中 $d$ 是模型隐藏维度，我们的方法产生一个 token 到专家的分配，由三个输出矩阵 $I$ ， $G$ 和 $P$ 表示。矩阵 $I$ 是一个索引矩阵，其中 $I[i,j]$ 指定 $j$ -th 选定的专家的 $i$ -th token。门控矩阵 $G\in\mathbb{R}^{e\times k}$ 表示为选定 token 的专家权重， $P\in\mathbb{R}^{e\times k\times n}$ 指的是一个 $I$ 的 one-hot 版本，将用于为每个专家收集 tokens。计算这些矩阵时使用的门控函数，

where $S$ denotes the token-to-expert affinity scores, $W_{g}\in\mathbb{R}^{d\times e}$ denotes the expert embeddings, and $TopK()$ selects the k largest entries for each row of $S^{\top}$.
其中 $S$ 表示 token 到专家的亲和度分数， $W_{g}\in\mathbb{R}^{d\times e}$ 表示专家嵌入， $TopK()$ 选择 $S^{\top}$ 中每行的 k 个最大条目。

Similar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by $X_{\text{in}}\in\mathbb{R}^{e\times k\times d}$, is produced using the permutation matrix $P$. Here $X_{\text{in}}[i]\in\mathbb{R}^{k\times d}$ denotes the input of the $i$-th expert. Similarly, let $W_{1}$ and $W_{2}$ denote the parameters of gated FFN in which $W_{1}[i]$ and $W_{2}[i]\in\mathbb{R}^{d\times d^{\prime}}$ denote the parameter matrices of the $i$-th expert. We compute the output of each expert $X_{e}[i]$ as follows,
类似于 Switch Transformer [10] 和 GShard [21]，我们在稠密前馈网络（FFN）层应用了专家混合和门控功能，因为这是基于 Transformer 的网络中计算量最大的一部分。标记为 $X_{\text{in}}\in\mathbb{R}^{e\times k\times d}$ 的门控 FFN 的输入是使用排列矩阵 $P$ 产生的。这里 $X_{\text{in}}[i]\in\mathbb{R}^{k\times d}$ 表示第 $i$ 位专家的输入。同样，设 $W_{1}$ 和 $W_{2}$ 表示门控 FFN 的参数，其中 $W_{1}[i]$ 和 $W_{2}[i]\in\mathbb{R}^{d\times d^{\prime}}$ 表示第 $i$ 位专家的参数矩阵。我们计算每个专家的输出 $X_{e}[i]$ 如下，

We omit the bias terms here for brevity. The finally output of the gated FFN layer $X_{\text{out}}\in\mathbb{R}^{n\times d}$ can be obtained given $X_{e}$, the permutation and gating matrices $P$ and $G$,
为简洁起见，我们在这里省略了偏置项。门控 FFN 层的最终输出 $X_{\text{out}}\in\mathbb{R}^{n\times d}$ 可以通过给定 $X_{e}$ 、排列和门控矩阵 $P$ 和 $G$ 获得。

Both $X_{e}$ and $X_{\text{out}}$ can be efficiently computed using Einstein summation (einsum) operations.
$X_{e}$ 和 $X_{\text{out}}$ 都可以通过爱因斯坦求和（einsum）操作有效计算。

We also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and fine-tuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.
我们还考虑通过限制每个 token 的最大专家数量来调整我们的专家选择路由。我们感兴趣的是添加这个约束是否可以改善预训练和微调结果。更重要的是，它有助于分析每个 token 使用可变数量专家对模型性能的影响程度。

Let $A\in\mathbb{R}^{e\times n}$ be a positive matrix where $A[i,j]$ represents whether the i-th expert selects j-th token. We solve the following entropy-regularized linear programming problem
设 $A\in\mathbb{R}^{e\times n}$ 为一个正矩阵，其中 $A[i,j]$ 表示第 i 个专家是否选择第 j 个 token。我们求解以下熵正则化线性规划问题

where $<S^{\top},A>$ denotes the inner product, $H(A)$ is the sum of element-wise entropy¹¹1$H(A)=\sum_{ij}-A[i,j]\log A[i,j]$, and $b>0$ is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints. We use Dykstra’s algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.²²2We use $\lambda=0.001$ and a maximum of 100 iterations. After $A$ is computed, the routing indices $I$ is selected using $TopK(A,k)$ instead.
其中 $<S^{\top},A>$ 表示内积， $H(A)$ 是逐元素熵 ¹ 的和，而 $b>0$ 是对每个 token 的选择的整数上界。添加一个小熵项可以给出一个接近整数的解，同时启用可以在 TPU 上运行的快速迭代求解器。具体来说，解空间是分别满足线性约束之一的三个凸集的交集。我们使用 Dykstra 算法 [9] 交替地将中间解投影到一个凸集上。在 $A$ 计算后，路由索引 $I$ 改为使用 $TopK(A,k)$ 选择。

At the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [21, 10]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5]. In the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.
在高层次上，我们采用稀疏激活的 Mixture-of-Experts（MoE）理念。我们使用 Transformer 架构，并在每隔一层的 Transformer 中将前馈组件替换为 MoE 层，遵循近期的实践经验。规律性 Transformer 层和 MoE 层的交错排列在实际中提升了模型性能和训练效率，这可能是因为在 MoE 层之间引入一些共享组件可以减轻跳过 token 所带来的负面影响。我们在实验中应用了一些近期工作中采用的额外修改。例如，我们用按层的相对位置偏置替代了标准位置嵌入。在非 MoE 的前馈子层（只有每隔一层是 MoE 层）中，我们用门控线性单元替换了第一个线性映射和激活函数，其计算输入的两次线性变换的逐元素乘积，随后是一个高斯误差线性单元激活函数。

As described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as ‘‘experts’’. The gating function in Eq. 2 uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [31, 10, 21]. During training, each MoE layer’s learnable gating network described in Eq. 2 is trained to use the input to activate the best subset of experts using a top-$k$ function along the token dimension. An ‘‘shuffle’’ stage and an ‘‘unshuffle’’ stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in LABEL:eq:permute and Eq. 4.
如前所述，每个 MoE 层由一组独立的前馈网络组成，称为“专家”。在方程 2 中使用门控函数，其采用 softmax 激活函数来对这些专家建模出一个概率分布。这个概率分布表示每个传入 token 对专家的偏好，这与传统的门控网络[31, 10, 21]计算方式类似。在训练过程中，在方程 2 中描述的每个 MoE 层的可学习门控网络被训练以使用输入激活使用 top- $k$ 函数沿 token 维度的最佳专家子集。插入一个“打乱”和一个“恢复”阶段到 MoE 层，其中第一阶段将 tokens 聚集到其指定的专家，而第二阶段将 tokens 重新排列回输入批次中的原始顺序。这一步阐述在 LABEL:eq:permute 和方程 4 中。

Similar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor $c$ in Eq. 1. Unless otherwise specified, we set $c=2$ such that our method can be directly compared to the top-2 token-choice gating in GShard.
类似于传统的 MoE 方法，MoE 层拥有更多参数。然而，因为在训练或推理期间，对于任何给定的 token，只激活有限的专家子集，所以每个 token 的激活模型大小可以与密集层相媲美。例如，Switch Transformer[10] 仅激活一个专家，而 GShard[21] 每个 token 使用两个专家。在我们的方法中，激活的专家数量对于每个 token 可以有所不同，但通过在方程 1 中固定容量因子 $c$ ，整体计算与基准结构保持相同。除非另有说明，我们设定 $c=2$ 以便我们的方法可以直接与 GShard 中的 top-2 token 选择门控进行比较。

We train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.
我们通过增加专家数量来训练几个变体的架构，在 100M 规模（即 100M 专家规模）下以理解我们方法的扩展效果。我们也训练了一个 8B 规模的 MoE 模型。这个大型 MoE 模型使用 GSPMD[36]中介绍的 2D 分片算法进行分区，充分利用 TPU 群集[19]的 2D 拓扑。在不同的规模和设置下，我们的方法优于相关工作，并在 GLUE 和 SuperGLUE 中选定的任务上展示了强大的下游任务性能。

Table 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we studied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While $n_{params}$ is the total number of trainable parameters, $n_{act-params}$ represents the number of activated parameters per token. $L$ is the total number of Transformer layers, $M$ is the model dimension, $H$ is the hidden dimension after the projection in each transformer layer, $n_{heads}$ is the number of attention heads, and $d_{head}$ is the hidden dimension of each attention head.
表 1 总结了不同 MoE 模型的超参数设置。作为参考点，我们还包括相应的稠密模型配置，这些配置在推理过程中每个 token 激活的参数数量相当。为了研究专家数量扩展的效果，我们研究了在保持每个专家大小为 1 亿参数的情况下变化专家数量。例如，0.1B/64E 表示一个大约 1 亿参数的稠密模型架构，其每隔一个层被一个 64 专家的 MoE 层替代。当每个 MoE 层只有一个专家时，MoE 模型退化为稠密 Transformer 架构。 $n_{params}$ 是可训练参数的总数， $n_{act-params}$ 代表每个 token 激活的参数数， $L$ 是 Transformer 层的总数， $M$ 是模型维度， $H$ 是每个 Transformer 层投影后的隐藏维度， $n_{heads}$ 是注意力头的数量， $d_{head}$ 是每个注意力头的隐藏维度。

Dataset: We use the high-quality dataset from GLaM [du2022glam] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.
数据集：我们使用了 GLaM [du2022glam] 提供的高质量数据集，其中包含了 1.6 万亿个 token，代表了广泛的自然语言用例。一个内部分类器被训练用来在一系列经过筛选的文本与其他网页之间进行分类并评估网页的内容质量。经过高质量过滤的网页子集被与书籍、维基百科页面、对话、论坛和新闻结合，创建了最终的数据集。数据和混合权重可在 GLaM 论文的表 3 中找到。

Model Training: Our model training follows the setups of GLaM [du2022glam] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay $\beta_{1}=0$ and second-moment decay $\beta_{2}=0.99$. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.
模型训练：我们的模型训练遵循 GLaM [du2022glam] 的设置，其中采用了最大序列长度为 1024 个 tokens。我们使用 Adafactor 优化器[32]，其中一阶动量衰减为 $\beta_{1}=0$ ，二阶动量衰减为 $\beta_{2}=0.99$ 。在前 10K 训练步骤中，我们保持学习率不变，然后使用反平方根计划衰减。与大多数相关工作不同，我们未施加任何用于负载平衡的附加损失，例如 Switch Transformer[10]和 GShard[21] 中所述。我们使用了 256K 词汇量的 SentencePiece 子词分词器。最大的模型(8B/64E)在 512 个 TPU V4 芯片上进行训练。训练时，我们的辍学率为 0，因为训练数据语料库中的 token 数量远大于训练期间的总 token 数量。

Model Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [35, 34].
模型评估：我们主要关注在 GLUE 和 SuperGLUE 基准中选择的 11 个任务上进行微调表现的评估。

We first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance where some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.
我们首先研究训练效率和收敛性。我们使用容量系数为 2 的专家选择（EC-CF2）来匹配 GShard top-2 门控在每个 token 基础上激活的模型大小和计算成本，并在固定的步骤数下运行两者。结果如图 2 (a)所示。相比 GShard top-2 门控，在评估数据集的困惑度和下游任务微调上均表现出更强性能，EC-CF2 在训练期间的收敛速度快了超过 2 倍。更具体地说，EC-CF2 在不到一半的步骤中达到了 GShard top-2 的相同困惑度，并且每一步 GShard top-2 的处理速度比我们的方法慢 20%。如 3.1 节所述，top-2 门控中步骤时间较慢是由于负载不平衡，其中一些专家接收到的 tokens 数量远超预期容量。因此，步骤延迟将受到负载最多的专家的瓶颈限制。

As presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.
如表 1 所示，增加专家数量可有效提高模型容量，而不会增加激活的模型规模。我们在固定专家规模为 1 亿参数的情况下，对专家选择（EC）和 GShard（Top-2）方法进行专家数量的扩展，发现这两种方法在预训练期间评估数据集上的困惑度表现良好。如图 2（b）所示，增加专家数量持续改善训练困惑度。

To validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.
为了验证改进的困惑度是否直接转化为下游任务的更好性能，我们对 GLUE 和 SuperGLUE 的 11 个选定任务进行微调。我们比较了三种 MoE 方法，包括 Switch Transformer top-1 gating（ST Top-1）、GShard top-2 gating（GS Top-2）和我们的方法（EC-CF2），该方法匹配了 GS Top-2 的激活内存大小和计算成本。由表 2 中的结果可以看出，我们的 EC-CF2 方法始终优于相关方法，并在较大的 8B/64E 环境下平均提高了超过 2%的准确率。表 3 进一步将我们的 8B/64E 模型与其密集对应模型进行比较。同样，我们的方法获得了更强的微调结果，平均得分提高了 3.4 分。

Interestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.
有趣的是，我们观察到 100M/32E 模型设置对于 GS Top-2 和 EC-CF2 都表现最佳，即使其有效模型容量小于 100M/64E 和 100M/128E。这一结果表明，良好的训练困惑度并不总是意味下游任务的更好表现。

Capped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.
限制专家选择：我们通过限制每个标记的最大专家数量，使用第 3.3 节中描述的方法来正则化专家选择。表 4 报告了 11 个选定数据集的平均准确率。EC-CAP2 是我们专家选择方法的一个变体，通过将每个标记的专家数量限制为 2。这样平均降低了微调准确率 0.8 分。此外，EC-CAP3 允许每个标记最多 3 个专家，并获得与普通专家选择方法相当的结果。这项消融研究证实了允许可变数量的专家对于每个标记确实有帮助。

Variable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.
可变标记专家：我们计算标记到专家路由的统计数据，特别是关于已路由到某个数量的专家的标记比例。根据图 3，大多数标记已路由到一位或两位专家，而 23%的标记已路由到三位或四位专家，只有约 3%的标记已路由到超过 4 位专家。该图验证了我们的假设，即我们的方法学会了为标记分配可变数量的专家，这对重要的标记可能是有益的。

In this section, we compare our method with Hash Layers [28]. We use $\mod{x}$ to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.
本节中，我们将我们的方法与 Hash Layers [28]进行比较。我们使用 $\mod{x}$ 将令牌 ID 映射到专家 ID。这确保了负载平衡并生成了专门的专家。微调结果在表 4 的最后一行中展示。基于哈希的路由在平均分数和方差方面表现不如专家选择。这表明仅靠负载平衡并不能产生所有的好处。

Capacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. 1, the capacity factor determines how many experts on average each token can be routed to, thus the bucket size $k$ of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating. We further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.
容量因子：我们在专家选择方法中研究了容量因子，并将训练困惑度与 Switch Transformer 中使用的基线顶级-1 门控方法进行比较。如公式 1 所述，容量因子决定了平均每个标记可以被路由到多少专家，因此确定了每个专家的桶大小 $k$ 。在我们之前的所有实验中，我们使用了容量因子为 2，这与 GShard 方法使用的顶级-2 门控的计算足迹相匹配。为了在每个标记的基础上公正地匹配与 Switch Transformer 中使用的顶级-1 门控的计算成本，我们将容量因子减至 1，并在图 4（a）中绘制了训练困惑度。不出所料，使用较小的容量因子会提高困惑度，但我们的方法仍显著优于顶级-1 门控。我们进一步将容量因子降低到 0.5，观察到它仍然优于顶级-1 门控。

Comparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in perplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance,  Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.
与密集模型在预训练中的比较：我们将我们的方法与密集模型在预训练阶段进行比较。如图 4（b）所示，我们的方法在困惑度和收敛时间上始终优于密集方法。对于参数量为 1 亿的较小专家规模，稀疏门控的优势更加显著。与图 2（b）中所展示的增加专家数量能够提高模型性能的结果不同，图 4（b）显示增加专家容量也能显著提升模型性能。