DeepSeek-V3 Technical Report
DeepSeek-V3 技术报告

DeepSeek-AIresearch@deepseek.com

Abstract 摘要

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788 M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available athttps://github.com/deepseek-ai/DeepSeek-V3
我们推出的DeepSeek-V3是一个强大的专家混合（MoE）语言模型，它拥有671B个总参数，每个标记有37B个激活参数。为了实现高效推理和低成本训练，DeepSeek-V3采用了多头潜意识（MLA）和DeepSeekMoE架构，这在DeepSeek-V2中得到了充分验证。此外，DeepSeek-V3 还率先采用了无辅助损失的负载均衡策略，并设定了多标记预测训练目标，以提高性能。我们在14.8万亿个不同的高质量代币上对DeepSeek-V3进行预训练，然后在监督微调和强化学习阶段充分发挥其能力。综合评估显示，DeepSeek-V3的性能优于其他开源模型，并可与领先的闭源模型相媲美。尽管性能卓越，DeepSeek-V3 的全部训练仅需 2.788 M H800 GPU 小时。此外，其训练过程也非常稳定。在整个训练过程中，我们没有遇到任何不可恢复的损失峰值，也没有进行任何回滚。模型检查点可从以下网址获取：https://github.com/deepseek-ai/DeepSeek-V3

Figure 1 | Benchmark performance of DeepSeek-V3 and its counterparts.
图 1：DeepSeek-V3 及其同类产品的基准性能。

Contents 目录

1 Introduction … 4 1 导言 ... 4
2 Architecture … 6
2 建筑 ... 6
2.1 Basic Architecture … 6
2.1 基本架构 ... 6
2.1.1 Multi-Head Latent Attention … 7
2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing … 8
2.2 Multi-Token Prediction … 10
2.2 多标记预测...... 10
3 Infrastructures … 11
3 基础设施 ... 11
3.1 Compute Clusters … 11
3.1 计算集群 ... 11
3.2 Training Framework … 12
3.2 培训框架 ... 12
3.2.1 DualPipe and Computation-Communication Overlap … 12
3.2.2 Efficient Implementation of Cross-Node All-to-All Communication … 13
3.2.3 Extremely Memory Saving with Minimal Overhead … 14
3.3 FP8 Training … 14
3.3 FP8 培训 ... 14
3.3.1 Mixed Precision Framework … 15
3.3.1 混合精度框架 ... 15
3.3.2 Improved Precision from Quantization and Multiplication … 16
3.3.3 Low-Precision Storage and Communication … 18
3.4 Inference and Deployment … 18
3.4.1 Prefilling … 19
3.4.1 预填充 ... 19
3.4.2 Decoding … 19
3.4.2 解码 ... 19
3.5 Suggestions on Hardware Design … 20
3.5.1 Communication Hardware … 20
3.5.1 通信硬件 ... 20
3.5.2 Compute Hardware … 20
3.5.2 计算硬件 ... 20
4 Pre-Training … 22
4 培训前 ... 22
4.1 Data Construction … 22
4.2 Hyper-Parameters … 22
4.2 超参数 ... 22
4.3 Long Context Extension … 23
4.4 Evaluations … 24
4.4.1 Evaluation Benchmarks … 24
4.4.1 评估基准 ... 24
4.4.2 Evaluation Results … 25
4.4.2 评估结果 ... 25
4.5 Discussion … 26
4.5 讨论...... 26
4.5.1 Ablation Studies for Multi-Token Prediction … 26
4.5.2 Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy … 27
4.5.3 Batch-Wise Load Balance VS. Sequence-Wise Load Balance … 27
4.5.3 批量负载平衡 VS.顺序负载平衡 ... 27
5 Post-Training … 28
5 培训后...... 28
5.1 Supervised Fine-Tuning … 28
5.2 Reinforcement Learning … 29
5.2 强化学习 ... 29
5.2.1 Reward Model … 29
5.2.1 奖励模式 ... 29
5.2.2 Group Relative Policy Optimization … 30
5.3 Evaluations … 30
5.3.1 Evaluation Settings … 30
5.3.1 评估设置 ... 30
5.3.2 Standard Evaluation … 32
5.3.2 标准评估...... 32
5.3.3 Open-Ended Evaluation … 33
5.3.4 DeepSeek-V3 as a Generative Reward Model … 33
5.4 Discussion … 34
5.4 讨论 ... 34
5.4.1 Distillation from DeepSeek-R1 … 34
5.4.1 来自 DeepSeek-R1 的蒸馏... 34
5.4.2 Self-Rewarding … 34
5.4.3 Multi-Token Prediction Evaluation … 35
6 Conclusion, Limitations, and Future Directions … 35
A Contributions and Acknowledgments … 45
贡献和鸣谢...... 45
B Ablation Studies for Low-Precision Training … 47
B 用于低精度训练的消融研究...... 47
B. 1 FP8 v.s. BF16 Training … 47
B. 2 Discussion About Block-Wise Quantization … 47
C Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models … 48
C 基于辅助损失模型和无辅助损失模型的 16B 专家专业化模式......................

1. Introduction 1.导言

In recent years, Large Language Models (LLMs) have been undergoing rapid iteration and evolution (Anthropic, 2024; Google, 2024; OpenAI, 2024a), progressively diminishing the gap towards Artificial General Intelligence (AGI). Beyond closed-source models, open-source models, including DeepSeek series (DeepSeek-AI, 2024a|b|c; Guo et al., 2024), LLaMA series (AI@Meta, 2024a b; Touvron et al., 2023a b), Qwen series (Qwen, 2023, 2024a b), and Mistral series (Jiang et al. 2023; Mistral, 2024), are also making significant strides, endeavoring to close the gap with their closed-source counterparts. To further push the boundaries of open-source model capabilities, we scale up our models and introduce DeepSeek-V3, a large Mixture-of-Experts (MoE) model with 671B parameters, of which 37B are activated for each token.
近年来，大型语言模型（LLMs）经历了快速的迭代和演进（Anthropic，2024；Google，2024；OpenAI，2024a），逐步缩小了与人工通用智能（AGI）的差距。除闭源模型外，开源模型包括 DeepSeek 系列（DeepSeek-AI, 2024a|b|c；Guo et al.，2024）、LLaMA 系列（AI@Meta, 2024a b；Touvron et al、2023a b）、Qwen 系列（Qwen, 2023, 2024a b）和 Mistral 系列（Jiang 等，2023；Mistral, 2024）也都取得了长足进步，努力缩小与闭源同行的差距。为了进一步推动开源模型能力的发展，我们扩大了模型的规模，并引入了 DeepSeek-V3，这是一个拥有 671B 个参数的大型专家混合物（MoE）模型，其中 37B 个参数针对每个标记被激活。

With a forward-looking perspective, we consistently strive for strong model performance and economical costs. Therefore, in terms of architecture, DeepSeek-V3 still adopts Multi-head Latent Attention (MLA) (DeepSeek-AI, 2024c) for efficient inference and DeepSeekMoE (Dai et al., 2024) for cost-effective training. These two architectures have been validated in DeepSeekV2 (DeepSeek-AI, 2024c), demonstrating their capability to maintain robust model performance while achieving efficient training and inference. Beyond the basic architecture, we implement two additional strategies to further enhance the model capabilities. Firstly, DeepSeek-V3 pioneers an auxiliary-loss-free strategy (Wang et al., 2024a) for load balancing, with the aim of minimizing the adverse impact on model performance that arises from the effort to encourage load balancing. Secondly, DeepSeek-V3 employs a multi-token prediction training objective, which we have observed to enhance the overall performance on evaluation benchmarks.
我们以前瞻性的视角，始终追求强大的模型性能和经济的成本。因此，在架构方面，DeepSeek-V3仍然采用多头潜意识（MLA）（DeepSeek-AI, 2024c）进行高效推理，并采用DeepSeekMoE（Dai等人，2024）进行低成本训练。这两种架构已在DeepSeekV2（DeepSeek-AI，2024c）中得到验证，证明它们有能力在实现高效训练和推理的同时保持模型的稳健性能。除基本架构外，我们还实施了两项额外策略，以进一步增强模型能力。首先，DeepSeek-V3 率先采用了一种无辅助损失策略（Wang 等人，2024a）来实现负载平衡，目的是尽量减少因鼓励负载平衡而对模型性能造成的不利影响。其次，DeepSeek-V3 采用了多标记预测训练目标，据我们观察，该目标提高了评估基准的整体性能。

In order to achieve efficient training, we support the FP8 mixed precision training and implement comprehensive optimizations for the training framework. Low-precision training has emerged as a promising solution for efficient training (Dettmers et al., 2022; Kalamkar et al., 2019, Narang et al., 2017, Peng et al., 2023b), its evolution being closely tied to advancements in hardware capabilities (Luo et al., 2024; Micikevicius et al., 2022; Rouhani et al., 2023a). In this work, we introduce an FP8 mixed precision training framework and, for the first time, validate its effectiveness on an extremely large-scale model. Through the support for FP8 computation and storage, we achieve both accelerated training and reduced GPU memory usage. As for the training framework, we design the DualPipe algorithm for efficient pipeline parallelism, which has fewer pipeline bubbles and hides most of the communication during training through computation-communication overlap. This overlap ensures that, as the model further scales up, as long as we maintain a constant computation-to-communication ratio, we can still employ fine-grained experts across nodes while achieving a near-zero all-to-all communication overhead. In addition, we also develop efficient cross-node all-to-all communication kernels to fully utilize InfiniBand (IB) and NVLink bandwidths. Furthermore, we meticulously optimize the memory footprint, making it possible to train DeepSeek-V3 without using costly tensor parallelism. Combining these efforts, we achieve high training efficiency.
为了实现高效训练，我们支持 FP8 混合精度训练，并对训练框架进行了全面优化。低精度训练已成为高效训练的一种有前途的解决方案（Dettmers等人，2022年；Kalamkar等人，2019年；Narang等人，2017年；Peng等人，2023年b），其发展与硬件能力的进步密切相关（Luo等人，2024年；Micikevicius等人，2022年；Rouhani等人，2023年a）。在这项工作中，我们引入了 FP8 混合精度训练框架，并首次在超大规模模型上验证了其有效性。通过支持 FP8 计算和存储，我们实现了加速训练和减少 GPU 内存使用。在训练框架方面，我们设计了高效流水线并行的 DualPipe 算法，该算法具有较少的流水线气泡，并通过计算-通信重叠隐藏了训练过程中的大部分通信。这种重叠确保了在模型进一步扩展时，只要我们保持恒定的计算与通信比率，就能在节点间使用细粒度专家，同时实现接近零的全对全通信开销。此外，我们还开发了高效的跨节点全对全通信内核，以充分利用 InfiniBand（IB）和 NVLink 带宽。此外，我们还精心优化了内存占用，使 DeepSeek-V3 的训练无需使用昂贵的张量并行。综合这些努力，我们实现了很高的训练效率。

During pre-training, we train DeepSeek-V3 on 14.8T high-quality and diverse tokens. The pre-training process is remarkably stable. Throughout the entire training process, we did not encounter any irrecoverable loss spikes or have to roll back. Next, we conduct a two-stage context length extension for DeepSeek-V3. In the first stage, the maximum context length is extended to 32 K , and in the second stage, it is further extended to 128 K . Following this, we conduct post-training, including Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on the base model of DeepSeek-V3, to align it with human preferences and further unlock its potential. During the post-training stage, we distill the reasoning capability from the DeepSeekR1 series of models, and meanwhile carefully maintain the balance between model accuracy
在预训练过程中，我们在 14.8T 高质量和多样化的标记上训练 DeepSeek-V3。预训练过程非常稳定。在整个训练过程中，我们没有遇到任何不可恢复的损失峰值，也无需回滚。接下来，我们对 DeepSeek-V3 进行了两阶段的上下文长度扩展。第一阶段，最大上下文长度扩展到 32 K，第二阶段进一步扩展到 128 K。之后，我们对 DeepSeek-V3 的基础模型进行了包括监督微调（SFT）和强化学习（RL）在内的后训练，使其符合人类偏好，进一步释放其潜力。在后训练阶段，我们从DeepSeekR1系列模型中提炼出推理能力，同时谨慎地保持模型准确性和推理能力之间的平衡。

Training Costs 培训费用	Pre-Training 培训前	Context Extension 背景扩展	Post-Training 培训后	Total 总计
in H800 GPU Hours 在 H800 GPU 小时数中	2664 K	119 K	5 K	2788 K
in USD 以美元计	$$ 5.328 M$	$$ 0.238 M$	$$ 0.01 M$	$$ 5.576 M$

Table 1 | Training costs of DeepSeek-V3, assuming the rental price of H800 is

$ 2

per GPU hour.
表 1：假设 H800 的租赁价格为

$ 2

每 GPU 小时，DeepSeek-V3 的培训成本。
and generation length. 和世代长度。
We evaluate DeepSeek-V3 on a comprehensive array of benchmarks. Despite its economical training costs, comprehensive evaluations reveal that DeepSeek-V3-Base has emerged as the strongest open-source base model currently available, especially in code and math. Its chat version also outperforms other open-source models and achieves performance comparable to leading closed-source models, including GPT-4o and Claude-3.5-Sonnet, on a series of standard and open-ended benchmarks.
我们在一系列基准测试中对 DeepSeek-V3 进行了评估。尽管训练成本低廉，但综合评估结果显示，DeepSeek-V3-Base 已成为目前最强大的开源基础模型，尤其是在代码和数学方面。在一系列标准和开放式基准测试中，其聊天版本的性能也优于其他开源模型，并可与 GPT-4o 和 Claude-3.5-Sonnet 等领先的闭源模型相媲美。

Lastly, we emphasize again the economical training costs of DeepSeek-V3, summarized in Table 1, achieved through our optimized co-design of algorithms, frameworks, and hardware. During the pre-training stage, training DeepSeek-V3 on each trillion tokens requires only 180 K H800 GPU hours, i.e., 3.7 days on our cluster with 2048 H800 GPUs. Consequently, our pretraining stage is completed in less than two months and costs 2664 K GPU hours. Combined with 119 K GPU hours for the context length extension and 5 K GPU hours for post-training, DeepSeek-V3 costs only 2.788 M GPU hours for its full training. Assuming the rental price of the H800 GPU is

$ 2

per GPU hour, our total training costs amount to only

$ 5.576 M

. Note that the aforementioned costs include only the official training of DeepSeek-V3, excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data.
最后，我们再次强调DeepSeek-V3的经济训练成本，总结如表1所示，这是通过我们对算法、框架和硬件的优化协同设计实现的。在预训练阶段，在每万亿个代币上训练DeepSeek-V3只需要180 K H800 GPU小时，也就是说，在我们拥有2048个H800 GPU的集群上只需要3.7天。因此，我们的预训练阶段在不到两个月的时间内完成，耗费了 2664 K GPU 小时。加上用于上下文长度扩展的 119 K GPU 小时和用于后期训练的 5 K GPU 小时，DeepSeek-V3 的全部训练成本仅为 2.788 M GPU 小时。假设 H800 GPU 的租赁价格为每 GPU 小时

$ 2

，那么我们的总训练成本仅为

$ 5.576 M

。请注意，上述成本仅包括 DeepSeek-V3 的官方训练费用，不包括之前对架构、算法或数据进行研究和消融实验的相关费用。

Our main contribution includes:
我们的主要贡献包括

Architecture: Innovative Load Balancing Strategy and Training Objective
架构：创新负载平衡策略和培训目标

On top of the efficient architecture of DeepSeek-V2, we pioneer an auxiliary-loss-free strategy for load balancing, which minimizes the performance degradation that arises from encouraging load balancing.
在DeepSeek-V2的高效架构基础上，我们首创了一种无辅助损失的负载均衡策略，最大限度地降低了鼓励负载均衡所带来的性能下降。
We investigate a Multi-Token Prediction (MTP) objective and prove it beneficial to model performance. It can also be used for speculative decoding for inference acceleration.
我们研究了多令牌预测（MTP）目标，并证明它有利于提高模型性能。它还可用于推测性解码，以加快推理速度。

Pre-Training: Towards Ultimate Training Efficiency
训练前：提高训练效率

We design an FP8 mixed precision training framework and, for the first time, validate the feasibility and effectiveness of FP8 training on an extremely large-scale model.
我们设计了一个 FP8 混合精度训练框架，并首次在一个超大规模模型上验证了 FP8 训练的可行性和有效性。
Through the co-design of algorithms, frameworks, and hardware, we overcome the communication bottleneck in cross-node MoE training, achieving near-full computationcommunication overlap. This significantly enhances our training efficiency and reduces the training costs, enabling us to further scale up the model size without additional overhead.
通过共同设计算法、框架和硬件，我们克服了跨节点 MoE 训练中的通信瓶颈，实现了近乎完全的计算-通信重叠。这大大提高了我们的训练效率，降低了训练成本，使我们能够在不增加额外开支的情况下进一步扩大模型规模。
At an economical cost of only 2.664 M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours.
我们仅用 2.664 M H800 GPU 小时的经济成本，就在 14.8T 代币上完成了 DeepSeek-V3 的预训练，生成了目前最强的开源基础模型。预训练后的后续训练阶段仅需 0.1 百万 GPU 小时。

Post-Training: Knowledge Distillation from DeepSeek-R1
培训后：从 DeepSeek-R1 中提炼知识

We introduce an innovative methodology to distill reasoning capabilities from the long-Chain-of-Thought (CoT) model, specifically from one of the DeepSeek R1 series models, into standard LLMs, particularly DeepSeek-V3. Our pipeline elegantly incorporates the
我们引入了一种创新方法，将长思链（CoT）模型（特别是 DeepSeek R1 系列模型之一）中的推理能力提炼到标准 LLMs 中，尤其是 DeepSeek-V3。我们的管道优雅地将
verification and reflection patterns of R1 into DeepSeek-V3 and notably improves its reasoning performance. Meanwhile, we also maintain control over the output style and length of DeepSeek-V3.
我们将R1的验证和反射模式引入DeepSeek-V3，显著提高了其推理性能。同时，我们还保持了对 DeepSeek-V3 输出样式和长度的控制。

Summary of Core Evaluation Results
核心评估结果摘要

Knowledge: (1) On educational benchmarks such as MMLU, MMLU-Pro, and GPQA, DeepSeek-V3 outperforms all other open-source models, achieving 88.5 on MMLU, 75.9 on MMLU-Pro, and 59.1 on GPQA. Its performance is comparable to leading closed-source models like GPT-4o and Claude-Sonnet-3.5, narrowing the gap between open-source and closed-source models in this domain. (2) For factuality benchmarks, DeepSeek-V3 demonstrates superior performance among open-source models on both SimpleQA and Chinese SimpleQA. While it trails behind GPT-4o and Claude-Sonnet-3.5 in English factual knowledge (SimpleQA), it surpasses these models in Chinese factual knowledge (Chinese SimpleQA), highlighting its strength in Chinese factual knowledge.
知识：（1）在MMLU、MMLU-Pro和GPQA等教育基准测试中，DeepSeek-V3的表现优于所有其他开源模型，在MMLU上达到88.5，在MMLU-Pro上达到75.9，在GPQA上达到59.1。其性能可与 GPT-4o 和 Claude-Sonnet-3.5 等领先的闭源模型相媲美，缩小了该领域开源模型与闭源模型之间的差距。(2) 在事实性基准方面，DeepSeek-V3 在 SimpleQA 和中文 SimpleQA 上的表现在开源模型中都属上乘。虽然它在英语事实知识（SimpleQA）方面落后于 GPT-4o 和 Claude-Sonnet-3.5，但在中文事实知识（Chinese SimpleQA）方面却超过了这两个模型，突出了它在中文事实知识方面的优势。
Code, Math, and Reasoning: (1) DeepSeek-V3 achieves state-of-the-art performance on math-related benchmarks among all non-long-CoT open-source and closed-source models. Notably, it even outperforms o1-preview on specific benchmarks, such as MATH-500, demonstrating its robust mathematical reasoning capabilities. (2) On coding-related tasks, DeepSeek-V3 emerges as the top-performing model for coding competition benchmarks, such as LiveCodeBench, solidifying its position as the leading model in this domain. For engineering-related tasks, while DeepSeek-V3 performs slightly below Claude-Sonnet-3.5, it still outpaces all other models by a significant margin, demonstrating its competitiveness across diverse technical benchmarks.
代码、数学和推理：（1）DeepSeek-V3在所有非长CoT开源和闭源模型中的数学相关基准测试中都取得了最先进的性能。值得注意的是，它在特定基准（如 MATH-500）上的表现甚至超过了 o1-preview，这证明了它强大的数学推理能力。(2）在编码相关任务方面，DeepSeek-V3 在编码竞赛基准（如 LiveCodeBench）上表现最佳，巩固了其在该领域领先模型的地位。在与工程相关的任务中，虽然 DeepSeek-V3 的性能略低于 Claude-Sonnet-3.5，但仍以显著优势超越所有其他模型，显示了其在各种技术基准中的竞争力。

In the remainder of this paper, we first present a detailed exposition of our DeepSeek-V3 model architecture (Section 2). Subsequently, we introduce our infrastructures, encompassing our compute clusters, the training framework, the support for FP8 training, the inference deployment strategy, and our suggestions on future hardware design. Next, we describe our pre-training process, including the construction of training data, hyper-parameter settings, longcontext extension techniques, the associated evaluations, as well as some discussions (Section 4). Thereafter, we discuss our efforts on post-training, which include Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), the corresponding evaluations, and discussions (Section 5). Lastly, we conclude this work, discuss existing limitations of DeepSeek-V3, and propose potential directions for future research (Section 6).
在本文的其余部分，我们将首先详细介绍 DeepSeek-V3 模型架构（第 2 节）。随后，我们介绍了我们的基础设施，包括我们的计算集群、训练框架、对 FP8 训练的支持、推理部署策略以及我们对未来硬件设计的建议。接下来，我们将介绍我们的预训练过程，包括训练数据的构建、超参数设置、长语境扩展技术、相关评估以及一些讨论（第 4 节）。之后，我们将讨论我们在后训练方面所做的努力，包括监督微调（SFT）、强化学习（RL）、相应的评估和讨论（第 5 节）。最后，我们总结了这项工作，讨论了 DeepSeek-V3 现有的局限性，并提出了未来研究的潜在方向（第 6 节）。

2. Architecture 2.建筑学

We first introduce the basic architecture of DeepSeek-V3, featured by Multi-head Latent Attention (MLA) (DeepSeek-AI, 2024c) for efficient inference and DeepSeekMoE (Dai et al., 2024) for economical training. Then, we present a Multi-Token Prediction (MTP) training objective, which we have observed to enhance the overall performance on evaluation benchmarks. For other minor details not explicitly mentioned, DeepSeek-V3 adheres to the settings of DeepSeekV2 (DeepSeek-AI, 2024c).
我们首先介绍DeepSeek-V3的基本架构，其特点是采用多头潜在注意力（MLA）（DeepSeek-AI, 2024c）进行高效推理，采用DeepSeekMoE（Dai等人，2024）进行经济训练。然后，我们提出了多标记预测（MTP）训练目标，据我们观察，该目标提高了评估基准的整体性能。至于其他未明确提及的小细节，DeepSeek-V3沿用了DeepSeekV2（DeepSeek-AI，2024c）的设置。

2.1. Basic Architecture 2.1.基本架构

The basic architecture of DeepSeek-V3 is still within the Transformer (Vaswani et al., 2017) framework. For efficient inference and economical training, DeepSeek-V3 also adopts MLA and DeepSeekMoE, which have been thoroughly validated by DeepSeek-V2. Compared with DeepSeek-V2, an exception is that we additionally introduce an auxiliary-loss-free load balancing
DeepSeek-V3 的基本架构仍采用 Transformer（Vaswani 等人，2017 年）框架。为了实现高效的推理和经济的训练，DeepSeek-V3 还采用了 MLA 和 DeepSeekMoE，它们已经在 DeepSeek-V2 中得到了充分验证。与 DeepSeek-V2 相比，我们额外引入了无辅助损失的负载平衡技术。

Figure 2 | Illustration of the basic architecture of DeepSeek-V3. Following DeepSeek-V2, we adopt MLA and DeepSeekMoE for efficient inference and economical training.
图 2：DeepSeek-V3 的基本架构示意图。继 DeepSeek-V2 之后，我们采用了 MLA 和 DeepSeekMoE 来实现高效推理和经济训练。
strategy (Wang et al., 2024a) for DeepSeekMoE to mitigate the performance degradation induced by the effort to ensure load balance. Figure 2 illustrates the basic architecture of DeepSeek-V3, and we will briefly review the details of MLA and DeepSeekMoE in this section.
MLA和DeepSeekMoE策略（Wang等人，2024a）可减轻因确保负载平衡而导致的性能下降。图2展示了DeepSeek-V3的基本架构，我们将在本节简要回顾MLA和DeepSeekMoE的细节。

2.1.1. Multi-Head Latent Attention
2.1.1.多头潜意识

For attention, DeepSeek-V3 adopts the MLA architecture. Let

d

denote the embedding dimension,

n_{h}

denote the number of attention heads,

d_{h}

denote the dimension per head, and

h_{t} \in R^{d}

denote the attention input for the

t

-th token at a given attention layer. The core of MLA is the low-rank joint compression for attention keys and values to reduce Key-Value (KV) cache during inference:
对于注意力，DeepSeek-V3 采用 MLA 架构。让

d

表示嵌入维度，

n_{h}

表示注意头的数量，

d_{h}

表示每个注意头的维度，

h_{t} \in R^{d}

表示给定注意层中

t

-th 标记的注意输入。MLA 的核心是对注意力键和值进行低级联合压缩，以减少推理过程中的键值（KV）缓存：

\begin{aligned} c_{t}^{K V} & = W^{D K V} h_{t}, \\ [k_{t, 1}^{C}; k_{t, 2}^{C}; \dots; k_{t, n_{h}}^{C}] = k_{t}^{C} & = W^{U K} c_{t}^{K V}, \\ k_{t}^{R} & = RoPE (W^{K R} h_{t}), \\ k_{t, i} & = [k_{t, i}^{C} {\dot{k}}_{t}^{R}], \\ [v_{t, 1}^{C}; v_{t, 2}^{C}; \dots; v_{t, n_{h}}^{C}] = v_{t}^{C} & = W^{U V} c_{t}^{K V}, \end{aligned}

where

c_{t}^{K V} \in R^{d_{c}}

is the compressed latent vector for keys and values;

d_{c} (≪ d_{h} n_{h})

indicates the

K V

compression dimension;

W^{D K V} \in R^{d_{c} \times d}

denotes the down-projection matrix;

W^{U K}, W^{U V} \in R^{d_{h} n_{h} \times d_{c}}

are the up-projection matrices for keys and values, respectively;

W^{K R} \in R_{h}^{d_{h}^{R} \times d}

is the matrix used to produce the decoupled key that carries Rotary Positional Embedding (RoPE) (Su et al., 2024);

RoPE (\cdot)

denotes the operation that applies RoPE matrices; and

[∵ \cdot]

denotes concatenation. Note that for MLA, only the blue-boxed vectors (i.e.,

c_{t}^{K V}

and

k_{t}^{R}

) need to be cached during generation, which results in significantly reduced KV cache while maintaining performance comparable to standard Multi-Head Attention (MHA) (Vaswani et al., 2017).
其中，

c_{t}^{K V} \in R^{d_{c}}

是密钥和值的压缩潜向量；

d_{c} (≪ d_{h} n_{h})

表示

K V

压缩维数；

W^{D K V} \in R^{d_{c} \times d}

表示向下投影矩阵；

W^{U K}, W^{U V} \in R^{d_{h} n_{h} \times d_{c}}

分别是密钥和值的向上投影矩阵；

W^{K R} \in R_{h}^{d_{h}^{R} \times d}

是用于生成解耦密钥的矩阵，该矩阵带有旋转位置嵌入 (RoPE) 功能（Su 等人，2024 年）；

RoPE (\cdot)

表示应用 RoPE 矩阵的操作；

[∵ \cdot]

表示连接、2024）；

RoPE (\cdot)

表示应用 RoPE 矩阵的操作；

[∵ \cdot]

表示连接。需要注意的是，对于 MLA，在生成过程中只需要缓存蓝框向量（即

c_{t}^{K V}

和

k_{t}^{R}

），这就大大减少了 KV 缓存，同时保持了与标准多头注意力 (MHA) 相当的性能（Vaswani 等人，2017 年）。

For the attention queries, we also perform a low-rank compression, which can reduce the activation memory during training:
对于注意力查询，我们还进行了低级压缩，这可以减少训练过程中的激活记忆：

\begin{aligned} c_{t}^{Q} & = W^{D Q} h_{t}, \\ [q_{t, 1}^{C}; q_{t, 2}^{C}; \dots; q_{t, n_{h}}^{C}] = q_{t}^{C} & = W^{U Q} c_{t}^{Q} \\ [q_{t, 1}^{R}; q_{t, 2}^{R}; \dots; q_{t, n_{h}}^{R}] = q_{t}^{R} & = RoPE (W^{Q R} c_{t}^{Q}), \\ q_{t, i} & = [q_{t, i}^{C}; q_{t, i}^{R}], \end{aligned}

where

c_{t}^{Q} \in R^{d_{c}^{'}}

is the compressed latent vector for queries;

d_{c}^{'} (≪ d_{h} n_{h})

denotes the query compression dimension;

W^{D Q} \in R^{d_{c}^{'} \times d}, W^{U Q} \in R^{d_{h} n_{h} \times d_{c}^{d}}

are the down-projection and up-projection matrices for queries, respectively; and

W^{Q R} \in R^{d_{h}^{R} n_{h} \times d_{c}^{'}}

is the matrix to produce the decoupled queries that carry RoPE.
其中，

c_{t}^{Q} \in R^{d_{c}^{'}}

是查询的压缩潜向量；

d_{c}^{'} (≪ d_{h} n_{h})

表示查询压缩维度；

W^{D Q} \in R^{d_{c}^{'} \times d}, W^{U Q} \in R^{d_{h} n_{h} \times d_{c}^{d}}

分别是查询的向下投影矩阵和向上投影矩阵；

W^{Q R} \in R^{d_{h}^{R} n_{h} \times d_{c}^{'}}

是产生携带 RoPE 的解耦查询的矩阵。

Ultimately, the attention queries

(q_{t, i})

, keys

(k_{j, i})

, and values

(v_{j, i}^{C})

are combined to yield the final attention output

u_{t}

:
最终，将注意力查询

(q_{t, i})

、键

(k_{j, i})

和值

(v_{j, i}^{C})

合并，得出最终注意力输出

u_{t}

：

\begin{aligned} o_{t, i} & = \sum_{j = 1}^{t} {Softmax}_{j} (\frac{q_{t, i}^{T} k_{j, i}}{\sqrt{d_{h} + d_{h}^{R}}}) v_{j, i}^{C} \\ u_{t} & = W^{O} [o_{t, i}; o_{t, 2}; \dots; o_{t, n_{h}}], \end{aligned}

where

W^{O} \in R^{d \times d_{h} n_{h}}

denotes the output projection matrix.
其中

W^{O} \in R^{d \times d_{h} n_{h}}

表示输出投影矩阵。

2.1.2. DeepSeekMoE with Auxiliary-Loss-Free Load Balancing
2.1.2.带有无辅助负载平衡功能的 DeepSeekMoE

Basic Architecture of DeepSeekMoE. For Feed-Forward Networks (FFNs), DeepSeek-V3 employs the DeepSeekMoE architecture (Dai et al., 2024). Compared with traditional MoE architectures like GShard (Lepikhin et al., 2021), DeepSeekMoE uses finer-grained experts and isolates some experts as shared ones. Let

u_{t}

denote the FFN input of the

t

-th token, we compute the FFN output

h_{t}^{'}

as follows:
DeepSeekMoE 的基本架构。对于前馈网络（FFN），DeepSeek-V3采用了DeepSeekMoE架构（Dai等人，2024年）。与GShard（Lepikhin等人，2021年）等传统MoE架构相比，DeepSeekMoE使用了更细粒度的专家，并将一些专家隔离为共享专家。让

u_{t}

表示第

t

个标记的 FFN 输入，我们按以下方式计算 FFN 输出

h_{t}^{'}

：

\begin{aligned} h_{t}^{'} = u_{t} + \sum_{i = 1}^{N_{s}} {FFN}_{i}^{(s)} (u_{t}) + \sum_{i = 1}^{N_{r}} g_{i, t} {FFN}_{i}^{(r)} (u_{t}), \\ g_{i, t} = \frac{g_{i, t}^{'}}{\sum_{j = 1}^{N_{r}} g_{j, t}^{'}}, \\ g_{i, t}^{'} = {\begin{cases} s_{i, t}, & s_{i, t} \in Topk ({s_{j, t} ∣ 1 ⩽ j ⩽ N_{r}}, K_{r}), \\ 0, & otherwise, \end{cases} \\ s_{i, t} = Sigmoid (u_{t}^{T} e_{i}), \end{aligned}

where

N_{s}

and

N_{r}

denote the numbers of shared experts and routed experts, respectively;

{FFN}_{i}^{(s)} (\cdot)

and

{FFN}_{i}^{(r)} (\cdot)

denote the

i

-th shared expert and the

i

-th routed expert, respectively;

K_{r}

denotes the number of activated routed experts;

g_{i, t}

is the gating value for the

i

-th expert;

s_{i, t}

is the token-to-expert affinity;

e_{i}

is the centroid vector of the

i

-th routed expert; and

Topk (\cdot, K)

denotes the set comprising

K

highest scores among the affinity scores calculated for the

t

-th token and all routed experts. Slightly different from DeepSeek-V2, DeepSeek-V3 uses the sigmoid function to compute the affinity scores, and applies a normalization among all selected affinity scores to produce the gating values.
其中

N_{s}

和

N_{r}

分别表示共享专家和路由专家的数量；

{FFN}_{i}^{(s)} (\cdot)

和

{FFN}_{i}^{(r)} (\cdot)

分别表示第

i

个共享专家和第

i

个路由专家；

K_{r}

表示已激活的路由专家的数量；

g_{i, t}

是第

i

个专家的门控值；

s_{i, t}

是令牌与专家的亲和度；

e_{i}

是第

i

个路由专家的中心向量；

Topk (\cdot, K)

表示由第

t

个令牌和所有路由专家计算的亲和度分数中得分最高的

K

所组成的集合。与 DeepSeek-V2 略有不同的是，DeepSeek-V3 使用 sigmoid 函数计算亲和度分数，并对所有选定的亲和度分数进行归一化处理，以生成门控值。

Auxiliary-Loss-Free Load Balancing. For MoE models, an unbalanced expert load will lead to routing collapse (Shazeer et al., 2017) and diminish computational efficiency in scenarios with expert parallelism. Conventional solutions usually rely on the auxiliary loss (Fedus et al., 2021; Lepikhin et al., 2021) to avoid unbalanced load. However, too large an auxiliary loss will impair the model performance (Wang et al., 2024a). To achieve a better trade-off between load balance and model performance, we pioneer an auxiliary-loss-free load balancing strategy (Wang et al., 2024a) to ensure load balance. To be specific, we introduce a bias term

b_{i}

for each expert and add it to the corresponding affinity scores

s_{i, t}

to determine the top-K routing:
无辅助损失负载均衡。对于 MoE 模型，不平衡的专家负载会导致路由崩溃（Shazeer 等人，2017 年），并降低专家并行场景下的计算效率。传统的解决方案通常依靠辅助损耗（Fedus 等人，2021 年；Lepikhin 等人，2021 年）来避免不平衡负载。然而，过大的辅助损失会损害模型性能（Wang 等，2024a）。为了更好地权衡负载平衡和模型性能，我们开创了一种无辅助损失负载平衡策略（Wang 等，2024a），以确保负载平衡。具体来说，我们为每个专家引入了一个偏置项

b_{i}

，并将其添加到相应的亲和分数

s_{i, t}

中，以确定 top-K 路由：

g_{i, t}^{'} = {\begin{cases} s_{i, t}, & s_{i, t} + b_{i} \in Topk ({s_{j, t} + b_{j} ∣ 1 ⩽ j ⩽ N_{r}}, K_{r}) \\ 0, & otherwise. \end{cases}

Note that the bias term is only used for routing. The gating value, which will be multiplied with the FFN output, is still derived from the original affinity score

s_{i, t}

. During training, we keep monitoring the expert load on the whole batch of each training step. At the end of each step, we will decrease the bias term by

γ

if its corresponding expert is overloaded, and increase it by

γ

if its corresponding expert is underloaded, where

γ

is a hyper-parameter called bias update speed. Through the dynamic adjustment, DeepSeek-V3 keeps balanced expert load during training, and achieves better performance than models that encourage load balance through pure auxiliary losses.
请注意，偏置项仅用于路由。与 FFN 输出相乘的门控值仍来自原始亲和度得分

s_{i, t}

。在训练过程中，我们会持续监控每个训练步骤中整批专家的负载情况。在每一步结束时，如果对应的专家负荷过重，我们会减少偏置项

γ

；如果对应的专家负荷不足，我们会增加偏置项

γ

，其中

γ

是一个超参数，称为偏置更新速度。通过动态调整，DeepSeek-V3 可以在训练过程中保持均衡的专家负载，与通过纯辅助损失来鼓励负载均衡的模型相比，DeepSeek-V3 能取得更好的性能。

Complementary Sequence-Wise Auxiliary Loss. Although DeepSeek-V3 mainly relies on the auxiliary-loss-free strategy for load balance, to prevent extreme imbalance within any single sequence, we also employ a complementary sequence-wise balance loss:
互补序列辅助损失。虽然 DeepSeek-V3 主要依靠无辅助损失策略来实现负载平衡，但为了防止任何单一序列内出现极端不平衡，我们还采用了补充序列平衡损失：

\begin{matrix} L_{Bal} = α \sum_{i = 1}^{N_{r}} f_{i} P_{i}, \\ f_{i} = \frac{N_{r}}{K_{r} T} \sum_{t = 1}^{T} 1 (s_{i, t} \in Topk ({s_{j, t} ∣ 1 ⩽ j ⩽ N_{r}}, K_{r})), \\ s_{i, t}^{'} = \frac{s_{i, t}}{\sum_{j r = 1}^{N_{r}} s_{j, t}}, \\ P_{i} = \frac{1}{T} \sum_{t = 1}^{T} s_{i, t}^{'} \end{matrix}

where the balance factor

α

is a hyper-parameter, which will be assigned an extremely small value for DeepSeek-V3;

1 (\cdot)

denotes the indicator function; and

T

denotes the number of tokens in a sequence. The sequence-wise balance loss encourages the expert load on each sequence to be balanced.
其中，平衡因子

α

是一个超参数，对于 DeepSeek-V3 来说，它将被赋予一个极小的值；

1 (\cdot)

表示指标函数；

T

表示序列中的标记数。序列平衡损失鼓励平衡每个序列上的专家负载。

Figure 3 | Illustration of our Multi-Token Prediction (MTP) implementation. We keep the complete causal chain for the prediction of each token at each depth.
图 3：多标记预测 (MTP) 实现示意图。我们为每个标记在每个深度的预测保留了完整的因果链。

Node-Limited Routing. Like the device-limited routing used by DeepSeek-V2, DeepSeek-V3 also uses a restricted routing mechanism to limit communication costs during training. In short, we ensure that each token will be sent to at most

M

nodes, which are selected according to the sum of the highest

\frac{K_{r}}{M}

affinity scores of the experts distributed on each node. Under this constraint, our MoE training framework can nearly achieve full computation-communication overlap.
节点限制路由。与DeepSeek-V2使用的设备限制路由一样，DeepSeek-V3也使用限制路由机制来限制训练期间的通信成本。简而言之，我们确保每个令牌将最多发送到

M

个节点，这些节点是根据分布在每个节点上的专家的最高

\frac{K_{r}}{M}

亲和分数之和来选择的。在此约束条件下，我们的 MoE 训练框架几乎可以实现计算与通信的完全重叠。

No Token-Dropping. Due to the effective load balancing strategy, DeepSeek-V3 keeps a good load balance during its full training. Therefore, DeepSeek-V3 does not drop any tokens during training. In addition, we also implement specific deployment strategies to ensure inference load balance, so DeepSeek-V3 also does not drop tokens during inference.
不丢弃令牌。由于采用了有效的负载平衡策略，DeepSeek-V3 在整个训练过程中保持了良好的负载平衡。因此，DeepSeek-V3在训练过程中不会丢弃任何令牌。此外，我们还实施了特定的部署策略来确保推理负载平衡，因此DeepSeek-V3在推理过程中也不会丢弃令牌。

2.2. Multi-Token Prediction
2.2.多令牌预测

Inspired by Gloeckle et al. (2024), we investigate and set a Multi-Token Prediction (MTP) objective for DeepSeek-V3, which extends the prediction scope to multiple future tokens at each position. On the one hand, an MTP objective densifies the training signals and may improve data efficiency. On the other hand, MTP may enable the model to pre-plan its representations for better prediction of future tokens. Figure 3 illustrates our implementation of MTP. Different from Gloeckle et al. (2024), which parallelly predicts

D

additional tokens using independent output heads, we sequentially predict additional tokens and keep the complete causal chain at each prediction depth. We introduce the details of our MTP implementation in this section.
受 Gloeckle 等人（2024 年）的启发，我们研究并设定了 DeepSeek-V3 的多标记预测（MTP）目标，将预测范围扩展到每个位置的多个未来标记。一方面，MTP 目标可压缩训练信号，提高数据效率。另一方面，MTP 可以使模型预先规划其表示，从而更好地预测未来的标记。图 3 展示了我们的 MTP 实现。与 Gloeckle 等人（2024 年）使用独立的输出头并行预测

D

附加标记不同，我们按顺序预测附加标记，并在每个预测深度保留完整的因果链。本节将介绍我们的 MTP 实现细节。

MTP Modules. To be specific, our MTP implementation uses

D

sequential modules to predict

D

additional tokens. The

k

-th MTP module consists of a shared embedding layer Emb

(\cdot)

, a shared output head OutHead

(\cdot)

, a Transformer block

{TRM}_{k} (\cdot)

, and a projection matrix

M_{k} \in R^{d \times 2 d}

. For the

i

-th input token

t_{i}

, at the

k

-th prediction depth, we first combine the representation of the

i

-th token at the (

k - 1

)-th depth

h_{i}^{k - 1} \in R^{d}

and the embedding of the (

i + k

)-th token

E m b (t_{i + k}) \in R^{d}

MTP 模块。具体来说，我们的 MTP 实现使用

D

顺序模块来预测

D

附加标记。

k

-th MTP 模块由一个共享嵌入层 Emb

(\cdot)

、一个共享输出头 OutHead

(\cdot)

、一个变换器块

{TRM}_{k} (\cdot)

和一个投影矩阵

M_{k} \in R^{d \times 2 d}

组成。对于

i

-th 输入标记

t_{i}

，在

k

-th 预测深度，我们首先将

i

-th 标记在 (

k - 1

)-th 深度

h_{i}^{k - 1} \in R^{d}

的表示和 (

i + k

)-th 标记

E m b (t_{i + k}) \in R^{d}

的嵌入结合起来。
with the linear projection:
的线性投影：

h_{i}^{' k} = M_{k} [RMSNorm (h_{i}^{k - 1}); RMSNorm (Emb (t_{i + k}))],

where

[∵ \cdot]

denotes concatenation. Especially, when

k = 1, h_{i}^{k - 1}

refers to the representation given by the main model. Note that for each MTP module, its embedding layer is shared with the main model. The combined

h_{i}^{' k}

serves as the input of the Transformer block at the

k

-th depth to produce the output representation at the current depth

h_{i}^{k}

:
其中

[∵ \cdot]

表示连接。特别是当

k = 1, h_{i}^{k - 1}

指的是主模型给出的表示法时。请注意，每个 MTP 模块的嵌入层都与主模型共享。合并后的

h_{i}^{' k}

将作为

k

-th 深度的变换器模块的输入，生成当前深度

h_{i}^{k}

的输出表示：

h_{1 : T - k}^{k} = {TRM}_{k} (h_{1 : T - k}^{' k}),

where

T

represents the input sequence length and

_{i : j}

denotes the slicing operation (inclusive of both the left and right boundaries). Finally, taking

h_{i}^{k}

as the input, the shared output head will compute the probability distribution for the

k

-th additional prediction token

P_{i + 1 + k}^{k} \in R^{V}

, where

V

is the vocabulary size:
其中，

T

表示输入序列长度，

_{i : j}

表示切片操作（包括左边界和右边界）。最后，以

h_{i}^{k}

作为输入，共享输出头将计算

k

-th 附加预测标记

P_{i + 1 + k}^{k} \in R^{V}

的概率分布，其中

V

是词汇量大小：

P_{i + k + 1}^{k} = OutHead (h_{i}^{k}) .

The output head OutHead (

\cdot

) linearly maps the representation to logits and subsequently applies the Softmax (•) function to compute the prediction probabilities of the

k

-th additional token. Also, for each MTP module, its output head is shared with the main model. Our principle of maintaining the causal chain of predictions is similar to that of EAGLE (Li et al., 2024b), but its primary objective is speculative decoding (Leviathan et al., 2023: Xia et al., 2023), whereas we utilize MTP to improve training.
输出头 OutHead (

\cdot

) 将表示线性映射到对数，然后应用 Softmax (-) 函数计算

k

-th 额外标记的预测概率。此外，每个 MTP 模块的输出头都与主模型共享。我们保持预测因果链的原则与 EAGLE 类似（Li 等人，2024b），但其主要目的是推测解码（Leviathan 等人，2023：Xia 等人，2023），而我们利用 MTP 来改进训练。

MTP Training Objective. For each prediction depth, we compute a cross-entropy loss

L_{MTP}^{k}

:
MTP 训练目标。对于每个预测深度，我们都要计算交叉熵损失

L_{MTP}^{k}

：

L_{MTP}^{k} = CrossEntropy (P_{2 + k : T + 1}^{k}, t_{2 + k : T + 1}) = - \frac{1}{T} \sum_{i = 2 + k}^{T + 1} \log P_{i}^{k} [t_{i}],

where

T

denotes the input sequence length,

t_{i}

denotes the ground-truth token at the

i

-th position, and

P_{i}^{k} [t_{i}]

denotes the corresponding prediction probability of

t_{i}

, given by the

k

-th MTP module. Finally, we compute the average of the MTP losses across all depths and multiply it by a weighting factor

λ

to obtain the overall MTP loss

L_{MTP}

, which serves as an additional training objective for DeepSeek-V3:
其中，

T

表示输入序列长度，

t_{i}

表示第

i

-th 位置的地面实况标记，

P_{i}^{k} [t_{i}]

表示第

k

-th MTP 模块给出的

t_{i}

的相应预测概率。最后，我们计算所有深度的 MTP 损失的平均值，并乘以加权因子

λ

，得到总体 MTP 损失

L_{MTP}

，作为 DeepSeek-V3 的额外训练目标：

L_{MTP} = \frac{λ}{D} \sum_{k = 1}^{D} L_{MTP}^{k}

MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.
推理中的 MTP。我们的 MTP 策略主要是为了提高主模型的性能，因此在推理过程中，我们可以直接丢弃 MTP 模块，让主模型独立正常运行。此外，我们还可以将这些 MTP 模块重新用于推测解码，以进一步改善生成延迟。

3. Infrastructures 3.基础设施

3.1. Compute Clusters 3.1.计算集群

DeepSeek-V3 is trained on a cluster equipped with 2048 NVIDIA H800 GPUs. Each node in the H800 cluster contains 8 GPUs connected by NVLink and NVSwitch within nodes. Across different nodes, InfiniBand (IB) interconnects are utilized to facilitate communications.
DeepSeek-V3 在配备 2048 个英伟达 H800 GPU 的集群上进行训练。H800集群的每个节点包含8个GPU，通过节点内的NVLink和NVSwitch连接。不同节点之间利用 InfiniBand（IB）互连来促进通信。

Figure 4 | Overlapping strategy for a pair of individual forward and backward chunks (the boundaries of the transformer blocks are not aligned). Orange denotes forward, green denotes “backward for input”, blue denotes “backward for weights”, purple denotes PP communication, and red denotes barriers. Both all-to-all and PP communication can be fully hidden.
图 4：一对独立的前向和后向块的重叠策略（变压器块的边界未对齐）。橙色表示前向，绿色表示 "后向输入"，蓝色表示 "后向权重"，紫色表示 PP 通信，红色表示障碍。全对全和 PP 通信都可以完全隐藏。

3.2. Training Framework 3.2.培训框架

The training of DeepSeek-V3 is supported by the HAI-LLM framework, an efficient and lightweight training framework crafted by our engineers from the ground up. On the whole, DeepSeek-V3 applies 16-way Pipeline Parallelism (PP) (Qi et al., 2023a), 64-way Expert Parallelism (EP) (Lepikhin et al., 2021) spanning 8 nodes, and ZeRO-1 Data Parallelism (DP) (Rajbhandari et al., 2020).
DeepSeek-V3 的训练由 HAI-LLM 框架提供支持，这是一个由我们的工程师从头开始精心设计的高效、轻量级训练框架。总体而言，DeepSeek-V3 应用了 16 路管道并行（PP）（Qi 等人，2023a）、跨 8 个节点的 64 路专家并行（EP）（Lepikhin 等人，2021）和 ZeRO-1 数据并行（DP）（Rajbhandari 等人，2020）。

In order to facilitate efficient training of DeepSeek-V3, we implement meticulous engineering optimizations. Firstly, we design the DualPipe algorithm for efficient pipeline parallelism. Compared with existing PP methods, DualPipe has fewer pipeline bubbles. More importantly, it overlaps the computation and communication phases across forward and backward processes, thereby addressing the challenge of heavy communication overhead introduced by cross-node expert parallelism. Secondly, we develop efficient cross-node all-to-all communication kernels to fully utilize IB and NVLink bandwidths and conserve Streaming Multiprocessors (SMs) dedicated to communication. Finally, we meticulously optimize the memory footprint during training, thereby enabling us to train DeepSeek-V3 without using costly Tensor Parallelism (TP).
为了促进 DeepSeek-V3 的高效训练，我们进行了细致的工程优化。首先，我们设计了高效管线并行的 DualPipe 算法。与现有的PP方法相比，DualPipe的管道气泡更少。更重要的是，它在前向和后向进程中重叠了计算和通信阶段，从而解决了跨节点专家并行所带来的巨大通信开销的挑战。其次，我们开发了高效的跨节点全对全通信内核，以充分利用 IB 和 NVLink 带宽，并节省专用于通信的流式多处理器（SM）。最后，我们精心优化了训练过程中的内存占用，从而使我们能够在不使用昂贵的张量并行（TP）的情况下训练 DeepSeek-V3。

3.2.1. DualPipe and Computation-Communication Overlap
3.2.1.双管道和计算-通信重叠

For DeepSeek-V3, the communication overhead introduced by cross-node expert parallelism results in an inefficient computation-to-communication ratio of approximately 1:1. To tackle this challenge, we design an innovative pipeline parallelism algorithm called DualPipe, which not only accelerates model training by effectively overlapping forward and backward computationcommunication phases, but also reduces the pipeline bubbles.
对于 DeepSeek-V3，跨节点专家并行所带来的通信开销导致计算与通信比约为 1:1，效率低下。为了应对这一挑战，我们设计了一种名为 DualPipe 的创新流水线并行算法，它不仅能通过有效重叠前向和后向计算通信阶段来加速模型训练，还能减少流水线气泡。

The key idea of DualPipe is to overlap the computation and communication within a pair of individual forward and backward chunks. To be specific, we divide each chunk into four components: attention, all-to-all dispatch, MLP, and all-to-all combine. Specially, for a backward chunk, both attention and MLP are further split into two parts, backward for input and backward for weights, like in ZeroBubble (Qi et al., 2023b). In addition, we have a PP communication component. As illustrated in Figure 4 for a pair of forward and backward chunks, we rearrange these components and manually adjust the ratio of GPU SMs dedicated to communication versus computation. In this overlapping strategy, we can ensure that both all-to-all and PP communication can be fully hidden during execution. Given the efficient overlapping strategy, the full DualPipe scheduling is illustrated in Figure5. It employs a bidirectional pipeline scheduling, which feeds micro-batches from both ends of the pipeline simultaneously and a significant portion of communications can be fully overlapped. This overlap also ensures that, as the model further scales up, as long as we maintain a constant computation-to-communication ratio, we can still employ fine-grained experts across nodes while achieving a near-zero all-to-all communication overhead.
DualPipe 的主要理念是在一对独立的前向和后向分块中重叠计算和通信。具体来说，我们将每个分块分为四个部分：注意力、全对全调度、MLP 和全对全组合。特别是，对于后向分块，注意力和 MLP 都被进一步拆分为两部分，后向输入和后向权重，就像在 ZeroBubble 中一样（Qi 等人，2023b）。此外，我们还有一个 PP 通信组件。如图 4 中一对前向和后向块所示，我们重新排列这些组件，并手动调整专用于通信和计算的 GPU SM 的比例。通过这种重叠策略，我们可以确保在执行过程中完全隐藏全对全通信和PP通信。考虑到高效的重叠策略，完整的双管道调度如图 5 所示。它采用双向流水线调度，从流水线两端同时馈送微批次，相当一部分通信可以完全重叠。这种重叠还能确保在模型进一步扩展时，只要我们保持恒定的计算与通信比率，就仍能跨节点使用细粒度专家，同时实现接近零的全对全通信开销。

Figure 5 | Example DualPipe scheduling for 8 PP ranks and 20 micro-batches in two directions. The micro-batches in the reverse direction are symmetric to those in the forward direction, so we omit their batch ID for illustration simplicity. Two cells enclosed by a shared black border have mutually overlapped computation and communication.
图 5：双管调度示例，8 个 PP 级和 20 个微批在两个方向上调度。反向的微批次与正向的微批次是对称的，因此为了便于说明，我们省略了它们的批次 ID。由黑色边框围成的两个单元的计算和通信相互重叠。

Method 方法	Bubble 气泡	Parameter 参数	Activation 激活
1F1B	$(P P - 1) (F + B)$	$1 \times$	$P P$
ZB1P	$(P P - 1) (F + B - 2 W)$	$1 \times$	$P P$
DualPipe (Ours) 双管（我们的）	$(\frac{P P}{2} - 1) (F & B + B - 3 W)$	$2 \times$	$P P + 1$

Table 2 | Comparison of pipeline bubbles and memory usage across different pipeline parallel methods.

F

denotes the execution time of a forward chunk,

B

denotes the execution time of a full backward chunk,

W

denotes the execution time of a “backward for weights” chunk, and

F & B

denotes the execution time of two mutually overlapped forward and backward chunks.
表 2 不同流水线并行方法的流水线气泡和内存使用情况比较

F

表示前向分块的执行时间，

B

表示完整后向分块的执行时间，

W

表示 "权重后向 "分块的执行时间，

F & B

表示两个相互重叠的前向和后向分块的执行时间。

In addition, even in more general scenarios without a heavy communication burden, DualPipe still exhibits efficiency advantages. In Table 2, we summarize the pipeline bubbles and memory usage across different PP methods. As shown in the table, compared with ZB1P (Qi) et al., 2023b) and 1F1B (Harlap et al., 2018), DualPipe significantly reduces the pipeline bubbles while only increasing the peak activation memory by

\frac{1}{P P}

times. Although DualPipe requires keeping two copies of the model parameters, this does not significantly increase the memory consumption since we use a large EP size during training. Compared with Chimera (Li and Hoefler, 2021), DualPipe only requires that the pipeline stages and micro-batches be divisible by 2, without requiring micro-batches to be divisible by pipeline stages. In addition, for DualPipe, neither the bubbles nor activation memory will increase as the number of micro-batches grows.
此外，即使在没有沉重通信负担的一般情况下，DualPipe 仍然具有效率优势。在表 2 中，我们总结了不同密钥方法的管道气泡和内存使用情况。如表所示，与 ZB1P（Qi）等人，2023b）和 1F1B（Harlap 等人，2018）相比，DualPipe 显著减少了管道气泡，而峰值激活内存只增加了

\frac{1}{P P}

倍。虽然 DualPipe 需要保留两份模型参数，但由于我们在训练过程中使用了较大的 EP 大小，这并不会显著增加内存消耗。与 Chimera（Li 和 Hoefler，2021 年）相比，DualPipe 只要求流水线阶段和微批次能被 2 整除，而不要求微批次能被流水线阶段整除。此外，对于 DualPipe，气泡和激活内存都不会随着微批次数量的增加而增加。

3.2.2. Efficient Implementation of Cross-Node All-to-All Communication
3.2.2.跨节点全对全通信的高效实现

In order to ensure sufficient computational performance for DualPipe, we customize efficient cross-node all-to-all communication kernels (including dispatching and combining) to conserve the number of SMs dedicated to communication. The implementation of the kernels is codesigned with the MoE gating algorithm and the network topology of our cluster. To be specific, in our cluster, cross-node GPUs are fully interconnected with IB, and intra-node communications are handled via NVLink. NVLink offers a bandwidth of

160 GB / s

, roughly 3.2 times that of IB (

50 GB / s

). To effectively leverage the different bandwidths of IB and NVLink, we limit each token to be dispatched to at most 4 nodes, thereby reducing IB traffic. For each token, when its routing decision is made, it will first be transmitted via IB to the GPUs with the same in-node index on its target nodes. Once it reaches the target nodes, we will endeavor to ensure that it is instantaneously forwarded via NVLink to specific GPUs that host their target experts, without being blocked by subsequently arriving tokens. In this way, communications via IB and NVLink are fully overlapped, and each token can efficiently select an average of 3.2 experts per node without incurring additional overhead from NVLink. This implies that, although DeepSeek-V3
为了确保 DualPipe 有足够的计算性能，我们定制了高效的跨节点全对全通信内核（包括调度和组合），以节省专用于通信的 SM 数量。内核的实现是根据 MoE 门控算法和我们集群的网络拓扑结构进行编码设计的。具体来说，在我们的集群中，跨节点 GPU 完全通过 IB 互联，节点内通信则通过 NVLink 处理。NVLink 的带宽为

160 GB / s

，大约是 IB 的 3.2 倍（

50 GB / s

）。为了有效利用 IB 和 NVLink 的不同带宽，我们限制每个令牌最多向 4 个节点发送，从而减少 IB 流量。对于每个令牌，在做出路由决定后，它将首先通过 IB 传输到目标节点上具有相同节点内索引的 GPU。一旦到达目标节点，我们将努力确保通过 NVLink 将其即时转发给承载其目标专家的特定 GPU，而不会被随后到达的令牌阻塞。这样，通过 IB 和 NVLink 进行的通信将完全重叠，每个令牌平均可为每个节点有效选择 3.2 名专家，而不会产生 NVLink 的额外开销。这意味着，尽管 DeepSeek-V3
selects only 8 routed experts in practice, it can scale up this number to a maximum of 13 experts ( 4 nodes

\times 3.2

experts/node) while preserving the same communication cost. Overall, under such a communication strategy, only 20 SMs are sufficient to fully utilize the bandwidths of IB and NVLink.
在实际应用中，我们只能选择 8 个路由专家，但在通信成本不变的情况下，我们最多可以将专家数量增加到 13 个（4 个节点

\times 3.2

专家/节点）。总体而言，在这种通信策略下，只需 20 个 SM 就足以充分利用 IB 和 NVLink 的带宽。

In detail, we employ the warp specialization technique (Bauer et al. 2014) and partition 20 SMs into 10 communication channels. During the dispatching process, (1) IB sending, (2) IB-to-NVLink forwarding, and (3) NVLink receiving are handled by respective warps. The number of warps allocated to each communication task is dynamically adjusted according to the actual workload across all SMs. Similarly, during the combining process, (1) NVLink sending, (2) NVLink-to-IB forwarding and accumulation, and (3) IB receiving and accumulation are also handled by dynamically adjusted warps. In addition, both dispatching and combining kernels overlap with the computation stream, so we also consider their impact on other SM computation kernels. Specifically, we employ customized PTX (Parallel Thread Execution) instructions and auto-tune the communication chunk size, which significantly reduces the use of the L2 cache and the interference to other SMs.
具体来说，我们采用了 warp 专业化技术（Bauer 等人，2014 年），将 20 个 SM 划分为 10 个通信通道。在调度过程中，(1) IB 发送、(2) IB 到 NVLink 转发和 (3) NVLink 接收分别由各自的 warp 处理。根据所有 SM 的实际工作量，动态调整分配给每个通信任务的 warps 数量。同样，在合并过程中，(1) NVLink 发送、(2) NVLink 到 IB 的转发和累积以及 (3) IB 接收和累积也由动态调整的 warps 处理。此外，调度和组合内核都与计算流重叠，因此我们还要考虑它们对其他 SM 计算内核的影响。具体来说，我们采用了定制的 PTX（并行线程执行）指令，并自动调整了通信分块大小，从而大大减少了对二级缓存的使用和对其他 SM 的干扰。

3.2.3. Extremely Memory Saving with Minimal Overhead
3.2.3.以最小的开销极大地节省内存

In order to reduce the memory footprint during training, we employ the following techniques.
为了减少训练过程中的内存占用，我们采用了以下技术。

Recomputation of RMSNorm and MLA Up-Projection. We recompute all RMSNorm operations and MLA up-projections during back-propagation, thereby eliminating the need to persistently store their output activations. With a minor overhead, this strategy significantly reduces memory requirements for storing activations.
重新计算 RMSNorm 和 MLA 向上投影。我们在反向传播过程中重新计算所有 RMSNorm 运算和 MLA 向上投影，因此无需持续存储其输出激活。这种策略只需少量开销，就能大大降低存储激活值所需的内存。

Exponential Moving Average in CPU. During training, we preserve the Exponential Moving Average (EMA) of the model parameters for early estimation of the model performance after learning rate decay. The EMA parameters are stored in CPU memory and are updated asynchronously after each training step. This method allows us to maintain EMA parameters without incurring additional memory or time overhead.
CPU 中的指数移动平均值。在训练过程中，我们会保留模型参数的指数移动平均值（EMA），以便在学习率衰减后及早估计模型性能。EMA 参数存储在 CPU 内存中，并在每个训练步骤后异步更新。通过这种方法，我们可以保留 EMA 参数，而不会产生额外的内存或时间开销。

Shared Embedding and Output Head for Multi-Token Prediction. With the DualPipe strategy, we deploy the shallowest layers (including the embedding layer) and deepest layers (including the output head) of the model on the same PP rank. This arrangement enables the physical sharing of parameters and gradients, of the shared embedding and output head, between the MTP module and the main model. This physical sharing mechanism further enhances our memory efficiency.
共享嵌入和输出头，实现多词组预测。利用双管道策略，我们将模型的最浅层（包括嵌入层）和最深层（包括输出头）部署在同一PP等级上。这种安排使得 MTP 模块和主模型之间能够共享嵌入层和输出头的参数和梯度。这种物理共享机制进一步提高了我们的内存效率。

3.3. FP8 Training 3.3.FP8 培训

Inspired by recent advances in low-precision training (Dettmers et al., 2022, Noune et al., 2022, Peng et al., 2023b), we propose a fine-grained mixed precision framework utilizing the FP8 data format for training DeepSeek-V3. While low-precision training holds great promise, it is often limited by the presence of outliers in activations, weights, and gradients (Fishman et al., 2024; He et al.; Sun et al., 2024). Although significant progress has been made in inference quantization (Frantar et al., 2022; Xiao et al., 2023), there are relatively few studies demonstrating successful application of low-precision techniques in large-scale language model
受低精度训练最新进展的启发（Dettmers等人，2022年；Noune等人，2022年；Peng等人，2023年b），我们提出了一种细粒度混合精度框架，利用FP8数据格式训练DeepSeek-V3。虽然低精度训练大有可为，但它往往受限于激活、权重和梯度中异常值的存在（Fishman 等，2024；He 等；Sun 等，2024）。虽然在推理量化方面取得了重大进展（Frantar 等人，2022 年；肖等人，2023 年），但在大规模语言模型中成功应用低精度技术的研究相对较少。

Figure 6 | The overall mixed precision framework with FP8 data format. For clarification, only the Linear operator is illustrated.
图 6：采用 FP8 数据格式的整体混合精度框架。为便于说明，仅对线性运算符进行了说明。
pre-training (Fishman et al. 2024). To address this challenge and effectively extend the dynamic range of the FP8 format, we introduce a fine-grained quantization strategy: tile-wise grouping with

1 \times N_{c}

elements or block-wise grouping with

N_{c} \times N_{c}

elements. The associated dequantization overhead is largely mitigated under our increased-precision accumulation process, a critical aspect for achieving accurate FP8 General Matrix Multiplication (GEMM). Moreover, to further reduce memory and communication overhead in MoE training, we cache and dispatch activations in FP8, while storing low-precision optimizer states in BF16. We validate the proposed FP8 mixed precision framework on two model scales similar to DeepSeek-V2-Lite and DeepSeekV2, training for approximately 1 trillion tokens (see more details in Appendix B.1. Notably, compared with the BF16 baseline, the relative loss error of our FP8-training model remains consistently below

0.25 %

, a level well within the acceptable range of training randomness.
预训练（Fishman 等人，2024 年）。为了应对这一挑战并有效扩展 FP8 格式的动态范围，我们引入了细粒度量化策略：使用

1 \times N_{c}

元素进行瓦片分组，或使用

N_{c} \times N_{c}

元素进行块分组。在我们提高精度的累加过程中，相关的去量化开销在很大程度上得到了缓解，而这正是实现精确的 FP8 通用矩阵乘法 (GEMM) 的关键所在。此外，为了进一步减少 MoE 训练中的内存和通信开销，我们在 FP8 中缓存和分派激活，同时在 BF16 中存储低精度优化器状态。我们在两个与 DeepSeek-V2-Lite 和 DeepSeekV2 类似的模型规模上验证了所提出的 FP8 混合精度框架，训练量约为 1 万亿个代币（详情见附录 B.1）。值得注意的是，与 BF16 基线相比，我们的 FP8 训练模型的相对损失误差始终保持在

0.25 %

以下，这一水平远在训练随机性的可接受范围之内。

3.3.1. Mixed Precision Framework
3.3.1.混合精度框架

Building upon widely adopted techniques in low-precision training (Kalamkar et al., 2019; Narang et al., 2017), we propose a mixed precision framework for FP8 training. In this framework, most compute-density operations are conducted in FP8, while a few key operations are strategically maintained in their original data formats to balance training efficiency and numerical stability. The overall framework is illustrated in Figure 6 .
在广泛采用的低精度训练技术（Kalamkar 等人，2019；Narang 等人，2017）基础上，我们提出了 FP8 训练的混合精度框架。在这个框架中，大多数计算密集型操作都在 FP8 中进行，而少数关键操作则战略性地保持原始数据格式，以平衡训练效率和数值稳定性。整体框架如图 6 所示。

Firstly, in order to accelerate model training, the majority of core computation kernels, i.e., GEMM operations, are implemented in FP8 precision. These GEMM operations accept FP8 tensors as inputs and produce outputs in BF16 or FP32. As depicted in Figure6, all three GEMMs associated with the Linear operator, namely Fprop (forward pass), Dgrad (activation backward pass), and Wgrad (weight backward pass), are executed in FP8. This design theoretically doubles the computational speed compared with the original BF16 method. Additionally, the FP8 Wgrad GEMM allows activations to be stored in FP8 for use in the backward pass. This significantly reduces memory consumption.
首先，为了加速模型训练，大部分核心计算内核，即 GEMM 运算，都是以 FP8 精度实现的。这些 GEMM 运算接受 FP8 张量作为输入，并以 BF16 或 FP32 输出。如图 6 所示，与线性算子相关的所有三个 GEMM，即 Fprop（前向传递）、Dgrad（激活后向传递）和 Wgrad（权重后向传递），均以 FP8 执行。与原始的 BF16 方法相比，这种设计在理论上将计算速度提高了一倍。此外，FP8 Wgrad GEMM 允许将激活存储在 FP8 中，以便在后向传递中使用。这大大减少了内存消耗。

Despite the efficiency advantage of the FP8 format, certain operators still require a higher precision due to their sensitivity to low-precision computations. Besides, some low-cost operators can also utilize a higher precision with a negligible overhead to the overall training cost. For this reason, after careful investigations, we maintain the original precision (e.g., BF16 or FP32) for the following components: the embedding module, the output head, MoE gating modules, normalization operators, and attention operators. These targeted retentions of high precision ensure stable training dynamics for DeepSeek-V3. To further guarantee numerical stability, we store the master weights, weight gradients, and optimizer states in higher precision. While
尽管 FP8 格式具有效率优势，但由于某些运算符对低精度计算的敏感性，它们仍然需要更高的精度。此外，一些低成本运算符也可以使用更高的精度，而且对总体训练成本的影响可以忽略不计。因此，经过仔细研究，我们为以下组件保留了原始精度（如 BF16 或 FP32）：嵌入模块、输出头、MoE 门控模块、归一化算子和注意力算子。这些有针对性的高精度保留确保了 DeepSeek-V3 稳定的训练动态。为了进一步保证数值稳定性，我们以更高的精度存储主权重、权重梯度和优化器状态。同时

Figure 7 | (a) We propose a fine-grained quantization method to mitigate quantization errors caused by feature outliers; for illustration simplicity, only Fprop is illustrated. (b) In conjunction with our quantization strategy, we improve the FP8 GEMM precision by promoting to CUDA Cores at an interval of

N_{C} = 128

elements MMA for the high-precision accumulation.
图 7 | (a) 我们提出了一种细粒度量化方法，以减少由特征异常值引起的量化误差；为便于说明，仅对 Fprop 进行了说明。(b) 结合我们的量化策略，我们以

N_{C} = 128

元素 MMA 的间隔提升至 CUDA 内核，从而提高 FP8 GEMM 的精度，实现高精度累积。
these high-precision components incur some memory overheads, their impact can be minimized through efficient sharding across multiple DP ranks in our distributed training system.
这些高精度组件会产生一定的内存开销，但在我们的分布式训练系统中，通过在多个 DP 级之间高效分片，可以将它们的影响降至最低。

3.3.2. Improved Precision from Quantization and Multiplication
3.3.2.通过量化和乘法提高精度

Based on our mixed precision FP8 framework, we introduce several strategies to enhance lowprecision training accuracy, focusing on both the quantization method and the multiplication process.
在混合精度 FP8 框架的基础上，我们介绍了几种提高低精度训练精度的策略，重点关注量化方法和乘法过程。

Fine-Grained Quantization. In low-precision training frameworks, overflows and underflows are common challenges due to the limited dynamic range of the FP8 format, which is constrained by its reduced exponent bits. As a standard practice, the input distribution is aligned to the representable range of the FP8 format by scaling the maximum absolute value of the input tensor to the maximum representable value of FP8 (Narang et al., 2017). This method makes lowprecision training highly sensitive to activation outliers, which can heavily degrade quantization accuracy. To solve this, we propose a fine-grained quantization method that applies scaling at a more granular level. As illustrated in Figure 7 (a), (1) for activations, we group and scale elements on a

1 \times 128

tile basis (i.e., per token per 128 channels); and (2) for weights, we group and scale elements on a

128 \times 128

block basis (i.e., per 128 input channels per 128 output channels). This approach ensures that the quantization process can better accommodate outliers by adapting the scale according to smaller groups of elements. In Appendix B.2, we further discuss the training instability when we group and scale activations on a block basis in the same way as weights quantization.
精细量化。在低精度训练框架中，由于 FP8 格式的动态范围有限，受其指数位数减少的限制，溢出和欠溢是常见的挑战。作为一种标准做法，通过将输入张量的最大绝对值缩放为 FP8 的最大可表示值，将输入分布调整为 FP8 格式的可表示范围（Narang 等人，2017 年）。这种方法使得低精度训练对激活异常值高度敏感，这会严重降低量化精度。为了解决这个问题，我们提出了一种细粒度量化方法，在更细的层次上应用缩放。如图 7 (a) 所示，(1) 对于激活，我们以

1 \times 128

瓦为基础对元素进行分组和缩放（即每 128 个通道中的每个标记）；(2) 对于权重，我们以

128 \times 128

块为基础对元素进行分组和缩放（即每 128 个输入通道和 128 个输出通道）。这种方法可确保量化过程根据较小的元素组调整比例，从而更好地适应异常值。在附录 B.2 中，我们将进一步讨论以块为单位分组和缩放激活时的训练不稳定性。

One key modification in our method is the introduction of per-group scaling factors along the inner dimension of GEMM operations. This functionality is not directly supported in the standard FP8 GEMM. However, combined with our precise FP32 accumulation strategy, it can
我们方法中的一个关键修改是在 GEMM 操作的内维度上引入了每组缩放因子。标准 FP8 GEMM 并不直接支持这一功能。不过，结合我们精确的 FP32 累加策略，它可以
be efficiently implemented.
有效实施。
Notably, our fine-grained quantization strategy is highly consistent with the idea of microscaling formats (Rouhani et al., 2023b), while the Tensor Cores of NVIDIA next-generation GPUs (Blackwell series) have announced the support for microscaling formats with smaller quantization granularity (NVIDIA, 2024a). We hope our design can serve as a reference for future work to keep pace with the latest GPU architectures.
值得注意的是，我们的细粒度量化策略与微缩格式的理念高度一致（Rouhani 等人，2023b），而英伟达下一代 GPU（Blackwell 系列）的张量核心已宣布支持量化粒度更小的微缩格式（英伟达，2024a）。我们希望我们的设计能为未来的工作提供参考，以跟上最新 GPU 架构的步伐。

Increasing Accumulation Precision. Low-precision GEMM operations often suffer from underflow issues, and their accuracy largely depends on high-precision accumulation, which is commonly performed in an FP32 precision (Kalamkar et al., 2019; Narang et al., 2017). However, we observe that the accumulation precision of FP8 GEMM on NVIDIA H800 GPUs is limited to retaining around 14 bits, which is significantly lower than FP32 accumulation precision. This problem will become more pronounced when the inner dimension K is large (Wortsman et al., 2023), a typical scenario in large-scale model training where the batch size and model width are increased. Taking GEMM operations of two random matrices with

K = 4096

for example, in our preliminary test, the limited accumulation precision in Tensor Cores results in a maximum relative error of nearly

2 %

. Despite these problems, the limited accumulation precision is still the default option in a few FP8 frameworks (NVIDIA, 2024b), severely constraining the training accuracy.
提高累加精度。低精度 GEMM 运算经常出现下溢问题，其精度在很大程度上取决于高精度累加，而高精度累加通常以 FP32 精度执行（Kalamkar 等人，2019 年；Narang 等人，2017 年）。然而，我们观察到，英伟达 H800 GPU 上 FP8 GEMM 的累加精度仅限于保留 14 位左右，明显低于 FP32 累加精度。当内维度 K 较大时，这个问题会变得更加突出（Wortsman 等人，2023 年），这也是大规模模型训练中批量大小和模型宽度增加的典型情况。以

K = 4096

两个随机矩阵的 GEMM 运算为例，在我们的初步测试中，Tensor Cores 中有限的累积精度导致最大相对误差接近

2 %

。尽管存在这些问题，有限累积精度仍然是一些 FP8 框架的默认选项（英伟达，2024b），严重限制了训练精度。

In order to address this issue, we adopt the strategy of promotion to CUDA Cores for higher precision (Thakkar et al. 2023). The process is illustrated in Figure 7(b). To be specific, during MMA (Matrix Multiply-Accumulate) execution on Tensor Cores, intermediate results are accumulated using the limited bit width. Once an interval of

N_{C}

is reached, these partial results will be copied to FP32 registers on CUDA Cores, where full-precision FP32 accumulation is performed. As mentioned before, our fine-grained quantization applies per-group scaling factors along the inner dimension K . These scaling factors can be efficiently multiplied on the CUDA Cores as the dequantization process with minimal additional computational cost.
为了解决这个问题，我们采用了向 CUDA 内核升级以获得更高精度的策略（Thakkar 等人，2023 年）。图 7(b) 展示了这一过程。具体来说，在张量核心上执行 MMA（矩阵乘积）期间，中间结果会利用有限的位宽进行累积。一旦达到

N_{C}

的间隔，这些部分结果将被复制到 CUDA 内核上的 FP32 寄存器，并在那里执行全精度 FP32 累加。如前所述，我们的细粒度量化技术沿内维度 K 应用了每组缩放因子。这些缩放因子可以在 CUDA 内核上高效地相乘，作为去量化过程，只需最小的额外计算成本。

It is worth noting that this modification reduces the WGMMA (Warpgroup-level Matrix Multiply-Accumulate) instruction issue rate for a single warpgroup. However, on the H800 architecture, it is typical for two WGMMA to persist concurrently: while one warpgroup performs the promotion operation, the other is able to execute the MMA operation. This design enables overlapping of the two operations, maintaining high utilization of Tensor Cores. Based on our experiments, setting

N_{C} = 128

elements, equivalent to 4 WGMMAs, represents the minimal accumulation interval that can significantly improve precision without introducing substantial overhead.
值得注意的是，这种修改降低了单个处理器组的 WGMMA（处理器组级矩阵乘积）指令发布率。不过，在 H800 架构上，两个 WGMMA 同时存在是很典型的情况：当一个程序集群执行晋升操作时，另一个程序集群可以执行 MMA 操作。这种设计可以使两个操作重叠，保持张量核心的高利用率。根据我们的实验，设置

N_{C} = 128

元素（相当于 4 个 WGMMA）代表了最小的累积间隔，可显著提高精度，同时不会带来大量开销。

Mantissa over Exponents. In contrast to the hybrid FP8 format adopted by prior work (NVIDIA, 2024b; Peng et al., 2023b; Sun et al., 2019b), which uses E4M3 (4-bit exponent and 3-bit mantissa) in Fprop and E5M2 (5-bit exponent and 2-bit mantissa) in Dgrad and Wgrad, we adopt the E4M3 format on all tensors for higher precision. We attribute the feasibility of this approach to our fine-grained quantization strategy, i.e., tile and block-wise scaling. By operating on smaller element groups, our methodology effectively shares exponent bits among these grouped elements, mitigating the impact of the limited dynamic range.
尾数大于指数。与之前的工作（英伟达，2024b；彭等人，2023b；孙等人，2019b）所采用的混合 FP8 格式（在 Fprop 中使用 E4M3（4 位指数和 3 位尾数），在 Dgrad 和 Wgrad 中使用 E5M2（5 位指数和 2 位尾数））不同，我们在所有张量上都采用 E4M3 格式，以获得更高精度。我们将这种方法的可行性归功于我们的细粒度量化策略，即平铺和分块缩放。通过对较小的元素组进行操作，我们的方法可以有效地在这些分组元素之间共享指数位，从而减轻有限动态范围的影响。

Online Quantization. Delayed quantization is employed in tensor-wise quantization frameworks (NVIDIA, 2024b; Peng et al., 2023b), which maintains a history of the maximum absolute
在线量化。张量式量化框架（NVIDIA，2024b；Peng 等人，2023b）中采用了延迟量化技术，该技术可保持最大绝对值的历史记录。
values across prior iterations to infer the current value. In order to ensure accurate scales and simplify the framework, we calculate the maximum absolute value online for each

1 \times 128

activation tile or

128 \times 128

weight block. Based on it, we derive the scaling factor and then quantize the activation or weight online into the FP8 format.
值来推断当前值。为了确保精确的比例并简化框架，我们会在线计算每个

1 \times 128

激活瓦片或

128 \times 128

权重块的最大绝对值。在此基础上，我们得出缩放因子，然后将激活或权重在线量化为 FP8 格式。

3.3.3. Low-Precision Storage and Communication
3.3.3.低精度存储和通信

In conjunction with our FP8 training framework, we further reduce the memory consumption and communication overhead by compressing cached activations and optimizer states into lower-precision formats.
结合 FP8 训练框架，我们将缓存的激活和优化器状态压缩为低精度格式，从而进一步降低了内存消耗和通信开销。

Low-Precision Optimizer States. We adopt the BF16 data format instead of FP32 to track the first and second moments in the AdamW (Loshchilov and Hutter, 2017) optimizer, without incurring observable performance degradation. However, the master weights (stored by the optimizer) and gradients (used for batch size accumulation) are still retained in FP32 to ensure numerical stability throughout training.
低精度优化器状态。我们采用 BF16 数据格式，而不是 FP32 格式，以跟踪 AdamW（Loshchilov 和 Hutter，2017 年）优化器中的第一和第二矩，而不会导致可观察到的性能下降。不过，主权重（由优化器存储）和梯度（用于批量大小累积）仍保留在 FP32 中，以确保整个训练过程中的数值稳定性。

Low-Precision Activation. As illustrated in Figure6 the Wgrad operation is performed in FP8. To reduce the memory consumption, it is a natural choice to cache activations in FP8 format for the backward pass of the Linear operator. However, special considerations are taken on several operators for low-cost high-precision training:
低精度激活。如图 6 所示，Wgrad 运算以 FP8 格式执行。为了减少内存消耗，在线性运算器的后向传递中以 FP8 格式缓存激活是一个自然的选择。不过，为了进行低成本高精度训练，我们对几个运算符进行了特别考虑：
(1) Inputs of the Linear after the attention operator. These activations are also used in the backward pass of the attention operator, which makes it sensitive to precision. We adopt a customized E5M6 data format exclusively for these activations. Additionally, these activations will be converted from an

1 \times 128

quantization tile to an

128 \times 1

tile in the backward pass. To avoid introducing extra quantization error, all the scaling factors are round scaled, i.e., integral power of 2.
(1) 在注意力运算器之后的线性输入。这些激活也用于注意力算子的后向传递，因此对精度非常敏感。我们为这些激活值专门定制了 E5M6 数据格式。此外，在后向传递中，这些激活将从

1 \times 128

量化瓦片转换为

128 \times 1

瓦片。为了避免引入额外的量化误差，所有的缩放因子都是整数缩放，即 2 的整数次幂。
(2) Inputs of the SwiGLU operator in MoE. To further reduce the memory cost, we cache the inputs of the SwiGLU operator and recompute its output in the backward pass. These activations are also stored in FP8 with our fine-grained quantization method, striking a balance between memory efficiency and computational accuracy.
(2) MoE 中 SwiGLU 算子的输入。为了进一步降低内存成本，我们缓存了 SwiGLU 算子的输入，并在后向传递中重新计算其输出。这些激活也通过我们的细粒度量化方法存储在 FP8 中，从而在内存效率和计算精度之间取得平衡。

Low-Precision Communication. Communication bandwidth is a critical bottleneck in the training of MoE models. To alleviate this challenge, we quantize the activation before MoE up-projections into FP8 and then apply dispatch components, which is compatible with FP8 Fprop in MoE up-projections. Like the inputs of the Linear after the attention operator, scaling factors for this activation are integral power of 2 . A similar strategy is applied to the activation gradient before MoE down-projections. For both the forward and backward combine components, we retain them in BF16 to preserve training precision in critical parts of the training pipeline.
低精度通信。通信带宽是 MoE 模型训练中的一个关键瓶颈。为了缓解这一难题，我们将 MoE 向上投影前的激活量化为 FP8，然后应用调度组件，这与 MoE 向上投影中的 FP8 Fprop 兼容。与注意力算子之后的线性输入一样，该激活的缩放因子也是 2 的整数次方。类似的策略也适用于 MoE 向下投影前的激活梯度。我们在 BF16 中保留了前向和后向组合成分，以保持训练管道关键部分的训练精度。

3.4. Inference and Deployment
3.4.推理和部署

We deploy DeepSeek-V3 on the H800 cluster, where GPUs within each node are interconnected using NVLink, and all GPUs across the cluster are fully interconnected via IB. To simultaneously ensure both the Service-Level Objective (SLO) for online services and high throughput, we employ the following deployment strategy that separates the prefilling and decoding stages.
我们在H800集群上部署DeepSeek-V3，每个节点内的GPU通过NVLink互连，集群中的所有GPU通过IB完全互连。为了同时确保在线服务的服务级目标（SLO）和高吞吐量，我们采用了以下部署策略，将预填充和解码阶段分开。

3.4.1. Prefilling 3.4.1.预填充

The minimum deployment unit of the prefilling stage consists of 4 nodes with 32 GPUs. The attention part employs 4-way Tensor Parallelism (TP4) with Sequence Parallelism (SP), combined with 8-way Data Parallelism (DP8). Its small TP size of 4 limits the overhead of TP communication. For the MoE part, we use 32-way Expert Parallelism (EP32), which ensures that each expert processes a sufficiently large batch size, thereby enhancing computational efficiency. For the MoE all-to-all communication, we use the same method as in training: first transferring tokens across nodes via IB, and then forwarding among the intra-node GPUs via NVLink. In particular, we use 1-way Tensor Parallelism for the dense MLPs in shallow layers to save TP communication.
预填充阶段的最小部署单元由 4 个节点和 32 个 GPU 组成。注意力部分采用 4 路张量并行（TP4）和序列并行（SP），并结合 8 路数据并行（DP8）。其 4 路 TP 规模较小，限制了 TP 通信的开销。对于 MoE 部分，我们使用 32 路专家并行（EP32），确保每个专家处理足够大的批量，从而提高计算效率。对于 MoE 的全对全通信，我们采用了与训练相同的方法：首先通过 IB 跨节点传输令牌，然后通过 NVLink 在节点内 GPU 之间转发。特别是，我们对浅层的密集 MLP 使用单向张量并行，以节省 TP 通信。

To achieve load balancing among different experts in the MoE part, we need to ensure that each GPU processes approximately the same number of tokens. To this end, we introduce a deployment strategy of redundant experts, which duplicates high-load experts and deploys them redundantly. The high-load experts are detected based on statistics collected during the online deployment and are adjusted periodically (e.g., every 10 minutes). After determining the set of redundant experts, we carefully rearrange experts among GPUs within a node based on the observed loads, striving to balance the load across GPUs as much as possible without increasing the cross-node all-to-all communication overhead. For the deployment of DeepSeek-V3, we set 32 redundant experts for the prefilling stage. For each GPU, besides the original 8 experts it hosts, it will also host one additional redundant expert.
为了实现 MoE 部分不同专家之间的负载平衡，我们需要确保每个 GPU 处理的令牌数量大致相同。为此，我们引入了冗余专家部署策略，即复制高负载专家并进行冗余部署。高负载专家是根据在线部署期间收集的统计数据检测出来的，并定期进行调整（例如每 10 分钟一次）。在确定冗余专家集后，我们会根据观察到的负载在节点内的 GPU 之间仔细重新安排专家，力求在不增加跨节点全对全通信开销的情况下尽可能平衡 GPU 之间的负载。在部署DeepSeek-V3时，我们为预填充阶段设置了32个冗余专家。每个 GPU 除了承载原有的 8 个专家外，还将承载一个额外的冗余专家。

Furthermore, in the prefilling stage, to improve the throughput and hide the overhead of all-to-all and TP communication, we simultaneously process two micro-batches with similar computational workloads, overlapping the attention and MoE of one micro-batch with the dispatch and combine of another.
此外，在预填充阶段，为了提高吞吐量并隐藏全对全和 TP 通信的开销，我们同时处理两个计算工作量相似的微批次，将一个微批次的注意力和 MoE 与另一个微批次的调度和组合重叠。

Finally, we are exploring a dynamic redundancy strategy for experts, where each GPU hosts more experts (e.g., 16 experts), but only 9 will be activated during each inference step. Before the all-to-all operation at each layer begins, we compute the globally optimal routing scheme on the fly. Given the substantial computation involved in the prefilling stage, the overhead of computing this routing scheme is almost negligible.
最后，我们正在探索一种专家动态冗余策略，即每个 GPU 承载更多专家（例如 16 个专家），但在每个推理步骤中只激活 9 个专家。在每一层的全对全操作开始之前，我们会即时计算全局最优路由方案。鉴于预填充阶段涉及大量计算，计算该路由方案的开销几乎可以忽略不计。

3.4.2. Decoding 3.4.2.解码

During decoding, we treat the shared expert as a routed one. From this perspective, each token will select 9 experts during routing, where the shared expert is regarded as a heavy-load one that will always be selected. The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320. For the MoE part, each GPU hosts only one expert, and 64 GPUs are responsible for hosting redundant experts and shared experts. All-to-all communication of the dispatch and combine parts is performed via direct point-to-point transfers over IB to achieve low latency. Additionally, we leverage the IBGDA (NVIDIA, 2022) technology to further minimize latency and enhance communication efficiency.
在解码过程中，我们将共享专家视为路由专家。从这个角度看，每个令牌在路由过程中将选择 9 个专家，其中共享专家被视为重载专家，将始终被选择。解码阶段的最小部署单元由 40 个节点和 320 个 GPU 组成。注意力部分采用带 SP 的 TP4 和 DP80，而 MoE 部分采用 EP320。对于 MoE 部分，每个 GPU 只托管一个专家，64 个 GPU 负责托管冗余专家和共享专家。调度和组合部分的所有通信都通过 IB 直接点对点传输来实现低延迟。此外，我们还利用 IBGDA（英伟达，2022 年）技术进一步减少延迟并提高通信效率。

Similar to prefilling, we periodically determine the set of redundant experts in a certain interval, based on the statistical expert load from our online service. However, we do not need to rearrange experts since each GPU only hosts one expert. We are also exploring the dynamic redundancy strategy for decoding. However, this requires more careful optimization of the algorithm that computes the globally optimal routing scheme and the fusion with the dispatch kernel to reduce overhead.
与预填充类似，我们会根据在线服务统计的专家负载情况，在一定时间间隔内定期确定冗余专家集。不过，由于每个 GPU 只承载一个专家，因此我们不需要重新安排专家。我们还在探索解码的动态冗余策略。不过，这需要对计算全局最优路由方案的算法以及与调度内核的融合进行更细致的优化，以减少开销。

Additionally, to enhance throughput and hide the overhead of all-to-all communication, we are also exploring processing two micro-batches with similar computational workloads simultaneously in the decoding stage. Unlike prefilling, attention consumes a larger portion of time in the decoding stage. Therefore, we overlap the attention of one micro-batch with the dispatch+MoE+combine of another. In the decoding stage, the batch size per expert is relatively small (usually within 256 tokens), and the bottleneck is memory access rather than computation. Since the MoE part only needs to load the parameters of one expert, the memory access overhead is minimal, so using fewer SMs will not significantly affect the overall performance. Therefore, to avoid impacting the computation speed of the attention part, we can allocate only a small portion of SMs to dispatch+MoE+combine.
此外，为了提高吞吐量并隐藏全对全通信的开销，我们还探索在解码阶段同时处理两个计算工作量相似的微批次。与预填充不同，注意力在解码阶段消耗的时间更多。因此，我们将一个微批次的注意力与另一个微批次的调度+MoE+组合重叠。在解码阶段，每个专家的批量规模相对较小（通常在 256 个令牌以内），瓶颈在于内存访问而非计算。由于 MoE 部分只需要加载一个专家的参数，内存访问开销很小，因此使用较少的 SM 不会明显影响整体性能。因此，为了避免影响注意力部分的计算速度，我们可以只分配一小部分 SM 给 dispatch+MoE+combine 部分。

3.5. Suggestions on Hardware Design
3.5.硬件设计建议

Based on our implementation of the all-to-all communication and FP8 training scheme, we propose the following suggestions on chip design to AI hardware vendors.
基于我们实现的全对全通信和 FP8 训练方案，我们向人工智能硬件供应商提出了以下芯片设计建议。

3.5.1. Communication Hardware
3.5.1.通信硬件

In DeepSeek-V3, we implement the overlap between computation and communication to hide the communication latency during computation. This significantly reduces the dependency on communication bandwidth compared to serial computation and communication. However, the current communication implementation relies on expensive SMs (e.g., we allocate 20 out of the 132 SMs available in the H800 GPU for this purpose), which will limit the computational throughput. Moreover, using SMs for communication results in significant inefficiencies, as tensor cores remain entirely -utilized.
在 DeepSeek-V3 中，我们实现了计算与通信之间的重叠，以隐藏计算过程中的通信延迟。与串行计算和通信相比，这大大降低了对通信带宽的依赖。然而，目前的通信实现依赖于昂贵的 SM（例如，我们为此从 H800 GPU 可用的 132 个 SM 中分配了 20 个），这将限制计算吞吐量。此外，使用 SM 进行通信会导致效率显著降低，因为张量内核仍被完全利用。

Currently, the SMs primarily perform the following tasks for all-to-all communication:
目前，SM 主要执行以下全对全通信任务：

Forwarding data between the IB (InfiniBand) and NVLink domain while aggregating IB traffic destined for multiple GPUs within the same node from a single GPU.
在 IB（InfiniBand）和 NVLink 域之间转发数据，同时汇聚从单个 GPU 发送到同一节点内多个 GPU 的 IB 流量。
Transporting data between RDMA buffers (registered GPU memory regions) and input/output buffers.
在 RDMA 缓冲区（注册 GPU 内存区域）和输入/输出缓冲区之间传输数据。
Executing reduce operations for all-to-all combine.
执行全到全组合的还原操作。
Managing fine-grained memory layout during chunked data transferring to multiple experts across the IB and NVLink domain.
在分块数据跨 IB 和 NVLink 域传输给多个专家时，管理细粒度内存布局。

We aspire to see future vendors developing hardware that offloads these communication tasks from the valuable computation unit SM, serving as a GPU co-processor or a network co-processor like NVIDIA SHARP Graham et al. (2016). Furthermore, to reduce application programming complexity, we aim for this hardware to unify the IB (scale-out) and NVLink (scale-up) networks from the perspective of the computation units. With this unified interface, computation units can easily accomplish operations such as read, write, multicast, and reduce across the entire IB-NVLink-unified domain via submitting communication requests based on simple primitives.
我们希望看到未来的供应商开发出能从有价值的计算单元 SM 中卸载这些通信任务的硬件，充当 GPU 协处理器或类似英伟达 SHARP 的网络协处理器 Graham 等人（2016）。此外，为了降低应用编程的复杂性，我们希望该硬件能从计算单元的角度统一 IB（横向扩展）和 NVLink（纵向扩展）网络。有了这个统一接口，计算单元就可以通过提交基于简单基元的通信请求，在整个 IB-NVLink 统一域中轻松完成读、写、组播和还原等操作。

3.5.2. Compute Hardware 3.5.2.计算硬件

Higher FP8 GEMM Accumulation Precision in Tensor Cores. In the current Tensor Core implementation of the NVIDIA Hopper architecture, FP8 GEMM (General Matrix Multiply) employs fixed-point accumulation, aligning the mantissa products by right-shifting based on the maximum exponent before addition. Our experiments reveal that it only uses the highest 14
在张量核中实现更高的 FP8 GEMM 累加精度。在英伟达™（NVIDIA®）Hopper 架构的当前张量核实现中，FP8 GEMM（通用矩阵乘法）采用定点累加法，在加法之前根据最大指数右移尾数乘积。我们的实验表明，它只使用最高的 14
bits of each mantissa product after sign-fill right shifting, and truncates bits exceeding this range. However, for example, to achieve precise FP32 results from the accumulation of 32 FP8

\times

FP8 multiplications, at least 34 -bit precision is required. Thus, we recommend that future chip designs increase accumulation precision in Tensor Cores to support full-precision accumulation, or select an appropriate accumulation bit-width according to the accuracy requirements of training and inference algorithms. This approach ensures that errors remain within acceptable bounds while maintaining computational efficiency.
位，并截断超出此范围的位。但是，例如，要通过 32 FP8

\times

FP8 乘法的累加实现精确的 FP32 结果，至少需要 34 位精度。因此，我们建议未来的芯片设计提高张量核的累加精度，以支持全精度累加，或根据训练和推理算法的精度要求选择适当的累加位宽。这种方法既能确保误差保持在可接受的范围内，又能保持计算效率。

Support for Tile- and Block-Wise Quantization. Current GPUs only support per-tensor quantization, lacking the native support for fine-grained quantization like our tile- and blockwise quantization. In the current implementation, when the

N_{C}

interval is reached, the partial results will be copied from Tensor Cores to CUDA cores, multiplied by the scaling factors, and added to FP32 registers on CUDA cores. Although the dequantization overhead is significantly mitigated combined with our precise FP32 accumulation strategy, the frequent data movements between Tensor Cores and CUDA cores still limit the computational efficiency. Therefore, we recommend future chips to support fine-grained quantization by enabling Tensor Cores to receive scaling factors and implement MMA with group scaling. In this way, the whole partial sum accumulation and dequantization can be completed directly inside Tensor Cores until the final result is produced, avoiding frequent data movements.
支持瓦片和块向量化。目前的 GPU 只支持按张量量化，缺乏对细粒度量化的本地支持，如我们的分块量化。在当前的实现中，当达到

N_{C}

间隔时，部分结果将从张量核心复制到 CUDA 核心，乘以缩放因子，然后添加到 CUDA 核心的 FP32 寄存器中。虽然结合我们精确的 FP32 累加策略，去量化开销大大减少，但 Tensor 内核和 CUDA 内核之间频繁的数据移动仍然限制了计算效率。因此，我们建议未来的芯片支持细粒度量化，让张量核接收缩放因子，并通过组缩放实现 MMA。这样，整个部分和的累加和去量化就可以直接在张量核内部完成，直到产生最终结果，避免了频繁的数据移动。

Support for Online Quantization. The current implementations struggle to effectively support online quantization, despite its effectiveness demonstrated in our research. In the existing process, we need to read 128 BF16 activation values (the output of the previous computation) from HBM (High Bandwidth Memory) for quantization, and the quantized FP8 values are then written back to HBM, only to be read again for MMA. To address this inefficiency, we recommend that future chips integrate FP8 cast and TMA (Tensor Memory Accelerator) access into a single fused operation, so quantization can be completed during the transfer of activations from global memory to shared memory, avoiding frequent memory reads and writes. We also recommend supporting a warp-level cast instruction for speedup, which further facilitates the better fusion of layer normalization and FP8 cast. Alternatively, a near-memory computing approach can be adopted, where compute logic is placed near the HBM. In this case, BF16 elements can be cast to FP8 directly as they are read from HBM into the GPU, reducing off-chip memory access by roughly

50 %

.
支持在线量化。尽管我们的研究证明了在线量化的有效性，但目前的实现仍难以有效支持在线量化。在现有流程中，我们需要从 HBM（高带宽内存）中读取 128 个 BF16 激活值（上一次计算的输出）进行量化，然后将量化后的 FP8 值写回 HBM，再次读取进行 MMA。为解决这一低效问题，我们建议未来的芯片将 FP8 投加和 TMA（张量内存加速器）访问整合为单个融合操作，这样就可以在将激活从全局内存传输到共享内存的过程中完成量化，避免频繁读写内存。我们还建议支持翘曲级铸模指令，以提高速度，从而进一步促进层归一化和 FP8 铸模的更好融合。另外，还可以采用近内存计算方法，即将计算逻辑置于 HBM 附近。在这种情况下，当 BF16 元素从 HBM 读取到 GPU 时，可直接将其转换为 FP8，从而将片外存储器访问减少约

50 %

。

Support for Transposed GEMM Operations. The current architecture makes it cumbersome to fuse matrix transposition with GEMM operations. In our workflow, activations during the forward pass are quantized into

1 \times 128

FP8 tiles and stored. During the backward pass, the matrix needs to be read out, dequantized, transposed, re-quantized into

128 \times 1

tiles, and stored in HBM. To reduce memory operations, we recommend future chips to enable direct transposed reads of matrices from shared memory before MMA operation, for those precisions required in both training and inference. Combined with the fusion of FP8 format conversion and TMA access, this enhancement will significantly streamline the quantization workflow.
支持转置 GEMM 运算。当前的架构使得矩阵转置与 GEMM 运算的融合变得非常麻烦。在我们的工作流程中，前向传递过程中的激活被量化为

1 \times 128

FP8 块并存储起来。在后向传递过程中，需要将矩阵读出、去量化、转置、重新量化为

128 \times 1

块并存储在 HBM 中。为了减少内存操作，我们建议未来的芯片能够在进行 MMA 操作前从共享内存中直接转置读取矩阵，以满足训练和推理所需的精度要求。结合 FP8 格式转换和 TMA 访问的融合，这一改进将大大简化量化工作流程。

4. Pre-Training 4.培训前

4.1. Data Construction 4.1.数据构建

Compared with DeepSeek-V2, we optimize the pre-training corpus by enhancing the ratio of mathematical and programming samples, while expanding multilingual coverage beyond English and Chinese. Also, our data processing pipeline is refined to minimize redundancy while maintaining corpus diversity. Inspired by Ding et al. (2024), we implement the document packing method for data integrity but do not incorporate cross-sample attention masking during training. Finally, the training corpus for DeepSeek-V3 consists of 14.8 T high-quality and diverse tokens in our tokenizer.
与DeepSeek-V2相比，我们优化了预训练语料库，提高了数学和编程样本的比例，同时将多语言覆盖范围扩大到中英文之外。此外，我们还改进了数据处理流程，在保持语料多样性的同时尽量减少冗余。受 Ding 等人（2024 年）的启发，我们采用了文档打包方法来实现数据完整性，但在训练过程中没有加入跨样本注意力屏蔽。最后，DeepSeek-V3 的训练语料库由 14.8 T 高质量、多样化的标记组成。

In the training process of DeepSeekCoder-V2 (DeepSeek-AI, 2024a), we observe that the Fill-in-Middle (FIM) strategy does not compromise the next-token prediction capability while enabling the model to accurately predict middle text based on contextual cues. In alignment with DeepSeekCoder-V2, we also incorporate the FIM strategy in the pre-training of DeepSeek-V3. To be specific, we employ the Prefix-Suffix-Middle (PSM) framework to structure data as follows:
在DeepSeekCoder-V2（DeepSeek-AI, 2024a）的训练过程中，我们观察到中间填充（FIM）策略不会影响下一标记词的预测能力，同时还能使模型根据上下文线索准确预测中间文本。为了与 DeepSeekCoder-V2 保持一致，我们还在 DeepSeek-V3 的预训练中加入了 FIM 策略。具体来说，我们采用前缀-后缀-中间（PSM）框架来构建数据，具体如下：

<∣ f i m_{-} begin | > f_{pre} < | fim_hole | > f_{suf} < | fim_end | > f_{middle} < | eos_token ∣> .

This structure is applied at the document level as a part of the pre-packing process. The FIM strategy is applied at a rate of 0.1 , consistent with the PSM framework.
这种结构应用于文件层面，是预打包流程的一部分。FIM 策略的应用率为 0.1，与 PSM 框架一致。

The tokenizer for DeepSeek-V3 employs Byte-level BPE (Shibata et al. 1999) with an extended vocabulary of 128 K tokens. The pretokenizer and training data for our tokenizer are modified to optimize multilingual compression efficiency. In addition, compared with DeepSeek-V2, the new pretokenizer introduces tokens that combine punctuations and line breaks. However, this trick may introduce the token boundary bias (Lundberg, 2023) when the model processes multi-line prompts without terminal line breaks, particularly for few-shot evaluation prompts. To address this issue, we randomly split a certain proportion of such combined tokens during training, which exposes the model to a wider array of special cases and mitigates this bias.
DeepSeek-V3 的标记化器采用字节级 BPE（Shibata 等人，1999 年），扩展词汇量为 128 K 个标记。为了优化多语言压缩效率，我们修改了令牌生成器的预令牌生成器和训练数据。此外，与 DeepSeek-V2 相比，新的预置器引入了结合标点符号和换行符的标记。然而，当模型处理没有终端换行符的多行提示时，尤其是在处理很少的评估提示时，这种技巧可能会引入标记边界偏差（Lundberg，2023）。为了解决这个问题，我们在训练过程中随机拆分了一定比例的此类组合标记符，从而让模型接触到更多的特殊情况，减轻了这种偏差。

4.2. Hyper-Parameters 4.2.超参数

Model Hyper-Parameters. We set the number of Transformer layers to 61 and the hidden dimension to 7168. All learnable parameters are randomly initialized with a standard deviation of 0.006 . In MLA, we set the number of attention heads

n_{h}

to 128 and the per-head dimension

d_{h}

to 128 . The KV compression dimension

d_{c}

is set to 512 , and the query compression dimension

d_{c}^{'}

is set to 1536 . For the decoupled queries and key, we set the per-head dimension

d_{h}^{R}

to 64 . We substitute all FFNs except for the first three layers with MoE layers. Each MoE layer consists of 1 shared expert and 256 routed experts, where the intermediate hidden dimension of each expert is 2048. Among the routed experts, 8 experts will be activated for each token, and each token will be ensured to be sent to at most 4 nodes. The multi-token prediction depth

D

is set to 1 , i.e., besides the exact next token, each token will predict one additional token. As DeepSeek-V2, DeepSeek-V3 also employs additional RMSNorm layers after the compressed latent vectors, and multiplies additional scaling factors at the width bottlenecks. Under this configuration, DeepSeek-V3 comprises 671B total parameters, of which 37B are activated for each token.
模型超参数。我们将变压器层数设为 61，隐藏维度设为 7168。所有可学习参数均以 0.006 的标准偏差随机初始化。在 MLA 中，我们将注意力头数

n_{h}

设置为 128，每个注意力头的维度

d_{h}

设置为 128。KV 压缩维度

d_{c}

设置为 512，查询压缩维度

d_{c}^{'}

设置为 1536。对于解耦查询和密钥，我们将每个头部的维度

d_{h}^{R}

设置为 64。除前三层外，我们将所有 FFN 替换为 MoE 层。每个 MoE 层由 1 个共享专家和 256 个路由专家组成，每个专家的中间隐藏维度为 2048。在路由专家中，每个令牌将激活 8 个专家，并确保每个令牌最多发送到 4 个节点。多令牌预测深度

D

设置为 1，即除了准确的下一个令牌外，每个令牌将预测一个额外的令牌。与 DeepSeek-V2 一样，DeepSeek-V3 也在压缩潜向量后采用了额外的 RMSNorm 层，并在宽度瓶颈处乘以额外的缩放因子。在这种配置下，DeepSeek-V3 总共包含 671B 个参数，其中 37B 个参数用于激活每个标记。

Training Hyper-Parameters. We employ the AdamW optimizer (Loshchilov and Hutter, 2017) with hyper-parameters set to

β_{1} = 0.9, β_{2} = 0.95

, and weight_decay

= 0.1

. We set the maximum sequence length to 4 K during pre-training, and pre-train DeepSeek-V3 on 14.8T tokens. As for
训练超参数。我们采用 AdamW 优化器（Loshchilov 和 Hutter，2017 年），超参数设置为

β_{1} = 0.9, β_{2} = 0.95

，权重衰减

= 0.1

。在预训练期间，我们将最大序列长度设置为 4 K，并在 14.8T 标记上对 DeepSeek-V3 进行预训练。至于
the learning rate scheduling, we first linearly increase it from 0 to

2.2 \times 10^{- 4}

during the first 2 K steps. Then, we keep a constant learning rate of

2.2 \times 10^{- 4}

until the model consumes 10T training tokens. Subsequently, we gradually decay the learning rate to

2.2 \times 10^{- 5}

in 4.3 T tokens, following a cosine decay curve. During the training of the final 500B tokens, we keep a constant learning rate of

2.2 \times 10^{- 5}

in the first 333B tokens, and switch to another constant learning rate of

7.3 \times 10^{- 6}

in the remaining 167B tokens. The gradient clipping norm is set to 1.0 . We employ a batch size scheduling strategy, where the batch size is gradually increased from 3072 to 15360 in the training of the first 469B tokens, and then keeps 15360 in the remaining training. We leverage pipeline parallelism to deploy different layers of a model on different GPUs, and for each layer, the routed experts will be uniformly deployed on 64 GPUs belonging to 8 nodes. As for the node-limited routing, each token will be sent to at most 4 nodes (i.e.,

M = 4

). For auxiliary-loss-free load balancing, we set the bias update speed

γ

to 0.001 for the first 14.3 T tokens, and to 0.0 for the remaining 500B tokens. For the balance loss, we set

α

to 0.0001 , just to avoid extreme imbalance within any single sequence. The MTP loss weight

λ

is set to 0.3 for the first 10T tokens, and to 0.1 for the remaining 4.8 T tokens.
在学习率调度方面，我们首先在前 2 K 步将学习率从 0 线性地增加到

2.2 \times 10^{- 4}

。然后，我们保持

2.2 \times 10^{- 4}

的恒定学习率，直到模型消耗掉 10T 个训练代币。随后，我们按照余弦衰减曲线，在 4.3 T 代币中将学习率逐渐衰减到

2.2 \times 10^{- 5}

。在最后 500B 标记的训练过程中，我们在前 333B 标记中保持

2.2 \times 10^{- 5}

的恒定学习率，并在剩余的 167B 标记中切换到

7.3 \times 10^{- 6}

的恒定学习率。梯度剪切规范设置为 1.0。我们采用批量大小调度策略，在训练前 469B 个代币时，批量大小从 3072 逐步增加到 15360，然后在剩余训练中保持 15360。我们利用管道并行性将模型的不同层部署在不同的 GPU 上，对于每一层，路由专家将统一部署在属于 8 个节点的 64 个 GPU 上。至于节点限制路由，每个令牌将最多发送到 4 个节点（即

M = 4

）。为了实现无辅助损失的负载平衡，我们将前 14.3 T 令牌的偏置更新速度

γ

设置为 0.001，将其余 500B 令牌的偏置更新速度设置为 0.0。对于平衡损失，我们将

α

设置为 0.0001，以避免任何单一序列内出现极端不平衡。前 10T 标记的 MTP 损失权重

λ

设置为 0.3，其余 4.8T 标记的 MTP 损失权重设置为 0.1。

Figure

8 ∣

Evaluation results on the “Needle In A Haystack” (NIAH) tests. DeepSeek-V3 performs well across all context window lengths up to 128 K .
图

8 ∣

"干草堆中的针"（NIAH）测试的评估结果。DeepSeek-V3 在最大 128 K 的所有上下文窗口长度上都表现出色。

4.3. Long Context Extension
4.3.长语境扩展

We adopt a similar approach to DeepSeek-V2 (DeepSeek-AI, 2024c) to enable long context capabilities in DeepSeek-V3. After the pre-training stage, we apply YaRN (Peng et al., 2023a) for context extension and perform two additional training phases, each comprising 1000 steps, to progressively expand the context window from 4 K to 32 K and then to 128 K . The YaRN configuration is consistent with that used in DeepSeek-V2, being applied exclusively to the decoupled shared key

k_{t}^{R}

. The hyper-parameters remain identical across both phases, with the scale

s = 40, α = 1, β = 32

, and the scaling factor

\sqrt{t} = 0.1 \ln s + 1

. In the first phase, the sequence length is set to 32 K , and the batch size is 1920 . During the second phase, the sequence length is increased to 128 K , and the batch size is reduced to 480 . The learning rate for both phases is set to

7.3 \times 10^{- 6}

, matching the final learning rate from the pre-training stage.
我们采用与DeepSeek-V2（DeepSeek-AI，2024c）类似的方法，在DeepSeek-V3中实现长语境功能。在预训练阶段之后，我们应用 YaRN（Peng 等人，2023a）进行上下文扩展，并执行两个额外的训练阶段，每个阶段包括 1000 步，将上下文窗口从 4 K 逐步扩展到 32 K，然后再扩展到 128 K。YaRN 配置与 DeepSeek-V2 中使用的配置一致，只应用于去耦共享密钥

k_{t}^{R}

。两个阶段的超参数保持一致，规模为

s = 40, α = 1, β = 32

，缩放因子为

\sqrt{t} = 0.1 \ln s + 1

。在第一阶段，序列长度设置为 32 K，批量大小为 1920。在第二阶段，序列长度增加到 128 K，批量大小减小到 480。两个阶段的学习率均设置为

7.3 \times 10^{- 6}

，与预训练阶段的最终学习率相匹配。

Through this two-phase extension training, DeepSeek-V3 is capable of handling inputs up to 128 K in length while maintaining strong performance. Figure 8 illustrates that DeepSeek-V3, following supervised fine-tuning, achieves notable performance on the “Needle In A Haystack” (NIAH) test, demonstrating consistent robustness across context window lengths up to 128 K .
通过这种两阶段扩展训练，DeepSeek-V3 能够处理长度达 128 K 的输入，同时保持强劲的性能。图 8 显示，经过监督微调后，DeepSeek-V3 在 "大海捞针"（Needle In A Haystack，NIAH）测试中取得了显著的性能，在高达 128 K 的上下文窗口长度上表现出了一致的鲁棒性。

4.4. Evaluations 4.4.评估

4.4.1. Evaluation Benchmarks
4.4.1.评估基准

The base model of DeepSeek-V3 is pretrained on a multilingual corpus with English and Chinese constituting the majority, so we evaluate its performance on a series of benchmarks primarily in English and Chinese, as well as on a multilingual benchmark. Our evaluation is based on our internal evaluation framework integrated in our HAI-LLM framework. Considered benchmarks are categorized and listed as follows, where underlined benchmarks are in Chinese and double-underlined benchmarks are multilingual ones:
DeepSeek-V3 的基础模型是在以英语和中文为主的多语言语料库上进行预训练的，因此我们在一系列以英语和中文为主的基准以及一个多语言基准上对其性能进行了评估。我们的评估基于集成在 HAI-LLM 框架中的内部评估框架。考虑的基准分类如下，其中下划线的基准为中文基准，双下划线的基准为多语言基准：

Multi-subject multiple-choice datasets include MMLU (Hendrycks et al., 2020), MMLURedux (Gema et al., 2024), MMLU-Pro (Wang et al., 2024b), MMMLU (OpenAI, 2024b), C-Eval (Huang et al., 2023), and CMMLU (Li et al., 2023).
多主体多项选择数据集包括MMLU（Hendrycks等人，2020年）、MMLURedux（Gema等人，2024年）、MMLU-Pro（Wang等人，2024年b）、MMMLU（OpenAI，2024年b）、C-Eval（Huang等人，2023年）和CMMLU（Li等人，2023年）。

Language understanding and reasoning datasets include HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), and BigBench Hard (BBH) (Suzgun et al., 2022).
语言理解和推理数据集包括 HellaSwag（Zellers 等人，2019 年）、PIQA（Bisk 等人，2020 年）、ARC（Clark 等人，2018 年）和 BigBench Hard（BBH）（Suzgun 等人，2022 年）。

Closed-book question answering datasets include TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019).
闭卷答题数据集包括 TriviaQA（Joshi 等人，2017 年）和 NaturalQuestions（Kwiatkowski 等人，2019 年）。

Reading comprehension datasets include RACE Lai et al. (2017), DROP (Dua et al., 2019),
阅读理解数据集包括 RACE Lai 等人（2017 年）、DROP（Dua 等人，2019 年）、
(Sun et al., 2019a), and CMRC (Cui et al., 2019).
(Sun 等人，2019a）和 CMRC（Cui 等人，2019）。
Reference disambiguation datasets include CLUEWSC (Xu et al., 2020) and WinoGrande Sakaguchi et al. (2019).
参考消歧数据集包括 CLUEWSC（Xu 等人，2020 年）和 WinoGrande Sakaguchi 等人（2019 年）。

Language modeling datasets include Pile (Gao et al., 2020).
语言建模数据集包括 Pile（Gao 等人，2020 年）。
Chinese understanding and culture datasets include CCPM (Li et al., 2021).
中国理解与文化数据集包括 CCPM（Li 等人，2021 年）。
Math datasets include GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MGSM (Shi et al., 2023), and CMath (Wei et al., 2023).
数学数据集包括 GSM8K（Cobbe 等人，2021 年）、MATH（Hendrycks 等人，2021 年）、MGSM（Shi 等人，2023 年）和 CMath（Wei 等人，2023 年）。

Code datasets include HumanEval (Chen et al., 2021), LiveCodeBench-Base (0801-1101) (Jain et al., 2024), MBPP (Austin et al., 2021), and CRUXEval (Gu et al., 2024).
代码数据集包括 HumanEval（Chen 等人，2021 年）、LiveCodeBench-Base (0801-1101) （Jain 等人，2024 年）、MBPP（Austin 等人，2021 年）和 CRUXEval（Gu 等人，2024 年）。

Standardized exams include AGIEval (Zhong et al., 2023). Note that AGIEval includes both English and Chinese subsets.
标准化考试包括 AGIEval（Zhong 等人，2023 年）。请注意，AGIEval 包括英语和中文两个子集。

Following our previous work (DeepSeek-AI, 2024b|c), we adopt perplexity-based evaluation for datasets including HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, MMLU-Redux, MMLU-Pro, MMMLU, ARC-Easy, ARC-Challenge, C-Eval, CMMLU, C3, and CCPM, and adopt generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, MGSM, HumanEval, MBPP, LiveCodeBench-Base, CRUXEval, BBH, AGIEval, CLUEWSC, CMRC, and CMath. In addition, we perform language-modeling-based evaluation for Pile-test and use Bits-Per-Byte (BPB) as the metric to guarantee fair comparison among models using different tokenizers.
根据我们之前的工作（DeepSeek-AI, 2024b|c），我们对包括HellaSwag、PIQA、WinoGrande、RACE-Middle、RACE-High、MMLU、MMLU-Redux、MMLU-Pro、MMMLU、ARC-Easy、ARC-Challenge在内的数据集采用基于易错性的评估、C-Eval、CMMLU、C3 和 CCPM，并对 TriviaQA、NaturalQuestions、DROP、MATH、GSM8K、MGSM、HumanEval、MBPP、LiveCodeBench-Base、CRUXEval、BBH、AGIEval、CLUEWSC、CMRC 和 CMath 采用基于生成的评估。此外，我们还对 Pile-test 进行了基于语言建模的评估，并使用 Bits-Per-Byte (BPB) 作为指标，以确保使用不同标记化器的模型之间的公平比较。

Benchmark (Metric) 基准（指标）

# Shots # 射击

DeepSeek-V2 Base DeepSeek-V2 底座

Qwen2.5

72B Base

LLaMA-3.1 405B 底座

LLaMA-3.1

405B Base

DeepSeek-V3 Base DeepSeek-V3 底座

Architecture 建筑学

MoE 教育部

Dense 密集

MoE 教育部

# Activated Params # 激活参数

21B

72B

405B

37B

# Total Params # 参数总数

236B

72B

405B

671B

English 英语

Pile-test (BPB) 桩基试验（BPB）

0.606

0.638

0.542

0.548

BBH (Em)

3-shot 三连发

78.8

79.8

82.9

87.5

MMLU (EM)

5-shot 五连发

78.4

85.0

84.4

87.1

MMLU-Redux (EM)

5 -shot 5 -拍摄

75.6

83.2

81.3

86.2

MMLU-Pro (em)

5 -shot 5 -拍摄

51.4

58.3

52.8

64.4

DROP (F1) 下降（F1）

3-shot 三连发

80.4

80.6

86.0

89.0

ARC-Easy (EM)

25-shot 25发

97.6

98.4

98.9

ARC-Challenge (EM) ARC 挑战赛（EM）

25-shot 25发

92.2

94.5

95.3

HellaSwag (EM)

10-shot 十连发

87.1

84.8

89.2

88.9

PIQA (Em)

0 -shot

83.9

82.6

85.9

84.7

WinoGrande (EM)

5-shot 五连发

86.3

82.3

85.2

84.9

RACE-Middle (em) 种族-中等（em）

5-shot 五连发

73.1

68.1

74.2

67.1

RACE-High (EM)

5 -shot 5 -拍摄

52.6

50.3

56.8

51.3

TriviaQA (EM)

5-shot 五连发

80.0

71.9

82.7

82.9

NaturalQuestions (EM) 自然问题（EM）

5-shot 五连发

38.6

33.2

41.5

40.0

AGIEval (EM)

0 -shot

57.5

75.8

60.6

79.6

Code 代码

HumanEval (Pass@1)

0 -shot

43.3

53.0

54.9

65.2

MBPP (Pass@1)

3-shot 三连发

65.0

72.6

68.4

75.4

LiveCodeBench-Base (Pass@1)

3-shot 三连发

11.6

12.9

15.5

19.4

CRUXEval-I (em)

2-shot 2 发

52.5

59.1

58.5

67.3

CRUXEval-O (EM)

2-shot 2 发

49.8

59.9

69.8

Math 数学

GSM8K (Em)

8 -shot 8 -拍摄

81.6

88.3

83.5

89.3

MATH (EM) 数学（EM）

4-shot 4 连发

43.4

54.4

49.0

61.6

MGSM (EM)

8 -shot 8 -拍摄

63.6

76.2

69.9

79.8

CMath (EM)

3 -shot

78.7

84.5

77.3

90.7

Chinese 中文

CLUEWSC (EM)

5-shot 五连发

82.0

82.5

83.0

82.7

C-Eval (EM)

5-shot 五连发

81.4

89.2

72.5

90.1

CMMLU (EM)

5-shot 五连发

84.0

89.5

73.7

88.8

CMRC (EM)

1-shot 1 发

77.4

75.8

76.0

76.3

C3 (Em)

0 -shot

77.4

76.7

79.7

78.6

CCPM (EM)

0 -shot

93.0

88.5

78.6

92.0

Multilingual 多种语言

MMMLU-non-English (EM) 多语言流动大学-非英语（EM）

5-shot 五连发

64.0

74.8

73.8

79.4

| | Benchmark (Metric) | # Shots | DeepSeek-V2 Base | Qwen2.5 <br> 72B Base | LLaMA-3.1 <br> 405B Base | DeepSeek-V3 Base | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | | Architecture | - | MoE | Dense | Dense | MoE | | | # Activated Params | - | 21B | 72B | 405B | 37B | | | # Total Params | - | 236B | 72B | 405B | 671B | | English | Pile-test (BPB) | - | 0.606 | 0.638 | 0.542 | 0.548 | | | BBH (Em) | 3-shot | 78.8 | 79.8 | 82.9 | 87.5 | | | MMLU (EM) | 5-shot | 78.4 | 85.0 | 84.4 | 87.1 | | | MMLU-Redux (EM) | 5 -shot | 75.6 | 83.2 | 81.3 | 86.2 | | | MMLU-Pro (em) | 5 -shot | 51.4 | 58.3 | 52.8 | 64.4 | | | DROP (F1) | 3-shot | 80.4 | 80.6 | 86.0 | 89.0 | | | ARC-Easy (EM) | 25-shot | 97.6 | 98.4 | 98.4 | 98.9 | | | ARC-Challenge (EM) | 25-shot | 92.2 | 94.5 | 95.3 | 95.3 | | | HellaSwag (EM) | 10-shot | 87.1 | 84.8 | 89.2 | 88.9 | | | PIQA (Em) | 0 -shot | 83.9 | 82.6 | 85.9 | 84.7 | | | WinoGrande (EM) | 5-shot | 86.3 | 82.3 | 85.2 | 84.9 | | | RACE-Middle (em) | 5-shot | 73.1 | 68.1 | 74.2 | 67.1 | | | RACE-High (EM) | 5 -shot | 52.6 | 50.3 | 56.8 | 51.3 | | | TriviaQA (EM) | 5-shot | 80.0 | 71.9 | 82.7 | 82.9 | | | NaturalQuestions (EM) | 5-shot | 38.6 | 33.2 | 41.5 | 40.0 | | | AGIEval (EM) | 0 -shot | 57.5 | 75.8 | 60.6 | 79.6 | | Code | HumanEval (Pass@1) | 0 -shot | 43.3 | 53.0 | 54.9 | 65.2 | | | MBPP (Pass@1) | 3-shot | 65.0 | 72.6 | 68.4 | 75.4 | | | LiveCodeBench-Base (Pass@1) | 3-shot | 11.6 | 12.9 | 15.5 | 19.4 | | | CRUXEval-I (em) | 2-shot | 52.5 | 59.1 | 58.5 | 67.3 | | | CRUXEval-O (EM) | 2-shot | 49.8 | 59.9 | 59.9 | 69.8 | | Math | GSM8K (Em) | 8 -shot | 81.6 | 88.3 | 83.5 | 89.3 | | | MATH (EM) | 4-shot | 43.4 | 54.4 | 49.0 | 61.6 | | | MGSM (EM) | 8 -shot | 63.6 | 76.2 | 69.9 | 79.8 | | | CMath (EM) | 3 -shot | 78.7 | 84.5 | 77.3 | 90.7 | | Chinese | CLUEWSC (EM) | 5-shot | 82.0 | 82.5 | 83.0 | 82.7 | | | C-Eval (EM) | 5-shot | 81.4 | 89.2 | 72.5 | 90.1 | | | CMMLU (EM) | 5-shot | 84.0 | 89.5 | 73.7 | 88.8 | | | CMRC (EM) | 1-shot | 77.4 | 75.8 | 76.0 | 76.3 | | | C3 (Em) | 0 -shot | 77.4 | 76.7 | 79.7 | 78.6 | | | CCPM (EM) | 0 -shot | 93.0 | 88.5 | 78.6 | 92.0 | | Multilingual | MMMLU-non-English (EM) | 5-shot | 64.0 | 74.8 | 73.8 | 79.4 |

Table 3 | Comparison among DeepSeek-V3-Base and other representative open-source base models. All models are evaluated in our internal framework and share the same evaluation setting. Scores with a gap not exceeding 0.3 are considered to be at the same level. DeepSeek-V3-Base achieves the best performance on most benchmarks, especially on math and code tasks.
表 3：DeepSeek-V3-Base 与其他具有代表性的开源基础模型的比较。所有模型均在我们的内部框架中进行评估，并共享相同的评估设置。差距不超过 0.3 的分数被视为处于同一水平。DeepSeek-V3-Base 在大多数基准测试中表现最佳，尤其是在数学和代码任务中。

4.4.2. Evaluation Results
4.4.2.评估结果

In Table 3, we compare the base model of DeepSeek-V3 with the state-of-the-art open-source base models, including DeepSeek-V2-Base (DeepSeek-AI, 2024c) (our previous release), Qwen2.5 72B Base (Qwen, 2024b), and LLaMA-3.1 405B Base (AI@Meta, 2024b). We evaluate all these models with our internal evaluation framework, and ensure that they share the same evaluation setting. Note that due to the changes in our evaluation framework over the past months, the performance of DeepSeek-V2-Base exhibits a slight difference from our previously reported results. Overall, DeepSeek-V3-Base comprehensively outperforms DeepSeek-V2-Base and Qwen2.5 72B Base, and surpasses LLaMA-3.1 405B Base in the majority of benchmarks, essentially becoming the strongest open-source model.
在表3中，我们将DeepSeek-V3的基础模型与最先进的开源基础模型进行了比较，包括DeepSeek-V2-Base（DeepSeek-AI，2024c）（我们之前的版本）、Qwen2.5 72B Base（Qwen，2024b）和LLaMA-3.1 405B Base（AI@Meta，2024b）。我们使用内部评估框架对所有这些模型进行评估，并确保它们共享相同的评估设置。请注意，由于我们的评估框架在过去几个月中发生了变化，DeepSeek-V2-Base 的性能与我们之前报告的结果略有不同。总体而言，DeepSeek-V3-Base全面超越DeepSeek-V2-Base和Qwen2.5 72B Base，并在大多数基准测试中超越LLaMA-3.1 405B Base，基本成为最强的开源模型。

From a more detailed perspective, we compare DeepSeek-V3-Base with the other open-source base models individually. (1) Compared with DeepSeek-V2-Base, due to the improvements in our model architecture, the scale-up of the model size and training tokens, and the enhancement of data quality, DeepSeek-V3-Base achieves significantly better performance as expected. (2) Compared with Qwen 2.5 72B Base, the state-of-the-art Chinese open-source model, with only half of the activated parameters, DeepSeek-V3-Base also demonstrates remarkable advantages, especially on English, multilingual, code, and math benchmarks. As for Chinese benchmarks, except for CMMLU, a Chinese multi-subject multiple-choice task, DeepSeek-V3-Base also shows better performance than Qwen2.5 72B. (3) Compared with LLaMA-3.1 405B Base, the largest open-source model with 11 times the activated parameters, DeepSeek-V3-Base also exhibits much better performance on multilingual, code, and math benchmarks. As for English and Chinese language benchmarks, DeepSeek-V3-Base shows competitive or better performance, and is especially good on BBH, MMLU-series, DROP, C-Eval, CMMLU, and CCPM.
从更详细的角度，我们将DeepSeek-V3-Base与其他开源基础模型分别进行比较。(1）与DeepSeek-V2-Base相比，由于我们对模型架构的改进、模型规模和训练代币的扩大以及数据质量的提高，DeepSeek-V3-Base取得了预期的明显更好的性能。(2）与目前最先进的中文开源模型Qwen 2.5 72B Base相比，DeepSeek-V3-Base的激活参数仅为Qwen 2.5 72B Base的一半，但在英语、多语种、代码和数学等基准测试中，DeepSeek-V3-Base也表现出了显著的优势。至于中文基准，除了 CMMLU 这一中文多主体选择任务外，DeepSeek-V3-Base 的性能也优于 Qwen2.5 72B。(3) 与拥有 11 倍激活参数的最大开源模型 LLaMA-3.1 405B Base 相比，DeepSeek-V3-Base 在多语言、代码和数学基准上也表现出更好的性能。在英文和中文基准测试中，DeepSeek-V3-Base的性能具有竞争力或更好，尤其是在BBH、MMLU系列、DROP、C-Eval、CMMLU和CCPM上。

Due to our efficient architectures and comprehensive engineering optimizations, DeepSeekV3 achieves extremely high training efficiency. Under our training framework and infrastructures, training DeepSeek-V3 on each trillion tokens requires only 180K H800 GPU hours, which is much cheaper than training 72B or 405B dense models.
由于我们高效的架构和全面的工程优化，DeepSeekV3实现了极高的训练效率。在我们的训练框架和基础设施下，在每万亿个代币上训练 DeepSeek-V3 只需要 180K H800 GPU 小时，这比训练 72B 或 405B 密集模型要便宜得多。

Benchmark (Metric) 基准（指标）

# Shots # 射击

小型教育部基线

Small MoE

Baseline

小型 MoE w/ MTP

Small MoE

w/ MTP

大型教育部门基线

Large MoE

Baseline

大型 MoE w/ MTP

Large MoE

w/ MTP

# Activated Params (Inference)
# 激活参数（推理）

2.4 B

20.9 B

20.9B # 参数总数（推理）

20.9B

# Total Params (Inference)

# Training Tokens # 培训代币

15.7 B

228.7 B

Pile-test (BPB) 桩基试验（BPB）

1.33 T

540 B

540B

BBH (EM)

0.729

0.729

0.658

0.657

MMLU (EM)

3-shot 三连发

39.0

41.4

70.0

70.7

DROP (F1) 下降（F1）

5-shot 五连发

50.0

53.3

67.5

66.6

TriviaQA (EM)

1-shot 1 发

39.2

41.3

68.5

70.6

NaturalQuestions (EM) 自然问题（EM）

5-shot 五连发

56.9

57.7

67.0

67.3

HumanEval (Pass@1)

5-shot 五连发

22.7

22.3

27.2

28.5

MBPP (Pass@1)

0-shot 0 发

20.7

26.8

44.5

53.7

GSM8K (EM)

3-shot 三连发

35.8

36.8

61.6

62.2

MATH (EM) 数学（EM）

8-shot 8 连发

25.4

31.4

72.3

74.0

4-shot 4 连发

10.7

12.6

38.6

39.8

Table 4 | Ablation results for the MTP strategy. The MTP strategy consistently enhances the model performance on most of the evaluation benchmarks.
表 4：MTP 策略的消融结果。在大多数评估基准上，MTP 策略都能持续提升模型性能。

4.5. Discussion 4.5.讨论情况

4.5.1. Ablation Studies for Multi-Token Prediction
4.5.1.多标记预测的消融研究

In Table 4, we show the ablation results for the MTP strategy. To be specific, we validate the MTP strategy on top of two baseline models across different scales. At the small scale, we train a baseline MoE model comprising 15.7B total parameters on 1.33T tokens. At the large scale, we train a baseline MoE model comprising 228.7B total parameters on 540B tokens. On top of them, keeping the training data and the other architectures the same, we append a 1 -depth MTP module onto them and train two models with the MTP strategy for comparison. Note that during inference, we directly discard the MTP module, so the inference costs of the compared models are exactly the same. From the table, we can observe that the MTP strategy consistently enhances the model performance on most of the evaluation benchmarks.
表 4 显示了 MTP 策略的消融结果。具体来说，我们在两个不同规模的基线模型上验证了 MTP 策略。在小尺度上，我们在 1.33T 标记上训练由 15.7B 总参数组成的基线 MoE 模型。在大规模模型中，我们在 540B 标记上训练一个总参数为 2287B 的基线 MoE 模型。在保持训练数据和其他架构不变的情况下，我们在它们之上附加了一个深度为 1 的 MTP 模块，并使用 MTP 策略训练两个模型进行比较。注意，在推理过程中，我们直接丢弃了 MTP 模块，因此比较模型的推理成本完全相同。从表中可以看出，在大多数评估基准上，MTP 策略都能持续提高模型性能。

Benchmark (Metric) 基准（指标）

# Shots # 射击

小型 MoE 辅助损失型

Small MoE

Aux-Loss-Based

小 MoE 辅助-无损失

Small MoE

Aux-Loss-Free

大型 MoE 辅助损失型

Large MoE

Aux-Loss-Based

大型无损辅助设备

Large MoE

Aux-Loss-Free

# Activated Params # 激活参数

2.4 B

20.9 B

# Total Params # 参数总数

15.7 B

228.7 B

# Training Tokens # 培训代币

1.33 T

578 B

578B

Pile-test (BPB) 桩基试验（BPB）

0.727

0.724

0.656

0.652

BBH (EM)

37.3

39.3

66.7

67.9

MMLU (EM)

3-shot 三连发

5-shot 五连发

38.0

51.8

68.3

DROP (F1) 下降（F1）

1-shot 1 发

38.3

59.0

67.2

TriviaQA (EM)

5-shot 五连发

23.2

23.4

67.1

67.1

NaturalQuestions (EM) 自然问题（EM）

5-shot 五连发

22.0

22.6

27.1

67.7

HumanEval (Pass@1)

0-shot 0 发

36.6

35.8

28.2

46.1

MBPP (Pass@1)

3-shot 三连发

27.1

29.6

59.2

40.7

GSM8K (EM)

8-shot 8 连发

10.9

11.1

37.2

61.2

MATH (EM) 数学（EM）

4-shot 4 连发

74.5

Table 5 | Ablation results for the auxiliary-loss-free balancing strategy. Compared with the purely auxiliary-loss-based method, the auxiliary-loss-free strategy consistently achieves better model performance on most of the evaluation benchmarks.
表 5 无辅助损耗平衡策略的消融结果与纯粹基于辅助损耗的方法相比，无辅助损耗策略在大多数评估基准上都取得了更好的模型性能。

4.5.2. Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy
4.5.2.无损耗辅助平衡策略的消融研究

In Table 5, we show the ablation results for the auxiliary-loss-free balancing strategy. We validate this strategy on top of two baseline models across different scales. At the small scale, we train a baseline MoE model comprising 15.7B total parameters on 1.33T tokens. At the large scale, we train a baseline MoE model comprising 228.7B total parameters on 578B tokens. Both of the baseline models purely use auxiliary losses to encourage load balance, and use the sigmoid gating function with top-K affinity normalization. Their hyper-parameters to control the strength of auxiliary losses are the same as DeepSeek-V2-Lite and DeepSeek-V2, respectively. On top of these two baseline models, keeping the training data and the other architectures the same, we remove all auxiliary losses and introduce the auxiliary-loss-free balancing strategy for comparison. From the table, we can observe that the auxiliary-loss-free strategy consistently achieves better model performance on most of the evaluation benchmarks.
表 5 显示了无辅助损耗平衡策略的消融结果。我们在两个不同规模的基线模型上验证了这一策略。在小尺度上，我们在 1.33T 标记上训练一个总参数为 15.7B 的基线 MoE 模型。在大规模模型中，我们在 578B 标记上训练了一个总参数为 2287B 的基线 MoE 模型。这两个基线模型都纯粹使用辅助损失来促进负载平衡，并使用具有 top-K 亲和归一化的 sigmoid 门控函数。它们控制辅助损失强度的超参数分别与 DeepSeek-V2-Lite 和 DeepSeek-V2 相同。在这两个基线模型的基础上，在保持训练数据和其他架构不变的情况下，我们去除所有辅助损失，引入无辅助损失平衡策略进行比较。从表中可以看出，在大多数评估基准上，无辅助损失策略始终能获得更好的模型性能。

4.5.3. Batch-Wise Load Balance VS. Sequence-Wise Load Balance
4.5.3.批量负载平衡 VS.顺序负载平衡

The key distinction between auxiliary-loss-free balancing and sequence-wise auxiliary loss lies in their balancing scope: batch-wise versus sequence-wise. Compared with the sequence-wise auxiliary loss, batch-wise balancing imposes a more flexible constraint, as it does not enforce in-domain balance on each sequence. This flexibility allows experts to better specialize in different domains. To validate this, we record and analyze the expert load of a 16B auxiliary-loss-based baseline and a 16B auxiliary-loss-free model on different domains in the Pile test set. As illustrated in Figure 9, we observe that the auxiliary-loss-free model demonstrates greater expert specialization patterns as expected.
无辅助损耗平衡和序列辅助损耗的主要区别在于它们的平衡范围：批次平衡和序列平衡。与按序列计算的辅助损失相比，按批次计算的平衡施加了更灵活的约束，因为它不对每个序列强制执行域内平衡。这种灵活性允许专家更好地专注于不同领域。为了验证这一点，我们记录并分析了 Pile 测试集中基于 16B 辅助损耗的基线模型和无辅助损耗的 16B 模型在不同领域的专家负载。如图 9 所示，我们观察到无辅助损耗模型显示出更强的专家专业化模式。

To further investigate the correlation between this flexibility and the advantage in model performance, we additionally design and validate a batch-wise auxiliary loss that encourages load balance on each training batch instead of on each sequence. The experimental results show that, when achieving a similar level of batch-wise load balance, the batch-wise auxiliary loss can also achieve similar model performance to the auxiliary-loss-free method. To be specific, in our experiments with 1B MoE models, the validation losses are: 2.258 (using a sequencewise auxiliary loss), 2.253 (using the auxiliary-loss-free method), and 2.253 (using a batch-wise
为了进一步研究这种灵活性与模型性能优势之间的相关性，我们还设计并验证了一种批次辅助损失方法，这种方法鼓励在每个训练批次而不是每个序列上实现负载平衡。实验结果表明，当达到类似的批次负载平衡水平时，批次辅助损失也能获得与无辅助损失方法类似的模型性能。具体来说，在我们使用 1B MoE 模型的实验中，验证损失为2.258（使用顺序辅助损耗）、2.253（使用无辅助损耗方法）和 2.253（使用批次辅助损耗）。

Figure 9 | Expert load of auxiliary-loss-free and auxiliary-loss-based models on three domains in the Pile test set. The auxiliary-loss-free model shows greater expert specialization patterns than the auxiliary-loss-based one. The relative expert load denotes the ratio between the actual expert load and the theoretically balanced expert load. Due to space constraints, we only present the results of two layers as an example, with the results of all layers provided in Appendix C .
图 9：无辅助损耗模型和基于辅助损耗模型在桩测试集中三个领域的专家负载。与基于辅助损失的模型相比，无辅助损失模型显示出更大的专家专业化模式。相对专家负载表示实际专家负载与理论上平衡的专家负载之间的比率。由于篇幅有限，我们仅以两个层的结果为例，所有层的结果见附录 C。
auxiliary loss). We also observe similar results on 3 B MoE models: the model using a sequencewise auxiliary loss achieves a validation loss of 2.085, and the models using the auxiliary-loss-free method or a batch-wise auxiliary loss achieve the same validation loss of 2.080 .
辅助损失）。我们还在 3 B MoE 模型上观察到类似的结果：使用序列辅助损失的模型的验证损失为 2.085，而使用无辅助损失方法或批量辅助损失的模型的验证损失同样为 2.080。

In addition, although the batch-wise load balancing methods show consistent performance advantages, they also face two potential challenges in efficiency: (1) load imbalance within certain sequences or small batches, and (2) domain-shift-induced load imbalance during inference. The first challenge is naturally addressed by our training framework that uses large-scale expert parallelism and data parallelism, which guarantees a large size of each micro-batch. For the second challenge, we also design and implement an efficient inference framework with redundant expert deployment, as described in Section 3.4, to overcome it.
此外，虽然分批负载平衡方法在性能上表现出一致的优势，但它们在效率上也面临两个潜在挑战：(1) 特定序列或小批量内的负载不平衡；(2) 推理过程中领域转移引起的负载不平衡。我们的训练框架采用了大规模专家并行和数据并行，保证了每个微批次的大容量，从而自然而然地解决了第一个挑战。对于第二个挑战，我们还设计并实施了一个高效的推理框架，该框架采用冗余专家部署（如第 3.4 节所述）来克服它。

5. Post-Training 5.培训后

5.1. Supervised Fine-Tuning
5.1.监督微调

We curate our instruction-tuning datasets to include 1.5 M instances spanning multiple domains, with each domain employing distinct data creation methods tailored to its specific requirements.
我们的指令调整数据集包括跨越多个领域的 150 万个实例，每个领域都根据其特定要求采用不同的数据创建方法。

Reasoning Data. For reasoning-related datasets, including those focused on mathematics, code competition problems, and logic puzzles, we generate the data by leveraging an internal DeepSeek-R1 model. Specifically, while the R1-generated data demonstrates strong accuracy, it suffers from issues such as overthinking, poor formatting, and excessive length. Our objective is to balance the high accuracy of R1-generated reasoning data and the clarity and conciseness of regularly formatted reasoning data.
推理数据。对于推理相关的数据集，包括数学、代码竞赛问题和逻辑谜题，我们利用 DeepSeek-R1 内部模型生成数据。具体来说，虽然 R1 生成的数据具有很高的准确性，但也存在思考过度、格式不佳和过长等问题。我们的目标是在 R1 生成的推理数据的高准确性和格式规范的推理数据的清晰简洁性之间取得平衡。

To establish our methodology, we begin by developing an expert model tailored to a specific domain, such as code, mathematics, or general reasoning, using a combined Supervised FineTuning (SFT) and Reinforcement Learning (RL) training pipeline. This expert model serves as a data generator for the final model. The training process involves generating two distinct types of SFT samples for each instance: the first couples the problem with its original response in the format of <problem, original response>, while the second incorporates a system prompt alongside the problem and the R1 response in the format of <system prompt, problem, R1 response>.
为了建立我们的方法论，我们首先使用监督微调（SFT）和强化学习（RL）相结合的训练管道，开发一个针对特定领域（如代码、数学或一般推理）的专家模型。该专家模型是最终模型的数据生成器。训练过程包括为每个实例生成两种不同类型的 SFT 样本：第一种样本以 < 问题、原始响应> 的格式将问题与原始响应结合在一起，第二种样本则以 < 系统提示、问题、R1 响应> 的格式将系统提示与问题和 R1 响应结合在一起。

The system prompt is meticulously designed to include instructions that guide the model toward producing responses enriched with mechanisms for reflection and verification. During the RL phase, the model leverages high-temperature sampling to generate responses that integrate patterns from both the R1-generated and original data, even in the absence of explicit system prompts. After hundreds of RL steps, the intermediate RL model learns to incorporate R1 patterns, thereby enhancing overall performance strategically.
系统提示是经过精心设计的，其中包括指导模型做出具有反思和验证机制的反应的指令。在 RL 阶段，即使没有明确的系统提示，模型也会利用高温采样来生成集成了 R1 生成数据和原始数据模式的响应。经过数百个 RL 步骤后，中间 RL 模型学会了整合 R1 模式，从而战略性地提高了整体性能。

Upon completing the RL training phase, we implement rejection sampling to curate highquality SFT data for the final model, where the expert models are used as data generation sources. This method ensures that the final training data retains the strengths of DeepSeek-R1 while producing responses that are concise and effective.
在完成 RL 训练阶段后，我们实施拒绝采样，为最终模型收集高质量的 SFT 数据，其中专家模型被用作数据生成源。这种方法可确保最终训练数据保留 DeepSeek-R1 的优势，同时生成简洁有效的响应。

Non-Reasoning Data. For non-reasoning data, such as creative writing, role-play, and simple question answering, we utilize DeepSeek-V2.5 to generate responses and enlist human annotators to verify the accuracy and correctness of the data.
非推理数据。对于非推理数据，如创意写作、角色扮演和简单的问题回答，我们利用 DeepSeek-V2.5 生成回答，并请人类注释员验证数据的准确性和正确性。

SFT Settings. We fine-tune DeepSeek-V3-Base for two epochs using the SFT dataset, using the cosine decay learning rate scheduling that starts at

5 \times 10^{- 6}

and gradually decreases to

1 \times 10^{- 6}

. During training, each single sequence is packed from multiple samples. However, we adopt a sample masking strategy to ensure that these examples remain isolated and mutually invisible.
SFT 设置。我们使用 SFT 数据集对 DeepSeek-V3-Base 进行了两个 epoch 的微调，使用余弦衰减学习率调度，从

5 \times 10^{- 6}

开始，逐渐降低到

1 \times 10^{- 6}

。在训练过程中，每个单一序列都由多个样本打包而成。不过，我们采用了样本屏蔽策略，以确保这些示例保持隔离和互不可见。

5.2. Reinforcement Learning
5.2.强化学习

5.2.1. Reward Model 5.2.1.奖励模式

We employ a rule-based Reward Model (RM) and a model-based RM in our RL process.
我们在 RL 流程中采用了基于规则的奖励模型（RM）和基于模型的奖励模型。

Rule-Based RM. For questions that can be validated using specific rules, we adopt a rulebased reward system to determine the feedback. For instance, certain math problems have deterministic results, and we require the model to provide the final answer within a designated format (e.g., in a box), allowing us to apply rules to verify the correctness. Similarly, for LeetCode problems, we can utilize a compiler to generate feedback based on test cases. By leveraging rule-based validation wherever possible, we ensure a higher level of reliability, as this approach is resistant to manipulation or exploitation.
基于规则的奖励机制。对于可以使用特定规则验证的问题，我们采用基于规则的奖励系统来确定反馈。例如，某些数学问题的结果是确定的，我们要求模型在指定格式内（如在方框内）提供最终答案，这样我们就可以应用规则来验证其正确性。同样，对于 LeetCode 问题，我们可以利用编译器根据测试用例生成反馈。通过尽可能利用基于规则的验证，我们可以确保更高水平的可靠性，因为这种方法不易被操纵或利用。

Model-Based RM. For questions with free-form ground-truth answers, we rely on the reward model to determine whether the response matches the expected ground-truth. Conversely, for questions without a definitive ground-truth, such as those involving creative writing, the reward model is tasked with providing feedback based on the question and the corresponding answer
基于模型的 RM。对于有自由形式基本真相答案的问题，我们依靠奖励模型来确定答案是否符合预期的基本真相。相反，对于那些没有确定基本事实的问题，如涉及创意写作的问题，奖励模型的任务是根据问题和相应的答案提供反馈。
as inputs. The reward model is trained from the DeepSeek-V3 SFT checkpoints. To enhance its reliability, we construct preference data that not only provides the final reward but also includes the chain-of-thought leading to the reward. This approach helps mitigate the risk of reward hacking in specific tasks.
作为输入。奖励模型是通过 DeepSeek-V3 SFT 检查点训练出来的。为了提高其可靠性，我们构建的偏好数据不仅提供最终奖励，还包括导致奖励的思维链。这种方法有助于降低特定任务中奖励被黑客攻击的风险。

5.2.2. Group Relative Policy Optimization
5.2.2.组相对策略优化

Similar to DeepSeek-V2 (DeepSeek-AI, 2024c), we adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024), which foregoes the critic model that is typically with the same size as the policy model, and estimates the baseline from group scores instead. Specifically, for each question

q

, GRPO samples a group of outputs

{o_{1}, o_{2}, \dots, o_{G}}

from the old policy model

π_{θ_{old}}

and then optimizes the policy model

π_{θ}

by maximizing the following objective:
与 DeepSeek-V2 (DeepSeek-AI, 2024c)类似，我们采用了组相对策略优化 (GRPO) (Shao 等人, 2024)，它放弃了通常与策略模型大小相同的批评者模型，而是从组分数中估计基线。具体来说，对于每个问题

q

，GRPO 从旧政策模型

π_{θ_{old}}

中抽取一组输出

{o_{1}, o_{2}, \dots, o_{G}}

，然后通过最大化以下目标来优化政策模型

π_{θ}

：

\begin{matrix} J_{G R P O} (θ) = E [q \sim P (Q), {o_{i}}_{i = 1}^{G} \sim π_{θ_{o l d}} (O ∣ q)] \\ \frac{1}{G} \sum_{i = 1}^{G} (min (\frac{π_{θ} (o_{i} ∣ q)}{π_{θ_{o l d}} (o_{i} ∣ q)} A_{i}, clip (\frac{π_{θ} (o_{i} ∣ q)}{π_{θ_{o l d}} (o_{i} ∣ q)}, 1 - ε, 1 + ε) A_{i}) - β D_{K L} (π_{θ} | | π_{r e f})), \\ D_{K L} (π_{θ} | | π_{r e f}) = \frac{π_{r e f} (o_{i} ∣ q)}{π_{θ} (o_{i} ∣ q)} - \log \frac{π_{r e f} (o_{i} ∣ q)}{π_{θ} (o_{i} ∣ q)} - 1, \end{matrix}

where

ε

and

β

are hyper-parameters;

π_{r e f}

is the reference model; and

A_{i}

is the advantage, derived from the rewards

{r_{1}, r_{2}, \dots, r_{G}}

corresponding to the outputs within each group:
其中，

ε

和

β

为超参数；

π_{r e f}

为参考模型；

A_{i}

为优势，由每组输出对应的奖励

{r_{1}, r_{2}, \dots, r_{G}}

得出：

A_{i} = \frac{r_{i} - mean ({r_{1}, r_{2}, \dots, r_{G}})}{std ({r_{1}, r_{2}, \dots, r_{G}})} .

We incorporate prompts from diverse domains, such as coding, math, writing, role-playing, and question answering, during the RL process. This approach not only aligns the model more closely with human preferences but also enhances performance on benchmarks, especially in scenarios where available SFT data are limited.
我们在 RL 过程中加入了来自不同领域的提示，如编码、数学、写作、角色扮演和问题解答。这种方法不仅能使模型更贴近人类偏好，还能提高基准性能，尤其是在可用 SFT 数据有限的情况下。

5.3. Evaluations 5.3.评估

5.3.1. Evaluation Settings
5.3.1.评估设置

Evaluation Benchmarks. Apart from the benchmark we used for base model testing, we further evaluate instructed models on IFEval (Zhou et al., 2023), FRAMES (Krishna et al. 2024), LongBench v2 (Bai et al., 2024), GPQA (Rein et al., 2023), SimpleQA (OpenAI, 2024c), CSimpleQA (He et al. 2024), SWE-Bench Verified (OpenAI, 2024d), Aider

^{1}

, LiveCodeBench (Jain et al. 2024) (questions from August 2024 to November 2024), Codeforces

^{2}

, Chinese National High School Mathematics Olympiad (CNMO 2024)

^{3}

, and American Invitational Mathematics Examination 2024 (AIME 2024) (MAA, 2024).
评估基准。2023）、SimpleQA（OpenAI，2024c）、CSimpleQA（He 等人，2024）、SWE-Bench Verified（OpenAI，2024d）、Aider

^{1}

、LiveCodeBench（Jain et al.2024）（2024 年 8 月至 2024 年 11 月的试题）、Codeforces

^{2}

、中国全国高中数学奥林匹克竞赛（CNMO 2024）

^{3}

和 2024 年美国数学邀请考试（AIME 2024）（MAA，2024）。

Compared Baselines. We conduct comprehensive evaluations of our chat model against several strong baselines, including DeepSeek-V2-0506, DeepSeek-V2.5-0905, Qwen2.5 72B Instruct, LLaMA-3.1 405B Instruct, Claude-Sonnet-3.5-1022, and GPT-4o-0513. For the DeepSeek-V2 model series, we select the most representative variants for comparison. For closed-source models, evaluations are performed through their respective APIs.
对比基线。我们将聊天模型与几个强大的基线进行了综合评估，包括 DeepSeek-V2-0506、DeepSeek-V2.5-0905、Qwen2.5 72B Instruct、LLaMA-3.1 405B Instruct、Claude-Sonnet-3.5-1022 和 GPT-4o-0513。对于 DeepSeek-V2 模型系列，我们选择最具代表性的变体进行比较。对于闭源模型，我们通过其各自的应用程序接口进行评估。

Detailed Evaluation Configurations. For standard benchmarks including MMLU, DROP, GPQA, and SimpleQA, we adopt the evaluation prompts from the simple-evals framework

^{4}

We utilize the Zero-Eval prompt format (Lin, 2024) for MMLU-Redux in a zero-shot setting. For other datasets, we follow their original evaluation protocols with default prompts as provided by the dataset creators. For code and math benchmarks, the HumanEval-Mul dataset includes 8 mainstream programming languages (Python, Java, Cpp, C#, JavaScript, TypeScript, PHP, and Bash) in total. We use CoT and non-CoT methods to evaluate model performance on LiveCodeBench, where the data are collected from August 2024 to November 2024. The Codeforces dataset is measured using the percentage of competitors. SWE-Bench verified is evaluated using the agentless framework (Xia et al., 2024). We use the “diff” format to evaluate the Aider-related benchmarks. For mathematical assessments, AIME and CNMO 2024 are evaluated with a temperature of 0.7 , and the results are averaged over 16 runs, while MATH-500 employs greedy decoding. We allow all models to output a maximum of 8192 tokens for each benchmark.
详细评估配置。对于包括 MMLU、DROP、GPQA 和 SimpleQA 在内的标准基准，我们采用了 simple-evals 框架中的评估提示

^{4}

。对于 MMLU-Redux，我们采用了 Zero-Eval 提示格式（Lin，2024 年），并进行了零镜头设置。对于其他数据集，我们按照数据集创建者提供的默认提示遵循其原始评估协议。在代码和数学基准测试方面，HumanEval-Mul 数据集共包含 8 种主流编程语言（Python、Java、Cpp、C#、JavaScript、TypeScript、PHP 和 Bash）。我们使用 CoT 和非 CoT 方法评估模型在 LiveCodeBench 上的性能，数据收集时间为 2024 年 8 月至 2024 年 11 月。Codeforces 数据集采用竞争者百分比进行衡量。SWE-Bench 验证使用无代理框架进行评估（Xia 等人，2024 年）。我们使用 "diff "格式评估与 Aider 相关的基准。在数学评估中，AIME 和 CNMO 2024 的评估温度为 0.7，结果取 16 次运行的平均值，而 MATH-500 则采用贪婪解码。我们允许所有模型在每个基准中最多输出 8192 个标记。

Benchmark (Metric) 基准（指标）

\begin{matrix} DeepSeek \\ V2-0506 \end{matrix}

DeepSeek

V2.5-0905

Qwen2.5 72B-Inst.

LLaMA-3.1 405B-Inst.

克劳德-3.5-十四行诗-1022

Claude-3.5-

Sonnet-1022

\begin{matrix} GPT-4o \\ 0513 \end{matrix}

\begin{matrix} DeepSeek \\ V3 \end{matrix}

Architecture 建筑学

MoE 教育部

Dense 密集

MoE 教育部

# Activated Params # 激活参数

21B

72B

405B

37B

# Total Params # 参数总数

236B

72B

405B

671B

English 英语

MMLU (EM)

78.2

80.6

85.3

88.6

88.3

87.2

88.5

MMLU-Redux (EM)

77.9

80.3

85.6

86.2

88.9

88.0

89.1

MMLU-Pro (EM)

58.5

66.2

71.6

73.3

78.0

72.6

75.9

DROP (3-shot F1) 下降（3 发 F1）

83.0

87.8

76.7

88.7

88.3

83.7

91.

IF-Eval (Prompt Strict) IF-Eval（严格提示）

57.7

80.6

84.1

86.0

86.5

84.3

86.1

GPQA-Diamond (Pass@1)

35.3

41.3

49.0

51.1

65.0

49.9

59.1

SimpleQA (Correct) 简单质量保证（正确）

9.0

10.2

9.1

17.1

28.4

38.2

24.9

FRAMES (Acc.) 框架 (Acc.)

66.9

65.4

69.8

70.0

72.5

80.5

73.3

LongBench v2 (Acc.)

31.6

35.4

39.4

36.1

41.0

48.1

48.7

Code 代码

HumanEval-Mul (Pass@1)

69.3

77.4

77.3

77.2

81.7

80.5

82.6

LiveCodeBench (Pass@1-COT)

18.8

29.2

31.1

28.4

36.3

33.4

40.5

LiveCodeBench (Pass@1)

20.3

28.4

28.7

30.1

32.8

34.2

37.6

Codeforces (Percentile) 代码力（百分位数）

17.5

35.6

24.8

25.3

20.3

23.6

51.6

SWE Verified (Resolved) SWE 已验证（已解决）

22.6

23.8

24.5

50.8

38.8

42.0

Aider-Edit (Acc.)

60.3

71.6

65.4

63.9

84.2

72.9

79.7

Aider-Polyglot (Acc.) 多面手助手（Acc.）

18.2

7.6

5.8

45.3

16.0

49.6

Math 数学

AIME 2024 (Pass@1)

4.6

16.7

23.3

16.0

9.3

39.2

MATH-500 (ЕМ)

56.3

74.7

80.0

73.8

78.3

74.6

90.2

CNMO 2024 (Pass@1)

2.8

10.8

15.9

6.8

13.1

10.8

43.2

Chinese 中文

CLUEWSC (EM)

89.9

90.4

91.4

84.7

85.4

87.9

90.9

C-Eval (EM)

78.6

79.5

86.1

61.5

76.7

76.0

86.5

C-SimpleQA (Correct) C-简单质量保证（正确）

48.5

54.1

48.4

50.4

51.3

59.3

64.8

| | Benchmark (Metric) | $\begin{array}{\|c} \text { DeepSeek } \\ \text { V2-0506 } \end{array}$ | DeepSeek <br> V2.5-0905 | Qwen2.5 72B-Inst. | LLaMA-3.1 405B-Inst. | Claude-3.5- <br> Sonnet-1022 | $\begin{gathered} \text { GPT-4o } \\ 0513 \end{gathered}$ | $\begin{array}{\|c} \text { DeepSeek } \\ \text { V3 } \end{array}$ | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | | Architecture | MoE | MoE | Dense | Dense | - | - | MoE | | | # Activated Params | 21B | 21B | 72B | 405B | - | - | 37B | | | # Total Params | 236B | 236B | 72B | 405B | - | - | 671B | | English | MMLU (EM) | 78.2 | 80.6 | 85.3 | 88.6 | 88.3 | 87.2 | 88.5 | | | MMLU-Redux (EM) | 77.9 | 80.3 | 85.6 | 86.2 | 88.9 | 88.0 | 89.1 | | | MMLU-Pro (EM) | 58.5 | 66.2 | 71.6 | 73.3 | 78.0 | 72.6 | 75.9 | | | DROP (3-shot F1) | 83.0 | 87.8 | 76.7 | 88.7 | 88.3 | 83.7 | 91. | | | IF-Eval (Prompt Strict) | 57.7 | 80.6 | 84.1 | 86.0 | 86.5 | 84.3 | 86.1 | | | GPQA-Diamond (Pass@1) | 35.3 | 41.3 | 49.0 | 51.1 | 65.0 | 49.9 | 59.1 | | | SimpleQA (Correct) | 9.0 | 10.2 | 9.1 | 17.1 | 28.4 | 38.2 | 24.9 | | | FRAMES (Acc.) | 66.9 | 65.4 | 69.8 | 70.0 | 72.5 | 80.5 | 73.3 | | | LongBench v2 (Acc.) | 31.6 | 35.4 | 39.4 | 36.1 | 41.0 | 48.1 | 48.7 | | Code | HumanEval-Mul (Pass@1) | 69.3 | 77.4 | 77.3 | 77.2 | 81.7 | 80.5 | 82.6 | | | LiveCodeBench (Pass@1-COT) | 18.8 | 29.2 | 31.1 | 28.4 | 36.3 | 33.4 | 40.5 | | | LiveCodeBench (Pass@1) | 20.3 | 28.4 | 28.7 | 30.1 | 32.8 | 34.2 | 37.6 | | | Codeforces (Percentile) | 17.5 | 35.6 | 24.8 | 25.3 | 20.3 | 23.6 | 51.6 | | | SWE Verified (Resolved) | - | 22.6 | 23.8 | 24.5 | 50.8 | 38.8 | 42.0 | | | Aider-Edit (Acc.) | 60.3 | 71.6 | 65.4 | 63.9 | 84.2 | 72.9 | 79.7 | | | Aider-Polyglot (Acc.) | - | 18.2 | 7.6 | 5.8 | 45.3 | 16.0 | 49.6 | | Math | AIME 2024 (Pass@1) | 4.6 | 16.7 | 23.3 | 23.3 | 16.0 | 9.3 | 39.2 | | | MATH-500 (ЕМ) | 56.3 | 74.7 | 80.0 | 73.8 | 78.3 | 74.6 | 90.2 | | | CNMO 2024 (Pass@1) | 2.8 | 10.8 | 15.9 | 6.8 | 13.1 | 10.8 | 43.2 | | Chinese | CLUEWSC (EM) | 89.9 | 90.4 | 91.4 | 84.7 | 85.4 | 87.9 | 90.9 | | | C-Eval (EM) | 78.6 | 79.5 | 86.1 | 61.5 | 76.7 | 76.0 | 86.5 | | | C-SimpleQA (Correct) | 48.5 | 54.1 | 48.4 | 50.4 | 51.3 | 59.3 | 64.8 |

Table 6 | Comparison between DeepSeek-V3 and other representative chat models. All models are evaluated in a configuration that limits the output length to 8 K . Benchmarks containing fewer than 1000 samples are tested multiple times using varying temperature settings to derive robust final results. DeepSeek-V3 stands as the best-performing open-source model, and also exhibits competitive performance against frontier closed-source models.
表 6：DeepSeek-V3 与其他代表性聊天模型的比较。所有模型都是在将输出长度限制为 8 K 的配置下进行评估的。包含少于 1000 个样本的基准测试使用不同的温度设置进行了多次测试，以得出稳健的最终结果。DeepSeek-V3 是性能最好的开源模型，与前沿的闭源模型相比也具有很强的竞争力。

5.3.2. Standard Evaluation
5.3.2.标准评估

Table 6 presents the evaluation results, showcasing that DeepSeek-V3 stands as the bestperforming open-source model. Additionally, it is competitive against frontier closed-source models like GPT-4o and Claude-3.5-Sonnet.
表 6 列出了评估结果，表明 DeepSeek-V3 是表现最佳的开源模型。此外，与 GPT-4o 和 Claude-3.5-Sonnet 等前沿闭源模型相比，它也很有竞争力。

English Benchmarks. MMLU is a widely recognized benchmark designed to assess the performance of large language models, across diverse knowledge domains and tasks. DeepSeek-V3 demonstrates competitive performance, standing on par with top-tier models such as LLaMA-3.1-405B, GPT-4o, and Claude-Sonnet 3.5, while significantly outperforming Qwen2.5 72B. Moreover, DeepSeek-V3 excels in MMLU-Pro, a more challenging educational knowledge benchmark, where it closely trails Claude-Sonnet 3.5. On MMLU-Redux, a refined version of MMLU with corrected labels, DeepSeek-V3 surpasses its peers. In addition, on GPQA-Diamond, a PhD-level evaluation testbed, DeepSeek-V3 achieves remarkable results, ranking just behind Claude 3.5 Sonnet and outperforming all other competitors by a substantial margin.
英语基准。MMLU 是一项广受认可的基准测试，旨在评估大型语言模型在不同知识领域和任务中的性能。DeepSeek-V3 的性能极具竞争力，可与 LLaMA-3.1-405B、GPT-4o 和 Claude-Sonnet 3.5 等顶级模型相媲美，同时明显优于 Qwen2.5 72B。此外，DeepSeek-V3 在更具挑战性的教育知识基准 MMLU-Pro 中表现出色，紧追 Claude-Sonnet 3.5。在MMLU-Redux上，DeepSeek-V3超越了其他同类产品。此外，在博士级评估测试平台GPQA-Diamond上，DeepSeek-V3也取得了不俗的成绩，排名仅次于克劳德-桑内特3.5，远远超过了其他所有竞争对手。

In long-context understanding benchmarks such as DROP, LongBench v2, and FRAMES, DeepSeek-V3 continues to demonstrate its position as a top-tier model. It achieves an impressive 91.6 F1 score in the 3-shot setting on DROP, outperforming all other models in this category. On FRAMES, a benchmark requiring question-answering over 100k token contexts, DeepSeekV3 closely trails GPT-4o while outperforming all other models by a significant margin. This demonstrates the strong capability of DeepSeek-V3 in handling extremely long-context tasks. The long-context capability of DeepSeek-V3 is further validated by its best-in-class performance on LongBench v2, a dataset that was released just a few weeks before the launch of DeepSeek V3. On the factual knowledge benchmark, SimpleQA, DeepSeek-V3 falls behind GPT-4o and Claude-Sonnet, primarily due to its design focus and resource allocation. DeepSeek-V3 assigns more training tokens to learn Chinese knowledge, leading to exceptional performance on the C-SimpleQA. On the instruction-following benchmark, DeepSeek-V3 significantly outperforms its predecessor, DeepSeek-V2-series, highlighting its improved ability to understand and adhere to user-defined format constraints.
在 DROP、LongBench v2 和 FRAMES 等长文本理解基准测试中，DeepSeek-V3 继续彰显其顶级机型的地位。它在 DROP 的 3shot 设置中取得了令人印象深刻的 91.6 F1 分数，超过了该类别中的所有其他模型。在FRAMES（一个需要回答超过10万个令牌上下文的问题的基准测试）上，DeepSeekV3紧随GPT-4o之后，同时以显著优势超越所有其他模型。这表明DeepSeek-V3在处理超长语境任务方面具有强大的能力。DeepSeek-V3 在 LongBench v2（DeepSeek V3 发布前几周刚刚发布的数据集）上的最佳表现进一步验证了 DeepSeek-V3 的长语境能力。在事实知识基准SimpleQA上，DeepSeek-V3落后于GPT-4o和Claude-Sonnet，主要原因是其设计重点和资源分配。DeepSeek-V3 分配了更多的训练代币来学习中文知识，因此在 C-SimpleQA 上表现优异。在指令跟随基准测试中，DeepSeek-V3的表现明显优于其前身DeepSeek-V2系列，这凸显了它在理解和遵守用户自定义格式约束方面的能力得到了提高。

Code and Math Benchmarks. Coding is a challenging and practical task for LLMs, encompassing engineering-focused tasks like SWE-Bench-Verified and Aider, as well as algorithmic tasks such as HumanEval and LiveCodeBench. In engineering tasks, DeepSeek-V3 trails behind Claude-Sonnet-3.5-1022 but significantly outperforms open-source models. The open-source DeepSeek-V3 is expected to foster advancements in coding-related engineering tasks. By providing access to its robust capabilities, DeepSeek-V3 can drive innovation and improvement in areas such as software engineering and algorithm development, empowering developers and researchers to push the boundaries of what open-source models can achieve in coding tasks. In algorithmic tasks, DeepSeek-V3 demonstrates superior performance, outperforming all baselines on benchmarks like HumanEval-Mul and LiveCodeBench. This success can be attributed to its advanced knowledge distillation technique, which effectively enhances its code generation and problem-solving capabilities in algorithm-focused tasks.
代码和数学基准。对于 LLMs 来说，编码是一项具有挑战性的实际任务，包括 SWE-Bench-Verified 和 Aider 等工程任务，以及 HumanEval 和 LiveCodeBench 等算法任务。在工程任务中，DeepSeek-V3落后于Claude-Sonnet-3.5-1022，但明显优于开源模型。开源 DeepSeek-V3 预计将促进编码相关工程任务的进步。通过提供对其强大功能的访问，DeepSeek-V3 可以推动软件工程和算法开发等领域的创新和改进，使开发人员和研究人员有能力突破开源模型在编码任务中所能达到的极限。在算法任务中，DeepSeek-V3表现出卓越的性能，在HumanEval-Mul和LiveCodeBench等基准测试中超过了所有基准测试。这一成功归功于其先进的知识提炼技术，该技术在以算法为核心的任务中有效增强了代码生成和问题解决能力。

On math benchmarks, DeepSeek-V3 demonstrates exceptional performance, significantly surpassing baselines and setting a new state-of-the-art for non-o1-like models. Specifically, on AIME, MATH-500, and CNMO 2024, DeepSeek-V3 outperforms the second-best model, Qwen2.5 72B, by approximately

10 %

in absolute scores, which is a substantial margin for such challenging benchmarks. This remarkable capability highlights the effectiveness of the distillation technique from DeepSeek-R1, which has been proven highly beneficial for non-o1-like models.
在数学基准测试中，DeepSeek-V3表现出了卓越的性能，大大超过了基准线，为非O1类模型树立了新的标杆。具体来说，在AIME、MATH-500和CNMO 2024上，DeepSeek-V3的绝对成绩超过了排名第二的Qwen2.5 72B模型约

10 %

，对于此类具有挑战性的基准来说，这是一个相当大的差距。这一卓越能力凸显了DeepSeek-R1中蒸馏技术的有效性，该技术已被证明对非O1类模型大有裨益。

Model 模型	Arena-Hard 竞技场-硬质	AlpacaEval 2.0
DeepSeek-V2.5-0905	76.2	50.5
Qwen2.5-72B-Instruct	81.2	49.1
LLaMA-3.1 405B	69.3	40.5
GPT-4o-0513	80.4	51.1
Claude-Sonnet-3.5-1022	85.2	52.0
DeepSeek-V3	$85.5$	$70.0$

Table 7 | English open-ended conversation evaluations. For AlpacaEval 2.0, we use the lengthcontrolled win rate as the metric.
表 7 英语开放式对话评价。在 AlpacaEval 2.0 中，我们使用长度控制胜率作为衡量标准。

Chinese Benchmarks. Qwen and DeepSeek are two representative model series with robust support for both Chinese and English. On the factual benchmark Chinese SimpleQA, DeepSeekV3 surpasses Qwen2.5-72B by 16.4 points, despite Qwen 2.5 being trained on a larger corpus compromising 18 T tokens, which are

20 %

more than the 14.8 T tokens that DeepSeek-V3 is pre-trained on.
中文基准。Qwen和DeepSeek是两个具有代表性的模型系列，对中文和英文都有强大的支持。在事实基准中文SimpleQA上，DeepSeekV3比Qwen2.5-72B高出16.4分，尽管Qwen2.5是在更大的语料库上训练的，有18 T词条，比DeepSeek-V3预先训练的14.8 T词条多

20 %

。

On C-Eval, a representative benchmark for Chinese educational knowledge evaluation, and CLUEWSC (Chinese Winograd Schema Challenge), DeepSeek-V3 and Qwen2.5-72B exhibit similar performance levels, indicating that both models are well-optimized for challenging Chinese-language reasoning and educational tasks.
在具有代表性的中文教育知识评估基准C-Eval和CLUEWSC（中文维诺格拉德模式挑战赛）上，DeepSeek-V3和Qwen2.5-72B表现出了相似的性能水平，表明这两个模型都针对具有挑战性的中文推理和教育任务进行了优化。

5.3.3. Open-Ended Evaluation
5.3.3.开放式评估

In addition to standard benchmarks, we also evaluate our models on open-ended generation tasks using LLMs as judges, with the results shown in Table 7. Specifically, we adhere to the original configurations of AlpacaEval 2.0 (Dubois et al., 2024) and Arena-Hard (Li et al., 2024a), which leverage GPT-4-Turbo-1106 as judges for pairwise comparisons. On Arena-Hard, DeepSeek-V3 achieves an impressive win rate of over

86 %

against the baseline GPT-4-0314, performing on par with top-tier models like Claude-Sonnet-3.5-1022. This underscores the robust capabilities of DeepSeek-V3, especially in dealing with complex prompts, including coding and debugging tasks. Furthermore, DeepSeek-V3 achieves a groundbreaking milestone as the first open-source model to surpass

85 %

on the Arena-Hard benchmark. This achievement significantly bridges the performance gap between open-source and closed-source models, setting a new standard for what open-source models can accomplish in challenging domains.
除了标准基准，我们还在以 LLMs 为评委的开放式生成任务中评估了我们的模型，结果如表 7 所示。具体来说，我们采用了 AlpacaEval 2.0（Dubois 等人，2024 年）和 Arena-Hard（Li 等人，2024a）的原始配置，利用 GPT-4-Turbo-1106 作为评委进行成对比较。在Arena-Hard上，DeepSeek-V3对基线GPT-4-0314的胜率超过

86 %

，与Claude-Sonnet-3.5-1022等顶级模型不相上下。这凸显了 DeepSeek-V3 的强大功能，尤其是在处理复杂提示（包括编码和调试任务）时。此外，作为首个在Arena-Hard基准测试中超越

85 %

的开源模型，DeepSeek-V3实现了一个开创性的里程碑。这一成就大大缩小了开源模型与闭源模型之间的性能差距，为开源模型在具有挑战性的领域所能达到的目标设定了新的标准。

Similarly, DeepSeek-V3 showcases exceptional performance on AlpacaEval 2.0, outperforming both closed-source and open-source models. This demonstrates its outstanding proficiency in writing tasks and handling straightforward question-answering scenarios. Notably, it surpasses DeepSeek-V2.5-0905 by a significant margin of

20 %

, highlighting substantial improvements in tackling simple tasks and showcasing the effectiveness of its advancements.
同样，DeepSeek-V3 在 AlpacaEval 2.0 上也表现出了卓越的性能，超过了闭源和开源模型。这表明它在编写任务和处理直接的问题解答场景方面表现出色。值得注意的是，它以

20 %

的显著优势超越了 DeepSeek-V2.5-0905，突显了其在处理简单任务方面的实质性改进，并展示了其进步的有效性。

5.3.4. DeepSeek-V3 as a Generative Reward Model
5.3.4.作为生成奖励模型的 DeepSeek-V3

We compare the judgment ability of DeepSeek-V3 with state-of-the-art models, namely GPT-4o and Claude-3.5. Table 8 presents the performance of these models in RewardBench (Lambert et al., 2024). DeepSeek-V3 achieves performance on par with the best versions of GPT-4o-0806 and Claude-3.5-Sonnet-1022, while surpassing other versions. Additionally, the judgment ability of DeepSeek-V3 can also be enhanced by the voting technique. Therefore, we employ DeepSeekV3 along with voting to offer self-feedback on open-ended questions, thereby improving the
我们将 DeepSeek-V3 的判断能力与最先进的模型（即 GPT-4o 和 Claude-3.5）进行了比较。表 8 列出了这些模型在 RewardBench（Lambert 等人，2024 年）中的表现。DeepSeek-V3 的性能与 GPT-4o-0806 和 Claude-3.5-Sonnet-1022 的最佳版本相当，同时超过了其他版本。此外，DeepSeek-V3 的判断能力还可以通过投票技术得到增强。因此，我们在使用DeepSeek-V3的同时还采用了投票技术，对开放式问题进行自我反馈，从而提高了DeepSeek-V3的判断能力。

Model 模型	Chat 聊天	Chat-Hard 聊天-困难	Safety 安全	Reasoning 推理	Average 平均
GPT-4o-0513	96.6	70.4	86.7	84.9	84.7
GPT-4o-0806	96.1	76.1	88.1	86.6	86.7
GPT-4o-1120	95.8	71.3	86.2	85.2	84.6
Claude-3.5-sonnet-0620	96.4	74.0	81.6	84.7	84.2
Claude-3.5-sonnet-1022	96.4	79.7	91.1	87.6	88.7
DeepSeek-V3	96.9	79.8	87.0	84.3	87.0
DeepSeek-V3 (maj@6)	96.9	82.6	89.5	89.2	89.6

Table 8 | Performances of GPT-4o, Claude-3.5-sonnet and DeepSeek-V3 on RewardBench.
表 8：GPT-4o、Claude-3.5-sonnet 和 DeepSeek-V3 在 RewardBench 上的性能。

Model 模型	LiveCodeBench-CoT		MATH-500
Model 模型	Pass@1 通过@1	Length 长度	Pass@1 通过@1	Length 长度
DeepSeek-V2.5 Baseline DeepSeek-V2.5 基线	31.1	718	74.6	769
DeepSeek-V2.5 +R1 Distill DeepSeek-V2.5 +R1 蒸馏器	37.4	783	83.2	1510

Table 9 | The contribution of distillation from DeepSeek-R1. The evaluation settings of LiveCodeBench and MATH-500 are the same as in Table 6
表 9：DeepSeek-R1 的蒸馏贡献。LiveCodeBench 和 MATH-500 的评估设置与表 6 相同
effectiveness and robustness of the alignment process.
协调进程的有效性和稳健性。

5.4. Discussion 5.4.讨论情况

5.4.1. Distillation from DeepSeek-R1
5.4.1.从 DeepSeek-R1 蒸馏

We ablate the contribution of distillation from DeepSeek-R1 based on DeepSeek-V2.5. The baseline is trained on short CoT data, whereas its competitor uses data generated by the expert checkpoints described above.
我们根据 DeepSeek-V2.5 删除了 DeepSeek-R1 中的蒸馏贡献。基准线是在短 CoT 数据上训练出来的，而它的竞争对手则使用上述专家检查点生成的数据。

Table 9 demonstrates the effectiveness of the distillation data, showing significant improvements in both LiveCodeBench and MATH-500 benchmarks. Our experiments reveal an interesting trade-off: the distillation leads to better performance but also substantially increases the average response length. To maintain a balance between model accuracy and computational efficiency, we carefully selected optimal settings for DeepSeek-V3 in distillation.
表 9 显示了蒸馏数据的有效性，在 LiveCodeBench 和 MATH-500 基准测试中均有显著改进。我们的实验揭示了一个有趣的权衡问题：蒸馏能带来更好的性能，但同时也大大增加了平均响应长度。为了在模型准确性和计算效率之间保持平衡，我们精心为 DeepSeek-V3 选择了最佳蒸馏设置。

Our research suggests that knowledge distillation from reasoning models presents a promising direction for post-training optimization. While our current work focuses on distilling data from mathematics and coding domains, this approach shows potential for broader applications across various task domains. The effectiveness demonstrated in these specific areas indicates that long-CoT distillation could be valuable for enhancing model performance in other cognitive tasks requiring complex reasoning. Further exploration of this approach across different domains remains an important direction for future research.
我们的研究表明，从推理模型中提炼知识是训练后优化的一个很有前景的方向。虽然我们目前的工作重点是提炼数学和编码领域的数据，但这种方法显示出在各种任务领域进行更广泛应用的潜力。在这些特定领域中表现出的有效性表明，在其他需要复杂推理的认知任务中，长CoT提炼对于提高模型性能可能很有价值。在不同领域进一步探索这种方法仍然是未来研究的一个重要方向。

5.4.2. Self-Rewarding 5.4.2.自我奖励

Rewards play a pivotal role in RL, steering the optimization process. In domains where verification through external tools is straightforward, such as some coding or mathematics scenarios, RL demonstrates exceptional efficacy. However, in more general scenarios, constructing a feedback
奖励在 RL 中发挥着关键作用，引导着优化过程。在通过外部工具直接验证的领域，如一些编码或数学场景，RL 显示出了非凡的功效。然而，在更一般的情况下，构建一个反馈
mechanism through hard coding is impractical. During the development of DeepSeek-V3, for these broader contexts, we employ the constitutional AI approach (Bai et al., 2022), leveraging the voting evaluation results of DeepSeek-V3 itself as a feedback source. This method has produced notable alignment effects, significantly enhancing the performance of DeepSeek-V3 in subjective evaluations. By integrating additional constitutional inputs, DeepSeek-V3 can optimize towards the constitutional direction. We believe that this paradigm, which combines supplementary information with LLMs as a feedback source, is of paramount importance. The LLM serves as a versatile processor capable of transforming unstructured information from diverse scenarios into rewards, ultimately facilitating the self-improvement of LLMs. Beyond self-rewarding, we are also dedicated to uncovering other general and scalable rewarding methods to consistently advance the model capabilities in general scenarios.
通过硬编码来实现这种机制是不切实际的。在DeepSeek-V3的开发过程中，针对这些更广泛的情况，我们采用了宪法人工智能方法（Bai等人，2022年），利用DeepSeek-V3本身的投票评估结果作为反馈源。这种方法产生了显著的对齐效果，大大提高了DeepSeek-V3在主观评估中的表现。通过整合额外的宪法输入，DeepSeek-V3 可以朝着宪法方向进行优化。我们认为，将补充信息与作为反馈源的 LLMs 结合起来的这一范例至关重要。LLM是一个多功能处理器，能够将来自不同场景的非结构化信息转化为奖励，最终促进LLMs的自我完善。除了自我奖励之外，我们还致力于发掘其他通用的、可扩展的奖励方法，以不断提高模型在一般场景中的能力。

5.4.3. Multi-Token Prediction Evaluation
5.4.3.多令牌预测评估

Instead of predicting just the next single token, DeepSeek-V3 predicts the next 2 tokens through the MTP technique. Combined with the framework of speculative decoding (Leviathan et al., 2023. Xia et al., 2023), it can significantly accelerate the decoding speed of the model. A natural question arises concerning the acceptance rate of the additionally predicted token. Based on our evaluation, the acceptance rate of the second token prediction ranges between

85 %

and

90 %

across various generation topics, demonstrating consistent reliability. This high acceptance rate enables DeepSeek-V3 to achieve a significantly improved decoding speed, delivering 1.8 times TPS (Tokens Per Second).
DeepSeek-V3 通过 MTP 技术预测下两个标记，而不是只预测下一个标记。结合推测解码框架（Leviathan 等人，2023；Xia 等人，2023），它可以显著加快模型的解码速度。一个自然而然的问题是附加预测标记的接受率。根据我们的评估，在不同的生成主题中，第二个标记预测的接受率介于

85 %

和

90 %

之间，显示了一致的可靠性。如此高的接受率使得 DeepSeek-V3 的解码速度大幅提高，达到了 1.8 倍的 TPS（每秒令牌数）。

6. Conclusion, Limitations, and Future Directions
6.结论、局限性和未来方向

In this paper, we introduce DeepSeek-V3, a large MoE language model with 671B total parameters and 37B activated parameters, trained on 14.8 T tokens. In addition to the MLA and DeepSeekMoE architectures, it also pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. The training of DeepSeek-V3 is cost-effective due to the support of FP8 training and meticulous engineering optimizations. The post-training also makes a success in distilling the reasoning capability from the DeepSeek-R1 series of models. Comprehensive evaluations demonstrate that DeepSeek-V3 has emerged as the strongest open-source model currently available, and achieves performance comparable to leading closed-source models like GPT-4o and Claude-3.5-Sonnet. Despite its strong performance, it also maintains economical training costs. It requires only 2.788 M H800 GPU hours for its full training, including pre-training, context length extension, and post-training.
本文介绍了 DeepSeek-V3，这是一个大型 MoE 语言模型，拥有 671B 总参数和 37B 激活参数，在 14.8T 标记上进行了训练。除了 MLA 和 DeepSeekMoE 架构外，它还首创了一种无辅助损失的负载均衡策略，并设定了一个多令牌预测训练目标，以获得更强的性能。由于支持 FP8 训练和精心的工程优化，DeepSeek-V3 的训练具有很高的成本效益。后期训练还成功地提炼了DeepSeek-R1系列模型的推理能力。综合评估表明，DeepSeek-V3已成为目前最强大的开源模型，其性能可与GPT-4o和Claude-3.5-Sonnet等领先的闭源模型相媲美。尽管性能强大，但它的培训成本也很经济。它的全部训练（包括预训练、上下文长度扩展和后训练）只需要 2.788 M H800 GPU 小时。

While acknowledging its strong performance and cost-effectiveness, we also recognize that DeepSeek-V3 has some limitations, especially on the deployment. Firstly, to ensure efficient inference, the recommended deployment unit for DeepSeek-V3 is relatively large, which might pose a burden for small-sized teams. Secondly, although our deployment strategy for DeepSeekV3 has achieved an end-to-end generation speed of more than two times that of DeepSeek-V2, there still remains potential for further enhancement. Fortunately, these limitations are expected to be naturally addressed with the development of more advanced hardware.
在肯定DeepSeek-V3强大性能和高性价比的同时，我们也认识到DeepSeek-V3存在一些局限性，尤其是在部署方面。首先，为了确保高效推理，DeepSeek-V3 的推荐部署单元相对较大，这可能会给小型团队带来负担。其次，尽管我们为DeepSeek-V3制定的部署策略使端到端生成速度达到了DeepSeek-V2的两倍多，但仍有进一步提升的潜力。幸运的是，随着更先进硬件的开发，这些局限性有望得到自然解决。

DeepSeek consistently adheres to the route of open-source models with longtermism, aiming to steadily approach the ultimate goal of AGI (Artificial General Intelligence). In the future, we plan to strategically invest in research across the following directions.
DeepSeek 始终坚持开源模式与长期主义相结合的路线，旨在稳步接近 AGI（人工通用智能）的终极目标。未来，我们计划在以下几个方向进行战略性的研究投入。

We will consistently study and refine our model architectures, aiming to further improve
我们将不断研究和完善我们的模型架构，旨在进一步改进
both the training and inference efficiency, striving to approach efficient support for infinite context length. Additionally, we will try to break through the architectural limitations of Transformer, thereby pushing the boundaries of its modeling capabilities.
在训练和推理效率方面，我们都在努力接近对无限上下文长度的高效支持。此外，我们还将尝试突破 Transformer 的架构限制，从而推动其建模能力的发展。
We will continuously iterate on the quantity and quality of our training data, and explore the incorporation of additional training signal sources, aiming to drive data scaling across a more comprehensive range of dimensions.
我们将不断改进训练数据的数量和质量，并探索纳入更多的训练信号源，目的是在更全面的维度上推动数据扩展。
We will consistently explore and iterate on the deep thinking capabilities of our models, aiming to enhance their intelligence and problem-solving abilities by expanding their reasoning length and depth.
我们将不断探索和迭代模型的深度思维能力，旨在通过扩展其推理长度和深度来提高其智力和解决问题的能力。
We will explore more comprehensive and multi-dimensional model evaluation methods to prevent the tendency towards optimizing a fixed set of benchmarks during research, which may create a misleading impression of the model capabilities and affect our foundational assessment.
我们将探索更全面、更多维的模型评估方法，以防止在研究过程中倾向于优化一套固定的基准，从而对模型的能力造成误解，影响我们的基础评估。

References 参考资料

AI@Meta. Llama 3 model card, 2024a. URL https://github.com/meta-llama/llama3/bl ob/main/MODEL_CARD.md
AI@Meta.Llama 3 模型卡，2024a。URL https://github.com/meta-llama/llama3/bl ob/main/MODEL_CARD.md

AI@Meta. Llama 3.1 model card, 2024b. URL https://github. com/meta-llama/llama-m odels/blob/main/models/llama3_1/MODEL_CARD.md.
AI@Meta.Llama 3.1 模型卡，2024b。URL https://github. com/meta-llama/llama-m odels/blob/main/models/llama3_1/MODEL_CARD.md.

Anthropic. Claude 3.5 sonnet, 2024. URL https://www. anthropic.com/news/claude-3 -5-sonnet.
人类。克劳德 3.5十四行诗，2024 年。URL https://www. anthropic.com/news/claude-3-5-sonnet。
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022.
Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, J. Tang, and J. Li. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024.
Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, J. Tang, and J. Li.LongBench v2：ArXiv preprint arXiv:2412.15204, 2024.
M. Bauer, S. Treichler, and A. Aiken. Singe: leveraging warp specialization for high performance on GPUs. In Proceedings of the 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP '14, page 119-130, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450326568. doi: 10.1145/2555243.2555258. URL https://doi.org/10.1145/2555243.2555258
M. Bauer, S. Treichler, and A. Aiken.Singe：利用翘曲特化在 GPU 上实现高性能。In Proceedings of the 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP '14, page 119-130, New York, NY, USA, 2014.美国计算机协会。ISBN9781450326568。DOI：10.1145/2555243.2555258。URL https://doi.org/10.1145/2555243.2555258
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432-7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi.PIQA：用自然语言推理物理常识。In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432-7439.AAAI出版社，2020年。DOI：10.1609/AAAI.v34i05.6239。URL https://doi.org/10.1609/aaai.v34i05.6239
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin,
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin、

Appendix 附录

A. Contributions and Acknowledgments
A.贡献和鸣谢

Research & Engineering  研究与工程
Aixin Liu  刘爱新
Bing Xue  薛冰
Bingxuan Wang  王炳萱
Bochao Wu  吴博超
Chengda Lu  成达路
Chenggang Zhao  赵成刚
Chengqi Deng
Chenyu Zhang*  张晨宇*
Chong Ruan  阮崇
Damai Dai  达迈-戴
Daya Guo  郭大雅
Dejian Yang  杨德建
Deli Chen
Erhang Li  李二航
Fangyun Lin  林芳云
Fucong Dai  戴福聪
Fuli Luo*  罗富礼*
Guangbo Hao  郝广博
Guanting Chen
Guowei Li  李国伟
H. Zhang  H. 张

Han Bao*  韩宝*
Hanwei Xu
Haocheng Wang*  王浩成*
Haowei Zhang
Honghui Ding  丁红辉
Huajian Xin*  华建新*
Huazuo Gao  高华作
Hui Qu
Jianzhong Guo  郭建中
Jiashi Li  李嘉石
Jiawei Wang*  王嘉伟*
Jingchang Chen  陈景昌
Jingyang Yuan  景阳园
Junjie Qiu  邱俊杰
Junlong Li  李俊龙
Junxiao Song
Kai Dong  董凯
Kai Hu*  胡凯*
Kaige Gao  高凯歌
Kang Guan
Kexin Huang  黄可欣
Kuai Yu
Lean Wang

Lecong Zhang  张乐聪
Liang Zhao  梁昭
Litong Wang  王立彤
Liyue Zhang  张丽悦
Mingchuan Zhang  张明川
Minghua Zhang  张明华
Minghui Tang  唐明辉
Panpan Huang  黄盼盼
Peiyi Wang  王佩仪
Qiancheng Wang
Qihao Zhu
Qinyu Chen
Qiushi Du  杜秋实
Ruiqi Ge
Ruisong Zhang  张瑞松
Ruizhe Pan  潘瑞哲
Runji Wang  王润吉
Runxin Xu  徐润新
Ruoyu Zhang  张若愚
Shanghao Lu
Shangyan Zhou
Shanhuang Chen  陈善煌
Shengfeng Ye  叶盛峰
Shirong Ma  马世荣
Shiyu Wang  王诗雨
Shuiping Yu  于水萍
Shunfeng Zhou  周顺风
Shuting Pan  潘淑婷
Tao Yun  陶云
Tian Pei  田沛
Wangding Zeng  曾望定
Wanjia Zhao*  赵万佳*
Wen Liu  刘雯
Wenfeng Liang  梁文峰
Wenjun Gao  高文俊
Wenqin Yu  于文琴
Wentao Zhang  张文涛
Xiao Bi  肖毕
Xiaodong Liu  刘小东
Xiaohan Wang  王晓涵
Xiaokang Chen  陈小康
Xiaokang Zhang  张小康
Xiaotao Nie  聂小涛
Xin Cheng
Xin Liu

Xin Xie
Xingchao Liu  刘兴超
Xingkai Yu  于兴凯
Xinyu Yang  杨新宇
Xinyuan Li  李新元
Xuecheng Su  苏学成
Xuheng Lin  林旭恒
Y.K. Li
Y.Q. Wang
Y.X. Wei

Yang Zhang  张扬
Yanhong Xu  徐艳红
Yao Li  姚莉
Yao Zhao
Yaofeng Sun  孙耀峰
Yaohui Wang  王耀辉
Yi Yu
Yichao Zhang  张轶超
Yifan Shi  施一凡
Yiliang Xiong  熊一亮
Ying He  何颖
Yishi Piao
Yisong Wang  王毅松
Yixuan Tan  谭怡璇
Yiyang Ma*  马逸阳*
Yiyuan Liu
Yongqiang Guo  郭永强
Yu Wu
Yuan Ou  袁欧
Yuduan Wang  王玉端
Yue Gong  龚玥
Yuheng Zou  邹宇恒
Yujia He  何雨佳
Yunfan Xiong  熊云帆
Yuxiang Luo  罗宇翔
Yuxiang You  游宇翔
Yuxuan Liu  刘宇轩
Yuyang Zhou  周宇阳
Z.F. Wu
Z.Z. Ren

Zehui Ren  任泽辉
Zhangli Sha
Zhe Fu  傅哲
Zhean Xu  徐哲安
Zhenda Xie
Zhengyan Zhang  张正艳
Zhewen Hao  郝哲文
Zhibin Gou  苟志斌
Zhicheng Ma  马志成

Zhigang Yan  颜志刚
Zhihong Shao  邵志宏
Zhiyu Wu  吴志宇
Zhuoshu Li  李卓舒
Zihui Gu
Zijia Zhu  朱子佳
Zijun Liu*  刘子军*
Zilin Li  李子林
Ziwei Xie
Ziyang Song  宋子扬
Ziyi Gao  高子怡
Zizheng Pan  潘子正

Data Annotation 数据注释
Bei Feng 北风
Hui Li
J.L. Cai

Jiaqi Ni  倪佳琪
Lei Xu  雷旭
Meng Li  李萌
Ning Tian  宁天
R.J. Chen
R.L. Jin

Ruyi Chen 陈如意
S.S. Li

Shuang Zhou 周爽
Tianyu Sun 孙天宇
X.Q. Li

Xiangyue Jin  金湘岳
Xiaojin Shen  沈晓津
Xiaosha Chen  陈小沙
Xiaowen Sun  孙晓文
Xiaoxiang Wang
Xinnan Song  宋茜楠
Xinyi Zhou
Y.X. Zhu

Yanhong Xu  徐艳红
Yanping Huang  黄艳平
Yaohui Li  李耀辉
Yi Zheng
Yuchen Zhu
Yunxian Ma  马云贤
Zhen Huang
Zhipeng Xu  徐志鹏
Zhongyu Zhang  张中玉

Business & Compliance 业务与合规
Dongjie Ji

Jian Liang 梁健	W.L. Xiao
Jin Chen 陈进	Wei An 魏安
Leyi Xia	Xianzu Wang 王显祖
Miaojun Wang 王妙君	Xinxia Shan 单新霞
Mingming Li 李明明	Ying Tang
Peng Zhang 张鹏	Yukun Zha 查玉坤
Shaoqing Wu 吴少青	Yuting Yan 严玉婷
Shengfeng Ye 叶盛峰	Zhen Zhang 张震
T. Wang T.王

Within each role, authors are listed alphabetically by the first name. Names marked with * denote individuals who have departed from our team.
在每个角色中，作者按姓名字母顺序排列。标有 * 的姓名表示已离开我们团队的人员。

B. Ablation Studies for Low-Precision Training
B.用于低精度训练的消融研究

Figure 10 | Loss curves comparison between BF16 and FP8 training. Results are smoothed by Exponential Moving Average (EMA) with a coefficient of 0.9.
图 10 BF16 和 FP8 训练的损失曲线对比。结果采用系数为 0.9 的指数移动平均法 (EMA) 进行平滑处理。

B.1. FP8 v.s. BF16 Training
B.1.FP8 对 BF16 培训

We validate our FP8 mixed precision framework with a comparison to BF16 training on top of two baseline models across different scales. At the small scale, we train a baseline MoE model comprising approximately 16 B total parameters on 1.33 T tokens. At the large scale, we train a baseline MoE model comprising approximately 230 B total parameters on around 0.9 T tokens. We show the training curves in Figure 10 and demonstrate that the relative error remains below

0.25 %

with our high-precision accumulation and fine-grained quantization strategies.
我们将 FP8 混合精度框架与 BF16 进行了比较，并在不同规模的两个基线模型之上进行了训练，从而验证了我们的 FP8 混合精度框架。在小规模上，我们在 1.33 T 标记上训练了一个基线 MoE 模型，该模型包含约 16 B 个总参数。在大尺度下，我们在约 0.9 T 标记上训练一个总参数约为 230 B 的基线 MoE 模型。我们在图 10 中显示了训练曲线，并证明使用我们的高精度累积和细粒度量化策略，相对误差仍然低于

0.25 %

。

B.2. Discussion About Block-Wise Quantization
B.2.关于全块量化的讨论

Although our tile-wise fine-grained quantization effectively mitigates the error introduced by feature outliers, it requires different groupings for activation quantization, i.e.,

1 \times 128

in forward pass and

128 \times 1

for backward pass. A similar process is also required for the activation gradient. A straightforward strategy is to apply block-wise quantization per

128 \times 128

elements like the way we quantize the model weights. In this way, only transposition is required for backward. Therefore, we conduct an experiment where all tensors associated with Dgrad are quantized on a block-wise basis. The results reveal that the Dgrad operation which computes the activation gradients and back-propagates to shallow layers in a chain-like manner, is highly sensitive to precision. Specifically, block-wise quantization of activation gradients leads to
虽然我们的瓦片式细粒度量化可以有效地减少特征异常值带来的误差，但它需要对激活量化进行不同的分组，即在前向传递中使用

1 \times 128

，在后向传递中使用

128 \times 1

。激活梯度也需要类似的过程。一种直接的策略是对

128 \times 128

元素进行分块量化，就像我们对模型权重进行量化一样。通过这种方式，只需要进行后向转置。因此，我们进行了一项实验，对与 Dgrad 相关的所有张量进行分块量化。结果显示，计算激活梯度并以链式方式反向传播到浅层的 Dgrad 操作对精度非常敏感。具体来说，分块量化激活梯度会导致
model divergence on an MoE model comprising approximately 16B total parameters, trained for around 300B tokens. We hypothesize that this sensitivity arises because activation gradients are highly imbalanced among tokens, resulting in token-correlated outliers (Xi et al., 2023). These outliers cannot be effectively managed by a block-wise quantization approach.
在一个总参数约为 16B 的 MoE 模型上，模型发散性是根据约 300B 标记训练的。我们推测，这种敏感性的产生是因为标记间的激活梯度高度不平衡，从而导致了与标记相关的异常值（Xi 等人，2023 年）。分块量化方法无法有效管理这些异常值。

C. Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-LossFree Models
C.基于辅助损失的 16B 模型和无辅助损失模型的专家专业化模式

We record the expert load of the 16B auxiliary-loss-based baseline and the auxiliary-loss-free model on the Pile test set. The auxiliary-loss-free model tends to have greater expert specialization across all layers, as demonstrated in Figure 10.
我们记录了基于 16B 辅助损耗的基线模型和无辅助损耗模型在 Pile 测试集上的专家负载。如图 10 所示，无辅助损耗模型在所有层级上的专家专业化程度往往更高。

Aux-Loss-Based Layer 1 基于辅助损失的第 1 层

Aux-Loss-Based Layer 4 基于辅助损失的第 4 层

Aux-Loss-Free Layer 4 无损耗辅助层 4

Aux-Loss-Based Layer 5 基于辅助损失的第 5 层

Aux-Loss-Free Layer 5 无损耗辅助层 5

Aux-Loss-Based Layer 6 基于辅助损失的第 6 层

Aux-Loss-Free Layer 6 无辅助损失层 6

(a) Layers 1-7 (a) 第 1-7 层

Aux-Loss-Based Layer 13 辅助损失层 13

Aux-Loss-Free Layer 14 无损耗辅助层 14

Aux-Loss-Free Layer 15 无损耗辅助层 15

Aux-Loss-Free Layer 16 无损辅助层 16

Aux-Loss-Free Layer 17 无损辅助层 17

Aux-Loss-Based Layer 18 辅助损失层 18
Wikipedia (en) 维基百科
Github

DM Mathematics DM 数学

Aux-Loss-Free Layer 18 无损耗辅助层 18

Aux-Loss-Free Layer 19 无损辅助层 19

Aux-Loss-Free Layer 20 无损耗辅助层 20

Aux-Loss-Based Layer 22 辅助损失层 22

Aux-Loss-Free Layer 22 无损辅助层 22

Aux-Loss-Free Layer 23 无损耗辅助层 23

Aux-Loss-Based Layer 24 辅助损失层 24

Wikipedia (en) 维基百科
Github
DM Mathematics DM 数学

(d) Layers 19-25 (d) 第 19-25 层

(e) Layers 25-27 (e) 第 25-27 层

Figure 10 | Expert load of auxiliary-loss-free and auxiliary-loss-based models on three domains in the Pile test set. The auxiliary-loss-free model shows greater expert specialization patterns than the auxiliary-loss-based one. The relative expert load denotes the ratio between the actual expert load and the theoretically balanced expert load.
图 10：无辅助损耗模型和基于辅助损耗模型在桩测试集中三个领域的专家负载。与基于辅助损失的模型相比，无辅助损失模型显示出更大的专家专业化模式。相对专家负载表示实际专家负载与理论上平衡的专家负载之间的比率。

$^{1}$ https://aider.chat
$^{2}$ https://codeforces.com
$^{3}$ https://www.cms.org.cn/Home/comp/comp/cid/12.html
$^{4}$ https://github.com/openai/simple-evals

DeepSeek-V3 Technical Report DeepSeek-V3 技术报告

Abstract 摘要

Contents 目录

1. Introduction 1.导言

Architecture: Innovative Load Balancing Strategy and Training Objective架构：创新负载平衡策略和培训目标

Pre-Training: Towards Ultimate Training Efficiency训练前：提高训练效率

Post-Training: Knowledge Distillation from DeepSeek-R1培训后：从 DeepSeek-R1 中提炼知识

Summary of Core Evaluation Results核心评估结果摘要

2. Architecture 2.建筑学

2.1. Basic Architecture 2.1.基本架构

2.1.1. Multi-Head Latent Attention2.1.1.多头潜意识

2.1.2. DeepSeekMoE with Auxiliary-Loss-Free Load Balancing2.1.2.带有无辅助负载平衡功能的 DeepSeekMoE

2.2. Multi-Token Prediction2.2.多令牌预测

3. Infrastructures 3.基础设施

3.1. Compute Clusters 3.1.计算集群

3.2. Training Framework 3.2.培训框架

3.2.1. DualPipe and Computation-Communication Overlap3.2.1.双管道和计算-通信重叠

3.2.2. Efficient Implementation of Cross-Node All-to-All Communication3.2.2.跨节点全对全通信的高效实现

3.2.3. Extremely Memory Saving with Minimal Overhead3.2.3.以最小的开销极大地节省内存

3.3. FP8 Training 3.3.FP8 培训

3.3.1. Mixed Precision Framework3.3.1.混合精度框架

3.3.2. Improved Precision from Quantization and Multiplication3.3.2.通过量化和乘法提高精度

3.3.3. Low-Precision Storage and Communication3.3.3.低精度存储和通信

3.4. Inference and Deployment3.4.推理和部署

3.4.1. Prefilling 3.4.1.预填充

3.4.2. Decoding 3.4.2.解码

3.5. Suggestions on Hardware Design3.5.硬件设计建议

3.5.1. Communication Hardware3.5.1.通信硬件

3.5.2. Compute Hardware 3.5.2.计算硬件

4. Pre-Training 4.培训前

4.1. Data Construction 4.1.数据构建

4.2. Hyper-Parameters 4.2.超参数

4.3. Long Context Extension4.3.长语境扩展

4.4. Evaluations 4.4.评估

4.4.1. Evaluation Benchmarks4.4.1.评估基准

4.4.2. Evaluation Results4.4.2.评估结果

4.5. Discussion 4.5.讨论情况

4.5.1. Ablation Studies for Multi-Token Prediction4.5.1.多标记预测的消融研究

4.5.2. Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy4.5.2.无损耗辅助平衡策略的消融研究

4.5.3. Batch-Wise Load Balance VS. Sequence-Wise Load Balance4.5.3.批量负载平衡 VS.顺序负载平衡

5. Post-Training 5.培训后

5.1. Supervised Fine-Tuning5.1.监督微调

5.2. Reinforcement Learning5.2.强化学习

5.2.1. Reward Model 5.2.1.奖励模式

5.2.2. Group Relative Policy Optimization5.2.2.组相对策略优化

5.3. Evaluations 5.3.评估

5.3.1. Evaluation Settings5.3.1.评估设置

5.3.2. Standard Evaluation5.3.2.标准评估

5.3.3. Open-Ended Evaluation5.3.3.开放式评估

5.3.4. DeepSeek-V3 as a Generative Reward Model5.3.4.作为生成奖励模型的 DeepSeek-V3

5.4. Discussion 5.4.讨论情况

5.4.1. Distillation from DeepSeek-R15.4.1.从 DeepSeek-R1 蒸馏

5.4.2. Self-Rewarding 5.4.2.自我奖励

5.4.3. Multi-Token Prediction Evaluation5.4.3.多令牌预测评估

6. Conclusion, Limitations, and Future Directions6.结论、局限性和未来方向

References 参考资料

Appendix 附录

A. Contributions and AcknowledgmentsA.贡献和鸣谢

B. Ablation Studies for Low-Precision TrainingB.用于低精度训练的消融研究

B.1. FP8 v.s. BF16 TrainingB.1.FP8 对 BF16 培训

B.2. Discussion About Block-Wise QuantizationB.2.关于全块量化的讨论

C. Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-LossFree ModelsC.基于辅助损失的 16B 模型和无辅助损失模型的专家专业化模式

Aux-Loss-Based Layer 1 基于辅助损失的第 1 层

Aux-Loss-Based Layer 4 基于辅助损失的第 4 层

Aux-Loss-Free Layer 19 无损辅助层 19

Aux-Loss-Based Layer 22 辅助损失层 22

Aux-Loss-Free Layer 22 无损辅助层 22

DeepSeek-V3 Technical Report
DeepSeek-V3 技术报告

Architecture: Innovative Load Balancing Strategy and Training Objective
架构：创新负载平衡策略和培训目标

Pre-Training: Towards Ultimate Training Efficiency
训练前：提高训练效率

Post-Training: Knowledge Distillation from DeepSeek-R1
培训后：从 DeepSeek-R1 中提炼知识

Summary of Core Evaluation Results
核心评估结果摘要

2.1.1. Multi-Head Latent Attention
2.1.1.多头潜意识

2.1.2. DeepSeekMoE with Auxiliary-Loss-Free Load Balancing
2.1.2.带有无辅助负载平衡功能的 DeepSeekMoE

2.2. Multi-Token Prediction
2.2.多令牌预测

3.2.1. DualPipe and Computation-Communication Overlap
3.2.1.双管道和计算-通信重叠

3.2.2. Efficient Implementation of Cross-Node All-to-All Communication
3.2.2.跨节点全对全通信的高效实现

3.2.3. Extremely Memory Saving with Minimal Overhead
3.2.3.以最小的开销极大地节省内存

3.3.1. Mixed Precision Framework
3.3.1.混合精度框架

3.3.2. Improved Precision from Quantization and Multiplication
3.3.2.通过量化和乘法提高精度

3.3.3. Low-Precision Storage and Communication
3.3.3.低精度存储和通信

3.4. Inference and Deployment
3.4.推理和部署

3.5. Suggestions on Hardware Design
3.5.硬件设计建议

3.5.1. Communication Hardware
3.5.1.通信硬件

4.3. Long Context Extension
4.3.长语境扩展

4.4.1. Evaluation Benchmarks
4.4.1.评估基准

4.4.2. Evaluation Results
4.4.2.评估结果

4.5.1. Ablation Studies for Multi-Token Prediction
4.5.1.多标记预测的消融研究

4.5.2. Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy
4.5.2.无损耗辅助平衡策略的消融研究

4.5.3. Batch-Wise Load Balance VS. Sequence-Wise Load Balance
4.5.3.批量负载平衡 VS.顺序负载平衡

5.1. Supervised Fine-Tuning
5.1.监督微调

5.2. Reinforcement Learning
5.2.强化学习

5.2.2. Group Relative Policy Optimization
5.2.2.组相对策略优化

5.3.1. Evaluation Settings
5.3.1.评估设置

5.3.2. Standard Evaluation
5.3.2.标准评估

5.3.3. Open-Ended Evaluation
5.3.3.开放式评估

5.3.4. DeepSeek-V3 as a Generative Reward Model
5.3.4.作为生成奖励模型的 DeepSeek-V3

5.4.1. Distillation from DeepSeek-R1
5.4.1.从 DeepSeek-R1 蒸馏

5.4.3. Multi-Token Prediction Evaluation
5.4.3.多令牌预测评估

6. Conclusion, Limitations, and Future Directions
6.结论、局限性和未来方向

A. Contributions and Acknowledgments
A.贡献和鸣谢

B. Ablation Studies for Low-Precision Training
B.用于低精度训练的消融研究

B.1. FP8 v.s. BF16 Training
B.1.FP8 对 BF16 培训

B.2. Discussion About Block-Wise Quantization
B.2.关于全块量化的讨论

C. Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-LossFree Models
C.基于辅助损失的 16B 模型和无辅助损失模型的专家专业化模式