In our pursuit of becoming a better full service research firm, we’ve moved off Substack. For any questions please read https://semianalysis.com/faq/#substack
在我们追求成为更好的全方位服务研究公司的过程中，我们已不再使用 Substack。如有任何问题，请阅读 https://semianalysis.com/faq/#substack

There has been an increasing amount of fear, uncertainty and doubt (FUD) regarding AI Scaling laws. A cavalcade of part-time AI industry prognosticators have latched on to any bearish narrative they can find, declaring the end of scaling laws that have driven the rapid improvement in Large Language Model (LLM) capabilities in the last few years. Journalists have joined the dogpile and have supported these narratives, armed with noisy leaks filled with vague information around the failure of models to scale successfully due to alleged underperformance. Other skeptics point to saturated benchmarks, with newer models showing little sign of improvement said benchmarks. Critics also point to the exhaustion of available training data and slowing hardware scaling for training.
关于人工智能扩展法则的恐惧、不确定性和怀疑（FUD）日益增加。一群兼职的人工智能行业预言家抓住了他们能找到的任何看跌叙述，宣称推动近年来大型语言模型（LLM）能力快速提升的扩展法则已走到尽头。记者们也加入了这一阵营，支持这些叙述，手中拿着充满模糊信息的嘈杂泄漏，声称模型未能成功扩展是由于所谓的表现不佳。其他怀疑者则指出基准测试的饱和，新模型在这些基准测试中几乎没有显示出改善的迹象。批评者还指出可用训练数据的枯竭和训练硬件扩展的放缓。

Despite this angst, large AI Labs and hyperscalers’ accelerating datacenter buildouts and capital expenditure speaks for itself. From Amazon investing considerable sums to accelerate its Trainium2 custom silicon and preparing 400k chips for Anthropic at an estimated cost of $6.5B in total IT and datacenter investment, to Meta’s 2GW datacenter plans for 2026 in Indiana, to OpenAI and Google’s aggressive multi-datacenter training plans to overcome single-site power limitations – key decision makers appear to be unwavering in their conviction that scaling laws are alive and well. Why?
尽管存在这种焦虑，大型人工智能实验室和超大规模数据中心的加速建设和资本支出不言而喻。从亚马逊投资大量资金加速其 Trainium2 定制硅片，并为 Anthropic 准备 40 万颗芯片，预计总 IT 和数据中心投资成本为 65 亿美元，到 Meta 在印第安纳州 2026 年的 2GW 数据中心计划，再到 OpenAI 和谷歌积极的多数据中心训练计划以克服单一地点的电力限制——关键决策者似乎对扩展法则的信念坚定不移。为什么？

Scaling Up Training, New and Old Paradigms Continue
扩大培训，新旧范式持续进行

The reality is that there are more dimensions for scaling beyond simply focusing on pre-training, which has been the sole focus of most of the part-time prognosticators. OpenAI’s o1 release has proved the utility and potential of reasoning models, opening a new unexplored dimension for scaling. This is not the only technique, however, that delivers meaningful improvements in model performance as compute is scaled up. Other areas that deliver model improvements with more compute include Synthetic Data Generation, Proximal Policy Optimization (PPO), Functional Verifiers, and other training infrastructure for reasoning. The sands of scaling are still shifting and evolving, and, with it, the entire AI development process has continued to accelerate.  
现实是，除了单纯关注预训练之外，还有更多的扩展维度，而这一直是大多数兼职预测者的唯一关注点。OpenAI 的 o1 版本证明了推理模型的实用性和潜力，为扩展开辟了一个新的未探索维度。然而，这并不是唯一一种在计算能力提升时能带来模型性能显著改善的技术。其他在计算能力提升时能改善模型的领域包括合成数据生成、近端策略优化（PPO）、功能验证器以及其他推理训练基础设施。扩展的沙子仍在不断变化和演变，整个 AI 开发过程也在持续加速。

Shifting from faulty benchmarks to more challenging ones will enable better measures of progress. In this report we will outline the old pre-training scaling trend as well as the new scaling trends for post-training and inference time. This includes how new methods will push the frontier – and will require even more training time compute scaling then thought before.
从错误的基准转向更具挑战性的基准将能够更好地衡量进展。在本报告中，我们将概述旧的预训练规模趋势以及新的后训练和推理时间的规模趋势。这包括新方法如何推动前沿——并且将需要比之前认为的更多的训练时间计算规模。

We will cover OpenAI o1 and o1 Pro’s architecture from both a training infrastructure and inference tokenomics perspective including cost, KVCache scaling, batching, and more. We will also dive into leading AI Lab synthetic data and RL infrastructure. Lastly, we want to set the record straight on Anthropic’s Claude 3.5 Opus and OpenAI’s Orion’s “failures,” and what scaling plans are going forward.
我们将从训练基础设施和推理代币经济学的角度，涵盖 OpenAI o1 和 o1 Pro 的架构，包括成本、KVCache 扩展、批处理等内容。我们还将深入探讨领先的 AI 实验室合成数据和强化学习基础设施。最后，我们希望澄清 Anthropic 的 Claude 3.5 Opus 和 OpenAI 的 Orion 的“失败”，以及未来的扩展计划。

Scaling Sings Odes to the Greatest Scaling Law of Computing, Moore’s Law
扩展颂歌：计算领域最伟大的扩展法则，摩尔定律

Today’s debate on AI scaling laws is not dissimilar to the decades-long debate around compute scaling and Moore’s law. Anyone who tries to measure CPU compute primarily by clock speed – a common metric used before the late 2000s around the time of the end of Dennard Scaling – would argue that we have not made any progress at all since then. In reality, compute has been advancing all along – when we hit a wall on processor clock speed, the focus shifted to multi-core architectures and other methods to drive performance, despite power density and cooling constraints.
今天关于人工智能扩展法则的辩论与围绕计算扩展和摩尔定律的数十年辩论并没有太大区别。任何试图主要通过时钟速度来衡量 CPU 计算的人——这是在 2000 年代末德纳德缩放结束时常用的指标——都会认为自那时以来我们没有任何进展。实际上，计算一直在不断进步——当我们在处理器时钟速度上遇到瓶颈时，焦点转向了多核架构和其他推动性能的方法，尽管存在功率密度和冷却限制。

The end of Moore’s Law is another wall that with which the semiconductor industry has contended, but this debate has been quieter lately as AI pioneers like Nvidia have provided massive compute gains by scaling along a few entirely new dimensions. Advanced packaging has enabling continued advances in compute by scaling input/output (I/Os) and enabling chips to harness a total silicon area beyond the reticle size limit. Parallel computing within and across chips and building larger high-bandwidth networking domains has enabled chips to work better together at scale, especially for inference.
摩尔定律的终结是半导体行业面临的另一个壁垒，但随着像英伟达这样的人工智能先驱通过在几个全新维度上扩展提供了巨大的计算增益，这场辩论最近变得更加平静。先进封装使得通过扩展输入/输出（I/Os）并使芯片能够利用超出掩模尺寸限制的总硅面积，继续推动计算的进步。在芯片内部和跨芯片的并行计算以及构建更大高带宽网络域，使得芯片能够在规模上更好地协同工作，特别是在推理方面。

As with computer enthusiasts in 2004, mainstream analysts and journalists are missing the forest for the trees: despite the slowing down of one trend, the industry collectively remains moving forward at a breakneck pace due to other new emerging paradigms that are ripe for scaling and expansion. It is possible to stack “scaling laws” – pre-training will become just one of the vectors of improvement, and the aggregate “scaling law” will continue scaling just like Moore’s Law has over last 50+ years.
与 2004 年的计算机爱好者一样，主流分析师和记者也未能看清大局：尽管某一趋势正在放缓，但由于其他新兴范式的出现，整个行业仍在以惊人的速度向前发展，这些新兴范式正适合扩展和规模化。“规模法则”可以叠加——预训练将成为改进的一个向量，而整体“规模法则”将继续扩展，就像摩尔定律在过去 50 多年中一样。

Challenges in Scaling Pre-training – Data wall, fault tolerance
在扩展预训练中的挑战 - 数据壁垒，容错性

Scaling pre-training has provided significant gains in model performance, but there are a few speed bumps that the industry is currently focusing on overcoming.
扩展预训练已显著提高模型性能，但行业目前正专注于克服一些障碍。

One obvious speed bump is that data is increasingly difficult to collect – while data on the internet is expanding quickly, it is not expanding at a rate proportional to compute. This is why today’s trillion parameter mega-models have been less than Chinchilla optimal – a much lower number of training tokens vs model parameters.
一个明显的障碍是数据越来越难以收集——虽然互联网上的数据快速增长，但其增长速度并未与计算能力成正比。这就是为什么今天的万亿参数超级模型的表现低于 Chinchilla 的最优水平——训练令牌的数量远低于模型参数。

Chinchilla scaling refers to the optimal increases in data versus parameter counts relative to increases in compute. Not enough data causes the model to generalize poorly, while too much data results in overtraining, which wastes compute resources. There are some instances where deviating from the optimal ratio makes sense: over-training models (e.g. GPT-4o and Llama) can decrease inference costs significantly and is preferrable for providers that have a larger user base to serve said model to.
奇努基拉缩放是指数据与参数数量相对于计算能力增加的最佳比例。数据不足会导致模型泛化能力差，而数据过多则会导致过度训练，浪费计算资源。在某些情况下，偏离最佳比例是有意义的：过度训练模型（例如 GPT-4o 和 Llama）可以显著降低推理成本，并且对于拥有更大用户基础的提供商来说，更加可取。

In January of 2023, before the launch of GPT-4, we wrote about the practical limits for scaling and how GPT-4 planned to break through them. Since then, models have ping-ponged from being more than Chinchilla Optimal (much greater data than model parameters) to less than Chinchilla Optimal (when data became constrained). The compute availability speedbump was overcome in the past when improvements in training and inference hardware alleviated constraints.
在 2023 年 1 月，GPT-4 发布之前，我们讨论了扩展的实际限制以及 GPT-4 计划如何突破这些限制。自那时以来，模型在超过 Chinchilla Optimal（数据远大于模型参数）和低于 Chinchilla Optimal（当数据受到限制时）之间反复波动。过去，计算可用性瓶颈通过训练和推理硬件的改进得以克服，从而缓解了限制。

With respect to today’s narrative around speed bumps – useful data sources such as textbooks and documentation are exhausted, and what remains is mostly lower-quality text data sources. Furthermore, web data is still a narrow distribution of data and models need more out of distribution data to continue to generalize. With models harder to scale in a way that is optimal, pre-training is becoming more challenging.
关于今天围绕速度障碍的叙述——有用的数据来源如教科书和文档已经用尽，剩下的主要是低质量的文本数据来源。此外，网络数据仍然是数据的狭窄分布，模型需要更多的分布外数据才能继续泛化。由于模型在以最佳方式扩展方面变得更加困难，预训练变得更加具有挑战性。

Also, if labs train models with an insufficient amount of data as they keep scaling, the models become over-parametrized, becoming inefficient and leading to heavy amounts of memorization rather than generalization. Labs have instead been turning to an increasing use of synthetic data to alleviate this problem.
此外，如果实验室在不断扩展时用不足的数据训练模型，这些模型会变得过度参数化，效率低下，导致大量记忆而非泛化。实验室因此越来越多地转向使用合成数据来缓解这个问题。

Though, this issue applies less to the main AI Labs. Meta alone has approximately 100x more data available to them than is on the public internet (if they can harness this data in a compliant manner). This may give them an edge in continuing to scale with fewer issues than others. YouTube has 720,000 new hours of video uploaded every day – and we think that AI Labs have only begun to contemplate training on the vast amount of data contained within video. This is in addition to their ability to generate quality synthetically generated data, which we discuss the architecture for later.
尽管这个问题对主要的人工智能实验室影响较小。Meta 单独拥有的可用数据大约是公共互联网数据的 100 倍（如果他们能够以合规的方式利用这些数据）。这可能使他们在继续扩展时面临的问题少于其他公司。YouTube 每天上传 72 万个小时的新视频——我们认为人工智能实验室仅仅开始考虑在视频中包含的大量数据上进行训练。这还不包括他们生成高质量合成数据的能力，我们将在后面讨论其架构。

To train on the quadrillions of alternative tokens available from video requires a huge continuation of scaling overall training FLOPs, which will be delivered by hardware innovation and systems engineering. For instance, scaling another order of magnitude on training FLOPs will require multi-datacenter training as the number of accelerators needed can no longer fit inside a single datacenter site. Project Rainier has Amazon providing Anthropic with 400k Tranium 2 chips, but, in raw FLOPs, that is less than 100k GB200s. Anthropic will have to produce significant engineering achievements to pull off training in such a cluster. Spreading accelerators across a large campus, or multiple campuses, itself leads to significant challenges posed by Amdahl’s law, though there are already more than a few posited solutions to address this challenge.
在视频中可用的数万亿替代令牌上进行训练需要大规模持续扩展整体训练 FLOPs，这将通过硬件创新和系统工程来实现。例如，训练 FLOPs 再扩大一个数量级将需要多数据中心训练，因为所需的加速器数量已无法容纳在单一数据中心内。Rainier 项目让亚马逊向 Anthropic 提供 40 万颗 Tranium 2 芯片，但在原始 FLOPs 上，这还不到 10 万 GB200。Anthropic 必须取得显著的工程成就才能在这样的集群中完成训练。将加速器分布在一个大型校园或多个校园中，带来了由阿姆达尔定律所带来的重大挑战，尽管已经提出了不少解决方案来应对这一挑战。

The other constraint with respect to scaling parameters is inference economics. AI Labs can capitalize vast sums of investment into training large models and amortize the model’s use both over a large and growing userbase, as well as for internal use cases, to develop further model iterations. When it comes to inference, they must be careful not to bring to market models that are too costly or uneconomical to serve.
与缩放参数相关的另一个限制是推理经济学。人工智能实验室可以将大量投资用于训练大型模型，并在一个庞大且不断增长的用户基础上摊销模型的使用，同时也用于内部用例，以开发进一步的模型迭代。在推理方面，他们必须小心不要将成本过高或经济上不合理的模型推向市场。

Evals are also not comprehensive; there are many capabilities or properties of models that existing evals do not cover well. Transfer learning, where the model gets better at a domain through learning about something else, and in-context learning are both areas where more evals need to be developed. Finally, there will always be end use cases that may be hard to predict in advance but provide an immense benefit to the end user.
评估也并不全面；现有的评估未能很好地覆盖模型的许多能力或特性。迁移学习，即模型通过学习其他事物在某个领域变得更好，以及上下文学习，都是需要开发更多评估的领域。最后，总会有一些最终使用案例可能难以提前预测，但对最终用户提供巨大的好处。

That which gets measured, improves.
被测量的事物会得到改善。

Newer, Harder Evals to Climb
更新、更严格的评估标准待提升

Newer evaluations have sprung up that aim to better differentiate models and focus on directly addressing specific useful applications. SWE-Bench is one of the most important evaluations today, aiming to have models solve human-reviewed GitHub issues from open-source Python repositories. The new Claude 3.5 Sonnet currently has achieved (State of the Art) on SWE-Bench Verified at 49%, but most models are much lower.
新的评估方法应运而生，旨在更好地区分模型，并专注于直接解决特定的有用应用。SWE-Bench 是目前最重要的评估之一，旨在让模型解决来自开源 Python 仓库的人类审核的 GitHub 问题。新的 Claude 3.5 Sonnet 目前在 SWE-Bench 上的验证状态达到了 49% 的（最先进水平），但大多数模型的表现要低得多。

Another example is a benchmark investigating AI R&D capabilities, which some describe as “the most important capability to track.” Research Engineering Benchmark (RE) consists of seven challenging and open-ended ML research environments. Humans generally perform better on evals over longer time horizons, but, on a 2-hour time horizon, the best AI agents achieved a score 4x higher than humans. Important tasks such as the above, in which humans currently dominate, are the perfect ground for scaling inference time compute. We expect that models that better leverage this form of scaling will outperform humans in the future.
另一个例子是一个基准调查 AI 研发能力，一些人将其描述为“最重要的跟踪能力”。研究工程基准（RE）由七个具有挑战性和开放性的机器学习研究环境组成。人类在较长时间范围内的评估表现通常更好，但在 2 小时的时间范围内，最佳 AI 代理的得分是人类的 4 倍。像上述重要任务，目前人类占主导地位，是扩展推理时间计算的完美基础。我们预计，能够更好利用这种扩展形式的模型将在未来超越人类。

Yet another trend is for evaluations to include extremely difficult expert-level questions. Two prominent examples are Graduate-Level Google-Proof Q&A Benchmark (GPQA) and Frontier Math. GPQA is made up of 448 multiple choice questions across chemistry, biology, and physics. For context, OpenAI found that expert-level humans (i.e. people with PhDs) scored ~70% on GPQA Diamond, with o1 scoring 78% on the same set. Last year, GPT-4 with search (and CoT on abstention) scored 39% on GPQA Diamond.
另一个趋势是评估中包含极其困难的专家级问题。两个显著的例子是研究生级谷歌防护问答基准（GPQA）和前沿数学。GPQA 由 448 个多项选择题组成，涵盖化学、生物学和物理学。作为背景，OpenAI 发现专家级人类（即拥有博士学位的人）在 GPQA Diamond 上的得分约为 70%，而 o1 在同一组题目上的得分为 78%。去年，带有搜索功能的 GPT-4（以及关于弃权的链式推理）在 GPQA Diamond 上的得分为 39%。

Another example of the trend towards using extremely tough questions is FrontierMath (FM). FM is a benchmark of hundreds of original math questions that can take humans hours and even up to days to solve. It covers a broad range of mathematical topics, including number theory, real analysis, etc. The special sauce with this eval is that it is not published, minimizing the risk of data contamination, and can be graded via an automated verifier – simplifying the evaluation process.
另一个使用极其困难问题的趋势例子是 FrontierMath（FM）。FM 是数百个原创数学问题的基准，这些问题可能需要人类花费数小时甚至数天来解决。它涵盖了广泛的数学主题，包括数论、实分析等。这个评估的特别之处在于它没有公开，最大限度地降低了数据污染的风险，并且可以通过自动验证器进行评分——简化了评估过程。

The best performing model on this benchmark comes in at 2%, but the labs expect this to dramatically improve. Anthropic has line of sight to hit 80% on FrontierMath over the medium term.
在这个基准上表现最好的模型为 2%，但实验室预计这一数字将大幅改善。Anthropic 有望在中期内在 FrontierMath 上达到 80%。

Post-training: a new scaling domain
后训练：一个新的扩展领域

Pre-training tends to be the focus of debates regarding scaling laws because it is easy to understand, but it is only one part of the AI lifecycle. Once a model is pre-trained, there is still considerable work to be done on getting it ready for use. The objective during pre-training is, very narrowly, to “predict the next token correctly.” Accomplishing this still leaves us well short of the end-goal of LLM development which is to “answer user prompts” or “do a task.”
预训练往往是关于规模法则辩论的焦点，因为它易于理解，但它只是人工智能生命周期的一部分。一旦模型经过预训练，仍然需要大量工作来使其准备好使用。预训练期间的目标非常狭窄，即“正确预测下一个标记”。实现这一目标仍然使我们远未达到LLM开发的最终目标，即“回答用户提示”或“完成任务”。

We will do an overview on Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Synthetic Data, before diving into how OpenAI’s O1 Pro model works and was created.
我们将对监督微调（SFT）、强化学习（RL）和合成数据进行概述，然后深入探讨 OpenAI 的 O1 Pro 模型是如何运作和创建的。

Supervised Fine-Tuning 监督微调

Supervised Fine-Tuning (SFT) is the most well-known type of post-training. A curated dataset of input and output pairs are shown to the model, with the “demonstration data” covering a specific domain (e.g. code, math, instruction following, etc.). Unlike with pre-training, the quality of fine-tuning data is much more important here than quantity. Given the lower quantity of data, that means it is less compute intensive.
监督微调（SFT）是最著名的后训练类型。一个经过精心策划的输入和输出对的数据集被展示给模型，“示范数据”涵盖特定领域（例如代码、数学、指令跟随等）。与预训练不同，微调数据的质量在这里比数量更为重要。由于数据量较少，这意味着它的计算强度较低。

The magic of GPT originally was using heavily curated samples of human generated and labeled data from firms like Scale AI. As time goes on, however, human generated data is struggling to scale.
GPT 的魔力最初在于使用来自 Scale AI 等公司的大量精心策划的人类生成和标记的数据。然而，随着时间的推移，人类生成的数据正面临扩展的困难。

Synthetic Data’s Integral Role in Post-training
合成数据在后训练中的重要作用

The most important challenge within SFT is constructing sufficiently large, high quality data sets in the desired domains. This allows the model to operate better in specific areas like code, math, reasoning, and due to transfer learning, has spillover effects making the model better in other domains too. Obviously, models with strong math and coding skills are better at general reasoning, but this extends to other areas – models trained on Chinese and English are better at English than those trained on English alone. Synthetic data has opened a dimension where high-quality data can be generated using a controlled, beyond scalable methodology to fine-tune models over any subject matter for which there exists a will to create it.
SFT 中最重要的挑战是构建足够大、高质量的数据集，以满足所需领域。这使得模型在特定领域如代码、数学、推理等方面表现更好，并且由于迁移学习的影响，使得模型在其他领域也变得更优秀。显然，具备强大数学和编码技能的模型在一般推理方面表现更佳，但这也扩展到其他领域——在中文和英文上训练的模型在英语方面的表现优于仅在英语上训练的模型。合成数据开辟了一个维度，可以使用受控的、超越可扩展的方法生成高质量数据，以便对任何有意创建的主题进行模型微调。

The heavy use of synthetic data also incentivizes a push toward better models. For example, OpenAI had GPT-4 before anyone else and could use it to generate better synthetic data sets than other model providers – until other providers had a model to match. One the primary reasons that many models in Open Source and at Chinese Labs caught up so fast was that they were trained on synthetic data from GPT-4.
合成数据的广泛使用也促使了对更好模型的推动。例如，OpenAI 在其他人之前就拥有了 GPT-4，并可以利用它生成比其他模型提供者更好的合成数据集——直到其他提供者有了可以匹配的模型。许多开源模型和中国实验室的模型迅速赶上的主要原因之一是它们是基于 GPT-4 的合成数据进行训练的。

The better the underlying model is at judging tasks, the better the dataset for training. Inherent in this are scaling laws of their own. This is how we got the “new Claude 3.5 Sonnet”. Anthropic finished training Claude 3.5 Opus and it performed well, with it scaling appropriately (ignore the scaling deniers who claim otherwise – this is FUD).
基础模型在判断任务方面越好，训练数据集就越好。这其中固有着自身的规模法则。这就是我们获得“新 Claude 3.5 Sonnet”的原因。Anthropic 完成了 Claude 3.5 Opus 的训练，并且表现良好，规模也适当（忽略那些声称相反的规模否认者——这只是恐惧、不确定和怀疑）。

Yet Anthropic didn’t release it. This is because instead of releasing publicly, Anthropic used Claude 3.5 Opus to generate synthetic data and for reward modeling to improve Claude 3.5 Sonnet significantly, alongside user data. Inference costs did not change drastically, but the model’s performance did. Why release 3.5 Opus when, on a cost basis, it does not make economic sense to do so, relative to releasing a 3.5 Sonnet with further post-training from said 3.5 Opus?
然而，Anthropic 并没有发布它。这是因为 Anthropic 没有选择公开发布，而是使用 Claude 3.5 Opus 生成合成数据，并进行奖励建模，以显著改善 Claude 3.5 Sonnet，此外还结合了用户数据。推理成本没有发生剧烈变化，但模型的性能却有所提升。为什么要发布 3.5 Opus，而在成本基础上，相比于发布经过进一步后训练的 3.5 Sonnet，这样做并没有经济意义呢？

With more synthetic data comes better models. Better models provide better synthetic data and act as better judges for filtering or scoring preferences. Inherent in the use of synthetic data are many smaller scaling laws that, collectively, push toward developing better models faster.
随着合成数据的增加，模型变得更好。更好的模型提供更好的合成数据，并作为更好的评判者来过滤或评分偏好。使用合成数据固有地包含许多较小的规模法则，这些法则共同推动更快地开发更好的模型。

Synthetic Data Examples 合成数据示例

Rejection Sampling 拒绝采样

An example of an area where synthetic data is heavily used is in generating datasets of code. This is typically done through designating a variety of programming tasks or prompts as seeds and prompting a model to generate questions relating to those tasks.
一个合成数据被广泛使用的领域示例是生成代码的数据集。这通常是通过将各种编程任务或提示指定为种子，并提示模型生成与这些任务相关的问题来完成的。

The model is then asked to generate a set of potential solutions. Solutions which pass the corresponding tests, or can execute correctly, are appended to the training dataset, effectively filtering out poor-quality samples in a process referred to as Rejection Sampling. Rejection Sampling is an instrumental part of the synthetic data generation process as it ensures that the dataset is of a sufficient quality to be valuable during Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). However, as a result, many of the generated tokens are thrown out – synthetic data generation takes a lot of compute.
模型随后被要求生成一组潜在解决方案。通过相应测试的解决方案，或能够正确执行的解决方案，将被附加到训练数据集中，有效地过滤掉低质量样本，这一过程被称为拒绝采样。拒绝采样是合成数据生成过程中的一个重要部分，因为它确保数据集的质量足够高，以便在监督微调（SFT）或强化学习（RL）期间具有价值。然而，结果是许多生成的标记被丢弃——合成数据生成需要大量计算。

This methodology for building a synthetic dataset for use in fine-tuning has been adopted by many of the large AI labs, and it is used for fine-tuning Gemini, GPT, Llama, and Claude.
这种用于构建合成数据集以进行微调的方法已被许多大型人工智能实验室采用，并用于微调 Gemini、GPT、Llama 和 Claude。

But Rejection Sampling can be more complicated than it appears. In Llama’s case, the model was prompted to revise its answer if the initial response was incorrect, and the model got the answer right on its second try 20% of the time. In another illustration of the usefulness of synthetic data, the Meta team translated Python code into PHP, ensuring quality via syntax parsing and execution, and fed this additional data into the SFT data set to account for the lack of public PHP code. This effectively demonstrates synthetic data being used to generate useful data reliably and predictably for underrepresented areas.
但拒绝采样可能比看起来更复杂。在 Llama 的案例中，如果初始响应不正确，模型会被提示修正其答案，而模型在第二次尝试时正确回答的概率为 20%。在另一个展示合成数据有用性的例子中，Meta 团队将 Python 代码翻译成 PHP，通过语法解析和执行确保质量，并将这些额外的数据输入到 SFT 数据集中，以弥补公共 PHP 代码的缺乏。这有效地展示了合成数据被用来可靠且可预测地生成有用数据，以满足代表性不足的领域。

Judgement by Model 模型判断

Another trend is to use another LLM as a judge. Meta used another, earlier version of Llama 3 as the rejection sampler, acting as the judge for code that was not strictly executable (i.e. pseudocode) and grading the output ‘pass’ or ‘fail’ on code correctness and style. In some instances, rejection sampling is done via a variety of models running concurrently to grade models. Although on net this is cheaper than human data, it is difficult to pull off such a chorus of automated judges. 
另一个趋势是使用另一个LLM作为评判者。Meta 使用了另一个早期版本的 Llama 3 作为拒绝采样器，充当不严格可执行代码（即伪代码）的评判者，并根据代码的正确性和风格对输出进行“通过”或“失败”的评分。在某些情况下，拒绝采样是通过多种模型并行运行来对模型进行评分。尽管总体上这比人工数据便宜，但要实现这样一组自动评判者是很困难的。

What is important to note here is that, across all methods of rejection sampling, code or not, the better the “judge” model, the higher the quality of the resulting data set. This feedback loop, while only just introduced in production for Meta this year, has been in use by Anthropic and OpenAI for a for a year or two prior to that.
需要注意的是，在所有拒绝采样的方法中，无论是否有代码，“评判”模型越好，生成的数据集质量就越高。这个反馈循环虽然今年刚刚在 Meta 的生产中引入，但在此之前，Anthropic 和 OpenAI 已经使用了一两年。

Long Context Datasets 长上下文数据集

Another example of synthetic data use is long context lengths. Models are pre-trained with capped context lengths (as most of the data is of a low context length already), but also because longer sequence lengths means a larger KV Cache to keep in memory – making the deployment of training infrastructure even harder than it already is. Models such as Gemini, GPT, and Claude are originally pre-trained with lower sequence lengths and then subsequently post-trained to add longer context lengths.
另一个合成数据使用的例子是长上下文长度。模型在限制的上下文长度下进行预训练（因为大多数数据的上下文长度已经很低），但更长的序列长度意味着需要在内存中保持更大的 KV 缓存——这使得训练基础设施的部署比现在更困难。像 Gemini、GPT 和 Claude 这样的模型最初是在较低的序列长度下进行预训练，然后随后进行后训练以添加更长的上下文长度。

It is generally difficult for humans to annotate long context examples in SFT data, as there are limitations of human resources of a sufficient talent level to provide quality annotation. Reading lengthy pieces of text is time consuming and tedious. Synthetic data has emerged as a useful, reliable way to ameliorate this problem.
人类在 SFT 数据中注释长上下文示例通常是困难的，因为提供高质量注释的人力资源有限。阅读冗长的文本既耗时又乏味。合成数据已成为改善这一问题的有用且可靠的方法。

One method to generate long context-length synthetic data is to use a model from an earlier checkpoint and have it summarize large pieces of text chunked into the size of its (currently small) context length. These summaries, or in other occasions, chats including simulated questions and answers, can then be used to help generate a body of synthetic data that can then be used in SFT.
生成长上下文长度合成数据的一种方法是使用早期检查点的模型，让它对大块文本进行摘要，这些文本被分块到其（当前较小的）上下文长度大小。这些摘要，或在其他情况下，包括模拟问答的聊天，可以用来帮助生成一批合成数据，然后可以在 SFT 中使用。

Other examples include generating synthetic data to make evals such as needle in haystack benchmarks pass. There are many more complex types of synthetic data to train the models to generalize and understand data in various parts of the extended context length.
其他例子包括生成合成数据以使评估通过，例如大海捞针基准测试。还有许多更复杂类型的合成数据用于训练模型，以便在扩展上下文长度的各个部分中进行泛化和理解数据。

Reinforcement Learning 强化学习

Reinforcement Learning (RL) is a leading method for alignment and model improvements.
强化学习（RL）是对齐和模型改进的主要方法。

Reinforcement Learning (RL) is when an Agent (for example, a Large Language Model) is taught to perform specific actions and seek certain outcomes by maximizing rewards that are given either for those specific actions or for achieving a given outcome. There are two axes to think about when it comes to RL: the source of the feedback, and how feedback is incorporated. The former is about how to source the signals, and the latter is about how to use those signals to update the model.
强化学习（RL）是指一个代理（例如，一个大型语言模型）被教导执行特定的动作并寻求某些结果，通过最大化为这些特定动作或实现给定结果而给予的奖励。考虑到 RL，有两个轴需要思考：反馈的来源，以及反馈是如何被纳入的。前者是关于如何获取信号，后者是关于如何使用这些信号来更新模型。

With reinforcement learning – the Large Language Model we are trying to optimize plays the role of an agent that can take a set of actions given an input or state and receive different rewards depending on the action it takes. We optimize this agent’s behavior with respect to our reinforcement learning goals by having the Agent learn the actions that can maximize the expected cumulative reward.
通过强化学习——我们试图优化的大型语言模型充当一个代理，能够根据输入或状态采取一系列行动，并根据其采取的行动获得不同的奖励。我们通过让代理学习能够最大化预期累积奖励的行动，来优化该代理的行为，以符合我们的强化学习目标。

There are a few main approaches to incorporate feedback and determine the action that an Agent takes – using Value-based methods or Policy-based methods such Direct Preference Optimization and Trust Region Policy Optimization (TRPO) as well as Actor-Critic methods that combine policy and value-based methods. Proximal Policy Optimization (PPO) is a prominent example of an actor-critic model, and more complex variations of it are the primary RL method at all major AI labs.
有几种主要方法来整合反馈并确定代理采取的行动——使用基于价值的方法或基于策略的方法，如直接偏好优化和信任区域策略优化（TRPO），以及结合策略和基于价值的方法的演员-评论家方法。近端策略优化（PPO）是演员-评论家模型的一个显著例子，其更复杂的变体是所有主要人工智能实验室的主要强化学习方法。

Value-based methods instead determine the value of getting to a given state and define values for each possible state. Each state is assigned a value based on the expected discounted return the agent can get if it starts in that state and then determines its action at each step based on the value of each action available to it. Historically, value-based methods were more commonly used in RL, but modern applications are much better served with Policy-based methods.
基于价值的方法则确定达到给定状态的价值，并为每个可能的状态定义价值。每个状态根据代理在该状态下开始时可以获得的预期折现回报被分配一个价值，然后代理根据可用每个动作的价值在每一步确定其行动。历史上，基于价值的方法在强化学习中更为常用，但现代应用更适合使用基于策略的方法。

In Policy-based methods, the Agent is driven by a policy function that identifies a set of actions that can be taken for a given state and assigns a probability distribution over those set of actions. Actions to be performed at a given state can be deterministic, meaning that being in each state will always lead to the same action, or stochastic, where a probability distribution instead describes potential actions at that given state. The policy function is then trained to direct the Agent towards actions that maximize expected reward.
在基于策略的方法中，代理由一个策略函数驱动，该函数识别出在给定状态下可以采取的一组行动，并为这些行动集分配一个概率分布。在给定状态下要执行的行动可以是确定性的，这意味着处于每个状态将始终导致相同的行动，或者是随机的，其中概率分布描述了在该给定状态下的潜在行动。然后，策略函数被训练以引导代理朝向最大化期望奖励的行动。

When employing policy-based methods during RL, a model can either evaluate the final result of a given task to determine the reward in the case of an Outcome Reward Model (ORM) or it can determine the reward by evaluating each individual step in a given process in the case of a Process Reward Model (PRM). Using a PRM can be particularly helpful when training reasoning models as while an ORM can detect that a chain of reasoning led to an incorrect answer, a PRM can tell you which step of the chain had the mistake.
在强化学习中使用基于策略的方法时，模型可以通过结果奖励模型（ORM）评估给定任务的最终结果以确定奖励，或者在过程奖励模型（PRM）的情况下，通过评估给定过程中的每个单独步骤来确定奖励。使用 PRM 在训练推理模型时特别有帮助，因为虽然 ORM 可以检测到一系列推理导致了错误答案，但 PRM 可以告诉你链中的哪个步骤出现了错误。

Because the policy function directs what the agent does at any given step – it is also an especially useful framework for optimizing the behavior of agents/models at intermediate steps of an inference process.
因为政策函数指引代理在任何给定步骤的行为——这也是一个特别有用的框架，用于优化推理过程中的中间步骤的代理/模型行为。

Outcome Reward Models and Process Reward Models are often used in Proximal Policy Optimization (PPO), an algorithm commonly used in reinforcement learning that iteratively improves a policy model to maximize cumulative rewards and optimize an LLM towards a given objective. Using ORMs and PRMs with PPO is particularly important when training multi-step reasoning models that are currently a key focus in the community. We will describe how this is done for o1 Pro below.
结果奖励模型和过程奖励模型通常用于近端策略优化（PPO），这是一种常用于强化学习的算法，通过迭代改进策略模型以最大化累积奖励并优化LLM以实现给定目标。在训练多步推理模型时，使用 ORM 和 PRM 与 PPO 特别重要，这些模型目前是社区的关键关注点。我们将描述如何为 o1 Pro 完成此操作。

Proximal Policy Optimization (PPO)
近端策略优化 (PPO)

Proximal Policy Optimization (PPO) can be used for both Alignment and Fine Tuning, but it is much better suited to and is used more often during Reinforcement Learning used during Alignment.
近端策略优化（PPO）可以用于对齐和微调，但在对齐过程中使用的强化学习中，它更为适合且使用得更为频繁。

For PPO, Policy refers to the abovementioned use of a policy model to dictate the actions of an agent or model, Proximal refers to the algorithm’s methodology of only gradually updating the policy, and Optimization refers to the process of iteratively improving the policy by providing feedback from a reward model to improve the policy model, thereby optimizing the expected cumulative reward. 
对于 PPO，政策指的是上述使用政策模型来决定代理或模型的行为，近端指的是算法的方法论，即仅逐步更新政策，优化指的是通过从奖励模型提供反馈来迭代改进政策的过程，从而优化预期的累积奖励。

We have mainly discussed Policy-based methods above, but PPO incorporates both Policy-based methods and Value-based methods in its implementation. As such, PPO can be said to use the Actor Critic method. An Actor is driven by a policy-based model that determines which action to take for a given state (i.e. Policy-based method) and there is a Critic that evaluates the action taken according to a value function (Value-based method). The Actor and Critic thus work together in an iterative fashion.
我们主要讨论了基于策略的方法，但 PPO 在其实现中结合了基于策略的方法和基于价值的方法。因此，可以说 PPO 使用了演员-评论家方法。演员由一个基于策略的模型驱动，该模型决定在给定状态下采取哪个行动（即基于策略的方法），而评论家根据价值函数评估所采取的行动（基于价值的方法）。因此，演员和评论家以迭代的方式协同工作。

Maximizing the PPO objective function will therefore push the policy in the direction of favoring actions that correspond to a higher value of the Advantage Function.
因此，最大化 PPO 目标函数将推动策略朝着有利于对应于更高优势函数值的动作的方向发展。

RLHF

Reinforcement Learning with Human Feedback (RLHF) has been a primary technique to align LLMs, make them useful, and was a leading factor for ChatGPT’s explosive growth. It typically utilizes policy-based learning, which is when a reward model that learns based on human feedback is used to update a policy that drives how a model behaves.  
人类反馈强化学习（RLHF）一直是对齐LLMs、使其有用的主要技术，并且是 ChatGPT 爆炸性增长的一个重要因素。它通常利用基于策略的学习，即使用基于人类反馈学习的奖励模型来更新驱动模型行为的策略。

With RLHF, human annotators review a sample of responses to prompts and rank their preference for one response over the other. The goal here is to amasses significant data on what responses humans would prefer. This preference data is then used to train a reward model, which attempts to guess the average labeler’s preference for a given output from a model. In other words, the trained reward model acts as a Critic in the Actor-Critic framework.
通过 RLHF，人类注释者审查一组对提示的响应样本，并对一个响应相对于另一个响应的偏好进行排名。这里的目标是积累关于人类偏好的重要数据。这些偏好数据随后用于训练奖励模型，该模型试图猜测平均标注者对模型给定输出的偏好。换句话说，训练后的奖励模型在演员-评论家框架中充当评论家。

The trained reward model evaluates this action against the human preferences it is trained on, and how much better or worse the action is compared to the average action. The feedback from this reward model then acts to align the Actor model, ensuring that it takes actions (generates tokens) in accordance with the desired policy.
经过训练的奖励模型将此行为与其训练所依据的人类偏好进行评估，并比较该行为与平均行为的优劣。来自该奖励模型的反馈随后用于调整演员模型，确保其采取的行动（生成的标记）符合期望的策略。

As discussed above, PPO is used to iteratively update the policy function of the language model. Allowing for stable learning while preventing drastic changes in policy. Large scale PPO for AI labs utilizes multiple weighted reward models for specific aspects like helpfulness, truthfulness, and safety.
如上所述，PPO 用于迭代更新语言模型的策略函数。允许稳定学习，同时防止策略的剧烈变化。大型 PPO 用于人工智能实验室，利用多个加权奖励模型来关注特定方面，如有用性、真实性和安全性。

Broadly speaking, RLHF allows models to perform better on tasks that real end users care about and have provided preference data on. Meta’s Llama 2-Chat achieved much better performance on factors such as helpfulness and harmlessness after rounds of RLHF. The paper demonstrates that the additional compute used to scale models during RL delivers clear results. The potential benefits from using synthetic data as opposed to human-generated feedback and relying more heaving on AI for feedback can also justify the use of even more compute.
从广义上讲，RLHF 使模型在真实终端用户关心并提供偏好数据的任务上表现更好。Meta 的 Llama 2-Chat 在有用性和无害性等因素上经过多轮 RLHF 后取得了更好的表现。论文表明，在 RL 过程中用于扩展模型的额外计算带来了明显的结果。使用合成数据而非人类生成反馈的潜在好处，以及更依赖 AI 进行反馈的做法，也可以证明使用更多计算资源的合理性。

However, there are significant limitations to RLHF. First – carrying out the entire lifecycle of RLHF can be very slow as one must take time to expose the various generated responses to human responders, usually through an AI company inserting such prompts for feedback when serving its models or human labelers.
然而，RLHF 存在显著的局限性。首先，执行 RLHF 的整个生命周期可能非常缓慢，因为必须花时间将各种生成的响应暴露给人类响应者，通常是通过 AI 公司在为其模型提供服务时插入此类提示以获取反馈或人类标注者。

Even with a large userbase, collecting a large amount of preference data is difficult and expensive – Meta spent $10-20 million dollars on preference data for Llama 2, more than the compute time itself.
即使拥有庞大的用户基础，收集大量偏好数据仍然困难且昂贵——Meta 在 Llama 2 的偏好数据上花费了 1000 万到 2000 万美元，甚至超过了计算时间本身的成本。

RLHF is inherently difficult to scale, especially in areas where there is not a huge amount of existing data. Human annotation is also expensive. This is why many AI companies are pivoting towards Reinforcement Learning with AI Feedback (RLAIF) during training.
RLHF 本质上难以扩展，尤其是在没有大量现有数据的领域。人工标注也很昂贵。这就是为什么许多人工智能公司在训练过程中转向使用带有人工反馈的强化学习（RLAIF）。

The larger AI companies have a clear advantage here. Claude, Gemini, and ChatGPT all ask users provide feedback on responses from models they host. For instance, on occasion, ChatGPT will explicitly ask you to select which one of two responses you prefer. This effectively gathers the best source of feedback (directly from users) for free. Because OpenAI has a huge customer base of more than 300M users, it can gather a lot of feedback, for improving models.
大型人工智能公司在这里具有明显优势。Claude、Gemini 和 ChatGPT 都要求用户对他们托管的模型的响应提供反馈。例如，ChatGPT 有时会明确要求您选择两个响应中您更喜欢哪一个。这有效地收集了最佳的反馈来源（直接来自用户），而且是免费的。由于 OpenAI 拥有超过 3 亿用户的庞大客户基础，它可以收集大量反馈，以改进模型。

Providers with fewer users or that operate a platform that is less conducive towards users providing feedback need to resort to other methods such as DPO instead of PPO. Direct Preference Optimization (DPO) is another technique often discussed with RLHF, though most do not technically categorize it as a Reinforcement Learning technique.
用户较少的提供商或运营一个不太利于用户提供反馈的平台需要采用其他方法，例如直接偏好优化（DPO），而不是偏好优化（PPO）。直接偏好优化（DPO）是另一种常与强化学习人类反馈（RLHF）讨论的技术，尽管大多数人并不将其技术上归类为强化学习技术。

DPO entirely forgoes training a reward model and instead uses optimization to directly adjust the policy to maximize the probability that the policy drives the model to produce the preferred outputs as based on the human preference data. The optimization works by using a binary cross-entropy loss that compares probability ratios between the current model and a reference model (generally the same model before fine tuning). DPO ensures the model learns to favor preferred responses while staying close to the reference model’s behavior.
DPO 完全放弃了训练奖励模型，而是使用优化直接调整策略，以最大化策略驱动模型产生基于人类偏好数据的首选输出的概率。优化通过使用二元交叉熵损失来工作，该损失比较当前模型与参考模型（通常是微调前的同一模型）之间的概率比率。DPO 确保模型学习偏好首选响应，同时保持接近参考模型的行为。

The simpler approach used in DPO can achieve comparable or better results than RLHF using a full reward model, while being less prone to crashes and easier to implement. A prominent example of this approach’s merits is that Llama 3 did not undergo RLHF and went through DPO. Meta found that in the case of Llama 3, DPO was more effective and stable than PPO and used less compute. However – using DPO means that the quality of the preference data set is paramount, meriting extra care and attention on how this data is gathered and processed.
DPO 中使用的更简单的方法可以实现与使用完整奖励模型的 RLHF 相当或更好的结果，同时不易崩溃且更易于实施。这个方法优点的一个显著例子是 Llama 3 没有经过 RLHF，而是采用了 DPO。Meta 发现，在 Llama 3 的情况下，DPO 比 PPO 更有效且更稳定，并且使用的计算资源更少。然而，使用 DPO 意味着偏好数据集的质量至关重要，需要额外关注和小心处理这些数据的收集和处理方式。

Meta eventually discovered the lesson the other labs already knew: DPO does not scale as well as PPO – and that they must turn to RLAIF to continue to improve their post training. This was shown in the release of the newest LLAMA 3.3.
Meta 最终发现了其他实验室已经知道的教训：DPO 的扩展性不如 PPO——他们必须转向 RLAIF 以继续改善训练后的表现。这在最新的 LLAMA 3.3 发布中得到了体现。

RLAIF

Instead of relying on human feedback to train a reward model, Reinforcement Learning with AI Feedback (RLAIF) replaces human feedback with another model. The reward model is trained based on AI-generated feedback – usually some form of scoring model or algorithm that will evaluate given completions and determine the reward accordingly.
与其依赖人类反馈来训练奖励模型，人工智能反馈强化学习（RLAIF）用另一个模型替代人类反馈。奖励模型是基于人工智能生成的反馈进行训练的——通常是一种评分模型或算法，用于评估给定的完成情况并相应地确定奖励。

Broadly, not much else is inherently different from RLHF, but RLAIF makes a dramatic difference. Annotations can be made quickly, and prompts can be generated synthetically to prompt the model undergoing reinforcement learning in areas where additional data or training is needed.
总体而言，RLHF 本身并没有太大不同，但 RLAIF 却带来了显著的变化。注释可以快速完成，提示可以合成生成，以促使正在进行强化学习的模型在需要额外数据或训练的领域进行学习。

In addition to providing feedback on typical math, science and general knowledge tasks, RLAIF also means that feedback to tackle more nuanced circumstances like ethical dilemmas, cultural norms, and social interactions can be generated quickly and ranked by another LLM. This enables more coverage in terms of topics to align the model over and also allows model trainers to quickly ramp training on those topics without waiting to gather human feedback.
除了提供对典型数学、科学和一般知识任务的反馈外，RLAIF 还意味着可以快速生成针对更微妙情况的反馈，例如伦理困境、文化规范和社会互动，并由另一个 LLM 进行排名。这使得在主题覆盖方面能够更好地对齐模型，同时也允许模型训练者快速开始这些主题的训练，而无需等待收集人类反馈。

A unique use of RLAIF is Anthropic’s constitutional AI. Constitutional AI works in two stages. In the first stage, a base model critiques and revises its own outputs in accordance with a set of constitutional principles written by humans. These initial responses that are evaluated can be toxic or unhelpful. The responses are then revised continuously using a variety of principles from the constitution. This creates a data set of revision and prompt pairs that are then used to fine tune a model through supervised fine-tuning (SFT).
RLAIF 的独特用法是 Anthropic 的宪法 AI。宪法 AI 分为两个阶段。在第一阶段，基础模型根据人类编写的一套宪法原则对其自身输出进行批评和修订。这些被评估的初始响应可能是有毒或无帮助的。然后，使用宪法中的各种原则不断修订这些响应。这创建了一组修订和提示对的数据集，随后用于通过监督微调（SFT）对模型进行微调。

The second stage of the process for Constitutional AI is similar to RLHF, but without the human preference data providing feedback regarding harmlessness. The AI evaluates pairs of responses from the previous stage’s model in accordance with constitutional principles which in effect are like multiple reward models. AI-generated preferences for harmlessness are combined with human feedback data for helpfulness to train a hybrid preference model (hybrid meaning it includes human data). Finally, the model from the first stage is fine-tuned using RL with this preference model as the reward signal.
过程的第二阶段对于宪法人工智能来说类似于 RLHF，但没有人类偏好数据提供关于无害性的反馈。人工智能根据宪法原则评估来自前一阶段模型的响应对，这些原则实际上就像多个奖励模型。人工智能生成的无害性偏好与人类反馈数据（关于有用性）结合，以训练一个混合偏好模型（混合意味着它包含人类数据）。最后，第一阶段的模型使用 RL 进行微调，以这个偏好模型作为奖励信号。

The most notable observation of this approach is that it’s scalable across many different domains – if there is a model that is good at ranking responses based on which one is more scientifically accurate in addition to being able to identify harmlessness, the model can be used to optimize for scientifically accurate responses as well.
这种方法最显著的观察是，它在许多不同领域具有可扩展性——如果有一个模型能够根据哪个响应在科学上更准确来对响应进行排名，并且能够识别无害性，那么该模型也可以用于优化科学准确的响应。

RL is also a key part of developing reasoning models that use Chain of Thought (CoT).
RL 也是开发使用思维链（CoT）的推理模型的关键部分。

Reasoning Models and Chain of Thought (CoT)
推理模型和思维链（CoT）

Math is the fundamental logic and reasoning of engineering, construction, and system design. Math stands out as a focus discipline for fine tuning models as model trainers lack sufficiently complex prompts at advanced difficulty levels. One way to overcome this problem is to pay highly skilled humans to craft prompts or generate them in house. Solving Math problems effectively through reasoning requires a clearly articulated and correct chain of thought that the model can learn from.
数学是工程、建筑和系统设计的基本逻辑和推理。数学作为一个重点学科，在微调模型方面表现突出，因为模型训练者在高级难度水平上缺乏足够复杂的提示。克服这个问题的一种方法是雇佣高技能的人类来制作提示或在内部生成提示。通过推理有效地解决数学问题需要一个清晰表达且正确的思维链，以便模型能够从中学习。

While some math capabilities can improve through tools like code interpreter access, allowing models to generate and execute code in languages like Python which can assist solving some math problems, code is not enough to solve many problems – particularly the most difficult math problems. A huge amount of effort is currently targeted at training reasoning models to solve complex math problems.
虽然一些数学能力可以通过像代码解释器访问这样的工具来提高，允许模型生成和执行像 Python 这样的语言中的代码，从而帮助解决一些数学问题，但代码并不足以解决许多问题——特别是最困难的数学问题。目前，巨大的努力正集中在训练推理模型以解决复杂的数学问题上。

Models can be prompted to generate chains of thought out of the box, but results can be unreliable since an error on one step of the chain will compound to the wrong end solution. Though, o1 Pro has multiple safeguards to prevent this. Another challenge is that even the latest models can hallucinate and fabricate information if there is uncertainty, which can easily compound error in one of the reasoning steps.
模型可以被提示生成超出常规的思维链，但结果可能不可靠，因为链中某一步的错误会导致错误的最终解决方案。不过，o1 Pro 有多重保护措施来防止这种情况。另一个挑战是，即使是最新的模型在存在不确定性时也可能产生幻觉和虚构信息，这可能会轻易加剧推理步骤中的错误。

A model has been aligned to conduct Reasoning using Chain of Thought can address many of the challenges above. In this approach, reinforcement learning is used to align the model’s behavior towards this Chain of Thought approach.
一个模型已经被调整以使用思维链进行推理，可以解决上述许多挑战。在这种方法中，使用强化学习来使模型的行为与这种思维链方法对齐。

This process applies reinforcement learning to align a base LLM’s behavior towards the Chain of Thought approach and improve its accuracy using several other separate models and LLMs.
该过程应用强化学习来使基础LLM的行为与思维链方法对齐，并利用其他几个独立模型和LLMs提高其准确性。

The first independent LLM to discuss is the Generator, which is trained to produce solutions that are reasoned out across multiple steps. The generator is typically separate from the base LLM as it is fine-tuned specifically for the task of generating these reasoning steps while the base LLM is usually fine-tuned for general tasks.
第一个独立的LLM要讨论的是生成器，它经过训练以产生经过多步推理得出的解决方案。生成器通常与基础LLM分开，因为它是专门针对生成这些推理步骤的任务进行微调的，而基础LLM通常是针对一般任务进行微调的。

Secondly is the Verifier Model, which is responsible for evaluating whether the solutions produced by the Generator are correct or not and provides a corresponding reward.
其次是验证模型，负责评估生成器产生的解决方案是否正确，并提供相应的奖励。

Verifier Models can be trained using either human annotation, through automatic process annotation or using automatic verifiers. Alternatively – verification In OpenAI’s paper, Let’s Verify Step by Step, researchers introduced the PRM800K process supervision dataset, in which human data-labelers annotate 800,000 process steps that form part of 75,000 solutions to 12,000 questions from the MATH Dataset that are output from a Generator as discussed in the paper.
验证模型可以通过人工标注、自动过程标注或使用自动验证器进行训练。或者——在 OpenAI 的论文《逐步验证》中，研究人员介绍了 PRM800K 过程监督数据集，其中人工数据标注者对 80 万个过程步骤进行标注，这些步骤构成了 75,000 个解决方案的一部分，涉及来自 MATH 数据集的 12,000 个问题，这些问题是由论文中讨论的生成器输出的。

The cost of gathering these annotations is not trivial. In the original Math paper, a few university students that were given an hour to complete 20 problems scored between 40% and 90%, with the 90% scorer being a three-time IMO gold medalist. The OpenAI paper cited cost as a reason that it would be impractical to build a large enough human annotated PRM-oriented dataset to match the order of magnitude larger ORM-oriented dataset to conduct apples-to-apples comparisons.
收集这些注释的成本并不低。在原始的数学论文中，几名大学生被给予一个小时的时间来完成 20 个问题，得分在 40%到 90%之间，其中 90%得分者是三届国际数学奥林匹克金牌得主。OpenAI 的论文引用成本作为理由，认为构建一个足够大的人工注释 PRM 导向数据集以匹配数量级更大的 ORM 导向数据集进行直接比较是不切实际的。

The alternatives are to use automatic process annotation, or to find automatic verifiers.
替代方案是使用自动过程注释，或寻找自动验证器。

Automatic verifiers are a system or model that can ideally quickly and easily verify whether the solution to a given problem is correct. For code, this could simply be the actual execution of the cost to test that it produces the desired results, while for Math it could be evaluating a given function or using prover like LEAN to check for correctness. However, using automatic verifiers might not be as “automatic” as it sounds – creating dependencies on external systems can add overhead which can detract from good training performance, while automatic verifiers can sometimes take time to run.
自动验证器是一种系统或模型，理想情况下可以快速而轻松地验证给定问题的解决方案是否正确。对于代码，这可能仅仅是实际执行成本以测试其是否产生期望的结果，而对于数学，这可能是评估给定函数或使用像 LEAN 这样的证明器来检查正确性。然而，使用自动验证器可能并不像听起来那么“自动”——对外部系统的依赖可能会增加开销，从而影响良好的训练性能，而自动验证器有时可能需要时间来运行。

Automatic process annotation can generate this step-by-step process annotation. Instead of having a human evaluate an intermediate step, the Completer is used to create multiple different paths of reasoning steps. The Math-Shepherd paper uses automatic process annotation – generating a number of paths, then evaluating these paths by either marking it as a good reasoning step if it leads to a correct final answer (i.e. Hard Estimation) or by assigning a score based on the frequency with which the step leads to the correct solution (i.e. Soft Estimation).
自动过程注释可以生成这个逐步过程注释。与其让人类评估一个中间步骤，不如使用 Completer 来创建多个不同的推理步骤路径。Math-Shepherd 论文使用自动过程注释——生成多个路径，然后通过标记为良好的推理步骤（如果它导致正确的最终答案，即困难估计）或根据该步骤导致正确解决方案的频率分配分数（即软估计）来评估这些路径。

The fourth model is the Reward Model, which is trained from the process annotation labels.
第四种模型是奖励模型，它是根据过程注释标签进行训练的。

To recap our earlier explanation, there are two types of reward models: ones which provide a reward based on the outcome, an Outcome Reward Model (ORM), and ones which provide a reward based on the process, Process Reward Models (PRM). ORMs typically work by ranking a variety of different answers that a model provides and then selecting the highest ranked one. In contrast, PRMs evaluate and assign a score to each step of the reasoning chain of thought and provide a reward based on this score and for this reason are generally preferred when training Chain of Thought models. The Let’s Verify Step by Step paper showcased stronger results for PRMs over ORMs. With that said, OpenAI relies more heavily on ORMs still.
回顾我们之前的解释，奖励模型有两种类型：一种是基于结果提供奖励的结果奖励模型（ORM），另一种是基于过程提供奖励的过程奖励模型（PRM）。ORM 通常通过对模型提供的各种不同答案进行排名，然后选择排名最高的答案来工作。相比之下，PRM 评估并为推理链中每一步分配分数，并根据该分数提供奖励，因此在训练思维链模型时通常更受欢迎。《逐步验证》论文展示了 PRM 相较于 ORM 的更强结果。尽管如此，OpenAI 仍然更依赖于 ORM。

In Math-Shepherd, Reinforcement Learning via Step-by-step Proximal Policy Optimization (PPO), is used to reinforce the final LLM to teach it to the desired reasoning chain of thought behavior.
在 Math-Shepherd 中，通过逐步近端策略优化（PPO）的强化学习被用来强化最终的LLM，以教导其达到期望的推理链思维行为。

Inference-time Scaling 推理时间缩放

The release of OpenAI o1 preview has brought the industry’s attention to the rise of a new scaling law – the greater the test-time compute (i.e. compute at inference time), the better the answer, and efforts to exploit this scaling dimension are at a major inflection point.
OpenAI o1 预览的发布引起了行业对一种新扩展法则的关注——测试时计算（即推理时的计算）越大，答案越好，利用这一扩展维度的努力正处于一个重大转折点。

When presented with queries, whether for simple or difficult questions, traditional LLMs will generate tokens continuously, without tracking intermediate steps, until they think they have reached the answer.
当面对查询时，无论是简单还是困难的问题，传统的LLMs会不断生成令牌，而不跟踪中间步骤，直到他们认为自己已经得到了答案。

In contrast, as explained above, Reasoning Models break the response into a discrete number of reasoning step called a Chain-of-Thought, before delivering a response to the user. Reasoning models can backtrack if they reach an illogical conclusion, recognizing that a mistake has been made or a certain approach has reached a dead end, revisiting earlier steps to put the chain of reasoning back on the right path.
相反，如上所述，推理模型将响应分解为称为思维链的离散推理步骤，然后再向用户提供响应。如果推理模型得出不合逻辑的结论，它可以回溯，意识到已经犯了错误或某种方法已走入死胡同，重新审视早期步骤，以将推理链重新引导回正确的路径。

There are two profound implications from the release of reasoning models – first, a meaningful improvement in model performance for challenging evaluations such as those oriented around coding, math, and science, and second, the realization that this improvement in model performance scales with test-time compute extends robustly to LLMs.
发布推理模型有两个深远的影响——首先，对于编码、数学和科学等具有挑战性的评估，模型性能有了显著提升；其次，意识到这种模型性能的提升与测试时的计算规模稳健地扩展到LLMs。

Test-time scaling is not a new concept. In board games and poker, the idea of expanding test-time compute has been around for some time. For example, AlphaGo, which is DeepMind’s system for playing Go, uses Monte Carlo Tree Search during test time to decide which moves to play. If stripped of its capabilities of searching during inference, it drops in Elo from ~5,200 to 3,000 (top humans are around ~3,800). Inference time compute allowed for superhuman achievements in Go.
测试时间扩展并不是一个新概念。在棋类游戏和扑克中，扩展测试时间计算的想法已经存在了一段时间。例如，AlphaGo 是 DeepMind 用于下围棋的系统，在测试时使用蒙特卡罗树搜索来决定下哪些棋。如果在推理过程中剥夺了其搜索能力，Elo 评分将从约 5200 降至 3000（顶级人类选手的评分约为 3800）。推理时间计算使得在围棋中实现超人类的成就成为可能。

With greater compute, reasoning models can think through more steps and increase the likelihood of reaching the right answer. Today, reasoning capabilities are bottlenecked by inference system capabilities as the long context lengths required for reasoning models significantly increase memory and compute requirements.
随着计算能力的增强，推理模型可以思考更多的步骤，并提高达到正确答案的可能性。如今，推理能力受到推理系统能力的瓶颈，因为推理模型所需的长上下文长度显著增加了内存和计算要求。

This means operators of inference systems for reasoning models are limiting the length of reasoning chains of thought to keep context lengths reasonable and prices down so as to serve an economical number of users at a reasonable token to token latency.  It follows that today’s reasoning models are performing with one arm tied behind their back and could scale very significantly in performance as more capable inference systems such as the GB200 NVL72 come to market. Once economical, allowing o1 to adjust the length of its reasoning chain and compute employed will be a technique to harness test-time compute scaling.
这意味着推理模型的推理系统运营商正在限制推理链的思维长度，以保持上下文长度合理并降低价格，从而以合理的令牌到令牌延迟为经济数量的用户提供服务。因此，今天的推理模型在性能上受到限制，随着更强大的推理系统如 GB200 NVL72 的上市，其性能可以显著提升。一旦经济可行，允许 o1 调整其推理链的长度和所使用的计算将是一种利用测试时计算扩展的技术。

As we see from evals and from the graph further down below, with one attempt, GPT-4o beats other models. The most naïve way to scale test-time compute is to simply increase the number of samples concurrently being run, effectively channeling the infinite monkey theorem. The paper Large Language Monkeys demonstrates that simply repeated sampling can scale inference time compute and can yield much better results. 
从评估和下面的图表中可以看出，GPT-4o 在一次尝试中超越了其他模型。测试时间计算的最简单扩展方式是简单地增加同时运行的样本数量，有效地体现了无限猴子定理。论文《大型语言猴子》表明，简单的重复采样可以扩展推理时间计算，并且可以产生更好的结果。

This is arguably one of the most basic ways of doing search. Generating more samples allows for greater coverage, which is defined as any of the samples getting the correct answer (i.e. pass@k). One could argue that simply enabling these smaller models to think over a problem many times may be more accurate and cheaper, though we will need to have an effective verifier to identify when we have successfully generated the metaphorical complete works of Shakespeare.
这可以说是进行搜索的最基本方式之一。生成更多样本可以实现更大的覆盖率，这被定义为任何样本获得正确答案（即 pass@k）。有人可能会认为，仅仅让这些较小的模型多次思考一个问题可能更准确且成本更低，尽管我们需要一个有效的验证者来识别何时成功生成了隐喻上的莎士比亚完整作品。

Scaling Inference Compute Through Search
通过搜索扩展推理计算

Search is another dimension of scaling that goes unharnessed with OpenAI o1 but is utilized in o1 Pro. o1 does not evaluate multiple paths of reasoning during test-time (i.e. during inference) or conduct any search at all. Sasha Rush’s video on Speculations on Test-Time Scaling (o1) provides a useful discussion and illustration of Search and other topics related to reasoning models.
搜索是扩展的另一个维度，在 OpenAI o1 中未被利用，但在 o1 Pro 中得到了应用。o1 在测试时（即推理期间）不评估多条推理路径，也不进行任何搜索。Sasha Rush 关于测试时间扩展（o1）的推测视频提供了对搜索和其他与推理模型相关主题的有用讨论和说明。

Self-Consistency / Majority Vote is one such search methodology in which we simply run the prompt through the model multiple times, thereby generating multiple responses, and then we pick the correct answer by choosing the response that appears most often among a given number of samples.
自一致性/多数投票是一种搜索方法，我们简单地将提示多次输入模型，从而生成多个响应，然后通过选择在给定数量的样本中出现最频繁的响应来选择正确答案。

Best-of-N Sampling is another idea in which we generate N solutions for a specific prompt and then use a verifier model to identify chains-of-thoughts that led to the correct answer. This method is generally restricted to areas that are amenable to verification (e.g., sudoku and not essays) and is limited by the effectiveness of the verifier model.
最佳 N 采样是另一个概念，我们为特定提示生成 N 个解决方案，然后使用验证模型识别导致正确答案的思维链。这种方法通常仅限于适合验证的领域（例如，数独而不是论文），并受到验证模型有效性的限制。

Monte Carlo roll-outs are a technique that build on Best-of-N. Here we evaluate a given intermediate step by generating multiple paths to complete the chain-of-thought starting from that intermediate step. This evaluation can help us decide whether to proceed with this step or move forward with prospective future step, improving our overall chain of thought.
蒙特卡罗滚动法是一种基于最佳选择的技术。在这里，我们通过生成多个路径来评估给定的中间步骤，以完成从该中间步骤开始的思维链。这种评估可以帮助我们决定是继续进行此步骤还是向前推进潜在的未来步骤，从而改善我们的整体思维链。

Now that we have discussed the basis of RL, Synthetic Data, Chain-of-Thought, Inference Time Compute and other concepts, let us go through what OpenAI has done with o1 and o1 Pro both during training and during inference. The construction of o1 is unique and doesn’t mirror the papers above. We will also discuss the tokenomics of inference time compute including cost, KV Cache scaling, batching, and more. Lastly, we will explain what OpenAI is doing next with Orion and why the narrative around it being a failure isn’t accurate.
现在我们已经讨论了强化学习的基础、合成数据、思维链、推理时间计算和其他概念，让我们来看看 OpenAI 在训练和推理过程中对 o1 和 o1 Pro 所做的工作。o1 的构建是独特的，并不与上述论文相似。我们还将讨论推理时间计算的代币经济学，包括成本、KV 缓存扩展、批处理等。最后，我们将解释 OpenAI 接下来在 Orion 上所做的工作，以及为什么关于它是失败的叙述并不准确。