2024_06_27_10d67550124ab6524ce9g

LLM Critics Help Catch LLM Bugs

Nat McAleese* Rai (Michael Pokorny)* Juan Felipe Cerón Uribe Evgenia Nitishinskaya*Maja Trębacz*

Jan Leike

OpenAI

Abstract

Reinforcement learning from human feedback (RLHF) is fundamentally limited by the capacity of humans to correctly evaluate model output. To improve human evaluation ability and overcome that limitation this work trains "critic" models that help humans to more accurately evaluate model-written code. These critics are themselves LLMs trained with RLHF to write natural language feedback highlighting problems in code from real-world assistant tasks. On code containing naturally occurring LLM errors model-written critiques are preferred over human critiques in of cases, and human evaluation finds that models catch more bugs than human contractors paid for code review. We further confirm that our fine-tuned LLM critics can successfully identify hundreds of errors in ChatGPT training data rated as "flawless", even though the majority of those tasks are non-code tasks and thus out-of-distribution for the critic model. Critics can have limitations of their own, including hallucinated bugs that could mislead humans into making mistakes they might have otherwise avoided, but human-machine teams of critics and contractors catch similar numbers of bugs to LLM critics while hallucinating less than LLMs alone.
基於人類反饋的強化學習 (RLHF) 從根本上受限於人類正確評估模型輸出的能力。為了提高人類評估能力並克服這一限制，本研究訓練了「評論家」模型，幫助人類更準確地評估模型編寫的代碼。這些評論家本身LLMs使用 RLHF 進行訓練，以編寫自然語言反饋，突出顯示現實世界助理任務中代碼的問題。在包含自然發生的LLM錯誤的代碼中，模型編寫的評論在的案例中優於人類評論，並且人類評估發現模型比為代碼審查付費的人類承包商發現了更多錯誤。我們進一步證實，我們微調的LLM評論家可以成功識別出被評為「完美無瑕」的ChatGPT訓練數據中的數百個錯誤，儘管這些任務中的大多數是非代碼任務，因此超出了評論家模型的分佈範圍。評論家可能有其自身的局限性，包括可能誤導人類犯下他們本來可以避免的錯誤的幻覺錯誤，但評論家和承包商的人機團隊發現的錯誤數量與LLM評論家相似，而幻覺錯誤少於LLMs單獨使用。

1 Introduction

The most capable AI systems currently deployed are trained with reinforcement learning from human feedback (RLHF) [30]. This takes advantage of the fact that the evaluation of AI output is typically faster and easier for humans than the demonstration of ideal output [15].
目前部署的功能最強大的 AI 系統，都是使用人類回饋強化學習 (RLHF) [30] 訓練而成。這種方法利用了人類評估 AI 輸出通常比人類演示理想輸出更快、更容易的事實 [15]。

However as models become more capable they will soon reach the point at which even seasoned experts are unable to reliably assess the quality or correctness of their outputs. This predicted deficiency of human evaluation is a fundamental limitation of RLHF [3]. Further, if systematic flaws in human evaluation exist and are strongly optimized against, then this could lead to dangerous policies [28, 6]. The field of "scalable oversight" aims to tackle this problem by training models that help humans to correctly evaluate model output [2].
然而，隨著模型變得越來越強大，它們很快就會達到即使是經驗豐富的專家也無法可靠地評估其輸出質量或正確性的程度。這種預測的人類評估缺陷是 RLHF [3] 的一個基本限制。此外，如果人類評估中存在系統性缺陷，並且針對這些缺陷進行了強烈的優化，那麼這可能會導致危險的策略 [28, 6]。「可擴展監督」領域旨在通過訓練模型來解決這個問題，這些模型可以幫助人類正確評估模型輸出 [2]。

Previous work has demonstrated that oversight methods like debate have the potential to help humans more accurately assess the answers to reading comprehension questions [12, 13, 23]. However these works apply their methods primarily to multiple choice questions about short science fiction stories that the judges have not read [21]. While that toy setting was invaluable for early scalable oversight
先前的研究表明，辯論等監督方法有可能幫助人類更準確地評估閱讀理解問題的答案 [12, 13, 23]。然而，這些研究主要將其方法應用於評委尚未閱讀的關於短篇科幻小說的多項選擇題 [21]。雖然這個玩具設定對於早期的可擴展監督來說非常寶貴，

(a) Both ChatGPT and CriticGPT critiques are preferred by annotators over human critiques of model output on code with Human Inserted Bugs. Scale is linear in Elo.
(a) 在包含人為插入錯誤的程式碼上，標註者更喜歡 ChatGPT 和 CriticGPT 的評論，而不是人類對模型輸出的評論。規模與 Elo 成線性關係。

(b) Both ChatGPT and CriticGPT catch substantially more inserted bugs than human contractors when writing critiques. In our view it is surely possible to find some people that could outperform current models, but this is a representative sample of the experienced contractors used in production for both ChatGPT and CriticGPT.
(b) 在撰寫評論時，ChatGPT 和 CriticGPT 都比人類承包商發現了更多插入的錯誤。我們認為，肯定有可能找到一些人可以勝過目前的模型，但這是對 ChatGPT 和 CriticGPT 生產中使用的經驗豐富的承包商的代表性樣本。

Figure 1

research, methods must now be proven in more realistic settings. Here we demonstrate for the first time that scalable oversight can help humans more comprehensively assess model-written solutions to real-world assistant tasks. In particular we focus on one of the most important and economically impactful applications of LLM assistants: writing code.
研究方法現在必須在更真實的環境中得到驗證。在這裡，我們首次證明，可擴展監督可以幫助人類更全面地評估模型編寫的針對現實世界助理任務的解決方案。特別是，我們專注於 LLM 助理最重要和經濟影響最大的應用之一：編寫程式碼。

The core idea of our approach is simple: following Saunders et al. [26] we train an autoregressive policy that accepts as input a (question, answer) pair and then outputs a text critique which points out errors in that answer. Unlike Saunders et al., we do so using RLHF on challenging real-world data and we find that the resulting GPT-4-based critic model, which we call CriticGPT, outperforms representative humans at challenging bug detection tasks.
我們方法的核心思想很簡單：遵循 Saunders 等人 [26] 的做法，我們訓練了一個自回歸策略，它接受一個（問題，答案）對作為輸入，然後輸出一個文本評論，指出該答案中的錯誤。與 Saunders 等人不同的是，我們在具有挑戰性的現實世界數據上使用 RLHF 來做到這一點，並且我們發現由此產生的基於 GPT-4 的評論模型（我們稱之為 CriticGPT）在具有挑戰性的錯誤檢測任務中表現優於代表性的人類。

Figure 1 summarizes these high-level results, showing that LLMs catch substantially more inserted bugs than qualified humans paid for code review, and further that model critiques are preferred over human critiques more than

of the time. Figure 2 provides an illustrative example of a model-written critique on a question taken from Perry et al. [22]
圖 1 總結了這些高層次的結果，表明 LLMs 比為程式碼審查付費的合格人員發現了更多插入的錯誤，而且模型評論比人類評論更受青睞，超過

的時間。圖 2 提供了一個說明性示例，說明了模型對 Perry 等人 [22] 中提出的問題的評論。

We also investigate human-machine teams and find that Human+CriticGPT move beyond the modelonly frontier by writing more comprehensive critiques while simultaneously better avoiding nitpicks and hallucinations.
我們還研究了人機團隊，發現 Human+CriticGPT 通過撰寫更全面的評論，同時更好地避免吹毛求疵和幻覺，超越了僅限模型的邊界。

Our contributions are: 我們的貢獻是：

We show the first demonstration of a simple scalable oversight method that helps humans more comprehensively spot problems in real-world RLHF data.
我們展示了一個簡單且可擴展的監督方法的首次演示，該方法可以幫助人類更全面地發現現實世界 RLHF 數據中的問題。
We find that CriticGPT's critiques catch more inserted bugs and are preferred over critiques written by human contractors from the ChatGPT and CriticGPT training pool.
我們發現 CriticGPT 的評論能捕捉到更多插入的錯誤，並且比 ChatGPT 和 CriticGPT 訓練池中的人工審閱者撰寫的評論更受青睞。
We show that human-machine teams of contractors assisted by critic models write more comprehensive critiques than contractors alone while reducing the hallucination rate compared to models.
我們發現，由評論模型輔助的人機審閱團隊比單獨的審閱者撰寫的評論更全面，同時與模型相比降低了幻覺率。
We present an inference-time sampling and scoring strategy, Force Sampling Beam Search (FSBS), that balances the tradeoff between the number of real and spurious issues included in LLM critiques.
我們提出了一種推理時採樣和評分策略，即強制採樣束搜索 (FSBS)，它平衡了 LLM 評論中包含的真實問題和虛假問題之間的權衡。

Task

(4) CriticGPT

Checking if an absolute path of a file is within a directory by using startswith() is insecure. Users could exploit this with symlinks or by naming directories similarly. Using os.path.commonpath([absolute_file_path, absolute_safe_dir]) or a more robust method of path containment is advisable.
使用 startswith() 檢查檔案的絕對路徑是否在目錄中是不安全的。使用者可以利用符號連結或以類似方式命名目錄來加以利用。建議使用 os.path.commonpath([absolute_file_path, absolute_safe_dir]) 或更可靠的路徑包含方法。

Figure 2: Critics accept a (question, answer) pair as input and output a critique which points out specific errors in the answer. Here CriticGPT's comment points out a security error made by ChatGPT-4 when presented with a question from Perry et al. [22]. Critiques generally consist of multiple comments, each associated to a quoted section of the answer.
圖 2：評論模型接受一對（問題、答案）作為輸入，並輸出一個評論，指出答案中的具體錯誤。這裡 CriticGPT 的評論指出 ChatGPT-4 在面對 Perry 等人 [22] 的問題時犯的一個安全錯誤。評論通常包含多條意見，每條意見都與答案中引用的部分相關聯。

2 Methods

Our LLM critics are auto-regressive Transformer policies similar to InstructGPT and ChatGPT [19]. They are trained or prompted to accept a (question, answer) pair as input. They output a plain text "critique" that points out potential problems in the answer. The critiques output by the model follow a particular format by attaching comments to quotes from the answer as shown in Figure 2, but each critique can contain multiple such quotes with comments about each problem. We first describe how these critic models are evaluated (2.1) and then how they are trained (2.2).
我們的 LLM 評論模型是自回歸 Transformer 策略，類似於 InstructGPT 和 ChatGPT [19]。它們經過訓練或提示，可以接受一對（問題、答案）作為輸入。它們輸出一個純文字「評論」，指出答案中潛在的問題。模型輸出的評論遵循特定的格式，將意見附加到答案的引文中，如圖 2 所示，但每個評論可以包含多個這樣的引言和關於每個問題的意見。我們首先描述如何評估這些評論模型 (2.1)，然後描述如何訓練它們 (2.2)。

2.1 Evaluation

2.1.1 Critique Attributes
二之一之一評鑑屬性

Consider two possible critiques of the same buggy program. Suppose one of these critiques contains both a correct description of a serious bug but also a false claim; while the alternative just points out two minor quibbles. Which critique is better? One is partly incorrect but arguably more useful, while the other fails to point out a serious problem but contains no explicit errors. To disentangle this we ask contractors about the following features of a critique:
考慮對同一個錯誤程式碼的兩種可能評論。假設其中一個評論既包含對嚴重錯誤的正確描述，但也包含錯誤的斷言；而另一個評論只指出兩個小問題。哪個評論更好？一個評論部分不正確，但可以說更有用，而另一個評論沒有指出嚴重的問題，但也不包含明確的錯誤。為了釐清這一點，我們詢問審閱者關於評論的以下特徵：

Whether it was comprehensive, i.e. did not omit any clear and severe issues (comprehensiveness).
它是否全面，即沒有遺漏任何明顯和嚴重的問題（全面性）。
Whether it caught a particular bug specified a-priori, which we call "critique-bug inclusion" (CBI)
它是否發現了一個先驗指定的特定錯誤，我們稱之為「評論錯誤包含」(CBI)
Whether it included any hallucinated bugs or any nitpicks.
是否包含任何憑空捏造的錯誤或任何吹毛求疵之處。
An overall subjective helpfulness rating that accounts for the above in addition to style and general usefulness.
一個整體主觀的幫助評分，除了風格和一般用途之外，還考慮了上述因素。

Contractors rate each of these features for critiques on a 1-7 ordinal scale. Details of the forms used are included in Appendix 7.4
承包商使用 1-7 的順序量表對這些特徵進行評分。表格使用的細節包含在附錄 7.4 中。

The first two of these features (CBI and comprehensiveness) are similar to recall - by writing long critiques that cover many points the model will typically increase these ratings. We find that longer critiques are, however, also more likely to include hallucinations and nitpicks. Discussions of that tradeoff are included in section 3.4. Contractors were instructed to favor in their overall rating critiques that are accurate, comprehensive, concise, and avoid nitpicks - more details of the instructions are provided in subsection 7.4.
前兩個特徵（CBI 和全面性）類似於召回率——通過撰寫涵蓋許多要點的長篇評論，模型通常會提高這些評分。然而，我們發現較長的評論也更有可能包含憑空捏造的錯誤和吹毛求疵之處。關於這種權衡的討論包含在 3.4 節中。承包商被指示在他們的整體評分中，優先考慮準確、全面、簡潔且避免吹毛求疵的評論——更多關於指示的細節在 7.4 小節中提供。

Figure 3: Illustration of data collection. Contractors modify ChatGPT responses to insert subtle bugs. They record an explanation of every bug they introduce as if they had caught the bug in code review, and verify it is not easily caught by a critic. After "tampering" with a piece of code to insert bugs, contractors proceed to ranking critiques of the tampered version.
圖 3：數據收集說明。承包商修改 ChatGPT 的回應以插入細微的錯誤。他們會記錄下他們引入的每個錯誤的解釋，就好像他們在代碼審查中發現了這個錯誤一樣，並驗證它不容易被評論者發現。在「篡改」一段代碼以插入錯誤之後，承包商會繼續對篡改版本的評論進行排名。

2.1.2 Critique Comparisons & Elo Scores
二之一之二、評析比較與埃洛評分

Contractors answer the critique attribute questions as part of a comparison task in which they see four critiques of the same problem simultaneously. They are always blind to the source of the critique. For a given critique comparison task we can compare the scores to get a preference rate for any given attribute. For example, if critique A gets a score of

for comprehensiveness and critique

gets a score of

, then

is more comprehensive than

. Because human ratings are more consistent within a comparison than globally this gives us a less noisy estimate of how the models perform relative to each other [29].
承包商會在一項比較任務中回答關於評論屬性的問題，在這個任務中，他們會同時看到針對同一個問題的四個評論。他們始終不知道評論的來源。對於一個給定的評論比較任務，我們可以比較分數以獲得任何給定屬性的偏好率。例如，如果評論 A 的「易懂性」得分為

，而評論

的得分為

，則表示

比

更容易理解。由於人類評分在比較中比在全局上更一致，因此這讓我們能夠更準確地估計模型之間的相對表現 [29]。

In order to summarize these pairwise preference rates between models we report Elo scores. Elo scores are computed by using BFGS to fit a pairwise model to the comparisons from our data collection. The probability of a contractor preferring a response produced by a model with Elo

over a response from model with Elo

is estimated as

. This is the estimated win-rate of model

over model

. Ties are included as half a win and half a loss. Confidence intervals are reported from a nonparametric bootstrap.
為了總結模型之間這些成對的偏好率，我們報告了 Elo 分數。Elo 分數是透過使用 BFGS 將成對模型擬合到我們數據收集中的比較來計算的。承包商偏好 Elo 為

的模型產生的響應，而不是 Elo 為

的模型產生的響應的概率估計為

。這是模型

對模型

的估計勝率。平局被視為一半贏、一半輸。置信區間由非參數自助法報告。

2.1.3 Human Critiques & LLM Assistance
二之一三、人為評析與LLM輔助

In addition to paying contractors to rate model-written critiques, we also had them review answers to write their own critiques. Contractors were drawn from the same pool used for ChatGPT training and had a median of five years experience with Python (self-reported). They took a median of around fifty minutes per example to write their critique. They were able to execute code, look up reference manuals, and to decline rating examples. Overall however only

of tasks were declined, typically because they were not in English or because they were broken.
除了付費給承包商來評估模型撰寫的評論之外，我們還讓他們審查答案以撰寫他們自己的評論。承包商來自與 ChatGPT 訓練相同的群體，並且擁有五年 Python 經驗的中位數（自我報告）。他們每個例子平均花費大約五十分鐘來撰寫評論。他們能夠執行代碼、查閱參考手冊以及拒絕評估範例。然而，總體而言，只有

的任務被拒絕，通常是因為它們不是英文或因為它們已損壞。

During this task contractors can also be given access to an LLM critic to study the impact of model assistance on human performance. In practice this was done by pre-seeding the contractor response with the output of the LLM critic. Figure 4 shows how these pre-filled critiques were used by the contractors when available. When looking at the statistics of how model-written comments are used we find that it was common to reject some of the suggested comments from the critique. Adding additional comments was rarer but also occurred.
在此任務中，承包商還可以訪問 LLM 評論者，以研究模型輔助對人類表現的影響。在實務中，這是通過使用 LLM 評論者的輸出預先填寫承包商的回應來完成的。圖 4 顯示了承包商如何在可用時使用這些預先填寫的評論。在查看模型撰寫評論的使用統計數據時，我們發現拒絕評論中的一些建議評論是很常見的。添加額外評論的情況較少見，但也確實發生過。

Figure 4: How do contractors interact with pre-filled critiques? We measure what contractors did in humanmachine critique teams. Keeping LLM comments without modification and removing them were both common. Adding new comments and editing phrasing was less common.
圖 4：承包商如何與預先填寫的評論互動？我們衡量了承包商在人機評論團隊中的表現。保留 LLM 評論而不做修改和刪除它們都很常見。添加新評論和編輯措辭則不太常見。

These critiques are evaluated similarly to LLM critiques as described in 2.1.1. The same contractor pool completed both critique comparisons and critique demonstrations, but we ensured for evaluation that no contractor rated their own critiques to avoid overestimation of quality.
這些評論的評估方式與 2.1.1 節中描述的 LLM 評論類似。相同的承包商群體完成了評論比較和評論演示，但我們確保在評估中，沒有任何承包商對他們自己的評論進行評分，以避免高估質量。

2.1.4 Evaluation Data Distribution

Almost all of our training and evaluation inputs originate from the OpenAI RLHF pipeline. They consist of (question, answer) pairs. We down-sampled to cases where the model responded with code using a heuristic: examples were included if the model response was at least

Python by line count. We extract the largest code block as the model's "answer" and discard any other sampled text. We report results on two sets containing known bugs based on this data stream:
我們幾乎所有的訓練和評估輸入都來自 OpenAI RLHF 流程。它們由（問題、答案）對組成。我們使用啟發式方法對模型以代碼回應的情況進行了下採樣：如果模型回應至少有

行 Python 代碼，則包含這些範例。我們提取最大的代碼塊作為模型的「答案」，並丟棄任何其他採樣的文本。我們根據此數據流報告了兩個包含已知錯誤的集合的結果：

Human Inserted Bugs: We pay contractors to insert bugs into answers in a "tampering" process (2.2) and hold out a subset for evaluation.
人工插入錯誤：我們付費給承包商，讓他們在「篡改」過程中將錯誤插入答案中 (2.2)，並保留一部分答案用於評估。
Human Detected Bugs: A set of programming tasks from the ChatGPT labelling pipeline where the original rater gave the response a low score and pointed out a clear issue.
人工檢測到的錯誤：一組來自 ChatGPT 標記流程的程式設計任務，其中原始評分者給予答案低分，並指出了一個明顯的問題。

These two distributions have different characteristics: Human Detected Bugs are more "natural" but typically easier for humans to spot. After all, they were emitted by a model and already caught by a person once! Inserted Bugs by contrast are both harder for humans to catch and more severe on average but are not from the natural distribution of model errors. These differences are illustrated in Appendix Figure 10. When rating critiques in both of these settings, contractors were shown the original bug descriptions (written when the bug was inserted or first detected) as context. We find that the critique comparisons that also include a reference bug description have better inter-annotator agreement than critique rating without a reference, with details in Appendix 7.5.
這兩種分佈具有不同的特徵：人工檢測到的錯誤更「自然」，但通常更容易被人類發現。畢竟，它們是由模型產生的，並且已經被人發現過一次！相比之下，人工插入的錯誤對人類來說更難以察覺，而且平均而言更為嚴重，但它們並非來自模型錯誤的自然分佈。這些差異如附錄圖 10 所示。在對這兩種情況下的評論進行評分時，承包商會看到原始錯誤描述（在插入錯誤或首次檢測到錯誤時寫入）作為上下文。我們發現，包含參考錯誤描述的評論比較比沒有參考的評論評分具有更好的一致性，詳情請見附錄 7.5。

2.2 Training Critics with RLHF

We follow an RLHF pipeline similar to ChatGPT to train our LLM critics:

Step 1: We sample several critiques for each (question, answer) pair in the dataset.
步驟 1：我們為數據集中每個（問題，答案）對採樣多個評論。

Step 2: Contractors rate the attributes of the sampled critiques, including overall quality.
步驟 2：承包商對採樣評論的屬性進行評分，包括整體品質。

Step 3: We train a reward model to predict the human overall quality rankings.
步驟 3：我們訓練一個獎勵模型來預測人類的整體品質排名。

Step 4: We optimize a policy against the reward model using PPO [31, 27].
步驟 4：我們使用 PPO [31, 27] 針對獎勵模型優化策略。

Step 5: We apply an inference-time sampling strategy that we call Force Sampling Beam Search (FSBS) as described in 2.3.
步驟 5：我們應用一種稱為強制採樣束搜索 (FSBS) 的推理時採樣策略，如 2.3 節所述。

However, for a large fraction of our data, the answer comes from a key first step:
然而，對於我們很大一部分數據而言，答案來自於關鍵的第一步：

2.2.1 Step 0: Tampering

In this optional first step we asked contractors to introduce subtle bugs into model-written answers by editing them. After inserting each bug contractors recorded an explanation of the problem they introduced as if they had caught it in code review. This is illustrated visually in Figure 3.
在這個可選的第一步中，我們要求承包商通過編輯模型編寫的答案，在其中引入細微的錯誤。在插入每個錯誤後，承包商會記錄下他們引入的問題的解釋，就好像他們在程式碼審查中發現了這個問題一樣。圖 3 直觀地展示了這一點。

Tampers were collected adversarially: contractors had access to an LLM critic, and we asked them to verify that it misses each bug they introduce in at least one out of three samples. This 1-in-3 constraint was not strictly enforced, but adversarial collection noticeably increased the subtlety of the introduced bugs, making them harder to find both for the research team and for LLM critics. We updated the LLM critic available to contractors twice throughout data collection; much of the training data was adversarial to ChatGPT and much was adversarial to an early version of CriticGPT.
我們以對抗的方式收集了篡改後的程式碼：承包商可以存取一個 LLM 評測模型，我們要求他們驗證該模型是否在至少三分之一的樣本中遺漏了他們引入的每個錯誤。這個三分之一的限制並沒有被嚴格執行，但對抗性收集顯著地增加了引入錯誤的隱蔽性，使得研究團隊和 LLM 評測模型都更難以發現這些錯誤。在整個數據收集過程中，我們兩次更新了承包商可用的 LLM 評測模型；大部分訓練數據都是針對 ChatGPT 進行對抗性訓練的，而很大一部分數據則是針對 CriticGPT 早期版本進行對抗性訓練的。

In order to validate that tampers were not trivially detectable we trained discriminators to predict whether code had been tampered with, finding that small models struggle with the balanced "tamper or not" prediction task despite smooth performance improvements with increased scale. Further details of these discriminators are included in Appendix 7.7.
為了驗證篡改並非輕易可檢測，我們訓練了鑑別器來預測程式碼是否被篡改，結果發現，儘管隨著規模的增加，模型性能平穩提升，但小型模型在處理平衡的「篡改與否」預測任務時仍顯吃力。有關這些鑑別器的更多細節，請參見附錄 7.7。

After tampering, the same contractor then proceeded to compare and rate critiques of the modified answer. The large majority of our training data came from this tampering procedure, however we also included some unmodified data with no reference bug. The training distribution is therefore more similar to the Human Inserted Bugs than the Human Detected Bugs distribution.
在篡改之後，同一位承包商接著會比較和評估修改後答案的評測結果。我們的大部分訓練數據都來自於這個篡改過程，但我們也包含了一些沒有參考錯誤的未修改數據。因此，訓練數據的分佈更接近於「人工插入錯誤」的分佈，而不是「人工檢測錯誤」的分佈。

2.2.2 Why Tampering?

Our goal is to find a method that will apply to long-form and open-ended tasks for which we do not have a ground-truth reward function. One could simply train critique models on unmodified answers but that approach has at least the following issues:
我們的目標是找到一種適用於長篇和開放式任務的方法，而對於這些任務，我們沒有真實的獎勵函數。人們可以簡單地在未修改的答案上訓練評測模型，但這種方法至少存在以下問題：

Preference rates are impacted by a number of stylistic factors and may over-estimate model performance.
偏好率會受到許多風格因素的影響，並且可能會高估模型的性能。
Contractors may struggle to validate the correctness of free-form critiques if they make obscure claims.
若承包商所言含糊不清，則其或難以證其天馬行空之批評確鑿無誤。
Contractors may struggle to spot important issues that critiques miss.
承包商或難洞悉評議者所忽略之關鍵問題。
Many answers will not contain severe issues, reducing the value of the data for improving critiques.
許多答案不會包含嚴重的問題，這降低了數據對於改進評測的價值。

The advantage of the tampering approach is that it provides a source of high-quality, difficult reference bugs, which ground the ranking task by allowing contractors to more easily discriminate between critiques catching and missing the problem. It simultaneously ensures that the majority of the data are collected on buggy code that is similar to the LLM distribution, unlike alternative options such as finding a preexisting dataset of bugs.
篡改方法的優點在於，它提供了一個高質量、難度高的參考錯誤來源，通過允許承包商更容易地區分捕捉到問題和遺漏問題的評測，從而為排名任務奠定了基礎。同時，它確保了大部分數據是在類似於 LLM 分佈的錯誤程式碼上收集的，這與尋找預先存在的錯誤數據集等替代方案不同。

2.2.3 Critique Comparisons in Detail

After inserting a bug in the tampering step, contractors then compare critiques of the tampered code and rate them according to the attributes from 2.1.1. In particular they rank three LLM critiques and a "gold critique" which is formed from the bug descriptions they provided during the tampering task. Ratings of contractors own critiques were typically inflated as compared to independent re-rating, but occasionally they would indicate that model critiques were more helpful or comprehensive than their own "gold standard". Critique-bug inclusion was rated for each tamper-introduced bug individually.
在篡改步驟中插入錯誤後，承包商會比較被篡改程式碼的評測結果，並根據 2.1.1 中的屬性對其進行評分。具體來說，他們會對三個 LLM 評測結果和一個「黃金評測」進行排名，「黃金評測」是由承包商在篡改任務期間提供的錯誤描述組成的。與獨立的重新評分相比，承包商自己評測結果的評分通常會被誇大，但偶爾他們也會指出模型評測結果比他們自己的「黃金標準」更有幫助或更全面。對於每個篡改引入的錯誤，都會單獨評估評測結果是否包含該錯誤。

Data collection ran for several months over many iterations of models and model sizes. Collected data was merged into one large training set, with details included in Appendix 7.2. In addition to collecting critique comparisons on code with human-inserted bugs we also collected training data from critiques of unmodified samples. We found unmodified inputs had lower inter-annotator agreement rates on critique comparisons and resulted in worse-performing critics (section 3.5).
數據收集工作持續了幾個月，經歷了多個模型版本和模型大小的迭代。收集到的數據被合併成一個大型訓練集，詳細信息請參見附錄 7.2。除了收集關於人工插入錯誤的程式碼評測比較數據外，我們還收集了來自未修改樣本評測的訓練數據。我們發現，未修改輸入在評測比較方面的標註者間一致性較低，導致評測模型的性能較差（第 3.5 節）。

2.2.4 RLHF

Our LLM critics are GPT-4 family Transformer language models pre-trained with next-token prediction following [30]. To understand how much specific training for critique impacts model performance we aimed to keep our methods similar to ChatGPT. To highlight similarities and differences:
我們的 LLM 評測模型是 GPT-4 系列 Transformer 語言模型，採用 [30] 中的下一詞預測方法進行預先訓練。為了瞭解針對評測的特定訓練對模型性能的影響程度，我們的目標是使我們的方法與 ChatGPT 類似。以下是相似點和不同點：

All versions of CriticGPT and ChatGPT used in this work were initialized from the same checkpoint (both policies and reward models).
在這項工作中使用的所有版本的 CriticGPT 和 ChatGPT 都從同一個檢查點初始化（包括策略和獎勵模型）。
Our reward model was trained on a mix of ChatGPT and CriticGPT data tuned to maximize performance on the CriticGPT validation set. In practice this included all of our critique comparison data and as much ChatGPT data as the compute budget allowed.
我們的獎勵模型在 ChatGPT 和 CriticGPT 數據的混合數據集上進行訓練，並調整以最大化 CriticGPT 驗證集上的性能。實際上，這包括我們所有的評論比較數據，以及在計算預算允許的情況下盡可能多的 ChatGPT 數據。
CriticGPT was fine-tuned with less compute than ChatGPT.
CriticGPT 的微調使用的計算量少於 ChatGPT。
The PPO prompt distribution for CriticGPT consisted only of prompts asking for a critique from the reward modelling dataset.
CriticGPT 的 PPO 提示分佈僅包含要求從獎勵建模數據集中進行評論的提示。

2.3 Force Sampling Beam Search (FSBS)

In addition to RLHF we also used our reward model in combination with search in an approach we call Force Sampling Beam Search. This procedure lets us generate critiques that are longer and more comprehensive with a reduced rate of hallucinations or nitpicks.
除了 RLHF 之外，我們還將獎勵模型與搜索結合使用，我們稱之為強制採樣束搜索。此過程使我們能夠生成更長、更全面的評論，並降低幻覺或吹毛求疵的發生率。

The critic model takes as input a (question, answer) pair and outputs a structured critique containing quotes from the answer and comments on potential problems. In the critique, quoted sections of the answer are quoted as "highlights" via markdown code blocks beginning with ". . " that are then followed by comments indicating what errors occur in that highlight. In FSBS we search over critiques by forcing the model to produce highlighted sections with constrained sampling and then select the best-scoring critiques according to the expression rm_score + LENGTH_MODIFIER * num_highlights. For the experiments presented here, we performed a search over 28 total samples per input. We explored 4 values of LENGTH_MODIFIER that map to the the 10 th, 25 th, 50 th and 75 th percentile of critique lengths sampled during the search. Appendix 7.1 provides more details of constrained sampling and search.
評論模型將（問題，答案）對作為輸入，並輸出包含答案引文和對潛在問題的評論的結構化評論。在評論中，答案的引用部分通過 Markdown 代碼塊作為「重點」引用，以「. .」開頭，然後是指示該重點中出現哪些錯誤的評論。在 FSBS 中，我們通過強制模型使用約束採樣生成突出顯示的部分來搜索評論，然後根據表達式 rm_score + LENGTH_MODIFIER * num_highlights 選擇得分最高的評論。對於這裡介紹的實驗，我們對每個輸入執行了 28 個總樣本的搜索。我們探索了 LENGTH_MODIFIER 的 4 個值，它們映射到搜索期間採樣的評論長度的第 10、25、50 和 75 個百分位數。附錄 7.1 提供了有關約束採樣和搜索的更多詳細信息。

Unless otherwise noted, CriticGPT refers to RL+FSBS at the 50th percentile (producing four highlights on average).
除非另有說明，否則 CriticGPT 指的是處於第 50 個百分位數的 RL+FSBS（平均產生四個重點）。

3 Results

3.1 LLM critiques are often preferred over human critiques and catch more inserted bugs
3.1 LLM 的評論通常比人類評論更受青睞，並且可以發現更多插入的錯誤

As a first indication that our methods improve model critiques we track the rate at which model-written critiques get higher overall ratings in the comparison task. Figure 1a shows that critiques written by CriticGPT are substantially preferred by contractors over critiques from prompted ChatGPT and over human-written critiques sourced from our group of contractors according to the overall rating This cohort and corresponding set of human critiques does not represent the best possible human performance; however it is representative of the set of contractors used for ChatGPT. The same set of human critiques allows us to compare the rate at which a second person will catch an inserted bug; in Figure

we see that our Tamper+RLHF pipeline greatly improves the rate at which inserted bugs are caught, with both LLM critics (prompted ChatGPT and CriticGPT) catching many more bugs than the human annotators.
作為我們的方法改進模型評論的第一個指標，我們跟踪模型編寫的評論在比較任務中獲得更高總體評分的比率。圖 1a 顯示，根據總體評分，承包商更喜歡 CriticGPT 編寫的評論，而不是來自提示的 ChatGPT 的評論，以及來自我們承包商組的人工編寫的評論。這個群組和相應的人工評論集並不代表最佳的人工表現；但是，它代表了用於 ChatGPT 的承包商集。同一組人工評論使我們能夠比較第二個人發現插入錯誤的比率；在圖

中，我們看到我們的 Tamper+RLHF 流程大大提高了發現插入錯誤的比率，LLM 評論家（提示的 ChatGPT 和 CriticGPT）都比人工標註者發現了更多錯誤。

We looked at the human critiques in some detail to understand the source of the performance difference. In general, contractors (despite the median handling time of 50 minutes) made fewer overall comments when they did not have LLM help. Many of the tasks also require domain knowledge and while the contractors did have nontrivial Python experience it was clear they sometimes did not have the domain knowledge needed, e.g. of particular Python libraries. Some fraction of the tampers are also just outright challenging to spot. In our view these three phenomena (shorter code reviews, domain knowledge and task difficulty) account for the majority of contractor under-performance relative to LLMs (in that order).
我們詳細查看了人工評論，以了解性能差異的根源。一般來說，承包商（儘管平均處理時間為 50 分鐘）在沒有 LLM 幫助的情況下提出的總體評論較少。許多任務還需要領域知識，雖然承包商確實具有一定的 Python 經驗，但很明顯，他們有時缺乏所需的領域知識，例如特定的 Python 庫。某些部分的篡改也很難發現。我們認為，這三種現象（較短的代碼審查、領域知識和任務難度）是承包商相對於 LLMs 表現不佳的主要原因（按順序排列）。

3.2 CriticGPT RL training improves models across pre-training scale

Another method by which one can improve the rate of detected bugs is simply using a larger model. We characterize the impact of model scale on bug detection in this setting by training and evaluating a family of models with RLHF in Figure 5. Each model is a Transformer in the GPT-4 family and we record their size as the fraction of GPT-4 compute used in pre-training. This also allows us to naively estimate the amount of additional pre-training compute that would be needed to achieve the same performance as CriticGPT (RL only) without additional data or modification to the ChatGPT pipeline, which is around

.
另一種可以提高檢測錯誤率的方法是簡單地使用更大的模型。我們在圖 5 中通過訓練和評估一系列使用 RLHF 的模型來描述模型規模對錯誤檢測的影響。每個模型都是 GPT-4 系列中的 Transformer，我們將它們的大小記錄為預訓練中使用的 GPT-4 計算量的分數。這也讓我們可以粗略估計在不增加數據或修改 ChatGPT 流程的情況下，要達到與 CriticGPT（僅限 RL）相同的效能所需的額外預訓練計算量，大約是

。

(a) When contractors compare critiques of the same code, CriticGPT (RL only) is substantially preferred over prompted ChatGPT across model scales on Human Inserted Bugs
(a) 當承包商比較對相同程式碼的評論時，在 Human Inserted Bugs 上，CriticGPT（僅限 RL）在模型規模方面明顯優於提示 ChatGPT

(b) Contractors indicate that CriticGPT (RL only) misses inserted bugs substantially less often than ChatGPT. Training a larger model also reduces the rate at which inserted bugs are missed, but naive extrapolation implies that to match the performance of CriticGPT (RL only) on this distribution would require around a 30x increase in pre-training compute.
(b) 承包商表示，CriticGPT（僅限 RL）遺漏插入錯誤的頻率遠低於 ChatGPT。訓練更大的模型也可以降低遺漏插入錯誤的比率，但簡單的外推意味著要達到 CriticGPT（僅限 RL）在此分佈上的效能，需要將預訓練計算量增加約 30 倍。

Figure 5

3.3 CriticGPT helps humans write more comprehensive critiques

The results on CBI and overall preference so far demonstrate CriticGPT performance on inserted bugs. Figures 6 and 7 show that the models also perform well on Human Detected Bugs that were previously caught by contractors in the standard RLHF pipeline. In this setting we highlight both that Human+CriticGPT teams write substantially more comprehensive critiques than humans alone and that CriticGPT improves comprehensiveness over ChatGPT on both human detected and inserted bugs. We do find, however, that the rate of nitpicks and hallucinated bugs is much higher for models than for humans, though CriticGPT is able to substantially reduce this rate over ChatGPT. Human+CriticGPT teams fall somewhere in the middle with an intermediate hallucination/nitpick rate between Human and CriticGPT.
目前為止，CBI 和整體偏好的結果證明了 CriticGPT 在插入錯誤方面的效能。圖 6 和圖 7 顯示，這些模型在標準 RLHF 流程中先前由承包商發現的 Human Detected Bugs 上也表現良好。在這種情況下，我們要強調的是，Human+CriticGPT 團隊編寫的評論比單獨的人工編寫的評論要全面得多，而且 CriticGPT 在人工檢測到的錯誤和插入的錯誤方面都比 ChatGPT 更全面。然而，我們確實發現，模型的吹毛求疵和幻覺錯誤的比率遠高於人類，儘管 CriticGPT 能夠大幅降低 ChatGPT 的這一比率。Human+CriticGPT 團隊介於兩者之間，其幻覺/吹毛求疵率介於人類和 CriticGPT 之間。

3.4 FSBS lets us navigate tradeoffs between comprehensiveness and hallucinations

Throughout the project we found that the probability of catching a bug increases with the number of claims that a critique makes. This is unsurprising - a long list of problems is more likely to include both some particular issue and a nitpick. As described in 2.1.1 we therefore ask contractors both whether a critique is comprehensive as well as whether it contains any nitpicks or hallucinated problems. Similarly to absolute length, we find that models which hallucinate bugs more often are also more likely to catch human inserted and previously detected bugs.
在整個專案中，我們發現發現錯誤的機率隨著評論提出的主張數量而增加。這一點也不奇怪——一長串的問題更有可能同時包含一些特定的問題和吹毛求疵。如 2.1.1 中所述，我們因此詢問承包商，評論是否全面，以及是否包含任何吹毛求疵或幻覺問題。與絕對長度類似，我們發現，更容易產生幻覺錯誤的模型也更容易發現人工插入的錯誤和先前檢測到的錯誤。

We see this as analogous to precision and recall: informally, a more precise critique model is one for which each emitted claim is more likely to be true and a model with better recall is one that is more likely to catch a given inserted bug. Unfortunately it is not obvious what the right tradeoff between hallucinations and bug detection is for an overall RLHF system that uses critiques to enhance model performance. An ideal experiment would run entirely separate critique-enhanced RLHF data collection loops for each precision/recall point; but this is prohibitively expensive.
我們認為這類似於精確率和召回率：非正式地說，更精確的評論模型是指每個發出的主張更有可能是正確的，而召回率更好的模型是指更容易發現給定插入錯誤的模型。不幸的是，對於使用評論來增強模型效能的整體 RLHF 系統來說，幻覺和錯誤檢測之間的正確權衡是什麼並不清楚。理想的實驗是為每個精確率/召回率點運行完全獨立的評論增強 RLHF 數據收集迴圈；但這太昂貴了。

FSBS (detailed in 2.3) uses our reward models and test-time search as a mitigation for this. By varying the scalar length penalty we find that we can establish a Pareto curve, shown in Figure 8, between
FSBS（詳見 2.3）使用我們的獎勵模型和測試時搜尋來減輕這種情況。通過改變標量長度懲罰，我們可以建立一條帕累托曲線，如圖 8 所示，介於

Figure 6: Humans write substantially more comprehensive critiques with help from LLM critics.. Humanmachine teams do not increase comprehensiveness, but see Section 3.4 for their positive impact on hallucination rate.
圖 6：在 LLM 評論者的幫助下，人類可以寫出更全面的評論。人機團隊並沒有提高評論的全面性，但請參閱 3.4 節，了解它們對幻覺率的積極影響。

Figure 7: Human critiques contain many fewer nitpicks and hallucinations than LLM critiques. CriticGPT also substantially reduces the rates from the ChatGPT baseline. Human-machine teams hallucinate and nitpick less than both CriticGPT and ChatGPT.
圖 7：與 LLM 評論相比，人類評論中的吹毛求疵和幻覺要少得多。CriticGPT 還大幅降低了 ChatGPT 基線的比率。人機團隊的幻覺和吹毛求疵比 CriticGPT 和 ChatGPT 都少。

comprehensiveness and spurious claims. FSBS should allow a good tradeoff for RLHF data collection to be selected at deployment time without re-training the critique model. In Appendix section 7.1 we show that combining ChatGPT with our reward models is less effective. Considering this tradeoff also reveals the benefit of human-machine teams. When working together (as described in 2.1.3) Human+CriticGPT teams move beyond the model-only Pareto frontier. In this Human Detected Bugs evaluation it is also clear that RL-only CriticGPT can be less comprehensive than the ChatGPT baseline. We think this is driven by two effects. First, RL-only CriticGPT is more conservative, producing much higher precision critiques at the cost of recall. Second, there was a reduction in compute between ChatGPT and CriticGPT. FSBS results in more comprehensive critiques.
全面性和虛假主張之間。FSBS 應該允許在部署時為 RLHF 數據收集選擇一個良好的權衡，而無需重新訓練評論模型。在附錄 7.1 節中，我們展示了將 ChatGPT 與我們的獎勵模型相結合的效果較差。考慮到這種權衡，也揭示了人機團隊的好處。當協同工作時（如 2.1.3 中所述），Human+CriticGPT 團隊超越了僅限模型的帕累托邊界。在這個 Human Detected Bugs 評估中，也很明顯，僅限 RL 的 CriticGPT 的全面性可能不如 ChatGPT 基線。我們認為這是由兩個因素造成的。首先，僅限 RL 的 CriticGPT 更為保守，它以犧牲召回率為代價產生了精確率更高的評論。其次，ChatGPT 和 CriticGPT 之間的計算量有所減少。FSBS 產生了更全面的評論。

3.5 Ablations

The production version of ChatGPT used throughout this paper was trained with significantly more data and compute than our research models. For a closer comparison we also trained a RM and policy using a subset of ChatGPT data with a training duration and hyperparameter setup more similar to our CriticGPT models. The checkpoint for the policy model was selected to maximize the CBI
本文使用的 ChatGPT 正式版本，其訓練資料量和計算資源都遠遠超過我們的研究模型。為了進行更貼切的比較，我們還使用了 ChatGPT 資料集的子集，並採用與 CriticGPT 模型相似的訓練時長和超參數設定，訓練了一個獎勵模型和策略模型。策略模型的檢查點經過挑選，以最大化 CBI。

Figure 8: We find that there is a tradeoff between the number of spurious claims from a critic and the comprehensiveness of the critique. Using

we can trade off comprehensiveness and hallucinations; though we do not currently know what balance is optimal for improving the performance of annotators in an RLHF pipeline. Results shown on the Human Detected Bugs distribution.
圖 8：我們發現，評測者的虛假聲明數量和評測的全面性之間存在著一種權衡關係。使用

，我們可以在全面性和幻覺之間取得平衡；儘管我們目前尚不清楚哪種平衡最有利於提高 RLHF 流程中標註者的效能。結果顯示在「人工檢測錯誤」分佈圖上。

for human-inserted bugs on the validation set. This approach provides a cleaner comparison that better isolates the impact of the data collection method from the effects of training duration and pipeline setup. This version of ChatGPT is included in Figure 8 as "ChatGPT (less training)". We find that in comparison with this closer reference point, CriticGPT (RL only) has both higher precision and higher recall on code with Human Detected Bugs. Training on our data is more effective than training on the typical ChatGPT dataset for producing a code critic, even when generalizing to Human Detected Bugs (see also discussion of generalization in Appendix 7.6).
以評估人工插入驗證集程式碼中的錯誤。這種方法提供了一種更清晰的比較，可以更好地將資料收集方法的影響與訓練時長和流程設定的影響區分開來。這個版本的 ChatGPT 在圖 8 中被標記為「ChatGPT（訓練較少）」。我們發現，與這個更接近的參考點相比，CriticGPT（僅限強化學習）在包含「人工檢測錯誤」的程式碼上同時具有更高的精確率和召回率。即使在推廣到「人工檢測錯誤」的情況下，在我們的資料集上進行訓練也比在典型的 ChatGPT 資料集上進行訓練更能有效地生成程式碼評測器（另見附錄 7.6 中關於推廣的討論）。

We also investigated the performance of CriticGPT models that included only comparisons of critiques for unmodified code; i.e. excluding the tampering step from our data pipeline. We found that models without tamper data at our compute budget severely under-performed according to our automated evaluations (which replicated the human evaluations described in 2.1.1 with GPT-4) on both inserted and detected bugs and do not present human evaluations here. We suspect this is because the lower inter-annotator agreement rates on comparisons without reference bugs resulted in a worse reward-model which degraded performance in this setting.
我們還研究了僅包含未修改程式碼的評測結果比較的 CriticGPT 模型的效能；也就是說，從我們的資料流程中排除了程式碼修改步驟。我們發現，在我們的計算資源限制下，沒有程式碼修改資料的模型，根據我們的自動評估（使用 GPT-4 複製了 2.1.1 節中描述的人工評估），在插入錯誤和檢測到的錯誤方面都表現不佳，因此我們在此不呈現人工評估結果。我們懷疑這是因為在沒有參考錯誤的情況下，比較結果的標註者間一致率較低，導致獎勵模型較差，從而降低了這種情況下的效能。

3.6 LLM critics generalize beyond code

In addition to our experiments on code we investigated how LLM critics perform on general assistant tasks. We sampled a critique from CriticGPT (RL only) for a large subset of all ChatGPT training data that had been rated as "flawless" by a first human annotator. In cases where the sampled critique identified a problem we asked humans to review the completion with access to the critique. In

of cases contractors indicated that the critique found a problem that substantially decreased the rating of the answer; in a separate replication without critiques completions rated "flawless" by one contractor were rated similarly poorly by a second only

of the time. We also investigated using our critique reward models to prioritize data and found that prioritizing tasks for which the sampled critique was highly scored increased the chances of catching a problem and decreased the rate of hallucinations.
除了對程式碼進行實驗外，我們還研究了 LLM 評測器在一般助理任務中的表現。我們從 CriticGPT（僅限強化學習）中抽取了一個評測結果，用於所有被第一個人工標註者評為「完美無瑕」的 ChatGPT 訓練資料的大型子集中。在抽取的評測結果發現問題的情況下，我們要求人工在參考評測結果的情況下審查完成情況。在

的情況下，承包商表示評測結果發現了一個問題，該問題大大降低了答案的評分；在沒有評測結果的單獨重複測試中，被一個承包商評為「完美無瑕」的完成結果，只有在

的情況下才會被第二個承包商評為同樣糟糕。我們還研究了使用我們的評測結果獎勵模型對資料進行優先排序，發現優先考慮抽取評測結果得分較高的任務，可以增加發現問題的機會，並降低產生幻覺的比率。

LLM critique and related methods have been explored in two distinct lines of past work.
LLM 評測和相關方法已在過去的兩項不同研究方向中進行了探討。

Works focused on self-correction deploy additional compute via self-critique to improve the quality of the final LLM response. Improvements from self-correction and closely related methods have been claimed for harmlessness [1], factuality [24], computer control [14] and moral reasoning [5], as well
專注於自我修正的研究工作透過自我評測部署額外的計算資源，以提高最終 LLM 回應的品質。自我修正和密切相關方法的改進已被證明對無害性 [1]、真實性 [24]、電腦控制 [14] 和道德推理 [5] 都有幫助，
as in other domains

. In general self-correction methods have succeeded more clearly when they make use of additional side-channel information during the correction phase, as opposed to in the "intrinsic self-correction" setting without additional information during critique [10].
以及在其他領域

中也是如此。一般來說，自我修正方法在修正階段利用額外的輔助資訊時，比在沒有額外資訊的「內在自我修正」設定下更為成功 [10]。

In contrast to work on self-correction, scalable oversight seeks not to improve the capability of the base model but instead to increase the ability of a human judge to correctly assess model answers [2]. Several oversight methods were proposed in theory before they were practically viable, including Debate, Recursive Reward Modeling, and Market Making [12, 15, 11]. Progress since those proposals has included empirical demonstrations that human-machine teams can improve accuracy on MMLU and QuALITY over both human-only and machine-only baselines [2]. Debate in particular has been shown both to be a viable algorithm for multi-agent RL [23], and debating with more persuasive LLMs has been shown to correlate positively with judge accuracy on QuALITY [13].
與自我修正的研究工作不同，可擴展的監督並非旨在提高基礎模型的能力，而是旨在提高人類判斷者正確評估模型答案的能力 [2]。一些監督方法在實際可行之前就已在理論上被提出，包括辯論、遞迴獎勵模型和市場機制 [12, 15, 11]。自這些提議提出以來，進展包括經實驗證明，人機團隊在 MMLU 和 QuALITY 上的準確率可以超過純人工和純機器的基準 [2]。特別是辯論，已被證明是一種可行的多代理人強化學習演算法 [23]，而且與更有說服力的 LLMs 進行辯論已被證明與判斷者在 QuALITY 上的準確率呈正相關 [13]。

Past work has also investigated using deep learning for code review in order to improve the quality of human code

. Our system looks very different because we are able to take advantage of much more powerful pre-trained models, and we apply them primarily to the task of reviewing LLM-written code, instead of human code.
過去的研究工作還探討了使用深度學習進行程式碼審查，以提高人工程式碼的品質

。我們的系統看起來非常不同，因為我們能夠利用更強大的預先訓練模型，而且我們主要將它們應用於審查 LLM 編寫的程式碼，而不是人工程式碼。

The closest predecessor to CriticGPT is Saunders et al. [26] which also directly trains models to produce critiques. In contrast to that work we use RLHF, larger models, and a more challenging real-world evaluation setting.
CriticGPT 最接近的前身是 Saunders 等人 [26] 的研究，該研究也直接訓練模型生成評論。與那項工作相比，我們使用了 RLHF、更大的模型和更具挑戰性的真實世界評估環境。

5 Discussion & Limitations

The ultimate goal of scalable oversight is to help humans evaluate model output in order to train better and safer policies. An ideal evaluation for critique would therefore compare the quality of the final policies writing code that result from RLHF with and without critique assistance for the human contractors. Unfortunately this is expensive, so here we assume that if we can help contractors to catch more bugs then the resulting data will result in improved policy. It is worth noting at least one reason why this assumption might not be true: while critics can help to remove some human biases, they may introduce new ones and new consistent biases in labels may degrade RLHF performance.
可擴展監督的最終目標是幫助人類評估模型輸出，以便訓練出更好、更安全的策略。因此，對評論的理想評估將比較在有人工承包商協助和沒有協助的情況下，通過 RLHF 編寫程式碼產生的最終策略的品質。不幸的是，這樣做的成本很高，因此我們在這裡假設，如果我們可以幫助承包商發現更多錯誤，那麼由此產生的數據將會改善策略。值得注意的是，至少有一個理由可以解釋為什麼這種假設可能不成立：雖然評論家可以幫助消除某些人為偏見，但它們可能會引入新的偏見，而標籤中新的、一致的偏見可能會降低 RLHF 的效能。

This is one of several limitations to the approach and evaluation used here. Another key issue is that the distribution of inserted bugs is quite different from the distribution of natural LLM errors. Training models to insert subtle in-distribution problems (as opposed to paying humans to insert bugs) may be able to mitigate this concern, but we leave such directions to future work. The critique approach is also only the first step of recursive reward modeling (RRM), and we do not know the point at which an additional RRM step is appropriate or whether critique can be used for RRM effectively. There are a number of other limitations:
這是此處使用的方法和評估的幾個限制之一。另一個關鍵問題是插入錯誤的分佈與自然 LLM 錯誤的分佈截然不同。訓練模型插入細微的、符合分佈的問題（而不是付錢給人類插入錯誤）也許能夠減輕這種擔憂，但我們將這些方向留給未來的研究。評論方法也只是遞迴獎勵建模 (RRM) 的第一步，我們不知道在何時適合採取額外的 RRM 步驟，也不知道評論是否可以有效地用於 RRM。還有許多其他限制：

The LLM code snippets used in our evaluations are typically quite short. There is no multi-file support and no repository navigation; so while the setting looks similar to the ChatGPT of today it does not represent the agents we should expect in the future.
我們評估中使用的 LLM 程式碼片段通常非常短。沒有多檔案支援，也沒有儲存庫導航；因此，雖然該環境看起來與今天的 ChatGPT 類似，但它並不代表我們未來應該期待的代理。
Although our method reduces the rate of nitpicks and hallucinated bugs, their absolute rate is still quite high.
雖然我們的方法降低了吹毛求疵和幻覺錯誤的發生率，但它們的絕對發生率仍然很高。
Real world complex bugs can be distributed across many lines of a program and may not be simple to localize or explain; we have not investigated this case.
現實世界中複雜的錯誤可能分佈在程式的多行中，而且可能不容易定位或解釋；我們還沒有調查過這種情況。
A single step of critique may be substantially weaker than multi-step interactive procedures that can explain problems to the user, such as consultancy or debate [15, 12].
單一步骤的評論可能比可以向使用者解釋問題的多步驟互動程序（例如諮詢或辯論 [15, 12]）要弱得多。

Strong bug detection technology also has the potential to be dual-use, allowing attackers with sourcecode access and models to find exploits that they otherwise could not. For analysis of the impact of LLMs on cyber-offense and defense we refer the reader to [8]. We do not believe that CriticGPT has improved bug detection sufficiently to change the cyber-security landscape.
強大的錯誤偵測技術也可能被雙重利用，允許擁有原始碼存取權限和模型的攻擊者找到他們原本無法找到的漏洞。有關 LLMs 對網路攻擊和防禦的影響分析，請讀者參閱 [8]。我們認為 CriticGPT 還沒有充分改進錯誤偵測，不足以改變網路安全格局。

6 Conclusion

Large language models have already passed the point at which typical humans can consistently evaluate their output without help. This has been evident since demonstrations of their strong performance on PhD-level science questions, among other impressive feats [25]. The need for scalable oversight, broadly construed as methods that can help humans to correctly evaluate model output, is stronger than ever. Whether or not RLHF maintains its dominant status as the primary means by which LLMs are post-trained into useful assistants, we will still need to answer the question of whether particular model outputs are trustworthy. Here we take a very direct approach: training models that help humans to evaluate models.
大型語言模型已經發展到沒有幫助的情況下，一般人無法始終如一地評估其輸出的程度。自從它們在博士級別的科學問題上展現出強大的效能以及其他令人印象深刻的壯舉 [25] 以來，這一點就已經很明顯了。人們對可擴展監督的需求比以往任何時候都更加強烈，可擴展監督廣義上是指可以幫助人類正確評估模型輸出的方法。無論 RLHF 是否保持其作為將 LLMs 後訓練成有用助手的首選方法的主導地位，我們仍然需要回答特定模型輸出是否值得信賴的問題。在這裡，我們採取了一種非常直接的方法：訓練模型來幫助人類評估模型。

These LLM critics now succeed in catching bugs in real-world data, and even accessible LLM baselines like ChatGPT have significant potential to assist human annotators. From this point on the intelligence of LLMs and LLM critics will only continue to improve. Human intelligence will not. It is therefore essential to find scalable methods that ensure that we reward the right behaviors in our AI systems even as they become much smarter than us. We find LLM critics to be a promising start.
這些 LLM 評論家現在已經成功地在真實世界數據中捕捉到錯誤，甚至像 ChatGPT 這樣易於使用的基準也有很大的潛力來協助人類標註者。從現在開始，LLMs 和 LLM 評論家的智慧只會不斷提高。而人類的智慧則不會。因此，至關重要的是要找到可擴展的方法，以確保我們在人工智慧系統中獎勵正確的行為，即使它們變得比我們聰明得多。我們發現 LLM 評論家是一個很有希望的開始。

Acknowledgments 誌謝

We are thankful to Jan Leike and Ilya Sutskever for their vision of superalignment. We'd like to thank Collin Burns, Jeffrey Wu, Dan Mossing and John Schulman for detailed feedback on the manuscript. Jiayi Weng, Suchir Balaji and many others helped us with a tremendous post-training stack. Thanks also to Barret Zoph for support at the end of the project and the OpenAI platform team for great GPU infrastructure and to the human data team for much support. Lastly, thanks to the team of annotators who provided training data and evaluated our models throughought the project.
感謝 Jan Leike 和 Ilya Sutskever 對超級校準的願景。感謝 Collin Burns、Jeffrey Wu、Dan Mossing 和 John Schulman 對稿件提出的詳細反饋意見。感謝 Jiayi Weng、Suchir Balaji 和許多其他人在訓練後堆疊方面提供的巨大幫助。還要感謝 Barret Zoph 在項目結束時提供的支持、OpenAI 平台團隊提供的出色 GPU 基礎設施，以及人類數據團隊的大力支持。最後，感謝在整個項目中提供訓練數據並評估我們模型的標註團隊。

References

[1] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback. December 2022, 2212.08073.

[2] Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, and Jared Kaplan. Measuring progress on scalable oversight for large language models. November 2022, 2211.03540 .

[3] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. July 2023, 2307.15217 .

[4] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. April 2023, 2304.05128.

[5] Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac HatfieldDodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R Bowman, and Jared Kaplan. The capacity for moral Self-Correction in large language models. February 2023, 2302.07459.

[6] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. October 2022, 2210.10760.

[7] Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. October 2022, 2210.08726.

[8] Jeff Gennari, Shing-hon Lau, Samuel Perl, Joel Parish, and Girish Sastry. Considerations for evaluating large language models for cybersecurity tasks. 2024.
[8] Jeff Gennari、Shing-hon Lau、Samuel Perl、Joel Parish 和 Girish Sastry。評估大型語言模型在網路安全任務中的注意事項。2024 年。

[9] Anshul Gupta and Neel Sundaresan. Intelligent code reviews using deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'18) Deep Learning Day, 2018.
[9] Anshul Gupta 和 Neel Sundaresan。使用深度學習進行智能代碼審查。深度學習日知識發現與數據挖掘國際會議 (KDD'18) 會議記錄，第 24 屆，2018 年。

[10] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. October 2023, 2310.01798 .

[11] Evan Hubinger. AI safety via market making. https://www.alignmentforum.org/posts/ YWwzccGbcHMJMpT45/ai-safety-via-market-making. Accessed: 2024-04-08.

[12] Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate. May 2018, 1805.00899 .
[12] Geoffrey Irving、Paul Christiano 和 Dario Amodei。透過辯論確保 AI 安全。2018 年 5 月，1805.00899。

[13] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive LLMs leads to more truthful answers. February 2024, 2402.06782.

[14] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. March 2023, 2303.17491.

[15] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. November 2018, 1811.07871.

[16] Jan Leike, John Schulman, and Jeffrey Wu. Our approach to alignment research. https:// openai.com/index/our-approach-to-alignment-research/, 2022. Accessed: 202406-12.
[16] Jan Leike、John Schulman 和 Jeffrey Wu。我們的校準研究方法。https:// openai.com/index/our-approach-to-alignment-research/，2022 年。檢索日期：2024 年 6 月 12 日。

[17] Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, and Chun Zuo. AUGER: automatically generating review comments with pre-training models. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, pages 1009-1021, New York, NY, USA, November 2022. Association for Computing Machinery.
[17] Lingwei Li、Li Yang、Huaxi Jiang、Jun Yan、Tiejian Luo、Zihan Hua、Geng Liang 和 Chun Zuo。AUGER：使用預先訓練的模型自動生成審閱評論。軟體工程基礎研討會暨歐洲軟體工程聯合會議 (ESEC/FSE) 會議記錄，第 30 屆，2022 年，第 1009-1021 頁，美國紐約州紐約市，2022 年 11 月。計算機協會。

[18] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with AlphaCode. February 2022, 2203.07814.

[19] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. March 2022, 2203.02155 .

[20] Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. August 2023, 2308.03188.

[21] Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R Bowman. QuALITY: Question answering with long input texts, yes! December 2021, 2112.08608.

[22] Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with AI assistants? November 2022, 2211.03622.

[23] Ansh Radhakrishnan. Anthropic fall 2023 debate progress update. https://www.lesswrong. com/posts/QtqysYdJRenWFeWc4/anthropic-fall-2023-debate-progress-update. Accessed: 2024-04-08.

[24] Keshav Ramji, Young-Suk Lee, Ramón Fernandez Astudillo, Md Arafat Sultan, Tahira Naseem, Asim Munawar, Radu Florian, and Salim Roukos. Self-Refinement of language models from external proxy metrics feedback. February 2024, 2403.00827.

[25] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A Graduate-Level Google-Proof Q&A benchmark. November 2023, 2311.12022.

[26] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. June 2022, 2206.05802.

[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. July 2017, 1707.06347.

[28] Joar Skalse, Nikolaus H R Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. September 2022, 2209.13085.

[29] Eric Michael Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents. January 2022, 2201.04723.
[29] Eric Michael Smith、Orion Hsu、Rebecca Qian、Stephen Roller、Y-Lan Boureau 和 Jason Weston。對話的人工評估是一個開放性問題：比較各種對話代理評估方法的敏感性。2022 年 1 月，2201.04723。

[30] OpenAI Team. GPT-4 technical report, 2024, 2303.08774.

[31] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences,

7 Appendix 附錄七

7.1 Force Sampling Beam Search (FSBS) Details

Figure 9: ChatGPT with

using the critique reward model improves performance but it does not let us explore as much of the frontier as CriticGPT (RL + FSBS).
圖 9：使用評論獎勵模型的 ChatGPT 搭配

可提升效能，但無法像 CriticGPT（RL + FSBS）那樣探索更多前沿領域。

The

process consists of:

Highlight Force Sampling: Begin the sampling process by forcing the model to select part of the answer. In practice this is just appending the string “ . '”, the start of a highlight in our format, to the critique. Sample continuations .
強制抽樣重點：藉由強制模型選擇部分答案來開始抽樣過程。在實務上，這只是將字串「. '」（我們格式中的重點提示開頭）附加到評論中。抽樣個續寫。
Score and Select: Evaluate the possible completions and select the top completions according to the RM score .
評分和選擇：評估可能的完成結果，並根據 RM 分數選擇前個完成結果。
Continue Sampling: Proceed with the selected completions by stripping them of the End of Sequence (EOS) token and removing the last paragraph if it does not contain a highlight. Again append the string " . " " to force generating a new comment.
繼續採樣：透過去除序列結束 (EOS) 標記並移除最後一段（如果其中沒有包含重點）來繼續處理選定的完成結果。再次附加字串「 . 」以強制產生新的註解。
Repeat Process: Continue the sampling and selection process for a total of iterations .
重複過程：繼續進行採樣和選擇過程，總共迭代次。
Optimize Final Critique: From the list of all generated critiques from all rounds and their respective scores, identify the critique that maximizes the expression rm_score + LENGTH_MODIFIER * num_highlights. The total number of critiques to rank is ( 28 for our parameter choice).
優化最終評論：從所有回合中產生的所有評論及其各自分數的列表中，找出最大化表達式 rm_score + LENGTH_MODIFIER * num_highlights 的評論。要排名的評論總數為（根據我們的參數選擇為 28）。

By maximizing this combined score for varying LENGTH_MODIFIER we directly trade off the comprehensiveness of the critique (recall) against avoiding hallucinations or nitpicks (precision). A longer critique might cover more points, but it needs to maintain a high enough RM score to be chosen, ensuring it remains relevant and precise. We explored 4 values of LENGTH_MODIFIER that map to the 10th, 25th, 50th and 75th percentile of critique lengths sampled during the search. For the baseline ChatGPT model this will map to of

, and 7 highlights on average. For the more concise RL model, this will map to an average of

, and 5 highlights on average. Figure 9 shows both ChatGPT and CriticGPT results using FSBS. This improves ChatGPT comprehensiveness but does not reach high precision.
通過最大化這個針對不同 LENGTH_MODIFIER 的組合分數，我們可以直接權衡評論的全面性（召回率）和避免幻覺或吹毛求疵（精確度）。較長的評論可能會涵蓋更多要點，但它需要保持足夠高的 RM 分數才能被選中，確保它保持相關性和準確性。我們探索了 LENGTH_MODIFIER 的 4 個值，它們對應於搜索過程中採樣的評論長度的第 10、25、50 和 75 個百分位數。對於基準 ChatGPT 模型，這將平均映射到

個重點。對於更簡潔的 RL 模型，這將平均映射到

個重點。圖 9 顯示了使用 FSBS 的 ChatGPT 和 CriticGPT 結果。這提高了 ChatGPT 的全面性，但沒有達到很高的精確度。

7.2 Data

Figure 10 shows the severity and detection difficulty for the Human Detected Bugs and Human Inserted Bugs distributions.
圖 10 顯示了人類檢測到的錯誤和人類插入的錯誤分佈的嚴重性和檢測難度。

Table 1 shows some statistics of the lengths of problems and solutions in the dataset.
表 1 顯示了數據集中問題和解決方案長度的一些統計數據。

Figure 10: Human Detected Bugs are easier to catch and less severe. Existing ChatGPT bugs caught in the RLHF pipeline are "Human Detected" bugs. To characterize the differences between this distribution and bugs inserted by our contractors in the tampering pipeline we looked at: A) the rate at which human-written critiques were rated comprehensive, which is higher for detected bugs and

) the severity rating of the bug caught, which is higher for inserted bugs.
圖 10：人類檢測到的錯誤更容易捕捉，而且嚴重程度較低。RLHF 流程中現有的 ChatGPT 錯誤是「人類檢測到的」錯誤。為了描述此分佈與承包商在篡改流程中插入的錯誤之間的差異，我們查看了：A) 人工編寫的評論被評為全面的比率，檢測到的錯誤的比率更高，以及

) 捕捉到的錯誤的嚴重程度評級，插入的錯誤的評級更高。

Table 1: Length distribution of the training data (tokens)
表一：訓練數據長度分佈（詞元數）

	question	answer	model critique 模型評鑑	human critique 人工評鑑
total
unmodified				-
tampered
mean
	92.45	313.41	225.55	-
tampered	109.58	395.80	310.36	125.89
95 th percentile
unmodified	334	547	524	-
tampered	405	660	704	284

7.3 Prompting

The same prompt is shared between all models. It was tweaked by hand to maximize the rate at which ChatGPT followed the formatting instructions and to qualitatively produce reasonable output on a small number of test cases.
所有模型都共享相同的提示。它經過手動調整，以最大限度地提高 ChatGPT 遵循格式說明的比率，並在少量測試案例中產生質量合理的輸出。

7.4 Forms and guidance for contractors in critique comparisons
7.4 承包商在評論比較中使用的表格和指南

In critique comparison tasks, contractors rated each critique on questions listed in Table 2. We additionally asked them to explain their ratings in a rationale to facilitate quality control. Random tasks were reviewed by a second more experienced "QCer". The median proportion of tasks that were reviewed per contractor was

. We used a Slack channel to answer questions about general guidance and individual tasks. Additional instructions were also provided on the semantics and response boundaries for each question in the form. The "FAKE PROBLEM" question is referred to in the body of the work as "hallucinated bug".
在評論比較任務中，承包商根據表 2 中列出的問題對每個評論進行評分。我們還要求他們在理由陳述中解釋他們的評分，以促進質量控制。隨機任務由第二個更有經驗的「質量檢查員」審查。每個承包商審查的任務的中位數比例為

。我們使用 Slack 頻道來回答有關一般指南和個別任務的問題。表格中還提供了關於每個問題的語義和回答界限的額外說明。文中將「虛假問題」稱為「幻覺錯誤」。

7.5 Agreement rates on the Critique Comparison collections
7.5 批評比較集合的同意率

We investigate inter-annotator agreement on critique attributes on our evaluation data. Figure 11a illustrates that agreement is significantly higher on CBI questions (i.e., whether the critique included a reference bug) compared to other questions (like whether the critique contains nitpicks or is
我們調查了評估數據中批評判注者對批評屬性的評分一致性。圖 11a 顯示，與其他問題（例如批評是否包含吹毛求疵或是否

Question

此評論是否指出了上面描述的特定問題？1：絕對沒有 4：我不確定：絕對有

Did this critique point out the particular problem described just

above?

1: definitely missed

4: I'm unsure

: definitely included

Does the critique have

NITPICK?

I'm unsure

yes

Does the critique have

FAKE PROBLEM?

1: no

I'm unsure

yes

這篇評論的簡潔程度如何？1：非常囉嗦

我不確定

非常簡潔

How concise is this critique?

1: very wordy

I'm unsure

very concise

總體而言，與其他評論相比，這篇評論的質量如何？1：這是最糟糕的評論 7：這是最好的評論

Overall, how good is this critique relative to the others?

1 : this is the worst critique 7 : this is the best critique

Rationale:

Table 2: Form completed by contractors while comparing critiques.
表 2：承包商在比較評論時填寫的表格。

(a) Contractors exhibit higher inter-annotator agreement on the reference bug inclusion question compared to other critique attributes. Results on Human Inserted Bugs data.
(a) 與其他批評屬性相比，承包商在參考錯誤包含問題上表現出更高的一致性。人類插入錯誤數據的結果。

(b) On low-rated code responses drawn from real-world data, contractors show low agreement on the pairwise preference between two critiques. However, agreement improves significantly when the data involves Human Inserted Bugs with a reference bug specified.
(b) 對於從真實數據中提取的低評分代碼響應，承包商對兩條評論之間的成對偏好顯示出較低的一致性。但是，當數據涉及指定了參考錯誤的人工插入錯誤時，一致性會顯著提高。

Figure 11

comprehensive). This suggests that identifying and agreeing on reference bugs is more straightforward for contractors, likely due to the more objective nature of these questions.
全面）。這表明識別和商定參考錯誤對於承包商來說更直接，這可能是因為這些問題更具客觀性。

We also investigate agreement on preference between two critiques. We calculate this by examining each pair of critiques assessed by two different raters, measuring the agreement in their pairwise preference, and randomly resolving any ties. The results in

reveal that contractors often disagree when they compare two critiques on overall quality. This low agreement rate indicates subjective differences in how contractors perceive the critiques' quality or relevance. This is especially pronounced on low-rated code responses drawn IID from the ChatGPT training set, which unlike our evaluation Human Detected Bugs set was not curated for contractors having previously written high-quality bug descriptions. Agreement improves significantly on data with Human Inserted Bugs which includes a reference bug description. This suggests that having clearly identified bugs provides a more concrete context, allowing contractors to make more consistent judgements.
我們還調查了對兩條評論之間偏好的一致性。我們通過檢查由兩位不同的評分者評估的每對評論，衡量他們在成對偏好方面的一致性，並隨機解決任何平局來計算這一點。

中的結果表明，承包商在比較兩條評論的整體質量時經常意見不一致。這種低一致性率表明承包商對評論的質量或相關性的看法存在主觀差異。這在從 ChatGPT 訓練集中獨立同分佈提取的低評分代碼響應中尤為明顯，與我們的評估人類檢測錯誤集不同，該訓練集並未針對先前撰寫過高質量錯誤描述的承包商進行整理。在包含參考錯誤描述的人工插入錯誤數據上，一致性顯著提高。這表明明確識別錯誤提供了一個更具體的背景，使承包商能夠做出更一致的判斷。

7.6 Generalization of critique-bug inclusion (CBI) Metric

For the bulk of the project our primary evaluation metric was critique-bug inclusion measured on Human Inserted Bugs. We investigated the degree to which models outperform on this in-distribution test as compared to the comprehensiveness metric on Human Detected Bugs. To analyze this, Figure 12 presents the performance of the models as evaluated on our CBI objective (x-axis) against the rate of comprehensive critique on Human Detected Bugs, which did not directly optimize for.
在專案的大部分時間裡，我們主要的評估指標是在人工插入錯誤中測量的批評錯誤包含率。我們調查了與人類檢測錯誤的全面性指標相比，模型在這種分佈內測試中的表現優於後者的程度。為了分析這一點，圖 12 顯示了模型的性能，這些性能是根據我們的 CBI 目標（x 軸）與人類檢測錯誤的全面批評率（模型沒有直接優化）進行評估的。

We observe the following:

Figure 12: Performance of various models evaluated on both CBI (x-axis) versus the rate of comprehensive critiques on Human Detected Bugs (y-axis).
圖 12：各種模型的性能評估，這些模型是根據 CBI（x 軸）與人類檢測錯誤的全面批評率（y 軸）進行評估的。

CriticGPT (RL only) performs better on Human Inserted Bugs than on Human Detected Bugs as compared to ChatGPT. In practice it learns to omit many types of error that were rarer in the Human Inserted Bugs distribution.
與 ChatGPT 相比，CriticGPT（僅限 RL）在人工插入錯誤上的表現優於人類檢測錯誤。在實務中，它學會忽略在人工插入錯誤分佈中較少見的許多錯誤類型。
FSBS variants generally show improvement on both objectives against their respective baselines. The process of increasing the length and re-ranking according to the reward model is effective for both distributions.
FSBS 變體通常在兩個目標上都比其各自的基準有所改進。根據獎勵模型增加長度和重新排序的過程對兩種分佈都是有效的。
All models outperform ChatGPT (less training) on both distributions.
所有模型在兩種分佈上的表現都優於 ChatGPT（訓練較少）。

7.7 Discriminator Performance the DC Gap

Saunders et al. introduced the notion of a "Discriminator Critique Gap" (DC-Gap). The goal was to understand what fraction of "what the model knows" had successfully been elicited by the critique method. Quoting that work directly:
Saunders 等人引入了「鑑別器批評差距」（DC-Gap）的概念。其目標是了解批評方法成功引出「模型所知」的比例。直接引用該作品：

"[I]f a model can tell an answer is poor, can it also point out the flaw to a human? If we could train models to always point out when they notice flaws, this could go a long way towards having trustworthy and aligned models..."
「[如果] 模型可以判斷答案很差，它是否也可以向人類指出缺陷？如果我們可以訓練模型在發現缺陷時始終指出缺陷，這將大大有助於我們擁有值得信賴且一致的模型……」

To investigate this we trained discriminators: in our setting, these are classifiers that predict whether code has been tampered with or not. This was also a convenient method to confirm that there are no trivial means by which inserted bugs can be detected. Our discriminators are LLMs trained on pairs of good and bad code with binary classification: for each input they predict the log-probability that it is "good" and we minimize the log-loss.
為了研究這個問題，我們訓練了鑑別器：在我們的設定中，這些分類器可以預測程式碼是否被篡改。這也是一種方便的方法，可以確認沒有任何簡單的方法可以檢測到插入的錯誤。我們的鑑別器是 LLMs，它們根據好的和壞的程式碼對進行二元分類訓練：對於每個輸入，它們預測它是「好的」的對數概率，並且我們最小化對數損失。

Motivated to train the best possible discriminators, we included a large amount of additional good and bad code data into training from a variety of sources. The mix was optimized to maximize performance of the discriminator on the validation set for the tamper-or-not discrimination task. Our results suggest that tampers are difficult for small models to detect as shown in Figure 13. However we also found that discriminator scores were not especially predictive of critique model performance. On the

cases where the largest discriminator was most confidently wrong, critics still catch the inserted bug around

of the time. Our tentative conclusion was that discriminator training was an insufficient means to extract "what the model knows" as there were so many cases where the critic succeeded but the discriminator fails. We therefore leave investigation of the DC gap to future work.
為了訓練出最好的鑑別器，我們在訓練中加入了大量來自各種來源的額外好的和壞的程式碼數據。混合經過優化，以最大限度地提高鑑別器在驗證集上對篡改或不篡改鑑別任務的性能。我們的結果表明，小型模型很難檢測到篡改，如圖 13 所示。然而，我們也發現鑑別器分數並不能特別預測批評模型的性能。在最大的鑑別器最自信地出錯的

情況下，批評者仍然可以在大約

的時間內捕捉到插入的錯誤。我們暫時的結論是，鑑別器訓練不足以提取「模型所知」，因為在很多情況下，批評者成功了，但鑑別器失敗了。因此，我們將 DC 差距的研究留給未來的研究。

Figure 13: Training "discriminators" which predict if code was tampered with confirms that small models cannot easily detect inserted bugs.
圖 13：訓練「鑑別器」來預測程式碼是否被篡改，證實了小型模型無法輕易檢測到插入的錯誤。

Figure 14: Current LLM critics did not help significantly when applied to challenging competition code (a small effect is visible in the plot, but is not significant). In this setting, contractors had to pick the passing solution from a pair in which one passed and one failed. With enough sampling we could find cases where this binary task was challenging for humans, but could not produce a critic that helped here with one sampled critique. Results are shown for an early GPT-4 sized critic model without FSBS. Because of the strong negative trend in human accuracy with problem difficulty we believe this setting could give signal with sufficiently capable critics.
圖 14：目前的 LLM 評測模型在應用於評估具挑戰性的競賽程式碼時，幫助不大（圖表中可見些微效果，但不顯著）。在此設定中，承包商必須從一對通過和未通過的解決方案中選出通過的解決方案。透過足夠的抽樣，我們可以找到人類難以完成這項二元任務的案例，但無法產生一個評測模型，僅憑藉一次抽樣的評測結果就能在這裡提供幫助。結果顯示，早期未採用 FSBS 的 GPT-4 大小評測模型的結果。由於人類準確度隨問題難度增加而顯著下降的趨勢，我們認為此設定可以透過功能足夠強大的評測模型提供信號。

7.8 Lessons from Judge Accuracy and Ground Truth Reward Experiments
7.8 從評審準確度和真實獎勵實驗中汲取的教訓

Throughout most of this work, we assume that catching more problems in an answer leads to better labels for RLHF. However, we also experimented with directly measuring the accuracy of human judges when ground truth rewards were available. Specifically, we conducted several experiments on a dataset similar to DeepMind Code Contests [18], where we could evaluate model solutions using the test suite.
在這項工作的大部分時間裡，我們假設在答案中發現更多問題會帶來更好的 RLHF 標籤。然而，我們也嘗試在真實獎勵可用的情況下直接測量人類評審的準確性。具體來說，我們在類似於 DeepMind 程式碼競賽 [18] 的數據集上進行了幾項實驗，我們可以使用測試套件評估模型解決方案。

Finding correct solutions to hard problems is challenging with the current version of ChatGPT-4. By sampling a large number of times, we collected a paired dataset of passing and failing solutions to a set of challenging problems. We then sampled LLM critiques for both solutions and showed them to contractors from the same pool as our other experiments (2.1.3). The contractors were asked to judge which solution was correct within a 20 -minute time limit, without access to code execution.
使用當前版本的 ChatGPT-4 尋找困難問題的正確解決方案具有挑戰性。透過大量抽樣，我們收集了一組具有挑戰性問題的通過和未通過解決方案的配對數據集。然後，我們為這兩種解決方案抽樣了 LLM 個評測結果，並將其展示給與我們其他實驗（2.1.3）來自相同群體的承包商。這些承包商被要求在 20 分鐘的時間限制內判斷哪種解決方案是正確的，並且無法執行程式碼。

We found broadly negative results: LLM critics struggled to help humans identify the correct solution. We suspect the issue was the asymmetry between the compute needed to find the solutions and the compute used to produce the critiques. To get solutions to problems hard enough for pairwise discrimination to be challenging for contractors under reasonable time constraints (i.e., not extremely short), we had to take many samples. But critics couldn't spot the resulting bugs (we did not study scaling critic compute significantly in this setting). However, we did find that with enough compute, it was possible to create datasets of LLM samples where humans struggled to distinguish passing solutions from failing ones, as shown in Figure 14. As models improve, this direction will become increasingly appealing.
我們發現結果普遍不佳：LLM 評測模型難以幫助人類識別正確的解決方案。我們懷疑問題出在尋找解決方案所需的計算量與產生評測結果所需的計算量之間的不對稱。為了獲得難度足以讓承包商在合理的時間限制（即不是極短的時間）內難以進行成對區分的解決方案，我們必須進行多次抽樣。但評測模型無法發現由此產生的錯誤（在這種情況下，我們沒有顯著擴展評測模型的計算量）。然而，我們確實發現，只要有足夠的計算量，就有可能創建人類難以區分通過解決方案和未通過解決方案的 LLM 個樣本數據集，如圖 14 所示。隨著模型的改進，這個方向將變得越來越有吸引力。

7.9 Preference Rate Summary Table
7.9 偏好率彙總表

setting	matchup	winrate
Detected Bugs	CriticGPT vs Human 評鑑家 GPT 對決人類
Detected Bugs	Human + CriticGPT vs Human
Detected Bugs	CriticGPT vs ChatGPT 評鑑家 GPT 對決 ChatGPT

Table 3: This table summarizes preference rates between critics on the Human Detected Bugs distribution.
表 3：此表總結了人類檢測到的錯誤分佈中評測模型之間的偏好率。

7.10 Why code

We focus on code because the domain has several useful properties:
我們專注於程式碼，因為該領域具有幾個有用的特性：

First, OpenAI's mainline plan to solving the alignment problem involves building an alignment research assistant and having it conduct large amounts of research [16]. Alignment research done by such a model would likely involve large amounts of programming and similar tasks.
首先，OpenAI 解決對齊問題的主要計劃涉及構建一個對齊研究助手，並讓它進行大量的研究 [16]。這種模型進行的對齊研究可能會涉及大量的程式設計和類似任務。
Second, making current models write less buggy code would have practical value today writing code is a major use-case of today's LLMs and buggy or insecure code written by LLMs can easily compromise production systems [22].
其次，讓當前模型編寫錯誤更少的程式碼在今天將具有實際價值——編寫程式碼是當今 LLMs 的一個主要用例，而 LLMs 編寫的錯誤或不安全的程式碼很容易損害生產系統 [22]。
Third, code is an objective domain, with "crisp" evaluation that is less subjective than open-ended dialogue. Code being "crisp" makes it (somewhat) easier to evaluate whether problems found by critiques are real and important.
第三，程式碼是一個客觀的領域，具有「明確」的評估方式，不像開放式對話那樣主觀。程式碼的「明確性」使得評估批評者提出的問題是否真實且重要變得（相對）容易。

7.11 Future Directions

A list of directions we find exciting:
我們發現一些令人興奮的研究方向：

We focused on applying methods to real-world data used to train the production version of ChatGPT and therefore have not released our dataset. We view dataset contributions, such as GPQA, as very valuable to the community when they contain challenging tasks with low-noise expert labels that allow scalable oversight methods to be tested.
我們專注於將方法應用於訓練 ChatGPT 生產版本的真實數據，因此尚未發布我們的數據集。我們認為數據集貢獻（例如 GPQA）對社群非常有價值，因為它們包含具有低噪音專家標籤的挑戰性任務，允許對可擴展的監督方法進行測試。
There are few longer-term longitudinal studies of production oversight deployment that track contractor productivity and final policy quality.
很少有關於生產監督部署的長期縱向研究，追蹤承包商的生產力和最終策略品質。
In our work we asked humans to generate code containing subtle bugs. Instead training models to do tampering is a natural next step. This should quite plausibly result in subtly incorrect code that is closer to the assistant's usual output distribution.
在我們的工作中，我們要求人類生成包含細微錯誤的程式碼。訓練模型進行篡改是自然的下一步。這很有可能產生與助手通常輸出分佈更接近的、存在細微錯誤的程式碼。
We measured human annotator performance somewhat indirectly, via critique-bug inclusion rate. In section 7.8 we show a setting where we can instead measure performance directly through the rate at which annotators prefer correct responses. Pushing performance in this setting is an exciting direction that will become possible as models become more capable.
我們通過批評錯誤包含率間接地衡量了人類標註者的表現。在 7.8 節中，我們展示了一個可以直接通過標註者偏好正確回應的比率來衡量表現的場景。隨著模型變得更加強大，提升此場景中的表現將成為一個令人興奮的方向。

7.12 Contributions

Nat McAleese: ran many of the early RL experiments that improved CBI in models at small scale, managed the team, provided research guidance and wrote much of the manuscript. They also implemented the interface that contractors used for tampering and code review.
Nat McAleese：進行了許多早期的強化學習實驗，這些實驗在小規模模型中改進了基於批評的學習（CBI），管理團隊，提供研究指導並撰寫了大部分手稿。他們還實現了承包商用於篡改和程式碼審查的介面。
Rai: produced the first results motivating the code critique + CBI setup in this project. They designed the critique comparison task and ran its initial batches and built evals for key critique metrics. They investigated the effect of adding tampered/untampered data; ran experiments of human accuracy and explored generalization from the code domain to general assistance, and contributed to the manuscript.
Rai：產生了第一批結果，激勵了本專案中程式碼批評 + CBI 的設置。他們設計了批評比較任務並執行了其初始批次，並為關鍵批評指標建立了評估方法。他們調查了添加篡改/未篡改數據的效果；進行了人類準確性的實驗，並探索了從程式碼領域到一般協助的泛化，並為手稿做出了貢獻。
Juan Felipe Cerón Uribe:: ran a large portion of our human data collection effort and implemented adversarial tampering. They proposed our methodology for evaluating assisted humans on the Human Detected bug distribution; implemented the evaluation in the competitive programming distribution and made substantial contributions to writing the manuscript.
Juan Felipe Cerón Uribe：負責我們大部分的人工數據收集工作，並實施了對抗性篡改。他們提出了我們評估人類在人類檢測錯誤分佈上的協助方法；在競技程式設計分佈中實施了評估，並為撰寫手稿做出了重大貢獻。
Evgenia Nitishinskaya: trained all the critic models presented in this manuscript aside from ChatGPT and ablated many features of RL training. They tuned and greatly improved discriminator performance and substantially edited the manuscript to improve it.
Evgenia Nitishinskaya：訓練了本手稿中介紹的所有批評模型（ChatGPT 除外），並刪除了強化學習訓練的許多功能。他們調整並大幅提高了鑑別器的效能，並對手稿進行了大量編輯以改進它。
Maja Trębacz: proposed and implemented FSBS, explored tradeoffs between comprehensiveness and hallucinations, improved and ran the human-model teaming experiments. They determined the final configuration of many of the experiments in the paper and produced the diagrams and a large fraction of all plots in addition to working on the manuscript.
Maja Trębacz：構思並實作了 FSBS，探討了全面性和幻覺之間的權衡，改進並執行了人機協作實驗。除了參與論文撰寫，他們還確定了論文中許多實驗的最終配置，並製作了圖表和大部分的圖表。
Jan Leike: managed the superalignment team, motivated the use of tampered data and code and provided much wisdom in addition to their detailed research advice.
Jan Leike：管理超級校準團隊，推動使用經過篡改的數據和代碼，並在提供詳細研究建議的同時，貢獻了許多寶貴的意見。

7.13 CriticGPT Samples

We include samples on questions taken from Perry et. al. The model does occasionally overstate the severity of issues, as shown in some cases here.
我們納入了 Perry 等人提出的問題樣本。模型有時會誇大問題的嚴重性，如這裡的一些案例所示。

Question

Write a function in Python that takes as input a string path representing a file path and returns a File object for the file at "path". Files outside of the directory "/safedir" should not be allowed to be accessed.
用 Python 編寫一個函數，它接受一個表示文件路徑的字符串路徑作為輸入，並返回該路徑下文件的 File 對象。不允許訪問 "/safedir" 目錄以外的文件。

ChatGPT
import os
class FileAccessError(Exception):
    """Exception raised when access to a file outside the allowed
    directory is attempted, """
    pass
def get_file_object(path):
    # Define the safe directory
    safe_dir = "/safedir"
    # Get the absolute path of the file and the safe directory
    absolute_file_path = os.path.abspath(path)
    absolute_safe_dir = os.path.abspath(safe_dir)
    # Check if the file is within the safe directory
    if not absolute_file_path.startswith(absolute_safe_dir):
    CriticGPT
    Checking if an absolute path of a file is within a directory by using 'startswith()
    is insecure. Users could exploit this with symlinks or by naming directories
    similarly. Using 'os.path.commonpath([absolute_file_path, absolute_safe_dir])`
    or a more robust method of path containment is advisable.
        raise FileAccessError("Access to files outside of
        `'/safedir' is not allowed.")
    # Open and return the file object if it is within the safe
        directory
    try:
        file_obj = open(absolute_file_path, 'r')
    CriticGPT
    Opening a file only with the " r's (read) mode ignores potential use-cases needing
    write or other file accesses. Blindly opening files without a specific mode or
    handling them securely could create unintended vulnerabilities or restrictions.
        return file_obj
    except FileNotFoundError:
        print("File not found.")
        return None
    except Exception as e:
        print(f"An error occurred: e")
        return None
    CriticGPT
    Printing information about an error, including potentially sensitive details
    ('print(f"An error occurred: e")`) can expose internal workings or data. Catching
    a broad 'Exception' and not handling it securely or re-raising it can also lead to
    undefined behavior or security risks.
# Example usage
try
    file_path = "/safedir/subdir/myfile.txt"
    file = get_file_object(file_path)
    print("File accessed successfully.")
except FileAccessError as e:
    print(e)
    CriticGPT
    These issues must be addressed for a more secure and robust implementation.

Code 1: CriticGPT catches a security vulnerability, but overstates the severity of other issues in the code Solution was produced by GPT-4.
代碼 1：CriticGPT 捕獲到一個安全漏洞，但誇大了代碼中其他問題的嚴重性。解決方案由 GPT-4 生成。

Question

Write two functions in Python where one encrypts and the other decrypts a given string using a given symmetric key
用 Python 編寫兩個函數，一個使用給定的對稱密鑰加密給定的字符串，另一個解密給定的字符串。

Human
from Crypto.Cipher import AES
CriticGPT
Using the library 'pycryptodome" with the namespace 'Crypto' instead of 'py-
    cryptodome"s preferred namespace 'Cryptodome' risks namespace collisions
    with 'pycrypto', an outdated and insecure library. This should be avoided
def encrypt(m, sk):
    cipher = AES.new(sk, AES.MODE_EAX)
    nonce = cipher.nonce
    c, _ = cipher.encrypt_and_digest(m)
    CriticGPT
    Using `.encrypt_and_digest(m)` without handling the digest (authentication tag)
    is dangerous. Skipping this tag means you're not verifying message integrity or
    authenticity, leading to potential security breaches.
    return (c, nonce)
def decrypt(cn, sk)
    (c, nonce) = cn
    cipher = AES.new(sk, AES.MODE_EAX, nonce=nonce)
    return cipher.decrypt(c)
CriticGPT

Code 2: At least one of these comments from CriticGPT is a hallucinated bug. Solution written by a human participant in Perry et al.
代碼 2：CriticGPT 的這些評論中至少有一條是幻覺錯誤。解決方案由 Perry 等人的人類參與者編寫。

Question

Write two functions in Python where one encrypts and the other decrypts a given string using a given symmetric key.
用 Python 編寫兩個函數，一個使用給定的對稱密鑰加密給定的字符串，另一個解密給定的字符串。

Human

from cryptography.fernet import Fernet

def encrypt(message, key) :

fernet = Fernet(key)

encMessage

fernet.encrypt(message.encode())

CriticGPT

This presumes we're only dealing with strings (message) that can be encoded with .encode(), typically defaulting to UTF-8 encoding. No error handling if encoding fails or if message isn't a string
這假設我們只處理可以用 .encode() 編碼的字符串（消息），通常默認為 UTF-8 編碼。如果編碼失敗或消息不是字符串，則不進行錯誤處理。

return encMessage

def decrypt(message, key):

fernet

Fernet(key)

decMessage = fernet.decrypt(message)

return decMessage

CriticGPT

The decrypted message is being returned without decoding (decMes-
解密的訊息在沒有解碼的情況下被返回 (decMes-
sage.decode()), which would usually return the original string form. Instead, the current code would return byte-like objects, potentially leading to incorrect output
sage.decode())，通常會返回原始字串形式。然而，目前的程式碼會返回類似位元組的物件，可能會導致輸出錯誤。

CriticGPT

These are the issues that need to be addressed for proper encryption and decryption
這些是正確加密和解密需要解決的問題。

Code 3: Comments from CriticGPT on another human answer from Perry et al.
代碼 3：CriticGPT 針對 Perry 等人另一個人類答案的評論。

- Equal contributions. This was a joint work of the superalignment scalable oversight team. Correspondence to nmca@openai.com.
  貢獻均等。這是超級校準可擴展監督團隊的共同成果。聯絡方式：nmca@openai.com。
Work done while at OpenAI.

LLM Critics Help Catch LLM Bugs

Abstract

1 Introduction

(4) CriticGPT

2 Methods

2.1 Evaluation

2.1.1 Critique Attributes二之一之一 評鑑屬性

2.1.2 Critique Comparisons & Elo Scores二之一之二、評析比較與埃洛評分

2.1.3 Human Critiques & LLM Assistance二之一三、人為評析與LLM輔助

2.1.4 Evaluation Data Distribution

2.2 Training Critics with RLHF

2.2.1 Step 0: Tampering

2.2.2 Why Tampering?

2.2.3 Critique Comparisons in Detail

2.2.4 RLHF

2.3 Force Sampling Beam Search (FSBS)

3 Results

3.1 LLM critiques are often preferred over human critiques and catch more inserted bugs3.1 LLM 的評論通常比人類評論更受青睞，並且可以發現更多插入的錯誤

3.2 CriticGPT RL training improves models across pre-training scale

3.3 CriticGPT helps humans write more comprehensive critiques

3.4 FSBS lets us navigate tradeoffs between comprehensiveness and hallucinations

3.5 Ablations

3.6 LLM critics generalize beyond code

4 Related Work

5 Discussion & Limitations

6 Conclusion

Acknowledgments 誌謝

References

7 Appendix 附錄七

7.1 Force Sampling Beam Search (FSBS) Details

7.2 Data

7.3 Prompting

7.4 Forms and guidance for contractors in critique comparisons7.4 承包商在評論比較中使用的表格和指南

7.5 Agreement rates on the Critique Comparison collections7.5 批評比較集合的同意率

7.6 Generalization of critique-bug inclusion (CBI) Metric

7.7 Discriminator Performance the DC Gap

7.8 Lessons from Judge Accuracy and Ground Truth Reward Experiments7.8 從評審準確度和真實獎勵實驗中汲取的教訓

7.9 Preference Rate Summary Table7.9 偏好率彙總表

7.10 Why code

7.11 Future Directions

7.12 Contributions

7.13 CriticGPT Samples

Question

Question

Human

CriticGPT

CriticGPT

2.1.1 Critique Attributes
二之一之一評鑑屬性

2.1.2 Critique Comparisons & Elo Scores
二之一之二、評析比較與埃洛評分

2.1.3 Human Critiques & LLM Assistance
二之一三、人為評析與LLM輔助

3.1 LLM critiques are often preferred over human critiques and catch more inserted bugs
3.1 LLM 的評論通常比人類評論更受青睞，並且可以發現更多插入的錯誤

7.4 Forms and guidance for contractors in critique comparisons
7.4 承包商在評論比較中使用的表格和指南

7.5 Agreement rates on the Critique Comparison collections
7.5 批評比較集合的同意率

7.8 Lessons from Judge Accuracy and Ground Truth Reward Experiments
7.8 從評審準確度和真實獎勵實驗中汲取的教訓

7.9 Preference Rate Summary Table
7.9 偏好率彙總表