This post relates an observation I've made in my work with GPT-2, which I have not seen made elsewhere.
这篇文章描述了我在与 GPT-2 工作中所做的一项观察，而我在其他地方没有见过相关的论述。

IMO, this observation sheds a good deal of light on how the GPT-2/3/etc models (hereafter just "GPT") work internally.
在我看来，这一观察为理解 GPT-2/3 等模型（以下简称“GPT”）的内部工作原理提供了很好的启示。

There is an accompanying Colab notebook which will let you interactively explore the phenomenon I describe here.
有一个配套的 Colab 笔记本，可以让你互动式地探索我在这里描述的现象。

[Edit: updated with another section on comparing to the inputs, rather than the outputs. This arguably resolves some of my confusion at the end. Thanks to algon33 and Gurkenglas for relevant suggestions here.]
[编辑：更新了另一个关于与输入而非输出进行比较的部分。这可以说解决了我最后的一些困惑。感谢 algon33 和 Gurkenglas 在这里提供的相关建议。]

[Edit 5/17/21: I've recently written a new Colab notebook which extends this post in various ways:
[编辑于 2021 年 5 月 17 日：我最近写了一个新的 Colab 笔记本，它在多个方面扩展了这篇文章。]

• trying the "lens" on various models from 125M to 2.7B parameters, including GPT-Neo and CTRL
  在 125M 到 2.7B 参数的各种模型上尝试“镜头”，包括 GPT-Neo 和 CTRL。
• exploring the contributions of the attention and MLP sub-blocks within transformer blocks/layers
  探索变换器块/层中注意力和多层感知器子块的贡献
• trying out a variant of the "decoder" used in this post, which dramatically helps with interpreting some models
  尝试一种在本帖中使用的“解码器”的变体，这对解释一些模型有显著帮助

]

overview  概述

• GPT's probabilistic predictions are a linear function of the activations in its final layer. If one applies the same function to the activations of intermediate GPT layers, the resulting distributions make intuitive sense.
  GPT 的概率预测是其最终层激活的线性函数。如果将相同的函数应用于中间 GPT 层的激活，所得到的分布是直观合理的。
• This "logit lens" provides a simple (if partial) interpretability lens for GPT's internals.
  这种“逻辑镜头”为 GPT 的内部提供了一种简单（如果不完全）的可解释性视角。
• Other work on interpreting transformer internals has focused mostly on what the attention is looking at. The logit lens focuses on what GPT "believes" after each step of processing, rather than how it updates that belief inside the step.
  对变压器内部进行解释的其他研究主要集中在注意力关注的对象上。Logit 透镜关注的是 GPT 在处理每个步骤后“相信”的内容，而不是它在步骤内部如何更新该信念。
• These distributions gradually converge to the final distribution over the layers of the network, often getting close to that distribution long before the end.
  这些分布逐渐收敛到网络层的最终分布，通常在结束前很久就接近该分布。
• At some point in the middle, GPT will have formed a "pretty good guess" as to the next token, and the later layers seem to be refining these guesses in light of one another.
  在某个中间阶段，GPT 将形成对下一个标记的“相当不错的猜测”，而后面的层似乎在相互考虑的基础上对这些猜测进行精炼。
• The general trend, as one moves from earlier to later layers, is
  一般趋势是，从早期层到后期层，
• "nonsense / not interpretable" (sometimes, in very early layers) -->
  "无意义 / 不可解释的" (有时在非常早的层次中) -->
• "shallow guesses (words that are the right part of speech / register / etc)" -->
  "肤浅的猜测（符合正确词性/用语风格等的词）"
• "better guesses"  "更好的猜测"
• ...though some of those phases are sometimes absent.
  ...虽然其中一些阶段有时会缺失。
• On the other hand, only the inputs look like the input tokens.
  另一方面，只有输入看起来像输入标记。
• In the logit lens, the early layers sometimes look like nonsense, and sometimes look like very simple guesses about the output. They almost never look like the input.
  在逻辑回归视角下，早期层有时看起来像是无意义的内容，有时看起来像是对输出的非常简单的猜测。它们几乎从不看起来像输入。
• Apparently, the model does not "keep the inputs around" for a while and gradually process them into some intermediate representation, then into a prediction.
  显然，该模型并没有“暂时保留输入”，而是直接将其逐步处理成某种中间表示，然后生成预测。
• Instead, the inputs are immediately converted to a very different representation, which is smoothly refined into the final prediction.
  相反，输入会立即被转换为一种非常不同的表示形式，并在此基础上平滑地细化为最终预测。
• This is reminiscent of the perspective in Universal Transformers which sees transformers as iteratively refining a guess.
  这让人想起了通用变压器中的观点，认为变压器是对一个猜测进行迭代优化的过程。
• However, Universal Transformers have both an encoder and decoder, while GPT is only a decoder. This means GPT faces a tradeoff between keeping around the input tokens, and producing the next tokens.
  然而，Universal Transformers 同时具有编码器和解码器，而 GPT 仅仅是一个解码器。这意味着 GPT 在保留输入标记和生成下一个标记之间面临权衡。
• Eventually it has to spit out the next token, so the longer it spends (in depth terms) processing something that looks like token i, the less time it has to convert it into token i+1. GPT has a deadline, and the clock is ticking.
  最终它必须输出下一个标记，因此它在处理看起来像标记 i 的东西上花费的时间越长，它转换成标记 i+1 的时间就越少。GPT 有一个截止日期，时间在不断流逝。
• More speculatively, this suggests that GPT mostly "thinks in predictive space," immediately converting inputs to predicted outputs, then refining guesses in light of other guesses that are themselves being refined.
  更具推测性的是，这表明 GPT 主要“在预测空间中思考”，立即将输入转换为预测的输出，然后根据其他正在被细化的猜测来完善自己的猜测。
• I think this might suggest there is some fundamentally better way to do sampling from GPT models? I'm having trouble writing out the intuition clearly, so I'll leave it for later posts.
  我认为这可能暗示从 GPT 模型中进行采样有某种根本上更好的方法？我在清晰地表达直觉方面遇到了困难，所以我会留到后面的帖子中再说。
• Caveat: I call this a "lens" because it is one way of extracting information from GPT's internal activations. I imagine there is other information present in the activations that cannot be understood by looking at logits over tokens. The logit lens show us some of what is going on, not all of it.
  警告：我称之为“透镜”，因为这是从 GPT 的内部激活中提取信息的一种方式。我想在激活中存在其他信息，无法通过查看令牌的对数几率来理解。对数几率透镜展示了我们所看到的一部分情况，而不是全部。

background on GPT's structure
GPT 的结构背景

You can skip or skim this if you already know it.
如果你已经知道这个内容，可以跳过或略读。

• Input and output  输入和输出
• As input, GPT takes a sequence of tokens. Each token is a single item from a vocabulary of N_v=50257 byte pairs (mostly English words).
  作为输入，GPT 接收一个令牌序列。每个令牌是来自一个包含 N_v=50257 个字节对（主要是英语单词）词汇表的单个项。
• As output, GPT returns a probability distribution over the vocabulary. It is trained so this distribution predicts the next token.
  作为输出，GPT 返回一个词汇表上的概率分布。它的训练使得这个分布能够预测下一个标记。
• That is, the model's outputs are shifted forward by one position relative to the inputs. The token at position i should, after flowing through the layers of the model, turn into the token at position i+1. (More accurately, a distribution over the token at position i+1.)
  也就是说，模型的输出相对于输入向前移动了一个位置。位置 i 的标记在经过模型的各层后应变成位置 i+1 的标记。（更准确地说，是对位置 i+1 的标记的分布。）
• Vocab and embedding spaces
  词汇和嵌入空间
• The vocab has size N_v=50257, but GPT works internally in a smaller "embedding" vector space, of dimension N_e.
  词汇表大小为 N_v=50257，但 GPT 在内部以更小的“嵌入”向量空间（维度为 N_e）进行工作。
• For example, in the GPT-2 1558M model size, N_e=1600. (Below, I'll often assume we're talking about GPT-2 1558M for concreteness.)
  例如，在 GPT-2 1558M 模型中，N_e=1600。 （下面，我会经常假设我们在谈论 GPT-2 1558M，以便更加具体。）
• There is an N_v-by-N_e embedding matrix W which is used to project the vocab space into the embedding space and vice versa.
  有一个 N_v 乘以 N_e 的嵌入矩阵 W，用于将词汇空间投影到嵌入空间，反之亦然。
• In, blocks, out  进，块，出
• The first thing that happens to the inputs is a multiplication by W, which projects them into the embedding space. [1]
  输入的第一件事是通过 W 进行乘法运算，将它们投影到嵌入空间中。[1]
• The resulting 1600-dimensional vector then passes through many neural network blocks, each of which returns another 1600-dimensional vector.
  结果得到的 1600 维向量随后通过许多神经网络块，每个块返回另一个 1600 维向量。
• At the end, the final 1600-dimensional vector is multiplied by W's transpose to project back into vocab space.
  最后，最终的 1600 维向量与 W 的转置相乘，以投影回词汇空间。
• The resulting 50257-dim vectors are treated as logits. Applying the softmax function to them gives you the output probability distribution.
  生成的 50257 维向量被视为对数几率。对它们应用 softmax 函数可以得到输出概率分布。

the logit lens  对数几率视角

As described above, GPT schematically looks like
如上所述，GPT 在结构上看起来像

• Project the input tokens from vocab space into the 1600-dim embedding space
  将输入的标记从词汇空间投影到 1600 维嵌入空间
• Modify this 1600-dim vector many times
  多次修改这个 1600 维的向量
• Project the final 1600-dim vector back into vocab space
  将最终的 1600 维向量投影回词汇空间

We have a "dictionary," W, that lets us convert between vocab space and embedding space at any point. We know that some vectors in embedding space make sense when converted into vocab space:
我们有一个“字典”，W，可以让我们在任何时刻在词汇空间和嵌入空间之间转换。我们知道，在嵌入空间中的某些向量在转换为词汇空间时是有意义的：

• The very first embedding vectors are just the input tokens (in embedding space)
  最初的嵌入向量只是输入的标记（在嵌入空间中）
• The very last embedding vectors are just the output logits (in embedding space)
  最后的嵌入向量就是输出的对数几率（在嵌入空间中）

What about the 1600-dim vectors produced in the middle of the network, say the output of the 12th layer or the 33rd? If we convert them to vocab space, do the results make sense? The answer is yes.
那么，网络中间产生的 1600 维向量，例如第 12 层或第 33 层的输出呢？如果我们将它们转换为词汇空间，结果是否有意义？答案是肯定的。

logits

For example: the plots below show the logit lens on GPT-2 as it predicts a segment of the abstract of the GPT-3 paper. (This is a segment in the middle of the abstract; it can see all the preceding text, but I'm not visualizing the activations for it.)
例如：下面的图显示了 GPT-2 在预测 GPT-3 论文摘要一部分时的 logit 镜头。（这是摘要中间的一段；它可以看到之前的所有文本，但我并没有为其可视化激活。）

For readability, I've made two plots showing two consecutive stretches of 10 tokens. Notes on how to read them:
为了便于阅读，我制作了两个图表，显示了两个连续的 10 个符号的延伸。关于如何阅读它们的说明：

• The input tokens are shown as 45-degree tilted axis labels at the bottom.
  输入的记号显示为底部的 45 度倾斜轴标签。
• The correct output (i.e. the input shifted by one) is likewise shown at the top.
  正确的输出（即输入向下移一位）同样显示在顶部。
• A (*) is added in these labels when the model's top guess matched the correct output.
  在这些标签中，当模型的最佳猜测与正确输出匹配时，会添加一个(*)。
• The vertical axis indexes the layers (or "blocks"), zero-indexed from 0 to 47. To make the plots less huge I skip every other intermediate layer. The Colab notebook lets you control this skipping as you like.
  垂直轴索引层（或“块”），从 0 到 47 进行零索引。为了使图表不那么庞大，我跳过每个中间层。Colab 笔记本允许您根据需要控制这种跳过。
• The top guess for each token, according to the model's activations at a given layer, is printed in each cell.
  根据模型在给定层的激活，每个单元格中打印了每个标记的最佳猜测。
• The colors show the logit associated with the top guess. These tend to increase steadily as the model converges on a "good guess," then get refined in the last layers.
  颜色显示与最佳猜测相关的对数几率。这些几率在模型收敛到“好猜测”时往往会持续增加，然后在最后几层中进行精细化调整。
• Cells are outlined when their top guess matches the final top guess.
  当细胞的最高猜测与最终的最高猜测相匹配时，细胞会被勾勒出来。
• For transformer experts: the "activations" here are the block outputs after layer norm, but before the learned point-wise transformation.
  对于变换器专家来说：“激活”是指层归一化后的块输出，但在学习的逐点变换之前。

There are various amusing and interesting things one can glimpse in these plots. The "early guesses" are generally wrong but often sensible enough in some way:
在这些情节中，能够窥见各种有趣和引人发笑的事物。“早期的猜测”通常是错误的，但在某种程度上往往是合理的。

• "We train GPT-3..." 000? (someday!)
  我们训练 GPT-3… 000？（总有一天！）
• "GPT-3, an..." enormous? massive? (not wrong!)
  GPT-3，一个...... 巨大的？庞大的？（没有错！）
• "We train GPT-3, an aut..." oreceptor? (later converges to the correct oregressive)
  我们训练 GPT-3，一个自动……受体？（最终收敛到正确的回归）
• "model with 175..." million? (later converges to a comma, not the correct billion)
  “模型有 1.75 亿？（后来收敛到一个逗号，而不是正确的十亿）”

ranks  排名

The view above focuses only on the top-1 guess at each layer, which is a reductive window on the full distributions.
上述观点仅关注每一层的 top-1 猜测，这只是对完整分布的一个简化窗口。

Another way to look at things: we still reduces the final output to the top-1 guess, but we compare other distributions to the final one by looking at the rank of the final top-1 guess.
另一种看待事物的方法：我们仍然将最终输出减少到排名第一的猜测，但我们通过查看最终排名第一猜测的排名，将其他分布与最终输出进行比较。

Even if the middle of the model hasn't yet converged to the final answer, maybe it's got that answer somewhere in its top 3, top 10, etc. That's a lot better than "top 50257."
即使模型的中间结果尚未收敛到最终答案，也许它在前 3 个、前 10 个等结果中有那个答案。这比“前 50257 个”要好得多。

Here's the same activations as ranks. (Remember: these are ranks of the model's final top-1 prediction, not the true token.)
这里是与排名相同的激活值。（请记住：这些是模型最终的顶级预测的排名，而不是实际的标记。）

In most cases, network's uncertainty has drastically reduced by the middle layers. The order of the top candidates may not be right, and the probabilities may not be perfectly calibrated, but it's got the gist already.
在大多数情况下，中间层显著减少了网络的不确定性。顶级候选者的顺序可能不正确，概率也可能没有完美地校准，但它已经抓住了要点。

KL divergence and input discarding
KL 散度及输入丢弃

Another way of comparing the similarity of two probability distributions is the KL divergence. Taking the KL divergence of the intermediate probabilities w/r/t the final probabilities, we get a more continuous view of how the distributions smoothly converge to the model's output.
比较两个概率分布相似性的另一种方法是 KL 散度。考虑中间概率与最终概率的 KL 散度，我们可以更连续地观察到分布如何平滑地收敛到模型的输出。

Because KL divergence is a more holistic measure of the similarity between two distributions than the ones I've used above, it's also my preferred metric for making the point that nothing looks like the input.
由于 KL 散度是衡量两个分布相似度的更全面的指标，优于我之前使用的指标，因此它也是我用来强调没有任何东西与输入相似的首选指标。

In the plots above, I've skipped the input layer (i.e. the input tokens in embedding space). Why? Because they're so different from everything else, they distract the eye!
在上面的图中，我省略了输入层（即嵌入空间中的输入标记）。为什么？因为它们与其他所有内容差异太大，容易分散注意力！

In the plots below, where color is KL divergence, I include the input as well. If we trust that KL divergence is a decent holistic way to compare two distributions (I've seen the same pattern with other metrics), then:
在下面的图中，颜色表示 KL 散度，我还包括了输入。如果我们相信 KL 散度是比较两个分布的一个合理的整体方式（我在其他指标中也见过相同的模式），那么：

• Immediately, after the very first layer, the input has been transformed into something that looks more like the final output (47 layers layer) than it does like the input.
  立即，在第一层之后，输入已经被转化为一种更接近最终输出（第 47 层）的形式，而不是更像输入。
• After this one discontinuous jump, the distribution progresses in a much more smooth way to the final output distribution.
  经过这一不连续的跳跃，分布以更加平滑的方式发展到最终输出分布。

other examples  其他例子

I show several other examples in the Colab notebook. I'll breeze through a few of them here.
我在 Colab 笔记本中展示了几个其他示例。这里我将简单介绍其中的一些。

copying a rare token  复制稀有代币

Sometimes it's clear that the next token should be a "copy" of an earlier token: whatever arbitrary thing was in that slot, spit it out again.
有时很明显下一个标记应该是之前某个标记的“复制”：无论那个位置上的任意内容是什么，都再次输出。

If this is a token with relatively low prior probability, one would think it would be useful to "keep it around" from the input so later positions can look at it and copy it. But as we saw, the input is never "kept around"!
如果这个标记的先验概率相对较低，人们可能会认为将其“保留”在输入中是有用的，以便后面的位点可以查看并复制它。但正如我们所看到的，输入从来没有被“保留”！

What happens instead? I tried this text:
发生了什么呢？我试着写了这段文本：

Sometimes, when people say plasma, they mean a state of matter. Other times, when people say plasma
有时，当人们提到等离子体时，他们指的是一种物质状态。其他时候，当人们提到等离子体时，

As shown below (truncated to the last few tokens for visibility), the model correctly predicts "plasma" at the last position, but only figures it out in the very last layers.
如下面所示（为了可视性截取到最后几个词元），模型在最后一个位置正确预测了“plasma”，但仅在最后几层中才弄清楚这一点。

Apparently it is keeping around a representation of the token "plasma" with enough resolution to copy it . . . but it only retrieves this representation at the end! (In the rank view, the rank of plasma is quite low until the very end.)
显然，它保持着对代币“等离子体”的一种表示，分辨率足够高以便复制……但它仅在最后才检索这个表示！(在排名视图中，等离子体的排名在最后之前相当低。)

This is surprising to me. The repetition is directly visible in the input: "when people say" is copied verbatim. If you just applied the rule "if input seems to be repeating, keep repeating it," you'd be good. Instead, the model scrambles away the pattern, then recovers it later through some other computational route.
这让我感到惊讶。重复在输入中是直接可见的：“当人们说”被逐字复制。如果你只应用规则“如果输入看起来在重复，就继续重复它”，那就好了。相反，模型扰乱了这种模式，然后通过其他计算途径在稍后恢复它。

extreme repetition  极端重复

We've all seen GPT sampling get into a loop where text repeats itself exactly, over and over. When text is repeating like this, where is the pattern "noticed"?
我们都见过 GPT 生成文本时进入循环的情况，文本一遍又一遍地完全重复。当文本像这样重复时，模式“被注意到”在哪里？

At least in the following example, it's noticed in the upper half of the network, while the lower half can't see it even after several rounds of repetition.
至少在以下例子中，网络的上半部分注意到了它，而下半部分即使经过多轮重复也看不见它。

why? / is this surprising?
为什么？ / 这让人惊讶吗？

First, some words about why this trick can even work at all.
首先，谈谈这个技巧为什么可以有效。

One can imagine models that perform the exact same computation as GPT-2, for which this trick would not work. For instance, each layer could perform some arbitrary vector rotation of the previous one before doing anything else to it. This would preserve all the information, but the change of basis would prevent the vectors from making sense when multiplied by W^T.
可以想象出一些模型，它们执行与 GPT-2 完全相同的计算，而这一技巧则不适用。例如，每一层可以在对前一层进行任何其他操作之前，执行某种任意的向量旋转。这将保留所有信息，但基底的改变会阻止向量在与 W^T 相乘时具有意义。

Why doesn't the model do this? Two relevant facts:
为什么模型不这样做？两个相关的事实：

1. Transformers are residual networks. Every connection in them looks like x + f(x) where f is the learned part. So the identity is very easy to learn.
变压器是残差网络。它们中的每个连接看起来都像 x + f(x)，其中 f 是学习的部分。因此，恒等式很容易学习。

This tends to keep things in the same basis across different layers, unless there's some reason to switch.
这通常会在不同层次之间保持相同的基础，除非有某种原因需要切换。

2. Transformers are usually trained with weight decay, which is almost the same thing as L2 regularization. This encourages learned weights to have small L2 norm.
2. 变换器通常使用权重衰减进行训练，这几乎与 L2 正则化相同。这鼓励学习到的权重具有较小的 L2 范数。

That means the model will try to "spread out" a computation across as many layers as possible (since the sum-of-squares is less than the square-of-sums). Given the task of turning an input into an output, the model will generally prefer changing the input a little, then a little more, then a little more, bit by bit.
这意味着模型将尽量在尽可能多的层之间“分散”计算（因为平方和小于和的平方）。在将输入转换为输出的任务中，模型通常会倾向于先稍微改变输入，然后再多一点，再多一点，一点一点地进行。

1+2 are a good story if you want to explain why the same vector basis is used across the network, and why things change smoothly. This story would render the whole thing unsurprising . . . except that the input is discarded in such a discontinuous way!
1+2 是一个不错的故事，如果你想解释为什么在整个网络中使用相同的向量基，以及为什么事物会平滑地变化。这个故事会使整个过程显得毫不意外……只是输入以如此不连续的方式被舍弃！

I would have expected a U-shaped pattern, where the early layers mostly look like the input, the late layers mostly look like the output, and there's a gradual "flip" in the middle between the two perspectives. Instead, the input space immediately vanishes, and we're in output space the whole way.
我原本预计会出现 U 型模式，其中早期层大多看起来像输入，晚期层大多看起来像输出，而在两种视角之间有一个渐进的“翻转”。然而，输入空间立即消失，我们始终处于输出空间中。

Maybe there is some math fact I'm missing here.
也许这里有一些我遗漏的数学事实。

Or, maybe there's some sort of "hidden" invertible relationship between
或者，也许存在某种“隐含”的可逆关系

• the embedding of a given token, and
  给定标记的嵌入，和
• the model's prior for what token comes after it (given no other information)
  该模型的先验知识，即在没有其他信息的情况下，预测其后接的标记

so that a token like "plasma" is kept around from the input -- but not in the form "the output is plasma," instead in the form "the output is [the kind of word that comes after plasma]."
因此，像“等离子体”这样的标记在输入中被保留——但不是以“输出是等离子体”的形式，而是以“输出是[在等离子体后面出现的词的种类]”的形式。

However, I'm not convinced by that story as stated. For one thing, GPT layers don't share their weights, so the mapping between these two spaces would have to be separately memorized by each layer, which seems costly. Additionally, if this were true, we'd expect the very early activations to look like naive context-less guesses for the next token. Often they are, but just as often they're weird nonsense like "Garland."
然而，我对这个故事并不信服。首先，GPT 层之间并不共享权重，因此这两个空间之间的映射必须由每一层单独记忆，这似乎是很耗费资源的。此外，如果这是真的，我们会期待非常早期的激活看起来像是对下一个标记的简单无上下文的猜测。它们往往是这样的，但同样常常也会是奇怪的无意义词汇，例如“花环”。

addendum: more on "input discarding"
附录：关于“输入丢弃”的更多信息

In comments, Gurkenglas noted that the plots showing KL(final || layer) don't tend the whole story.
在评论中，Gurkenglas 指出，显示 KL(final || layer)的图表并没有讲述整个故事。

The KL divergence is not a metric: it is not symmetric and does not obey the triangle inequality. Hence my intuitive picture of the distribution "jumping" from the input to the first layer, then smoothly converging to the final layer, is misleading: it implies we are measuring distances along a path through some space, but KL divergence does not measure distance in any space.
KL 散度不是一种度量：它不是对称的，并且不遵循三角不等式。因此，我对分布“跳跃”从输入层到第一层，然后平滑地收敛到最终层的直观想象是误导性的：它暗示我们正在沿着某个空间中的路径测量距离，但 KL 散度并不在任何空间中测量距离。

Gurkenglas and algon33 suggested plotting the KL divergences of everything w/r/t the input rather than the output: KL(input || layer).
Gurkenglas 和 algon33 建议绘制相对于输入而非输出的所有 KL 散度：KL(input || layer)。

Note that the input is close to a distribution that just assigns probability 1 to the input token ("close" because W * W^T is not invertible), so this is similar to asking "how probable is the input token, according to each layer?" That's a question which is also natural to answer by plotting ranks: what rank is assigned to the input token by each layer?
请注意，输入接近于一个仅为输入标记分配概率 1 的分布（“接近”是因为 W * W^T 是不可逆的），因此这类似于在询问“根据每一层，输入标记的概率有多大？”这是一个自然可以通过绘制排名来回答的问题：每一层给输入标记分配了什么排名？

Below, I show both: KL(input || layer), and the rank of the input token according to later layers.
下面，我展示了两者：KL（输入 || 层），以及根据后续层的输入标记的等级。

• For KL(input || layer), I use the same color scale as in the plots for KL(final || layer), so the two are comparable.
  对于 KL(input || layer)，我使用与 KL(final || layer) 图中相同的颜色尺度，因此二者是可比较的。
• For the ranks, I do not use the same color scale: I have the colors bottom out at rank 1000 instead of rank 100. This gives more visual insight into where the model could be preserving input information.
  对于排名，我不使用相同的颜色比例：我将颜色的底部设定在排名 1000 而不是排名 100。这提供了更多的视觉洞察，帮助理解模型可能在哪些地方保留输入信息。

• There is still a fast jump in KL(input || layer) after the input.
  输入后在 KL(input || layer)中仍然有一个快速跳转。
• However, it's far smaller than the jump in KL(output || layer) at the same point.
  然而，在同一点上，KL(output || layer)的增长要大得多。
• Note that the darkest color, meaning KL=30 does not appear on the plot of KL(input || layer).
  注意到最深的颜色，即 KL=30 并没有出现在 KL(输入 || 层) 的图中。
• On the plot of KL(output || layer), however, the maximum values were in fact much greater than 30; I cut off the color scale at 30 so other distinctions were perceptible at all.
  在 KL（输出 || 层）的图中，然而，最大值实际上远大于 30；我将颜色尺度截断在 30，以便其他区别能够清晰可见。
• Likewise, while ranks jump quickly after the input, they often stay relatively high in the context of a ~50K vocab.
  同样，在输入后排名迅速上升，但在大约 50K 词汇的情况下，它们通常保持相对较高的水平。
• I am curious about the differences here: some tokens are "preserved" much more in this sense than others.
  我对这里的差异很感兴趣：某些词元在这个意义上"保留"得比其他词元多得多。
• This is apparently contextual, not just based on the token itself. Note the stark differences between the rank trajectories of the first, second, and third commas in the passage.
  这显然是情境性的，而不仅仅是基于符号本身。请注意段落中第一、第二和第三个逗号的排名轨迹之间的显著差异。

It's possible that the relatively high ranks -- in the 100s or 1000s, but not the 10000s -- of input tokens in many cases is (related to) the mechanism by which the model "keeps around" rarer tokens in order to copy them later.
输入标记相对较高的排名——在 100 或 1000 范围内，但不超过 10000——在许多情况下可能与模型“保留”稀有标记的机制有关，以便后续复制。

As some evidence for this, I will show plots like the above for the plasma example. Here, I show a segment including the first instance of "plasma," rather than the second which copies it.
作为对此的一些证据，我将展示类似于上述的图形，针对等离子体的例子。在这里，我展示一个包含“等离子体”第一次出现的片段，而不是第二个重复它的片段。

The preservation of "plasma" here is striking.
这里“等离子体”的保存是显著的。

My intuitive guess is that the rarity, or (in some sense) "surprisingness," of the token causes early layers to preserve it: this would provide a mechanism for providing raw access to rare tokens in the later layers, which otherwise only be looking at more plausible tokens that GPT had guessed for the corresponding positions.
我直觉的猜测是，标记的稀有性，或者说在某种意义上的“出人意料”，导致早期层对此进行保留：这为后续层提供了一个机制，以原始方式访问稀有标记，而后者则只能查看 GPT 为相应位置猜测的更可信标记。

On the other hand, this story has trouble explaining why "G" and "PT" are not better preserved in the GPT3 abstract plots just above. This is the first instance of "GPT" in the full passage, so the model can't rely on copies of these at earlier positions. That said, my sense of scale for "well-preservedness" is a wild guess, and these particular metrics may not be ideal for capturing it anyway.
另一方面，这个故事很难解释为什么“G”和“PT”在上面的 GPT3 抽象图中没有得到更好的保留。这是全文中“GPT”的第一次出现，因此模型无法依赖于早期位置的这些复制品。尽管如此，我对“良好保留度”的尺度感知仅是个大胆的猜测，这些特定的指标可能也不适合捕捉这一点。

1. Right after this, positional embeddings are added. I'm ignoring positional embeddings in the post, but mention them in this footnote for accuracy. ↩︎
   紧接着，位置嵌入被添加。我在文中忽略了位置嵌入，但为了准确性在此脚注中提及。