Coherent Gradients: An Approach to
Understanding Generalization
in Gradient Descent-based Optimization
"Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization" 翻译为 "一致梯度：理解基于梯度下降优化的泛化的方法"

For our baseline, we use the standard MNIST dataset of 60,000 training examples and 10,000 test examples. Each example is a 28x28 pixel grayscale handwritten digit along with a label (‘0’–‘9’). We train a fully connected network on this dataset.
作为我们的基准，我们使用了标准的 MNIST 数据集，其中包括 60,000 个训练样本和 10,000 个测试样本。每个样本是一个 28x28 像素的灰度手写数字，附带一个标签（'0'-'9'）。我们在这个数据集上训练了一个全连接网络。
The network has one hidden layer with 2048 ReLUs and an output layer with a 10-way softmax. We initialize it with Xavier and train using vanilla SGD (i.e., no momentum) using cross entropy loss with a constant learning rate of 0.1 and a minibatch size of 100 for $10^{5}$ steps (i.e., about 170 epochs). We do not use any explicit regularizers.
网络有一个包含 2048 个 ReLU 的隐藏层和一个包含 10 个输出的 softmax 层。我们使用 Xavier 进行初始化，并使用交叉熵损失的普通 SGD（即没有动量），学习率为 0.1，小批量大小为 100，进行了 $10^{5}$ 步（大约 170 个 epochs）的训练。我们没有使用任何显式的正则化器。

We perturb the baseline by modifying only the dataset and keeping all other aspects of the architecture and learning algorithm fixed. The dataset is modified by adding various amounts of noise (25%, 50%, 75%, and 100%) to the labels of the training set (but not the test set). This noise is added by taking, say in the case of 25% label noise, 25% of the examples at random and randomly permuting their labels.
我们通过仅修改数据集并保持架构和学习算法的所有其他方面不变来扰动基准。数据集通过向训练集的标签添加不同数量的噪声（25%，50%，75% 和 100%）来进行修改（但不包括测试集）。这种噪声是通过随机选择，然后随机排列它们的标签来添加的，例如在 25% 标签噪声的情况下，随机选择 25% 的样本并随机排列它们的标签。
Thus, when we add 25% label noise, we still expect about 75% + 0.1 * 25%, i.e., 77.5% of the examples to have unchanged (i.e. “correct”) labels which we call the proper accuracy of the modified dataset. In what follows, we call examples with unchanged labels, pristine, and the remaining, corrupt. Also, from this perspective, it is convenient to refer to the original MNIST dataset as having 0% label noise.
因此，当我们添加 25%的标签噪声时，我们仍然期望大约 75% + 0.1 * 25%，即 77.5%的示例具有不变的（即“正确的”）标签，我们称之为修改后数据集的正确准确度。接下来，我们称具有不变标签的示例为原始的，其余的为损坏的。此外，从这个角度来看，方便将原始的 MNIST 数据集称为具有 0%标签噪声。

We use a fully connected architecture instead of a convolutional one to mitigate concerns that some of the difference in generalization between the original MNIST and the noisy variants could stem from architectural inductive bias. We restrict ourselves to only 1 hidden layer to have the gradients be as well-behaved as possible. Finally, the network width, learning rate, and the number of training steps are chosen to ensure that exactly the same procedure is usually able to fit all 5 variants to 100% training accuracy.
我们使用全连接的架构而不是卷积架构，以减轻原始 MNIST 和嘈杂变体之间一些泛化差异可能源自架构归纳偏差的担忧。我们限制自己只有 1 个隐藏层，以确保梯度尽可能地良好。最后，网络宽度、学习率和训练步数的选择是为了确保通常能够将所有 5 个变体都训练到 100%的准确度。

Before looking at the experimental results, it is useful to consider what Coherent Gradients can qualitatively say about this setup.
在查看实验结果之前，考虑一下相干梯度对这个设置有什么定性的影响是很有用的。
In going from 0% label noise to 100% label noise, as per experiment design, we expect examples in the training set to become more dissimilar (no matter what the current notion of similarity is). Therefore, we expect the per-example gradients to be less aligned with each other.
从 0%标签噪声到 100%标签噪声的过程中，根据实验设计，我们期望训练集中的示例变得更不相似（无论当前的相似性概念是什么）。因此，我们期望每个示例的梯度彼此之间的对齐度更低。
This in turn causes the overall gradient to become more diffuse, i.e., stronger directions become relatively weaker, and consequently, we expect it to take longer to reach a given level of accuracy as label noise increases, i.e., to have a lower realized learning rate.
这反过来导致整体梯度变得更加扩散，即更强的方向相对变得更弱，因此，我们期望随着标签噪声的增加，达到给定精度水平需要更长的时间，即实现较低的学习速率。

This can be made more precise by considering the following heuristic argument. Let $\theta_{t}$ be the vector of trainable parameters of the network at training step $t$. Let $\mathcal{L}$ denote the loss function of the network (over all training examples). Let $g_{t}$ be the gradient of $\mathcal{L}$ at $\theta_{t}$ and let $\alpha$ denote the learning rate. By Taylor expansion, to first order, the change $\Delta\mathcal{L}_{t}$ in the loss function due to a small gradient descent step $h_{t}=-\alpha\cdot g_{t}$ is given by
这可以通过考虑以下启发式论证来更加精确。让 $\theta_{t}$ 表示网络在训练步骤 $t$ 的可训练参数向量。让 $\mathcal{L}$ 表示网络的损失函数（针对所有训练样本）。让 $g_{t}$ 表示 $\mathcal{L}$ 在 $\theta_{t}$ 处的梯度，让 $\alpha$ 表示学习率。通过泰勒展开，一阶近似下，由于小梯度下降步骤 $h_{t}=-\alpha\cdot g_{t}$ 导致损失函数 $\Delta\mathcal{L}_{t}$ 的变化为。

where $\|\cdot\|$ denotes the $l_{2}$-norm. Now, let $g_{te}$ denote the gradient of training example $e$ at step $t$. Since the overall gradient is the sum of the per-example gradients, we have,
在这里，" $\|\cdot\|$ " 表示 $l_{2}$ 范数。现在，让 " $g_{te}$ " 表示在步骤 " $t$ " 上的训练样本 " $e$ " 的梯度。由于总体梯度是每个样本梯度的总和，我们有，

Now, heuristically, let us assume that all the $\|g_{te}\|$ are roughly the same and equal to $\|g_{t}^{\circ}\|$ which is not entirely unreasonable (at least at the start of training, if the network has no a priori reason to treat different examples very differently). If all the per-example gradients are approximately orthogonal (i.e., $\langle g_{te},g_{te^{\prime}}\rangle\approx 0$ for $e\neq e^{\prime}$), then $\|g_{t}\|^{2}\approx m\cdot\|g_{t}^{\circ}\|^{2}$ where $m$ is the number of examples. On the other hand, if they are approximately the same (i.e., $\langle g_{te},g_{te^{\prime}}\rangle\approx\|g_{t}^{\circ}\|^{2}$), then $\|g_{t}\|^{2}\approx m^{2}\cdot\|g_{t}^{\circ}\|^{2}$. Thus, we expect that greater the agreement in per-example gradients, the faster loss should decrease.
现在，从启发式的角度来看，让我们假设所有的 $\|g_{te}\|$ 大致相同，并且等于 $\|g_{t}^{\circ}\|$ 这在训练开始阶段是不完全不合理的（至少在网络没有先验理由对不同的示例进行非常不同的处理时）。如果所有的每个示例的梯度大致正交（即， $\langle g_{te},g_{te^{\prime}}\rangle\approx 0$ 对于 $e\neq e^{\prime}$ ），那么 $\|g_{t}\|^{2}\approx m\cdot\|g_{t}^{\circ}\|^{2}$ 其中 $m$ 是示例的数量。另一方面，如果它们大致相同（即， $\langle g_{te},g_{te^{\prime}}\rangle\approx\|g_{t}^{\circ}\|^{2}$ ），那么 $\|g_{t}\|^{2}\approx m^{2}\cdot\|g_{t}^{\circ}\|^{2}$ 。因此，我们期望每个示例梯度的一致性越大，损失下降得越快。

Finally, for datasets that have a significant fractions of pristine and corrupt examples (i.e., the 25%, 50%, and 75% noise) we can make a more nuanced prediction. Since, in those datasets, the pristine examples as a group are still more similar than the corrupt ones, we expect the pristine gradients to continue to align well and sum up to a strong gradient. Therefore, we expect them to be learned faster than the corrupt examples, and at a rate closer to the realized learning rate in the 0% label noise case. Likewise, we expect the realized learning rate on the corrupt examples to be closer to the 100% label noise case. Finally, as the proportion of pristine examples falls with increasing noise, we expect the realized learning rate for pristine examples to degrade.
最后，对于具有大量原始和损坏示例的数据集（即，25%，50% 和 75% 的噪声），我们可以做出更细致的预测。因为在这些数据集中，原始示例作为一组仍然比损坏示例更相似，我们期望原始梯度继续良好地对齐并汇总为强梯度。因此，我们期望它们学习速度比损坏示例更快，并且接近于 0% 标签噪声情况下的实际学习速率。同样，我们期望损坏示例上的实际学习速率接近于 100% 标签噪声情况。最后，随着原始示例比例随噪声增加而下降，我们期望原始示例的实际学习速率下降。

Note that this provides an explanation for the observation in the literature that that networks can learn even when the number of examples with noisy labels greatly outnumber the clean examples as long as the number of clean examples is sufficiently large (Rolnick et al., 2017) since with too few clean examples the pristine gradients are not strong enough to dominate.
请注意，这解释了文献中的观察结果，即只要干净示例的数量足够大（Rolnick 等人，2017），即使带有嘈杂标签的示例数量远远超过干净示例，网络也可以学习，因为干净示例太少时，原始梯度不足以占主导地位。

Figure 1(a) and (b) show the training and test curves for the baseline and the 4 variants. We note that for all 5 variants, at the end of training, we achieve 100% training accuracy but different amounts of generalization.
图1(a)和(b)展示了基线和4个变体的训练和测试曲线。我们注意到对于所有5个变体，在训练结束时，我们实现了100%的训练准确率，但是泛化程度不同。
As expected, SGD is able to fit random labels, yet when trained on real data, generalizes well. Figure 1(c) shows the reduction in training loss over the course of training, and Figure 1(d) shows the fraction of pristine and corrupt labels learned as training processes.
如预期，随机梯度下降（SGD）能够拟合随机标签，但在真实数据上训练时，能够很好地泛化。图1(c)显示了训练过程中训练损失的减少，图1(d)显示了在训练过程中学习的原始标签和错误标签的比例。

The results are in agreement with the qualitative predictions made above:
结果与上述定性预测一致：

1. 1.

   In general, as noise increases, the time taken to reach a given level of accuracy (i.e., realized learning rate) increases.

   
   1. 一般来说，随着噪声的增加，达到特定精度水平所需的时间（即实现的学习速率）也会增加。
2. 2.

   Pristine examples are learned faster than corrupt examples. They are learned at a rate closer to the 0% label noise rate whereas the corrupt examples are learned at a rate closer to the 100% label noise rate.

   
   2. 纯净的样本比损坏的样本学习得更快。它们的学习速率接近于 0%标签噪声率，而损坏的样本的学习速率接近于 100%标签噪声率。
3. 3.

   With fewer pristine examples, their learning rate reduces. This is most clearly seen in the first few steps of training by comparing say 0% noise with 25% noise.

   
   3. 随着纯净样本数量的减少，它们的学习速率也会降低。这在训练的最初几步中最为明显，比如将 0%噪声与 25%噪声进行比较。

Using Equation 1, note that the magnitude of the slope of the training loss curve is a good measure of the square of the $l_{2}$-norm of the overall gradient. Therefore, from the loss curves of Figure 1(c), it is clear that in early training, the more the noise, the weaker the $l_{2}$-norm of the gradient. If we assume that the per-example $l_{2}$-norm is the same in all variants at start of training, then from Equation 2, it is clear that with greater noise, the gradients are more dissimilar.
使用方程 1，注意训练损失曲线斜率的大小是整体梯度的二范数的良好度量。因此，从图 1(c)的损失曲线可以清楚地看出，在早期训练中，噪声越大，梯度的二范数越弱。如果我们假设每个示例的二范数在训练开始时在所有变体中是相同的，那么从方程 2 可以清楚地看出，噪声越大，梯度越不相似。

Finally, we note that this experiment is an instance where early stopping (e.g., Caruana et al. (2000)) is effective. Coherent gradients and the discussion in §2.2 provide some insight into this: Strong gradients both generalize well (they are stable since they are supported by many examples) and they bring the training loss down quickly for those examples. Thus early stopping maximizes the use of strong gradients and limits the impact of weak gradients. (The experiment in the §3 discusses a different way to limit the impact of weak gradients and is an interesting point of comparison with early stopping.)
最后，我们注意到这个实验是早停的一个例子（例如，Caruana 等人（2000））。连贯的梯度和第 2.2 节的讨论为此提供了一些见解：强梯度既具有良好的泛化能力（它们稳定，因为它们由许多示例支持），又能快速降低这些示例的训练损失。因此，早停最大限度地利用了强梯度，并限制了弱梯度的影响。（第 3 节的实验讨论了限制弱梯度影响的不同方法，与早停进行了有趣的比较。）

Within each noisy dataset, we expect the pristine examples to be more similar to each other and the corrupt ones to be less similar. In turn, based on the training curves (particularly, Figure 1 (d)), during the initial part of training, this should mean that the gradients from the pristine examples should be stronger than the gradients from the corrupt examples. We can study this effect via a different decomposition of square of the $l_{2}$-norm of the gradient (of equivalently upto a constant, the change in the loss function):
在每个嘈杂的数据集中，我们期望原始示例彼此更相似，而损坏示例则更不相似。因此，基于训练曲线（特别是图 1（d）），在训练的初始阶段，这意味着来自原始示例的梯度应该比来自损坏示例的梯度更强。我们可以通过对梯度的二次分解（或者等效地，损失函数的变化）来研究这种效应：

where $g_{t}^{p}$ and $g_{t}^{c}$ are the sum of the gradients of the pristine examples and corrupt examples respectively. (We cannot decompose the overall norm into a sum of norms of pristine and corrupt due to the cross terms $\langle g_{t}^{p},g_{t}^{c}\rangle$. With this decomposition, we attribute the cross terms equally to both.) Now, set $f_{t}^{p}=\frac{\langle g_{t},g_{t}^{p}\rangle}{<g_{t},g_{t}>}$ and $f_{t}^{c}=\frac{\langle g_{t},g_{t}^{c}\rangle}{\langle g_{t},g_{t}\rangle}$. Thus, $f_{t}^{p}$ and $f_{t}^{c}$ represent the fraction of the loss reduction due to pristine and corrupt at each time step respectively (and we have $f_{t}^{p}+f_{t}^{c}=1$), and based on the foregoing, we expect the pristine fraction to be a larger fraction of the total when training starts and to diminish as training progresses and the pristine examples are fitted.
其中 $g_{t}^{p}$ 和 $g_{t}^{c}$ 分别是原始示例和损坏示例的梯度之和。（由于交叉项 $\langle g_{t}^{p},g_{t}^{c}\rangle$ ，我们无法将总体范数分解为原始和损坏范数的和。通过这种分解，我们将交叉项平均分配给两者。）现在，设定 $f_{t}^{p}=\frac{\langle g_{t},g_{t}^{p}\rangle}{<g_{t},g_{t}>}$ 和 $f_{t}^{c}=\frac{\langle g_{t},g_{t}^{c}\rangle}{\langle g_{t},g_{t}\rangle}$ 。因此， $f_{t}^{p}$ 和 $f_{t}^{c}$ 分别代表每个时间步的原始和损坏造成的损失减少的比例（我们有 $f_{t}^{p}+f_{t}^{c}=1$ ），根据前述，我们期望在训练开始时原始部分占总体的比例更大，并且随着训练的进行和原始示例的拟合而减少。

The first row of Figure 2 shows a plot of estimates of $f_{t}^{p}$ and $f_{t}^{c}$ for 25%, 50% and 75% noise. These quantities were estimated by recording a sample of 400 per-example gradients for 600 weights (300 from each layer) in the network. We see that for 25% and 50% label noise, $f_{t}^{p}$ initially starts off higher than $f_{t}^{c}$ and after a few steps they cross over. This happens because at that point all the pristine examples have been fitted and for most of the rest of training the corrupt examples need to be fitted and so they largely contribute to the $l_{2}$-norm of the gradient (or equivalently by Equation 1 to loss reduction). Only at the end when the corrupt examples have also been fit, the two curves reach parity. In the case of 75% noise, we see that the cross over doesn’t happen, but there is a slight slope downwards for the contribution from pristine examples.
Figure 2的第一行显示了25%、50%和75%噪声下 $f_{t}^{p}$ 和 $f_{t}^{c}$ 的估计图。这些数量是通过记录网络中每个权重的400个样本梯度（每层300个）来估计的。我们看到，在25%和50%标签噪声下， $f_{t}^{p}$ 最初比 $f_{t}^{c}$ 高，几步之后它们交叉。这是因为在那一点上，所有原始示例都已经适应，而在大部分训练的其余时间里，需要适应损坏的示例，因此它们在很大程度上对梯度的 $l_{2}$ -范数（或者根据方程1等效地对损失减少）做出贡献。只有在最后，当损坏的示例也被适应时，这两条曲线才会达到平衡。在75%噪声的情况下，我们看到交叉不会发生，但是来自原始示例的贡献略微下降。
We believe this is because of the sheer number of corrupt examples, and so even though the individual corrupt example gradients are weak, their sum dominates.
我们认为这是因为腐败样本的数量庞大，因此即使单个腐败样本的梯度较弱，它们的总和也会占主导地位。

To get a sense of statistical significance in our hypothesis that there is a difference between the pristine and corrupt examples as a group, in the remaining rows of Figure 2, we construct a null world where there is no difference between pristine and corrupt. We do that by randomly permuting the “corrupt” and “pristine” designations among the examples (instead of using the actual designations) and reploting.
为了在我们的假设中获得统计显著性，即在图 2 的其余行中，我们构建一个空世界，在这个世界中，原始和腐败样本之间没有差异。我们通过在示例之间随机置换“腐败”和“原始”标识（而不是使用实际的标识）并重新绘制来实现这一点。
Although the null pristine and corrupt curves are mirror images (as they must be even in the null world since each example is given one of the two designations), we note that for 25% and 50% they do not cross over as they do with the real data. This increases our confidence that the null may be rejected.
尽管空的原始和腐败曲线是镜像（因为即使在空世界中，每个示例都被赋予两个标识，它们也必须是镜像的），但我们注意到在 25%和 50%时，它们不像真实数据那样交叉。这增加了我们对空假设可能被拒绝的信心。
The 75% case is weaker but only the real data shows the slight downward slope in pristine which none of the nulls typically show.
75% 的情况较弱，但只有真实数据显示了原始数据中微弱的下降趋势，而典型的空值数据中则没有显示出这种情况。
However, all the nulls do show that corrupt is more than pristine which increases our confidence that this is due to the significantly differing sizes of the two sets.
然而，所有的空值数据都显示了腐败数据多于原始数据，这增加了我们对于这是由于两组数据大小显著不同而引起的信心。
(Note that this happens in reverse in the 25% case: pristine is always above corrupt, but they never cross over in the null worlds.)
（请注意，这在 25% 的情况中是相反的：原始数据始终高于腐败数据，但在空值世界中它们永远不会交叉。）

To get a stronger signal for the difference between pristine and corrupt in the 75% case, we can look at a different statistic that adjusts for the different sizes of the pristine and corrupt sets. Let $|p|$ and $|c|$ be the number of pristine and corrupt examples respectively. Define
为了在 75%的情况下获得更强的原始和损坏之间的差异信号，我们可以查看一种不同的统计数据，该统计数据调整了原始和损坏集的不同大小。让 $|p|$ 和 $|c|$ 分别表示原始和损坏示例的数量。定义

which represents to a first order and upto a scale factor ($\alpha$) the mean cumulative contribution of a pristine or corrupt example up until that point in training (since the total change in loss from the start of training to time $t$ is approximately the sum of first order changes in the loss at each time step).
这代表了一个一阶和一个比例因子（ $\alpha$ ）的意义，即在训练过程中直到该点的一个原始或损坏示例的平均累积贡献（因为从训练开始到时间 $t$ 的总损失变化大约是在每个时间步的损失的一阶变化的总和）。

The first row of Figure 3 shows $i_{t}^{p}$ and $i_{t}^{c}$ for the first 10 steps of training where the difference between pristine and corrupt is the most pronounced.
图 3 的第一行显示了训练的前 10 步的 $i_{t}^{p}$ 和 $i_{t}^{c}$ ，在这里原始和损坏之间的差异最为显著。
As before, to give a sense of statistical significance, the remaining rows show the same plots in null worlds where we randomly permute the pristine or corrupt designations of the examples. The results appear somewhat significant but not overwhelmingly so.
与之前一样，为了给出统计显著性的感觉，其余行显示了在空白世界中的相同图，其中我们随机排列示例的原始或损坏标识。结果似乎有些显著，但并非压倒性。
It would be interesting to redo this on the entire population of examples and trainable parameters instead of a small sample.
这将是有趣的，如果能够在整个样本和可训练参数上重新进行，而不是在一个小样本上。

Our baseline setup is the same as before (§2.1) but we add a new dimension by modifying SGD to update each parameter with a “winsorized” gradient where we clip the most extreme values (outliers) among all the per-example gradients. Formally, let $g_{we}$ be the gradient for the trainable parameter $w$ for example $e$. The usual gradient computation for $w$ is
我们的基准设置与之前相同（§2.1），但我们通过修改 SGD 来添加一个新的维度，以使用“winsorized”梯度来更新每个参数，其中我们剪切所有每个示例梯度中的最极端值（离群值）。形式上，设 $g_{we}$ 为可训练参数 $w$ 在示例 $e$ 上的梯度。对于 $w$ 的常规梯度计算为

Now let $c\in[0,50]$ be a hyperparameter that controls the level of winsorization. Define $l_{w}$ to be the $c$-th percentile of $g_{we}$ taken over the examples. Similarly, let $u_{w}$ be the $(100-c)$-th percentile. Now, compute the $c$-winsorized gradient for $w$ (denoted by $g_{w}^{c}$) as follows:
现在，设 $c\in[0,50]$ 为控制 winsorization 程度的超参数。定义 $l_{w}$ 为在示例中取得的 $c$ 百分位数。类似地，设 $u_{w}$ 为 $(100-c)$ 百分位数。现在，计算 $w$ 的 $c$ -winsorized 梯度（用 $g_{w}^{c}$ 表示）如下：

The change to gradient descent is to simply use $g_{w}^{c}$ instead of $g_{w}$ when updating $w$ at each step.
对梯度下降的改变就是在每一步更新 $w$ 时，简单地使用 $g_{w}^{c}$ 而不是 $g_{w}$ 。

Note that although this is conceptually a simple change, it is computationally very expensive due to the need for per-example gradients. To reduce the computational cost we only use the examples in the minibatch to compute $l_{w}$ and $u_{w}$. Furthermore, instead of using 1 hidden layer of 2048 ReLUs, we use a smaller network with 3 hidden layers of 256 ReLUs each, and train for 60,000 steps (i.e., 100 epochs) with a fixed learning rate of 0.1. We train on the baseline dataset and the 4 noisy variants with $c\in\{0,1,2,4,8\}$. Since we have 100 examples in each minibatch, the value of $c$ immediately tells us how many outliers are clipped in each minibatch. For example, $c=2$ means the 2 largest and 2 lowest values of the per-example gradient are clipped (independently for each trainable parameter in the network), and $c=0$ corresponds to unmodified SGD.
请注意，尽管这在概念上是一个简单的改变，但由于需要每个示例的梯度，计算上非常昂贵。为了降低计算成本，我们只使用小批量中的示例来计算 $l_{w}$ 和 $u_{w}$ 。此外，我们不再使用 2048 个 ReLU 的 1 个隐藏层，而是使用 3 个隐藏层，每个隐藏层有 256 个 ReLU，并且使用固定学习率为 0.1 进行 60,000 步（即 100 个 epochs）的训练。我们在基准数据集和 4 个带有 $c\in\{0,1,2,4,8\}$ 的嘈杂变体上进行训练。由于每个小批量中有 100 个示例， $c$ 的值立即告诉我们每个小批量中剪切了多少异常值。例如， $c=2$ 表示剪切了每个可训练参数的梯度的 2 个最大值和 2 个最小值（对于网络中的每个可训练参数独立进行剪切）， $c=0$ 对应于未修改的 SGD。

If the Coherent Gradient hypothesis is right, then the strong gradients are responsible for making changes to the network that generalize well since they improve many examples simultaneously.
如果连贯梯度假设是正确的，那么强梯度负责对网络进行改变，从而使其具有良好的泛化能力，因为它们同时改进了许多示例。
On the other hand, the weak gradients lead to overfitting since they only improve a few examples. By winsorizing each coordinate, we suppress the most extreme values and thus ensure that a parameter is only updated in a manner that benefits multiple examples. Therefore:
另一方面，弱梯度会导致过拟合，因为它们只会改善少数示例。通过对每个坐标进行 winsorizing，我们抑制了最极端的值，从而确保参数只在有利于多个示例的情况下进行更新。因此：

• •

  Since $c$ controls which examples are considered extreme, the larger $c$ is, the less we expect the network to overfit.

  
  • 由于 $c$ 控制着哪些示例被认为是极端的， $c$ 越大，我们就越不希望网络过拟合。
• •

  But this also makes it harder for the network to fit the training data, and so we expect the training accuracy to fall as well.

  
  • 但这也使网络更难拟合训练数据，因此我们也预计训练准确度会下降。
• •

  Winsorization will not completely eliminate the weak directions. For example, for small values of $c$ we should still expect overfitting to happen over time though at a reduced rate since only the most egregious outliers are suppressed.

  
  • Winsorization 不能完全消除弱方向。例如，对于较小的 $c$ 值，我们仍然应该期望过拟合会随着时间的推移而发生，尽管发生的速率会降低，因为只有最严重的异常值被抑制。

The resulting training and test curves shown in Figure 4. The columns correspond to different amounts of label noise and the rows to different amounts of winsorization. In addition to the training and test accuracies (${\sf ta}$ and ${\sf va}$, respectively), we show the level of overfit which is defined as ${\sf ta}-[\epsilon\cdot\frac{1}{10}+(1-\epsilon)\cdot{\sf va}]$ to account for the fact that the test labels are not randomized.
图 4 中显示的训练和测试曲线。列对应不同标签噪声的数量，行对应不同的 winsorization 数量。除了训练和测试准确率（分别为 ${\sf ta}$ 和 ${\sf va}$ ），我们还展示了过拟合水平，定义为 ${\sf ta}-[\epsilon\cdot\frac{1}{10}+(1-\epsilon)\cdot{\sf va}]$ ，以考虑测试标签未随机化的事实。

We see that the experimental results are in agreement with the predictions above. In particular,
我们发现实验结果与上述预测一致。特别是，

• •

  For $c>1$, training accuracies do not exceed the proper accuracy of the dataset, though they may fall short, specially for large values of $c$.

  
  • 对于 $c>1$ ，训练准确率不会超过数据集的正确准确率，尽管它们可能会稍微不足，特别是在 $c$ 的大值情况下。
• •

  The rate at which the overfit curve grows goes down with increasing $c$.

  
  • 过拟合曲线增长速度随着 $c$ 的增加而下降。

Additionally, we notice that with a large amount of winsorization, the training and test accuracies reach a maximum and then go down. Part of the reason is that as a result of winsorization, each step is no longer in a descent direction, i.e., this is no longer gradient descent.
另外，我们注意到大量修剪后，训练和测试准确率达到最大值，然后下降。部分原因是由于修剪的结果，每一步不再是下降方向，即不再是梯度下降。