Measuring the Intrinsic Dimension
of Objective Landscapes
测量客观景观的内禀维度

The above example had a simple enough form that we obtained $d_{\mathrm{int}}=10$ by calculation. But in general we desire a method to measure or approximate $d_{\mathrm{int}}$ for more complicated problems, including problems with data-dependent objective functions, e.g. neural network training. Random subspace optimization provides such a method.
上述例子形式足够简单，我们通过计算得到了 $d_{\mathrm{int}}=10$ 。但在一般情况下，我们希望有一种方法来测量或近似更复杂问题中的 $d_{\mathrm{int}}$ ，包括具有数据相关目标函数的问题，例如神经网络训练。随机子空间优化提供了一种这样的方法。

Standard optimization, which we will refer to hereafter as the direct method of training, entails evaluating the gradient of a loss with respect to $\theta^{(D)}$ and taking steps directly in the space of $\theta^{(D)}$. To train in a random subspace, we instead define $\theta^{(D)}$ in the following way:
标准优化，我们在此之后将其称为直接训练方法，包括评估损失相对于 $\theta^{(D)}$ 的梯度，并在 $\theta^{(D)}$ 的空间中直接采取步骤。要在随机子空间中进行训练，我们定义 $\theta^{(D)}$ 如下：

where $P$ is a randomly generated $D\times d$ projection matrix¹¹1This projection matrix can take a variety of forms each with different computational considerations. In later sections we consider dense, sparse, and implicit $P$ matrices.
这个投影矩阵可以有多种形式，每种形式都有不同的计算考虑。在后面的章节中，我们考虑了稠密、稀疏和隐式 $P$ 矩阵。
$P$ 是随机生成的 $D\times d$ 投影矩阵 ¹ and $\theta^{(d)}$ is a parameter vector in a generally smaller space $\mathbb{R}^{d}$. $\theta^{(D)}_{0}$ and $P$ are randomly generated and frozen (not trained), so the system has only $d$ degrees of freedom. We initialize $\theta^{(d)}$ to a vector of all zeros, so initially $\theta^{(D)}=\theta^{(D)}_{0}$. This convention serves an important purpose for neural network training: it allows the network to benefit from beginning in a region of parameter space designed by any number of good initialization schemes (Glorot & Bengio, 2010; He et al., 2015) to be well-conditioned, such that gradient descent via commonly used optimizers will tend to work well.²²2A second, more subtle reason to start away from the origin and with a randomly oriented subspace is that this puts the subspace used for training and the solution set in general position with respect to each other. Intuitively, this avoids pathological cases where both the solution set and the random subspace contain structure oriented around the origin or along axes, which could bias toward non-intersection or toward intersection, depending on the problem.
第二个、更为微妙的原因是，从原点开始并以随机方向选择子空间，这样做可以使用于训练的子空间和一般解集相互处于一般位置。直观上，这避免了病态情况，即解集和随机子空间都包含围绕原点或沿轴的结构，这可能会根据问题偏向非交集或偏向交集。
并且 $\theta^{(d)}$ 是一个在通常更小的空间 $\mathbb{R}^{d}$ 中的参数向量。 $\theta^{(D)}_{0}$ 和 $P$ 是随机生成并冻结（未训练）的，因此系统只有 $d$ 个自由度。我们将 $\theta^{(d)}$ 初始化为零向量，所以最初 $\theta^{(D)}=\theta^{(D)}_{0}$ 。这种约定对于神经网络训练具有重要意义：它允许网络从任何数量的良好初始化方案（Glorot & Bengio，2010；He 等，2015）设计的参数空间区域开始，以便条件良好，这样通过常用优化器进行的梯度下降通常会工作得很好。 ²

Training proceeds by computing gradients with respect to $\theta^{(d)}$ and taking steps in that space. Columns of $P$ are normalized to unit length, so steps of unit length in $\theta^{(d)}$ chart out unit length motions of $\theta^{(D)}$. Columns of $P$ may also be orthogonalized if desired, but in our experiments we relied simply on the approximate orthogonality of high dimensional random vectors. By this construction $P$ forms an approximately orthonormal basis for a randomly oriented $d$ dimensional subspace of $\mathbb{R}^{D}$, with the origin of the new coordinate system at $\theta^{(D)}_{0}$. Fig. 1 (left and middle) shows an illustration of the related vectors.
通过计算相对于 $\theta^{(d)}$ 的梯度并在该空间中采取步骤来推进训练。 $P$ 的列被归一化到单位长度，因此在 $\theta^{(d)}$ 图中的单位长度步骤描绘了 $\theta^{(D)}$ 的单位长度运动。如果需要， $P$ 的列也可以正交化，但在我们的实验中，我们仅仅依赖于高维随机向量的近似正交性。通过这种构造， $P$ 形成了一个对于随机取向的 $d$ 维 $\mathbb{R}^{D}$ 子空间的近似正交基，新坐标系的起点在 $\theta^{(D)}_{0}$ 。图 1（左侧和中间）展示了相关向量的示意图。

Consider a few properties of this training approach. If $d=D$ and $P$ is a large identity matrix, we recover exactly the direct optimization problem. If $d=D$ but $P$ is instead a random orthonormal basis for all of $\mathbb{R}^{D}$ (just a random rotation matrix), we recover a rotated version of the direct problem. Note that for some “rotation-invariant” optimizers, such as SGD and SGD with momentum, rotating the basis will not change the steps taken nor the solution found, but for optimizers with axis-aligned assumptions, such as RMSProp (Tieleman & Hinton, 2012) and Adam (Kingma & Ba, 2014), the path taken through $\theta^{(D)}$ space by an optimizer will depend on the rotation chosen. Finally, in the general case where $d<D$ and solutions exist in $D$, solutions will almost surely (with probability 1) not be found if $d$ is less than the codimension of the solution. On the other hand, when $d\geq D-s$, if the solution set is a hyperplane, the solution will almost surely intersect the subspace, but for solution sets of arbitrary topology, intersection is not guaranteed. Nonetheless, by iteratively increasing $d$, re-running optimization, and checking for solutions, we obtain one estimate of $d_{\mathrm{int}}$. We try this sweep of $d$ for our toy problem laid out in the beginning of this section, measuring (by convention as described in the next section) the positive performance (higher is better) instead of loss.³³3For this toy problem we define $\mathrm{performance}=\exp({-\mathrm{loss}})$, bounding performance between 0 and 1, with 1 being a perfect solution.
为此玩具问题，我们定义 $\mathrm{performance}=\exp({-\mathrm{loss}})$ ，将性能限制在 0 到 1 之间，其中 1 代表完美解。
考虑这种训练方法的一些性质。如果 $d=D$ 和 $P$ 是一个大的单位矩阵，我们就能恢复出精确的直接优化问题。如果 $d=D$ 但 $P$ 是一个随机正交基，对所有 $\mathbb{R}^{D}$ （只是一个随机旋转矩阵），我们就能恢复出直接问题的旋转版本。注意，对于一些“旋转不变”的优化器，如 SGD 和带有动量的 SGD，旋转基不会改变采取的步骤或找到的解，但对于具有轴对齐假设的优化器，如 RMSProp（Tieleman & Hinton，2012）和 Adam（Kingma & Ba，2014），优化器在 $\theta^{(D)}$ 空间中采取的路径将取决于选择的旋转。最后，在一般情况中，如果 $d<D$ 和解存在于 $D$ ，当 $d$ 小于解的余维数时，几乎肯定（概率为 1）找不到解。另一方面，当 $d\geq D-s$ ，如果解集是一个超平面，解几乎肯定会与子空间相交，但对于任意拓扑的解集，相交并不保证。尽管如此，通过迭代增加 $d$ ，重新运行优化并检查解，我们获得 $d_{\mathrm{int}}$ 的一个估计值。 我们尝试对本节开头提出的玩具问题进行 $d$ 的扫描，按照下一节所述的惯例测量正性能（越高越好），而不是损失。 ³ As expected, the solutions are first found at $d=10$ (see Fig. 1, right), confirming our intuition that for this problem, $d_{\mathrm{int}}=10$.
如预期，解决方案首先在 $d=10$ 找到（见图 1 右侧），证实了我们的直觉，对于这个问题， $d_{\mathrm{int}}=10$ 。

In the rest of this paper, we measure intrinsic dimensions for particular neural network problems and draw conclusions about the associated objective landscapes and solution sets. Because modeling real data is more complex than the above toy example, and losses are generally never exactly zero, we first choose a heuristic for classifying points on the objective landscape as solutions vs. non-solutions. The heuristic we choose is to threshold network performance at some level relative to a baseline model, where generally we take as baseline the best directly trained model. In supervised classification settings, validation accuracy is used as the measure of performance, and in reinforcement learning scenarios, the total reward (shifted up or down such that the minimum reward is 0) is used. Accuracy and reward are preferred to loss to ensure results are grounded to real-world performance and to allow comparison across models with differing scales of loss and different amounts of regularization included in the loss.
在本文的其余部分，我们测量特定神经网络问题的内在维度，并就相关的目标景观和解集得出结论。由于对真实数据进行建模比上述玩具示例更复杂，并且损失通常永远不会精确为零，我们首先选择一种启发式方法来将目标景观上的点分类为解或非解。我们选择的启发式方法是相对于基线模型在某个水平上阈值化网络性能，其中我们通常将最佳直接训练模型作为基线。在监督分类设置中，验证准确率被用作性能的衡量标准，而在强化学习场景中，总奖励（向上或向下调整，使得最小奖励为 0）被使用。准确率和奖励比损失更受欢迎，以确保结果基于现实世界的性能，并允许在不同损失规模和损失中包含不同量正则化的模型之间进行比较。

We define $d_{\mathrm{int100}}$ as the intrinsic dimension of the “100%” solution: solutions whose performance is statistically indistinguishable from baseline solutions. However, when attempting to measure $d_{\mathrm{int100}}$, we observed it to vary widely, for a few confounding reasons: $d_{\mathrm{int100}}$ can be very high — nearly as high as $D$ — when the task requires matching a very well-tuned baseline model, but can drop significantly when the regularization effect of restricting parameters to a subspace boosts performance by tiny amounts. While these are interesting effects, we primarily set out to measure the basic difficulty of problems and the degrees of freedom needed to solve (or approximately solve) them rather than these subtler effects.
我们定义 $d_{\mathrm{int100}}$ 为“100%”解决方案的内在维度：其性能在统计学上与基线解决方案无法区分的解决方案。然而，当试图测量 $d_{\mathrm{int100}}$ 时，我们发现它变化很大，原因有几个： $d_{\mathrm{int100}}$ 可以非常高——几乎与 $D$ 相当——当任务需要匹配一个非常调优的基线模型时，但可以在将参数限制到子空间以微小幅度提升性能的正则化效应下显著下降。虽然这些是有趣的效果，但我们主要的目标是测量问题的基本难度以及解决（或近似解决）它们所需的自由度，而不是这些更微妙的效果。

Thus, we found it more practical and useful to define and measure $d_{\mathrm{int90}}$ as the intrinsic dimension of the “90%” solution: solutions with performance at least 90% of the baseline.
因此，我们认为将 $d_{\mathrm{int90}}$ 定义为“90%”解决方案的内禀维度更为实用和有用：即性能至少达到基准线 90%的解决方案。

We chose 90% after looking at a number of dimension vs. performance plots (e.g. Fig. 2) as a reasonable trade off between wanting to guarantee solutions are as good as possible, but also wanting measured $d_{\mathrm{int}}$ values to be robust to small noise in measured performance. If too high a threshold is used, then the dimension at which performance crosses the threshold changes a lot for only tiny changes in accuracy, and we always observe tiny changes in accuracy due to training noise.
我们观察了多个维度与性能的对比图（例如图 2），选择了 90%作为在保证解尽可能好的同时，也希望测量的 $d_{\mathrm{int}}$ 值对测量性能中的微小噪声具有鲁棒性的合理折中。如果使用过高的阈值，那么性能超过阈值的维度会因为精度的小幅度变化而发生很大变化，而我们总是观察到由于训练噪声导致的精度微小变化。

If a somewhat different (higher or lower) threshold were chosen, we expect most of conclusions in the rest of the paper to remain qualitatively unchanged. In the future, researchers may find it useful to measure $d_{\mathrm{int}}$ using higher or lower thresholds.
如果选择一个略有不同（更高或更低）的阈值，我们预计本文其余部分的结论在定性上保持不变。未来，研究人员可能会发现使用更高或更低的阈值来测量 $d_{\mathrm{int}}$ 是有用的。

We begin by analyzing a fully connected (FC) classifier trained on MNIST. We choose a network with layer sizes 784–200–200–10, i.e. a network with two hidden layers of width 200; this results in a total number of parameters $D=199,210$. A series of experiments with gradually increasing subspace dimension $d$ produce monotonically increasing performances, as shown in Fig. 2 (left). By checking the subspace dimension at which performance crosses the 90% mark, we measure this network’s intrinsic dimension $d_{\mathrm{int90}}$ at about 750.
我们从分析在 MNIST 上训练的全连接（FC）分类器开始。我们选择了一个具有 784-200-200-10 层大小的网络，即具有两个宽度为 200 的隐藏层；这导致参数总数为 $D=199,210$ 。一系列随着子空间维度逐渐增加的实验产生了单调递增的性能，如图 2（左）所示。通过检查性能超过 90%的子空间维度，我们测量出这个网络的内在维度 $d_{\mathrm{int90}}$ 约为 750。

We scale to larger supervised classification problems by considering CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Russakovsky et al., 2015). When scaling beyond MNIST-sized networks with $D$ on the order of 200k and $d$ on the order of 1k, we find it necessary to use more efficient methods of generating and projecting from random subspaces. This is particularly true in the case of ImageNet, where the direct network can easily require millions of parameters. In Sec. S7, we describe and characterize scaling properties of three methods of projection: dense matrix projection, sparse matrix projection (Li et al., 2006), and the remarkable Fastfood transform (Le et al., 2013). We generally use the sparse projection method to train networks on CIFAR-10 and the Fastfood transform for ImageNet.
我们通过考虑 CIFAR-10（Krizhevsky & Hinton，2009）和 ImageNet（Russakovsky 等人，2015）来扩展到更大的监督分类问题。当扩展到 MNIST 大小的网络， $D$ 大约为 200k， $d$ 大约为 1k 时，我们发现使用更高效的从随机子空间生成和投影的方法是必要的。这在 ImageNet 的情况下尤其如此，直接的网络可能需要数百万个参数。在第 S7 节中，我们描述并分析了三种投影方法的扩展特性：密集矩阵投影、稀疏矩阵投影（Li 等人，2006）以及令人瞩目的 Fastfood 变换（Le 等人，2013）。我们通常使用稀疏投影方法在 CIFAR-10 上训练网络，以及 Fastfood 变换用于 ImageNet。

Measured $d_{\mathrm{int90}}$ values for CIFAR-10 and are ImageNet given in Table 1, next to all previous MNIST results and RL results to come. For CIFAR-10 we find qualitatively similar results to MNIST, but with generally higher dimension (9k vs. 750 for FC and 2.9k vs. 290 for LeNet). It is also interesting to observe the difference of $d_{\mathrm{int90}}$ across network architectures. For example, to achieve a global $>\!\!50\%$ validation accuracy on CIFAR-10, FC, LeNet and ResNet approximately requires $d_{\mathrm{int90}}=9$k, $2.9$k and $1$k, respectively, showing that ResNets are more efficient. Full results and experiment details are given in Sec. S8 and Sec. S9. Due to limited time and memory issues, training on ImageNet has not yet given a reliable estimate for $d_{\mathrm{int90}}$ except that it is over 500k.
测量了 CIFAR-10 的 $d_{\mathrm{int90}}$ 值和 ImageNet 的值，如表 1 所示，紧邻所有之前的 MNIST 结果和 RL 结果。对于 CIFAR-10，我们发现与 MNIST 的结果在定性上相似，但通常维度更高（FC 为 9k 比 750，LeNet 为 2.9k 比 290）。观察不同网络架构之间的 $d_{\mathrm{int90}}$ 差异也很有趣。例如，为了在 CIFAR-10 上实现全局 $>\!\!50\%$ 验证准确率，FC、LeNet 和 ResNet 分别大约需要 $d_{\mathrm{int90}}=9$ k、 $2.9$ k 和 $1$ k，这表明 ResNets 更有效率。完整结果和实验细节见第 S8 节和第 S9 节。由于时间和内存限制，ImageNet 上的训练尚未为 $d_{\mathrm{int90}}$ 提供一个可靠的估计，除了它超过 500k。

Measuring intrinsic dimension allows us to perform some comparison across the divide between supervised learning and reinforcement learning. In this section we measure the intrinsic dimension of three control tasks of varying difficulties using both value-based and policy-based algorithms. The value-based algorithm we evaluate is the Deep Q-Network (DQN) (Mnih et al., 2013), and the policy-based algorithm is Evolutionary Strategies (ES) (Salimans et al., 2017). Training details are given in Sec. S6.2. For all tasks, performance is defined as the maximum-attained (over training iterations) mean evaluation reward (averaged over 30 evaluations for a given parameter setting). In Fig. 5, we show results of ES on three tasks: $\mathtt{InvertedPendulum\!-\!v1}$, $\mathtt{Humanoid\!-\!v1}$ in MuJoCo (Todorov et al., 2012), and $\mathtt{Pong\!-\!v0}$ in Atari. Dots in each plot correspond to the (noisy) median of observed performance values across many runs for each given $d$, and the vertical uncertainty bar shows the maximum and minimum observed performance values. The dotted horizontal line corresponds to the usual 90% baseline derived from the best directly-trained network (the solid horizontal line). A dot is darkened signifying the first $d$ that allows a satisfactory performance. We find that the inverted pendulum task is surprisingly easy, with $d_{\mathrm{int100}}=d_{\mathrm{int90}}=4$, meaning that only four parameters are needed to perfectly solve the problem (see Stanley & Miikkulainen (2002) for a similarly small solution found via evolution). The walking humanoid task is more difficult: solutions are found reliably by dimension 700, a similar complexity to that required to model MNIST with an FC network, and far less than modeling CIFAR-10 with a convnet. Finally, to play Pong on Atari (directly from pixels) requires a network trained in a 6k dimensional subspace, making it on the same order of modeling CIFAR-10. For an easy side-by-side comparison we list all intrinsic dimension values found for all problems in Table 1. For more complete ES results see Sec. S6.2, and Sec. S6.1 for DQN results.
测量内在维度使我们能够在监督学习和强化学习之间进行一些比较。在本节中，我们使用基于价值和基于策略的算法来测量三个难度不同的控制任务的内在维度。我们评估的基于价值的算法是深度 Q 网络（DQN）（Mnih 等人，2013 年），而基于策略的算法是进化策略（ES）（Salimans 等人，2017 年）。训练细节见第 S6.2 节。对于所有任务，性能定义为训练迭代中达到的最大平均评估奖励（对于给定的参数设置，平均 30 次评估）。在图 5 中，我们展示了 ES 在三个任务上的结果： $\mathtt{InvertedPendulum\!-\!v1}$ 在 MuJoCo（Todorov 等人，2012 年）中， $\mathtt{Humanoid\!-\!v1}$ 在 Atari 中。每个图中的点对应于多次运行中观察到的性能值的（有噪声的）中位数，垂直不确定性条表示观察到的最大和最小性能值。虚线水平线对应于从最佳直接训练网络得出的常规 90%基线（实线水平线）。 一个点被加深表示第一个允许满意性能的 $d$ 。我们发现倒立摆任务出奇地简单， $d_{\mathrm{int100}}=d_{\mathrm{int90}}=4$ 意味着只需要四个参数就能完美解决问题（参见 Stanley & Miikkulainen (2002)中通过进化找到的类似的小规模解决方案）。行走类人任务更困难：通过 700 维找到可靠解决方案，与用全连接网络模拟 MNIST 的复杂度相似，远低于用卷积网络模拟 CIFAR-10。最后，要在 Atari 上玩 Pong（直接从像素中）需要在一个 6k 维子空间中训练的网络，这使得它与模拟 CIFAR-10 的复杂度相当。为了方便对比，我们在表 1 中列出所有问题的内在维度值。更完整的 ES 结果见 S6.2 节，DQN 结果见 S6.1 节。

In the main paper, we attempted to find $d_{\mathrm{int90}}$ across 20 FC networks with various depths and widths. A grid sweep of number of hidden layers from {1,2,3,4,5} and width of each hidden layer from {50,100,200,400} is performed, and all 20 plots are shown in Fig. S6. For each $d$ we take 3 runs and plot the mean and variance with blue dots and blue error bars. $d_{\mathrm{int90}}$ is indicated in plots (darkened blue dots) by the dimension at which the median of the 3 runs passes 90% performance threshold. The variance of $d_{\mathrm{int90}}$ is estimated using 50 bootstrap samples. Note that the variance of both accuracy and measured $d_{\mathrm{int90}}$ for a given hyper-parameter setting are generally small, and the mean of performance monotonically increases (very similar to the single-run result) as $d$ increases. This illustrates that the difference between lucky vs. unlucky random projections have little impact on the quality of solutions, while the subspace dimensionality has a great impact. We hypothesize that the variance due to different $P$ matrices will be smaller than the variance due to different random initial parameter vectors $\theta^{(D)}_{0}$ because there are $dD$ i.i.d. samples used to create $P$ (at least in the dense case) but only $D$ samples used to create $\theta^{(D)}_{0}$, and aspects of the network depending on smaller numbers of random samples will exhibit greater variance. Hence, in some other experiments we rely on single runs to estimate the intrinsic dimension, though slightly more accurate estimates could be obtained via multiple runs.
在主论文中，我们尝试在 20 个具有不同深度和宽度的 FC 网络上寻找 $d_{\mathrm{int90}}$ 。对隐藏层数量从{1,2,3,4,5}和每个隐藏层宽度从{50,100,200,400}进行网格扫描，所有 20 个图均显示在图 S6 中。对于每个 $d$ ，我们进行 3 次运行，并用蓝色点表示均值和蓝色误差线绘制。 $d_{\mathrm{int90}}$ 通过 3 次运行的中位数超过 90%性能阈值的维度在图中表示（深蓝色点）。 $d_{\mathrm{int90}}$ 的方差使用 50 个自举样本估计。请注意，对于给定的超参数设置，准确性和测量的 $d_{\mathrm{int90}}$ 的方差通常很小，随着 $d$ 的增加，性能均值单调递增（与单次运行结果非常相似）。这表明幸运与不幸的随机投影之间的差异对解的质量影响很小，而子空间维度有重大影响。我们假设由于不同的 $P$ 矩阵引起的方差将小于由于不同的随机初始参数向量 $\theta^{(D)}_{0}$ 引起的方差，因为存在 $dD$ 独立同分布的。 样本用于创建 $P$ （至少在密集情况下），但仅使用 $D$ 样本创建 $\theta^{(D)}_{0}$ ，网络的一些方面依赖于更少的随机样本，将表现出更大的方差。因此，在一些其他实验中，我们依赖于单次运行来估计内在维度，尽管通过多次运行可以获得略微更准确的估计。

In similar manner to the above, in Fig. S7 we show the relationship between $d_{\mathrm{int90}}$ and $D$ across 20 networks but using a per-model, directly trained baseline. Most baselines are slightly below 100% accuracy. This is in contrast to Fig. 3, which used a simpler global baseline of 100% across all models. Results are qualitatively similar but with slightly lower intrinsic dimension due to slightly lower thresholds.
与上述类似，在图 S7 中，我们展示了 20 个网络中 $d_{\mathrm{int90}}$ 与 $D$ 之间的关系，但使用的是每个模型直接训练的基线。大多数基线的准确率略低于 100%。这与图 3 形成对比，图 3 使用了一个更简单的 100%的全球基线。结果在定性上相似，但由于阈值略低，内在维度略低。

Two kinds of shuffled MNIST datasets are considered:
两种打乱顺序的 MNIST 数据集被考虑：

• •

  The shuffled pixel dataset: the label for each example remains the same as the normal dataset, but a random permutation of pixels is chosen once and then applied to all images in the training and test sets. FC networks solve the shuffled pixel datasets exactly as easily as the base dataset, because there is no privileged ordering of input dimension in FC networks; all orderings are equivalent.

  
  • 混洗像素数据集：每个示例的标签与正常数据集相同，但选择一次随机的像素排列，然后应用于训练集和测试集中的所有图像。全连接（FC）网络可以像处理基本数据集一样轻松地解决混洗像素数据集，因为在全连接网络中没有输入维度的特权顺序；所有排列都是等效的。
• •

  The shuffled label dataset: the images remain the same as the normal dataset, but labels are randomly shuffled for the entire training set. Here, as in (Zhang et al., 2017), we only evaluate training accuracy, as test set accuracy remains forever at chance level (the training set $X$ and $y$ convey no information about test set $p(y|X)$, because the shuffled relationship in test is independent of that of training).

  
  • 混洗标签数据集：图像与正常数据集相同，但标签在整个训练集中随机打乱。在此，如同(Zhang 等人，2017 年)中所述，我们只评估训练准确率，因为测试集准确率始终保持在随机水平（训练集 $X$ 和 $y$ 无法提供关于测试集 $p(y|X)$ 的信息，因为测试集中的混洗关系与训练集中的关系是独立的）。

On the full shuffled label MNIST dataset (50k images), we trained an FC network ($L=5,W=400$, which had $d_{\mathrm{int90}}=750$ on standard MNIST), it yields $d_{\mathrm{int90}}=190$k. We can interpret this as requiring 3.8 floats to memorize each random label (at 90% accuracy). Wondering how this scales with dataset size, we estimated $d_{\mathrm{int90}}$ on shuffled label versions of MNIST at different scales and found curious results, shown in Table S2 and Fig. S8. As the dataset memorized becomes smaller, the number of floats required to memorize each label becomes larger. Put another way, as dataset size increases, the intrinsic dimension also increases, but not as fast as linearly. The best interpretation is not yet clear, but one possible interpretation is that networks required to memorize large training sets make use of shared machinery for memorization. In other words, though performance does not generalize to a validation set, generalization within a training set is non-negligible even though labels are random.
在完整的打乱标签的 MNIST 数据集（50k 张图像）上，我们训练了一个全连接网络（ $L=5,W=400$ ，在标准 MNIST 上的 $d_{\mathrm{int90}}=750$ ），它产生了 $d_{\mathrm{int90}}=190$ k。我们可以将其解释为需要 3.8 个浮点数来记忆每个随机标签（90%的准确率）。想知道这如何随着数据集大小而变化，我们在不同规模的 MNIST 打乱标签版本上估计了 $d_{\mathrm{int90}}$ ，并发现了有趣的结果，如表 S2 和图 S8 所示。随着记忆的数据集变小，记忆每个标签所需的浮点数也变大。换句话说，随着数据集大小的增加，内在维度也增加，但增加的速度并不像线性那样快。最佳解释尚不明确，但一种可能的解释是，用于记忆大型训练集的网络利用了记忆的共享机制。换句话说，尽管性能不能推广到验证集，但即使在标签随机的情况下，训练集内的泛化也是不可忽视的。

An interesting tangential observation is that random subspace training can in some cases make optimization more stable. First, it helps in the case of deeper networks. Fig. S9 shows training results for FC networks with up to 10 layers. SGD with step 0.1, and ReLUs with He initialization is used. Multiple networks failed at depths 4, and all failed at depths higher than 4, despite the activation function and initialization designed to make learning stable (He et al., 2015). Second, for MNIST with shuffled labels, we noticed that it is difficult to reach high training accuracy using the direct training method with SGD, though both subspace training with SGD and either type of training with Adam reliably reach 100% memorization as $d$ increases (see Fig. S8).
一个有趣的旁征博引的观察是，在某些情况下，随机子空间训练可以使优化更加稳定。首先，它有助于深层网络的情况。图 S9 显示了具有多达 10 层的 FC 网络的训练结果。使用步长为 0.1 的 SGD 和 He 初始化的 ReLUs。多个网络在深度 4 处失败，并且所有网络在深度大于 4 时都失败了，尽管激活函数和初始化是为了使学习稳定（He 等人，2015）。其次，对于打乱标签的 MNIST，我们注意到，尽管使用 SGD 的直接训练方法难以达到高训练精度，但使用 SGD 的子空间训练和 Adam 的任何一种训练类型随着 $d$ 的增加都能可靠地达到 100%的记忆（见图 S8）。

Because each random basis vector projects across all $D$ direct parameters, the optimization problem may be far better conditioned in the subspace case than in the direct case. A related potential downside is that projecting across $D$ parameters which may have widely varying scale could result in ignoring parameter dimensions with tiny gradients. This situation is similar to that faced by methods like SGD, but ameliorated by RMSProp, Adam, and other methods that rescale per-dimension step sizes to account for individual parameter scales. Though convergence of the subspace approach seems robust, further work may be needed to improve network amenability to subspace training: for example by ensuring direct parameters are similarly scaled by clever initialization or by inserting a pre-scaling layer between the projected subspace and the direct parameters themselves.
因为每个随机基向量都投影到所有 0#直接参数上，所以在子空间情况下，优化问题可能比直接情况下条件要好得多。相关潜在缺点是，投影到可能具有广泛不同尺度的 1#参数可能会导致忽略具有微小梯度的参数维度。这种情况类似于 SGD 等方法所面临的情况，但通过 RMSProp、Adam 等方法得到了改善，这些方法通过按维度调整步长来考虑单个参数的尺度。尽管子空间方法的收敛性看起来很稳健，但可能还需要进一步的工作来提高网络对子空间训练的适应性：例如，通过通过巧妙的初始化确保直接参数具有相似的尺度，或者在投影子空间和直接参数本身之间插入一个预缩放层。

Another finding through our experiments with MNIST FC networks has to do with the role of optimizers. The same set of experiments are run with both SGD (learning rate 0.1) and ADAM (learning rate 0.001), allowing us to investigate the impact of stochastic optimizers on the intrinsic dimension achieved.
通过我们在 MNIST FC 网络上的实验，另一个发现与优化器的作用有关。相同的实验集在 SGD（学习率 0.1）和 ADAM（学习率 0.001）下运行，使我们能够研究随机优化器对达到的内禀维数的影响。

The intrinsic dimension $d_{\mathrm{int90}}$ are reported in Fig. S10 (a)(b). In addition to two optimizers we also use two baselines: Global baseline that is set up as 90% of best performance achieved across all models, and individual baseline that is with regards to the performance of the same model in direct training.
图 S10（a）（b）报告了固有维度 $d_{\mathrm{int90}}$ 。除了两个优化器外，我们还使用了两个基线：全局基线设置为所有模型中最佳性能的 90%，以及与直接训练中相同模型的性能相关的个体基线。

We carry out with ES 3 RL tasks: $\mathtt{InvertedPendulum\!-\!v1}$, $\mathtt{Humanoid\!-\!v1}$, $\mathtt{Pong\!-\!v0}$. The hyperparameter settings for training are in Table S3.
我们执行了 ES 3 RL 任务： $\mathtt{InvertedPendulum\!-\!v1}$ ， $\mathtt{Humanoid\!-\!v1}$ ， $\mathtt{Pong\!-\!v0}$ 。训练的超参数设置见表 S3。