Representation Learning: A Review and New Perspectives
表示学习：综述与新的视角

In Bengio and LeCun (2007), one of us introduced the notion of AI-tasks, which are challenging for current machine learning algorithms, and involve complex but highly structured dependencies. One reason why explicitly dealing with representations is interesting is because they can be convenient to express many general priors about the world around us, i.e., priors that are not task-specific but would be likely to be useful for a learning machine to solve AI-tasks. Examples of such general-purpose priors are the following:
在 Bengio 和 LeCun（2007）的研究中，我们其中一人引入了 AI 任务的观念，这些任务对当前机器学习算法来说具有挑战性，并涉及复杂但高度结构化的依赖关系。处理表示形式的一个有趣之处在于，它们可以方便地表达关于我们周围世界的许多一般先验，即不是特定于任务的先验，但对于学习机器解决 AI 任务可能是有用的。这类通用先验的例子包括以下内容：
$\bullet$ Smoothness: assumes the function to be learned $f$ is s.t. $x\approx y$ generally implies $f(x)\approx f(y)$. This most basic prior is present in most machine learning, but is insufficient to get around the curse of dimensionality, see Section 3.2.
平滑性：假设要学习函数 $f$ 满足 $x\approx y$ ，这通常意味着 $f(x)\approx f(y)$ 。这种最基本的先验在大多数机器学习中都存在，但不足以克服维度灾难，参见第 3.2 节。
$\bullet$ Multiple explanatory factors: the data generating distribution is generated by different underlying factors, and for the most part what one learns about one factor generalizes in many configurations of the other factors. The objective to recover or at least disentangle these underlying factors of variation is discussed in Section 3.5. This assumption is behind the idea of distributed representations, discussed in Section 3.3 below.
多个解释因素：数据生成分布由不同的潜在因素产生，并且对于大多数情况，关于一个因素的学习可以推广到许多其他因素的配置中。在 3.5 节中讨论了恢复或至少解耦这些潜在变异因素的目标。这一假设是下面 3.3 节中讨论的分布式表示理念背后的。
$\bullet$ A hierarchical organization of explanatory factors: the concepts that are useful for describing the world around us can be defined in terms of other concepts, in a hierarchy, with more abstract concepts higher in the hierarchy, defined in terms of less abstract ones. This assumption is exploited with deep representations, elaborated in Section 3.4 below.
$\bullet$ 解释因素的层次结构：描述我们周围世界的有用概念可以用其他概念来定义，形成一个层次结构，其中更抽象的概念位于层次结构的高层，用较不抽象的概念来定义。这一假设在下面的第 3.4 节中通过深度表示得到利用。
$\bullet$ Semi-supervised learning: with inputs $X$ and target $Y$ to predict, a subset of the factors explaining $X$’s distribution explain much of $Y$, given $X$. Hence representations that are useful for $P(X)$ tend to be useful when learning $P(Y|X)$, allowing sharing of statistical strength between the unsupervised and supervised learning tasks, see Section 4.
半监督学习：给定输入 $X$ 和目标 $Y$ 进行预测，解释 $X$ 分布的因子子集在很大程度上解释了 $Y$ ，在 $X$ 的条件下。因此，对 $P(X)$ 有用的表示在学习 $P(Y|X)$ 时也往往有用，允许无监督学习和监督学习任务之间共享统计强度，参见第 4 节。
$\bullet$ Shared factors across tasks: with many $Y$’s of interest or many learning tasks in general, tasks (e.g., the corresponding $P(Y|X,{\rm task})$) are explained by factors that are shared with other tasks, allowing sharing of statistical strengths across tasks, as discussed in the previous section (Multi-Task and Transfer Learning, Domain Adaptation).
任务间的共享因素：在许多感兴趣的任务或一般的学习任务中，任务（例如，相应的任务）由与其他任务共享的因素所解释，允许在任务间共享统计优势，如前节（多任务学习和迁移学习，领域自适应）所述。
$\bullet$ Manifolds: probability mass concentrates near regions that have a much smaller dimensionality than the original space where the data lives. This is explicitly exploited in some of the auto-encoder algorithms and other manifold-inspired algorithms described respectively in Sections 7.2 and 8.
流形：概率质量集中在比数据所在原始空间维度小得多的区域附近。这一点在 7.2 节和 8 节中分别描述的一些自动编码器算法和其他流形启发算法中被明确利用。
$\bullet$ Natural clustering: different values of categorical variables such as object classes are associated with separate manifolds. More precisely, the local variations on the manifold tend to preserve the value of a category, and a linear interpolation between examples of different classes in general involves going through a low density region, i.e., $P(X|Y=i)$ for different $i$ tend to be well separated and not overlap much. For example, this is exploited in the Manifold Tangent Classifier discussed in Section 8.3. This hypothesis is consistent with the idea that humans have named categories and classes because of such statistical structure (discovered by their brain and propagated by their culture), and machine learning tasks often involves predicting such categorical variables.
自然聚类：不同类别的分类变量，如对象类别，与不同的流形相关联。更准确地说，流形上的局部变化倾向于保持类别值，并且在不同类别示例之间进行线性插值通常涉及穿过低密度区域，即对于不同的类别，它们往往很好地分离且重叠不多。例如，这在第 8.3 节讨论的流形切线分类器中得到了利用。这个假设与人类因为这样的统计结构（由他们的大脑发现并由他们的文化传播）而对类别和进行命名和分类的想法一致，机器学习任务通常涉及预测这样的分类变量。
$\bullet$ Temporal and spatial coherence: consecutive (from a sequence) or spatially nearby observations tend to be associated with the same value of relevant categorical concepts, or result in a small move on the surface of the high-density manifold. More generally, different factors change at different temporal and spatial scales, and many categorical concepts of interest change slowly. When attempting to capture such categorical variables, this prior can be enforced by making the associated representations slowly changing, i.e., penalizing changes in values over time or space. This prior was introduced in Becker and Hinton (1992) and is discussed in Section 11.3.
时间与空间一致性：连续（来自序列）或空间邻近的观测值往往与相关分类概念的同值相关联，或者在高度密集流形表面产生小的移动。更普遍地，不同因素在不同时间和空间尺度上发生变化，许多感兴趣的分类概念变化缓慢。在尝试捕捉此类分类变量时，可以通过使相关表示缓慢变化来强制执行此先验，即惩罚时间和空间上值的改变。此先验在 Becker 和 Hinton（1992）中提出，并在第 11.3 节中讨论。
$\bullet$ Sparsity: for any given observation $x$, only a small fraction of the possible factors are relevant. In terms of representation, this could be represented by features that are often zero (as initially proposed by Olshausen and Field (1996)), or by the fact that most of the extracted features are insensitive to small variations of $x$. This can be achieved with certain forms of priors on latent variables (peaked at 0), or by using a non-linearity whose value is often flat at 0 (i.e., 0 and with a 0 derivative), or simply by penalizing the magnitude of the Jacobian matrix (of derivatives) of the function mapping input to representation. This is discussed in Sections 6.1.1 and 7.2.
稀疏性：对于任何给定的观察值 $x$ ，只有一小部分可能的因素是相关的。在表示方面，这可以通过通常为零的特征（如 Olshausen 和 Field（1996）最初提出的）来表示，或者通过大多数提取的特征对 $x$ 的小变化不敏感的事实来表示。这可以通过对潜在变量（峰值在 0 处）的某些形式先验，或者使用值通常在 0 处平坦的非线性（即 0 及其导数为 0），或者简单地通过惩罚将输入映射到表示的函数的雅可比矩阵（导数的矩阵）的幅度来实现。这将在第 6.1.1 节和第 7.2 节中讨论。
$\bullet$ Simplicity of Factor Dependencies: in good high-level representations, the factors are related to each other through simple, typically linear dependencies. This can be seen in many laws of physics, and is assumed when plugging a linear predictor on top of a learned representation.
因子依赖的简洁性：在良好的高级表示中，因子之间通过简单、通常是线性的依赖关系相互关联。这在许多物理定律中都可以看到，并且在将线性预测器叠加在学到的表示之上时被假设。

We can view many of the above priors as ways to help the learner discover and disentangle some of the underlying (and a priori unknown) factors of variation that the data may reveal. This idea is pursued further in Sections 3.5 and 11.4.
我们可以将上述许多先验视为帮助学习者发现和区分数据可能揭示的一些潜在（且先验未知）的变异因素的方法。这一想法在 3.5 节和 11.4 节中得到进一步探讨。

For AI-tasks, such as vision and NLP, it seems hopeless to rely only on simple parametric models (such as linear models) because they cannot capture enough of the complexity of interest unless provided with the appropriate feature space. Conversely, machine learning researchers have sought flexibility in local⁶⁶6local in the sense that the value of the learned function at $x$ depends mostly on training examples $x^{(t)}$’s close to $x$ non-parametric learners such as kernel machines with a fixed generic local-response kernel (such as the Gaussian kernel). Unfortunately, as argued at length by Bengio and Monperrus (2005); Bengio et al. (2006a); Bengio and LeCun (2007); Bengio (2009); Bengio et al. (2010), most of these algorithms only exploit the principle of local generalization, i.e., the assumption that the target function (to be learned) is smooth enough, so they rely on examples to explicitly map out the wrinkles of the target function. Generalization is mostly achieved by a form of local interpolation between neighboring training examples. Although smoothness can be a useful assumption, it is insufficient to deal with the curse of dimensionality, because the number of such wrinkles (ups and downs of the target function) may grow exponentially with the number of relevant interacting factors, when the data are represented in raw input space. We advocate learning algorithms that are flexible and non-parametric⁷⁷7We understand non-parametric as including all learning algorithms whose capacity can be increased appropriately as the amount of data and its complexity demands it, e.g. including mixture models and neural networks where the number of parameters is a data-selected hyper-parameter. but do not rely exclusively on the smoothness assumption. Instead, we propose to incorporate generic priors such as those enumerated above into representation-learning algorithms. Smoothness-based learners (such as kernel machines) and linear models can still be useful on top of such learned representations. In fact, the combination of learning a representation and kernel machine is equivalent to learning the kernel, i.e., the feature space. Kernel machines are useful, but they depend on a prior definition of a suitable similarity metric, or a feature space in which naive similarity metrics suffice. We would like to use the data, along with very generic priors, to discover those features, or equivalently, a similarity function.
对于 AI 任务，如视觉和 NLP，仅依赖简单的参数模型（如线性模型）似乎毫无希望，因为除非提供适当的特征空间，否则它们无法捕捉到足够的感兴趣复杂度。相反，机器学习研究人员寻求在具有固定通用局部响应核（如高斯核）的非参数学习器（如核机）中的灵活性。不幸的是，正如 Bengio 和 Monperrus（2005）；Bengio 等人（2006a）；Bengio 和 LeCun（2007）；Bengio（2009）；Bengio 等人（2010）详细论证的那样，这些算法中的大多数仅利用局部泛化原理，即目标函数（要学习的）足够平滑的假设，因此它们依赖于示例来明确映射出目标函数的皱纹。泛化主要通过相邻训练示例之间的局部插值形式来实现。 尽管平滑性可以是一个有用的假设，但它不足以处理维度灾难，因为当数据在原始输入空间中表示时，这种皱纹（目标函数的起伏）的数量可能会随着相关相互作用因素数量的指数增长。我们提倡学习灵活且非参数化的算法，但不应仅依赖于平滑性假设。相反，我们建议将上述列举的通用先验纳入表示学习算法中。基于平滑性的学习器（如核机）和线性模型在上述学习表示之上仍然有用。事实上，学习表示和核机的组合等同于学习核，即特征空间。核机是有用的，但它们依赖于一个合适的相似性度量或特征空间的先验定义，或者在这个空间中，原始相似性度量就足够了。我们希望使用数据，以及非常通用的先验，来发现这些特征，或者说，一个相似性函数。

Good representations are expressive, meaning that a reasonably-sized learned representation can capture a huge number of possible input configurations. A simple counting argument helps us to assess the expressiveness of a model producing a representation: how many parameters does it require compared to the number of input regions (or configurations) it can distinguish? Learners of one-hot representations, such as traditional clustering algorithms, Gaussian mixtures, nearest-neighbor algorithms, decision trees, or Gaussian SVMs all require $O(N)$ parameters (and/or $O(N)$ examples) to distinguish $O(N)$ input regions. One could naively believe that one cannot do better. However, RBMs, sparse coding, auto-encoders or multi-layer neural networks can all represent up to $O(2^{k})$ input regions using only $O(N)$ parameters (with $k$ the number of non-zero elements in a sparse representation, and $k=N$ in non-sparse RBMs and other dense representations). These are all distributed ⁸⁸8 Distributed representations: where $k$ out of $N$ representation elements or feature values can be independently varied, e.g., they are not mutually exclusive. Each concept is represented by having $k$ features being turned on or active, while each feature is involved in representing many concepts. or sparse⁹⁹9Sparse representations: distributed representations where only a few of the elements can be varied at a time, i.e., $k<N$. representations. The generalization of clustering to distributed representations is multi-clustering, where either several clusterings take place in parallel or the same clustering is applied on different parts of the input, such as in the very popular hierarchical feature extraction for object recognition based on a histogram of cluster categories detected in different patches of an image (Lazebnik et al., 2006; Coates and Ng, 2011a). The exponential gain from distributed or sparse representations is discussed further in section 3.2 (and Figure 3.2) of  Bengio (2009). It comes about because each parameter (e.g. the parameters of one of the units in a sparse code, or one of the units in a Restricted Boltzmann Machine) can be re-used in many examples that are not simply near neighbors of each other, whereas with local generalization, different regions in input space are basically associated with their own private set of parameters, e.g., as in decision trees, nearest-neighbors, Gaussian SVMs, etc. In a distributed representation, an exponentially large number of possible subsets of features or hidden units can be activated in response to a given input. In a single-layer model, each feature is typically associated with a preferred input direction, corresponding to a hyperplane in input space, and the code or representation associated with that input is precisely the pattern of activation (which features respond to the input, and how much). This is in contrast with a non-distributed representation such as the one learned by most clustering algorithms, e.g., k-means, in which the representation of a given input vector is a one-hot code identifying which one of a small number of cluster centroids best represents the input ¹⁰¹⁰10As discussed in (Bengio, 2009), things are only slightly better when allowing continuous-valued membership values, e.g., in ordinary mixture models (with separate parameters for each mixture component), but the difference in representational power is still exponential (Montufar and Morton, 2012). The situation may also seem better with a decision tree, where each given input is associated with a one-hot code over the tree leaves, which deterministically selects associated ancestors (the path from root to node). Unfortunately, the number of different regions represented (equal to the number of leaves of the tree) still only grows linearly with the number of parameters used to specify it (Bengio and Delalleau, 2011). .
优秀的表示方法是富有表现力的，这意味着一个合理大小的学习表示可以捕捉大量的可能输入配置。一个简单的计数论据帮助我们评估产生表示的模型的表现力：与它能区分的输入区域（或配置）数量相比，它需要多少参数？例如，学习独热表示的学习者，如传统的聚类算法、高斯混合模型、最近邻算法、决策树或高斯 SVM，都需要 $O(N)$ 个参数（和/或 $O(N)$ 个示例）来区分 $O(N)$ 个输入区域。人们可能会天真地认为无法做得更好。然而，RBM、稀疏编码、自编码器或多层神经网络都可以仅使用 $O(N)$ 个参数（其中 $k$ 是稀疏表示中非零元素的数量， $k=N$ 是非稀疏 RBM 和其他密集表示中的数量）来表示多达 $O(2^{k})$ 个输入区域。这些都是分布式 ⁸ 或稀疏 ⁹ 表示。 聚类推广到分布式表示是多聚类，要么并行进行多个聚类，要么对输入的不同部分应用相同的聚类，例如在基于图像不同区域检测到的聚类类别直方图的非常流行的层次特征提取中进行对象识别（Lazebnik 等，2006；Coates 和 Ng，2011a）。关于分布式或稀疏表示的指数级增益在 Bengio（2009）的第 3.2 节（以及图 3.2）中进一步讨论。这是因为每个参数（例如稀疏码中一个单元的参数，或者受限玻尔兹曼机中的一个单元）可以在许多非简单邻近的例子中重复使用，而局部泛化中，输入空间的不同区域基本上与它们自己的私有参数集相关联，例如在决策树、最近邻、高斯 SVM 等中。在分布式表示中，针对给定的输入可以激活指数级大量的特征或隐藏单元的可能子集。 在一个单层模型中，每个特征通常与一个首选输入方向相关联，对应于输入空间中的一个超平面，与该输入相关的代码或表示正是激活模式（哪些特征对输入有响应，以及响应程度）。这与大多数聚类算法（如 k-means）学习到的非分布式表示形成对比，其中给定输入向量的表示是一个一热代码，用于标识少量聚类中心中哪一个最能代表输入。

Depth is a key aspect to representation learning strategies we consider in this paper. As we will discuss, deep architectures are often challenging to train effectively and this has been the subject of much recent research and progress. However, despite these challenges, they carry two significant advantages that motivate our long-term interest in discovering successful training strategies for deep architectures. These advantages are: (1) deep architectures promote the re-use of features, and (2) deep architectures can potentially lead to progressively more abstract features at higher layers of representations (more removed from the data).
深度是我们在这篇论文中考虑的表示学习策略的关键方面。正如我们将讨论的，深度架构通常难以有效训练，这已成为近期研究和进展的主题。然而，尽管存在这些挑战，它们具有两个显著的优势，这促使我们长期致力于发现深度架构的成功训练策略。这些优势是：（1）深度架构促进了特征的复用；（2）深度架构有可能在表示的更高层（更远离数据）产生越来越抽象的特征。

Feature re-use. The notion of re-use, which explains the power of distributed representations, is also at the heart of the theoretical advantages behind deep learning, i.e., constructing multiple levels of representation or learning a hierarchy of features. The depth of a circuit is the length of the longest path from an input node of the circuit to an output node of the circuit. The crucial property of a deep circuit is that its number of paths, i.e., ways to re-use different parts, can grow exponentially with its depth. Formally, one can change the depth of a given circuit by changing the definition of what each node can compute, but only by a constant factor. The typical computations we allow in each node include: weighted sum, product, artificial neuron model (such as a monotone non-linearity on top of an affine transformation), computation of a kernel, or logic gates. Theoretical results clearly show families of functions where a deep representation can be exponentially more efficient than one that is insufficiently deep (Håstad, 1986; Håstad and Goldmann, 1991; Bengio et al., 2006a; Bengio and LeCun, 2007; Bengio and Delalleau, 2011). If the same family of functions can be represented with fewer parameters (or more precisely with a smaller VC-dimension), learning theory would suggest that it can be learned with fewer examples, yielding improvements in both computational efficiency (less nodes to visit) and statistical efficiency (less parameters to learn, and re-use of these parameters over many different kinds of inputs).
特征重用。重用概念解释了分布式表示的强大之处，也是深度学习背后理论优势的核心，即构建多级表示或学习特征层次。电路的深度是从电路输入节点到输出节点的最长路径长度。深度电路的关键特性是其路径数，即重用不同部分的方式，可以随着深度的增加而指数增长。形式上，可以通过改变每个节点可以计算的定义来改变给定电路的深度，但只能通过一个常数因子。我们允许在每个节点中进行的典型计算包括：加权求和、乘积、人工神经元模型（如叠加在仿射变换上的单调非线性），核计算或逻辑门。理论结果清楚地表明，在函数族中，深度表示可以比深度不足的表示指数级更有效（Håstad，1986；Håstad 和 Goldmann，1991；Bengio 等，2006a；Bengio 和 LeCun，2007；Bengio 和 Delalleau，2011）。 如果同一族函数可以用更少的参数（或者更精确地说，更小的 VC 维数）来表示，学习理论将表明它可以用更少的示例来学习，从而在计算效率（需要访问的节点更少）和统计效率（需要学习的参数更少，以及这些参数在许多不同类型的输入中重复使用）方面带来改进。

Abstraction and invariance. Deep architectures can lead to abstract representations because more abstract concepts can often be constructed in terms of less abstract ones. In some cases, such as in the convolutional neural network (LeCun et al., 1998b), we build this abstraction in explicitly via a pooling mechanism (see section 11.2). More abstract concepts are generally invariant to most local changes of the input. That makes the representations that capture these concepts generally highly non-linear functions of the raw input. This is obviously true of categorical concepts, where more abstract representations detect categories that cover more varied phenomena (e.g. larger manifolds with more wrinkles) and thus they potentially have greater predictive power. Abstraction can also appear in high-level continuous-valued attributes that are only sensitive to some very specific types of changes in the input. Learning these sorts of invariant features has been a long-standing goal in pattern recognition.
抽象和不变性。深度架构可以导致抽象表示，因为更抽象的概念通常可以用较不抽象的概念来构建。在某些情况下，例如在卷积神经网络（LeCun 等，1998b）中，我们通过池化机制（见第 11.2 节）明确地构建这种抽象。更抽象的概念通常对输入的大部分局部变化是不变的。这使得捕捉这些概念的表现形式通常是原始输入的高度非线性函数。这在分类概念中显然是正确的，其中更抽象的表示检测到覆盖更多现象的类别（例如，具有更多褶皱的更大流形），因此它们可能具有更大的预测能力。抽象还可以出现在高级连续值属性中，这些属性只对输入中某些非常特定的变化敏感。在模式识别中学习这类不变特征一直是长期目标。

Beyond being distributed and invariant, we would like our representations to disentangle the factors of variation. Different explanatory factors of the data tend to change independently of each other in the input distribution, and only a few at a time tend to change when one considers a sequence of consecutive real-world inputs.
超越分布和不变性，我们希望我们的表示能够解耦变化的因素。数据的不同解释因素往往在输入分布中相互独立地变化，而当考虑一系列连续的现实世界输入时，只有少数因素会同时变化。

Complex data arise from the rich interaction of many sources. These factors interact in a complex web that can complicate AI-related tasks such as object classification. For example, an image is composed of the interaction between one or more light sources, the object shapes and the material properties of the various surfaces present in the image. Shadows from objects in the scene can fall on each other in complex patterns, creating the illusion of object boundaries where there are none and dramatically effect the perceived object shape. How can we cope with these complex interactions? How can we disentangle the objects and their shadows? Ultimately, we believe the approach we adopt for overcoming these challenges must leverage the data itself, using vast quantities of unlabeled examples, to learn representations that separate the various explanatory sources. Doing so should give rise to a representation significantly more robust to the complex and richly structured variations extant in natural data sources for AI-related tasks.
复杂数据源于众多来源的丰富交互。这些因素在一个复杂的网络中相互作用，可能会使人工智能相关的任务，如物体分类变得复杂。例如，一张图像由一个或多个光源、物体形状以及图像中各种表面的材料特性之间的交互构成。场景中物体的阴影可以以复杂的模式相互重叠，创造出实际上并不存在的物体边界，并极大地影响感知到的物体形状。我们如何应对这些复杂交互？如何解开物体及其阴影？最终，我们相信我们采用的克服这些挑战的方法必须利用数据本身，使用大量未标记的示例，来学习将各种解释来源分离的表示。这样做应该会产生一个对自然数据源中存在的复杂和丰富结构的变异具有显著鲁棒性的表示。

It is important to distinguish between the related but distinct goals of learning invariant features and learning to disentangle explanatory factors. The central difference is the preservation of information. Invariant features, by definition, have reduced sensitivity in the direction of invariance. This is the goal of building features that are insensitive to variation in the data that are uninformative to the task at hand. Unfortunately, it is often difficult to determine a priori which set of features and variations will ultimately be relevant to the task at hand. Further, as is often the case in the context of deep learning methods, the feature set being trained may be destined to be used in multiple tasks that may have distinct subsets of relevant features. Considerations such as these lead us to the conclusion that the most robust approach to feature learning is to disentangle as many factors as possible, discarding as little information about the data as is practical. If some form of dimensionality reduction is desirable, then we hypothesize that the local directions of variation least represented in the training data should be first to be pruned out (as in PCA, for example, which does it globally instead of around each example).
区分学习不变特征和学习解耦解释因素的相关但不同的目标非常重要。核心区别在于信息的保留。根据定义，不变特征在不变性的方向上具有降低的敏感性。这是构建对数据中与任务无关的变异不敏感的特征的目标。不幸的是，通常很难事先确定哪些特征集和变异最终与当前任务相关。此外，正如在深度学习方法背景下经常发生的那样，正在训练的特征集可能注定要用于多个可能具有不同相关特征子集的任务。考虑到这些因素，我们得出结论，特征学习的最稳健方法是尽可能解耦尽可能多的因素，同时尽可能少地丢弃关于数据的任何信息。 如果希望进行某种形式的降维，那么我们假设在训练数据中代表性最少的局部变化方向应该首先被剪枝（例如，PCA 就是这样做的，它是全局而不是围绕每个示例进行）。

One of the challenges of representation learning that distinguishes it from other machine learning tasks such as classification is the difficulty in establishing a clear objective, or target for training. In the case of classification, the objective is (at least conceptually) obvious, we want to minimize the number of misclassifications on the training dataset. In the case of representation learning, our objective is far-removed from the ultimate objective, which is typically learning a classifier or some other predictor. Our problem is reminiscent of the credit assignment problem encountered in reinforcement learning. We have proposed that a good representation is one that disentangles the underlying factors of variation, but how do we translate that into appropriate training criteria? Is it even necessary to do anything but maximize likelihood under a good model or can we introduce priors such as those enumerated above (possibly data-dependent ones) that help the representation better do this disentangling? This question remains clearly open but is discussed in more detail in Sections 3.5 and 11.4.
代表学习的一个挑战，区别于其他机器学习任务如分类，在于难以确立一个明确的训练目标。在分类的情况下，目标（至少在概念上）是明显的，我们希望最小化训练数据集中的错误分类数量。在代表学习的情况下，我们的目标与最终目标相去甚远，最终目标通常是学习一个分类器或其他预测器。我们的问题与强化学习中遇到的信用分配问题相似。我们提出，一个好的表示能够解开潜在变化因素，但如何将其转化为适当的训练标准呢？是否只需在好的模型下最大化似然性就足够了，或者我们可以引入上述（可能依赖于数据的）先验，帮助表示更好地进行这种解开？这个问题显然仍然悬而未决，但在第 3.5 节和第 11.4 节中进行了更详细的讨论。

Directed latent factor models separately parametrize the conditional likelihood $p(x\mid h)$ and the prior $p(h)$ to construct the joint distribution, $p(x,h)=p(x\mid h)p(h)$. Examples of this decomposition include: Principal Components Analysis (PCA) (Roweis, 1997; Tipping and Bishop, 1999), sparse coding (Olshausen and Field, 1996), sigmoid belief networks (Neal, 1992) and the newly introduced spike-and-slab sparse coding model (Goodfellow et al., 2011).
有向潜在因子模型分别对条件似然函数进行参数化 $p(x\mid h)$ 和对先验分布进行参数化 $p(h)$ ，以构建联合分布 $p(x,h)=p(x\mid h)p(h)$ 。这种分解的例子包括：主成分分析（PCA）（Roweis，1997；Tipping 和 Bishop，1999）、稀疏编码（Olshausen 和 Field，1996）、S 型信念网络（Neal，1992）以及新引入的尖峰和平板稀疏编码模型（Goodfellow 等，2011）。

Undirected graphical models, also called Markov random fields (MRFs), parametrize the joint $p(x,h)$ through a product of unnormalized non-negative clique potentials:
无向图模型，也称为马尔可夫随机场（MRFs），通过非归一化非负团势能的乘积来参数化联合分布：

where $\psi_{i}(x)$, $\eta_{j}(h)$ and $\nu_{k}(x,h)$ are the clique potentials describing the interactions between the visible elements, between the hidden variables, and those interaction between the visible and hidden variables respectively. The partition function $Z_{\theta}$ ensures that the distribution is normalized. Within the context of unsupervised feature learning, we generally see a particular form of Markov random field called a Boltzmann distribution with clique potentials constrained to be positive:
$\psi_{i}(x)$ 、 $\eta_{j}(h)$ 和 $\nu_{k}(x,h)$ 分别表示描述可见元素之间、隐藏变量之间以及可见变量与隐藏变量之间相互作用的团势能。配分函数 $Z_{\theta}$ 确保分布是归一化的。在无监督特征学习的背景下，我们通常看到一种特定的马尔可夫随机场形式，称为具有正约束的团势能的玻尔兹曼分布：

where ${\cal E}_{\theta}(x,h)$ is the energy function and contains the interactions described by the MRF clique potentials and $\theta$ are the model parameters that characterize these interactions.
${\cal E}_{\theta}(x,h)$ 是能量函数，包含由 MRF 团簇势描述的相互作用，而 $\theta$ 是表征这些相互作用的模型参数。

The Boltzmann machine was originally defined as a network of symmetrically-coupled binary random variables or units. These stochastic units can be divided into two groups: (1) the visible units $x\in\{0,1\}^{d_{x}}$ that represent the data, and (2) the hidden or latent units $h\in\{0,1\}^{d_{h}}$ that mediate dependencies between the visible units through their mutual interactions. The pattern of interaction is specified through the energy function:
玻尔兹曼机最初被定义为对称耦合的二进制随机变量或单元的网络。这些随机单元可以分为两组：（1）表示数据的可见单元 $x\in\{0,1\}^{d_{x}}$ ，以及（2）通过相互交互在可见单元之间传递依赖关系的隐藏或潜在单元 $h\in\{0,1\}^{d_{h}}$ 。交互模式通过能量函数指定：

The Boltzmann machine energy function specifies the probability distribution over $[x,h]$, via the Boltzmann distribution, Eq. 6, with the partition function $Z_{\theta}$ given by:
Boltzmann 机能量函数通过 Boltzmann 分布（公式 6）指定了 $[x,h]$ 的概率分布，其中配分函数 $Z_{\theta}$ 由以下给出：

This joint probability distribution gives rise to the set of conditional distributions of the form:
这个联合概率分布产生了形式为的条件分布集合：

In general, inference in the Boltzmann machine is intractable. For example, computing the conditional probability of $h_{i}$ given the visibles, $P(h_{i}\mid x)$, requires marginalizing over the rest of the hiddens, which implies evaluating a sum with $2^{d_{h}-1}$ terms:
一般来说，玻尔兹曼机中的推理是不可行的。例如，给定可见变量 $h_{i}$ ，计算条件概率需要对其他隐藏变量进行边缘化，这意味着要评估一个包含 $2^{d_{h}-1}$ 项的和：

However with some judicious choices in the pattern of interactions between the visible and hidden units, more tractable subsets of the model family are possible, as we discuss next.
然而，通过在可见单元和隐藏单元之间的交互模式中进行一些明智的选择，我们可能得到模型家族中更易于处理的子集，正如我们接下来要讨论的。

Restricted Boltzmann Machines (RBMs). The RBM is likely the most popular subclass of Boltzmann machine (Smolensky, 1986). It is defined by restricting the interactions in the Boltzmann energy function, in Eq. 7, to only those between $h$ and $x$, i.e. ${\cal E}_{\theta}^{\mathrm{RBM}}$ is ${\cal E}_{\theta}^{\mathrm{BM}}$ with $U=\mathbf{0}$ and $V=\mathbf{0}$. As such, the RBM can be said to form a bipartite graph with the visibles and the hiddens forming two layers of vertices in the graph (and no connection between units of the same layer). With this restriction, the RBM possesses the useful property that the conditional distribution over the hidden units factorizes given the visibles:
受限玻尔兹曼机（RBMs）。RBM 可能是玻尔兹曼机（Smolensky，1986）中最受欢迎的子类。它通过限制玻尔兹曼能量函数（方程 7）中的相互作用，仅限于 $h$ 和 $x$ 之间，即 ${\cal E}_{\theta}^{\mathrm{RBM}}$ 是 ${\cal E}_{\theta}^{\mathrm{BM}}$ 与 $U=\mathbf{0}$ 和 $V=\mathbf{0}$ 的函数。因此，RBM 可以被认为形成一个二分图，其中可见性和隐藏性形成图中的两个顶点层（并且同一层单元之间没有连接）。在这种限制下，RBM 具有有用的性质，即给定可见性，隐藏单元的条件分布可以分解：

Likewise, the conditional distribution over the visible units given the hiddens also factorizes:
同样，给定隐藏单元的可见单元的条件分布也进行分解：

This makes inferences readily tractable in RBMs. For example, the RBM feature representation is taken to be the set of posterior marginals $P(h_{i}\mid x)$, which, given the conditional independence described in Eq. 12, are immediately available. Note that this is in stark contrast to the situation with popular directed graphical models for unsupervised feature extraction, where computing the posterior probability is intractable.
这使得在 RBMs 中进行推断变得容易处理。例如，RBM 的特征表示被假定为后验边缘集 $P(h_{i}\mid x)$ ，根据式(12)中描述的条件独立性，这些边缘可以直接获得。请注意，这与用于无监督特征提取的流行有向图模型的情况形成鲜明对比，在这些模型中，计算后验概率是不可行的。

Importantly, the tractability of the RBM does not extend to its partition function, which still involves summing an exponential number of terms. It does imply however that we can limit the number of terms to $\min\{2^{d_{x}},2^{d_{h}}\}$. Usually this is still an unmanageable number of terms and therefore we must resort to approximate methods to deal with its estimation.
重要的是，RBM 的可处理性并不扩展到其配分函数，该函数仍然涉及求和指数级数项。然而，这确实意味着我们可以将项数限制为 $\min\{2^{d_{x}},2^{d_{h}}\}$ 。通常，这仍然是一个难以管理的项数，因此我们必须求助于近似方法来处理其估计。

It is difficult to overstate the impact the RBM has had to the fields of unsupervised feature learning and deep learning. It has been used in a truly impressive variety of applications, including fMRI image classification (Schmah et al., 2009), motion and spatial transformations (Taylor and Hinton, 2009; Memisevic and Hinton, 2010), collaborative filtering (Salakhutdinov et al., 2007) and natural image modeling (Ranzato and Hinton, 2010; Courville et al., 2011b).
RBM 对无监督特征学习和深度学习领域的影响难以言过其实。它已被应用于真正令人印象深刻的多种应用，包括 fMRI 图像分类（Schmah 等人，2009 年）、运动和空间变换（Taylor 和 Hinton，2009 年；Memisevic 和 Hinton，2010 年）、协同过滤（Salakhutdinov 等人，2007 年）和自然图像建模（Ranzato 和 Hinton，2010 年；Courville 等人，2011b）。

Important progress has been made in the last few years in defining generalizations of the RBM that better capture real-valued data, in particular real-valued image data, by better modeling the conditional covariance of the input pixels. The standard RBM, as discussed above, is defined with both binary visible variables $v\in\{0,1\}$ and binary latent variables $h\in\{0,1\}$. The tractability of inference and learning in the RBM has inspired many authors to extend it, via modifications of its energy function, to model other kinds of data distributions. In particular, there has been multiple attempts to develop RBM-type models of real-valued data, where $x\in{\mathbb{R}}^{d_{x}}$. The most straightforward approach to modeling real-valued observations within the RBM framework is the so-called Gaussian RBM (GRBM) where the only change in the RBM energy function is to the visible units biases, by adding a bias term that is quadratic in the visible units $x$. While it probably remains the most popular way to model real-valued data within the RBM framework, Ranzato and Hinton (2010) suggest that the GRBM has proved to be a somewhat unsatisfactory model of natural images. The trained features typically do not represent sharp edges that occur at object boundaries and lead to latent representations that are not particularly useful features for classification tasks. Ranzato and Hinton (2010) argue that the failure of the GRBM to adequately capture the statistical structure of natural images stems from the exclusive use of the model capacity to capture the conditional mean at the expense of the conditional covariance. Natural images, they argue, are chiefly characterized by the covariance of the pixel values, not by their absolute values. This point is supported by the common use of preprocessing methods that standardize the global scaling of the pixel values across images in a dataset or across the pixel values within each image.
在过去的几年里，在定义 RBM 的推广方面取得了重要进展，这些推广能更好地捕捉实值数据，特别是实值图像数据，通过更好地建模输入像素的条件协方差。如上所述，标准 RBM 由二元可见变量 $v\in\{0,1\}$ 和二元潜在变量 $h\in\{0,1\}$ 定义。RBM 中推理和学习的可处理性激励了许多作者通过修改其能量函数来扩展它，以模拟其他类型的数据分布。特别是，已经尝试开发多种 RBM 型实值数据模型，其中 $x\in{\mathbb{R}}^{d_{x}}$ 。在 RBM 框架内建模实值观察的最直接方法被称为高斯 RBM（GRBM），其中 RBM 能量函数的唯一变化是对可见单元偏置，通过添加一个在可见单元中二次的偏置项 $x$ 。虽然这可能是 RBM 框架内建模实值数据最流行的方法，但 Ranzato 和 Hinton（2010）建议，GRBM 已被证明是自然图像的一个不太令人满意的模型。 训练特征通常不表示出现在物体边界处的锐利边缘，导致潜在表示不是特别有用的分类任务特征。Ranzato 和 Hinton（2010）认为，GRBM 未能充分捕捉自然图像的统计结构，源于模型容量仅用于捕捉条件均值，而忽略了条件协方差。他们认为，自然图像主要特征是像素值的协方差，而不是它们的绝对值。这一点得到了预处理方法的普遍使用支持，这些方法在数据集图像的全局缩放或每个图像内部像素值之间标准化像素值。

These kinds of concerns about the ability of the GRBM to model natural image data has lead to the development of alternative RBM-based models that each attempt to take on this objective of better modeling non-diagonal conditional covariances.  (Ranzato and Hinton, 2010) introduced the mean and covariance RBM (mcRBM). Like the GRBM, the mcRBM is a 2-layer Boltzmann machine that explicitly models the visible units as Gaussian distributed quantities. However unlike the GRBM, the mcRBM uses its hidden layer to independently parametrize both the mean and covariance of the data through two sets of hidden units. The mcRBM is a combination of the covariance RBM (cRBM) (Ranzato et al., 2010a), that models the conditional covariance, with the GRBM that captures the conditional mean. While the GRBM has shown considerable potential as the basis of a highly successful phoneme recognition system (Dahl et al., 2010), it seems that due to difficulties in training the mcRBM, the model has been largely superseded by the mPoT model. The mPoT model (mean-product of Student’s T-distributions model)  (Ranzato et al., 2010b) is a combination of the GRBM and the product of Student’s T-distributions model (Welling et al., 2003). It is an energy-based model where the conditional distribution over the visible units conditioned on the hidden variables is a multivariate Gaussian (non-diagonal covariance) and the complementary conditional distribution over the hidden variables given the visibles are a set of independent Gamma distributions.
这些关于 GRBM 建模自然图像数据能力的担忧导致了基于 RBM 的替代模型的开发，每个模型都试图承担更好地建模非对角条件协方差的目标。（Ranzato 和 Hinton，2010）引入了均值和协方差 RBM（mcRBM）。与 GRBM 类似，mcRBM 是一个两层玻尔兹曼机，明确地将可见单元建模为高斯分布的量。然而，与 GRBM 不同，mcRBM 使用其隐藏层通过两组隐藏单元独立地参数化数据的均值和协方差。mcRBM 是条件协方差 RBM（cRBM）（Ranzato 等人，2010a）和捕捉条件均值的 GRBM 的结合。虽然 GRBM 已显示出作为高度成功的音素识别系统基础的巨大潜力（Dahl 等人，2010），但似乎由于训练 mcRBM 的困难，该模型已被 mPoT 模型所取代。mPoT 模型（学生 t 分布的均值乘积模型）（Ranzato 等人。distributions. 它是一种基于能量的模型，其中在隐藏变量条件下的可见单元的条件分布是多元高斯分布（非对角协方差），而给定可见变量的隐藏变量的互补条件分布是一组独立的伽马分布。

The PoT model has recently been generalized to the mPoT model (Ranzato et al., 2010b) to include nonzero Gaussian means by the addition of GRBM-like hidden units, similarly to how the mcRBM generalizes the cRBM. The mPoT model has been used to synthesize large-scale natural images (Ranzato et al., 2010b) that show large-scale features and shadowing structure. It has been used to model natural textures (Kivinen and Williams, 2012) in a tiled-convolution configuration (see section 11.2).
The PoT 模型最近被推广到 mPoT 模型（Ranzato 等，2010b），通过添加类似 GRBM 的隐藏单元来包含非零高斯均值，类似于 mcRBM 如何推广 cRBM。mPoT 模型已被用于合成大规模自然图像（Ranzato 等，2010b），这些图像显示出大规模特征和阴影结构。它已被用于在镶嵌卷积配置中模拟自然纹理（Kivinen 和 Williams，2012）（见第 11.2 节）。

Another recently introduced RBM-based model with the objective of having the hidden units encode both the mean and covariance information is the spike-and-slab Restricted Boltzmann Machine (ssRBM) (Courville et al., 2011a, b). The ssRBM is defined as having both a real-valued “slab” variable and a binary “spike” variable associated with each unit in the hidden layer. The ssRBM has been demonstrated as a feature learning and extraction scheme in the context of CIFAR-10 object classification (Krizhevsky and Hinton, 2009) from natural images and has performed well in the role (Courville et al., 2011a, b). When trained convolutionally (see Section 11.2) on full CIFAR-10 natural images, the model demonstrated the ability to generate natural image samples that seem to capture the broad statistical structure of natural images better than previous parametric generative models, as illustrated with the samples of Figure 2.
另一种近期提出的基于 RBM 的模型，其目标是让隐藏单元编码均值和协方差信息，是尖峰-板状受限玻尔兹曼机（ssRBM）（Courville 等，2011a，b）。ssRBM 被定义为每个隐藏层单元都关联一个实值“板状”变量和一个二进制“尖峰”变量。ssRBM 已被证明是 CIFAR-10 物体分类（Krizhevsky 和 Hinton，2009）的自然图像特征学习和提取方案，并在该角色中表现出色（Courville 等，2011a，b）。当在完整的 CIFAR-10 自然图像上以卷积方式训练（见第 11.2 节）时，该模型展示了生成自然图像样本的能力，这些样本似乎比之前的参数生成模型更好地捕捉了自然图像的广泛统计结构，如图 2 所示样本所示。

The mcRBM, mPoT and ssRBM each set out to model real-valued data such that the hidden units encode not only the conditional mean of the data but also its conditional covariance. Other than differences in the training schemes, the most significant difference between these models is how they encode their conditional covariance. While the mcRBM and the mPoT use the activation of the hidden units to enforce constraints on the covariance of $x$, the ssRBM uses the hidden unit to pinch the precision matrix along the direction specified by the corresponding weight vector. These two ways of modeling conditional covariance diverge when the dimensionality of the hidden layer is significantly different from that of the input. In the overcomplete setting, sparse activation with the ssRBM parametrization permits variance only in the select directions of the sparsely activated hidden units. This is a property the ssRBM shares with sparse coding models (Olshausen and Field, 1996; Grosse et al., 2007). On the other hand, in the case of the mPoT or mcRBM, an overcomplete set of constraints on the covariance implies that capturing arbitrary covariance along a particular direction of the input requires decreasing potentially all constraints with positive projection in that direction. This perspective would suggest that the mPoT and mcRBM do not appear to be well suited to provide a sparse representation in the overcomplete setting.
mcRBM、mPoT 和 ssRBM 都旨在对实值数据进行建模，使得隐藏单元不仅编码数据的条件均值，还编码其条件协方差。除了训练方案的不同之外，这些模型之间最显著的区别在于它们如何编码条件协方差。虽然 mcRBM 和 mPoT 使用隐藏单元的激活来对 $x$ 的协方差施加约束，而 ssRBM 则使用隐藏单元在对应权重向量指定的方向上夹紧精度矩阵。这两种建模条件协方差的方法在隐藏层维度与输入维度显著不同时会产生分歧。在过完备设置中，使用 ssRBM 参数化的稀疏激活只允许稀疏激活的隐藏单元在选定的方向上有方差。这是 ssRBM 与稀疏编码模型（Olshausen 和 Field，1996；Grosse 等，2007）共有的性质。 另一方面，在 mPoT 或 mcRBM 的情况下，对协方差施加的过完备约束集意味着要捕捉输入特定方向的任意协方差，需要降低在该方向上具有正投影的所有约束。这种观点表明，mPoT 和 mcRBM 似乎不适合在过完备设置中提供稀疏表示。

Many of the RBM training methods we discuss here are applicable to more general undirected graphical models, but are particularly practical in the RBM setting. Freund and Haussler (1994) proposed a learning algorithm for harmoniums (RBMs) based on projection pursuit. Contrastive Divergence (Hinton, 1999; Hinton et al., 2006) has been used most often to train RBMs, and many recent papers use Stochastic Maximum Likelihood (Younes, 1999; Tieleman, 2008).
许多我们在这里讨论的 RBM 训练方法适用于更一般的无向图模型，但在 RBM 设置中尤其实用。Freund 和 Haussler（1994）提出了一种基于投影追踪的谐波（RBM）学习算法。对比散度（Hinton，1999；Hinton 等，2006）最常被用来训练 RBM，许多近期论文使用了随机最大似然（Younes，1999；Tieleman，2008）。

As discussed in Sec. 6.1, in training probabilistic models parameters are typically adapted in order to maximize the likelihood of the training data (or equivalently the log-likelihood, or its penalized version, which adds a regularization term). With $T$ training examples, the log likelihood is given by:
如第 6.1 节所述，在训练概率模型时，参数通常会被调整以最大化训练数据的似然（或等价地，对数似然，或其惩罚版本，该版本添加了一个正则化项）。对于 $T$ 个训练样本，对数似然由以下公式给出：

where we have the expectations with respect to $p(h^{(t)}\mid x^{(t)})$ in the “clamped” condition (also called the positive phase), and over the full joint $p(x,h)$ in the “unclamped” condition (also called the negative phase). Intuitively, the gradient acts to locally move the model distribution (the negative phase distribution) toward the data distribution (positive phase distribution), by pushing down the energy of $(h,x^{(t)})$ pairs (for $h\sim P(h|x^{(t)})$) while pushing up the energy of $(h,x)$ pairs (for $(h,x)\sim P(h,x)$) until the two forces are in equilibrium, at which point the sufficient statistics (gradient of the energy function) have equal expectations with $x$ sampled from the training distribution or with $x$ sampled from the model.
在“夹紧”条件（也称为正相）下对 $p(h^{(t)}\mid x^{(t)})$ 的期望，以及在“未夹紧”条件（也称为负相）下对整个关节 $p(x,h)$ 的期望。直观上，梯度通过降低 $(h,x^{(t)})$ 对（对于 $h\sim P(h|x^{(t)})$ ）的能量并提高 $(h,x)$ 对（对于 $(h,x)\sim P(h,x)$ ）的能量，使模型分布（负相分布）局部移动到数据分布（正相分布），直到两种力达到平衡，此时充分统计量（能量函数的梯度）与从训练分布中抽取的 $x$ 或从模型中抽取的 $x$ 具有相同的期望。

The RBM conditional independence properties imply that the expectation in the positive phase of Eq. 15 is tractable. The negative phase term – arising from the partition function’s contribution to the log-likelihood gradient – is more problematic because the computation of the expectation over the joint is not tractable. The various ways of dealing with the partition function’s contribution to the gradient have brought about a number of different training algorithms, many trying to approximate the log-likelihood gradient.
RBM 的条件独立性属性意味着方程 15 的正相位的期望是可处理的。负相项——源于配分函数对对数似然梯度的贡献——更为复杂，因为对联合分布的期望计算不可处理。处理配分函数对梯度贡献的各种方法导致了多种不同的训练算法，许多算法试图近似对数似然梯度。

To approximate the expectation of the joint distribution in the negative phase contribution to the gradient, it is natural to again consider exploiting the conditional independence of the RBM in order to specify a Monte Carlo approximation of the expectation over the joint:
为了近似负梯度贡献的联合分布的期望，自然再次考虑利用 RBM 的条件独立性来指定联合期望的蒙特卡洛近似：