Efficient Estimation of Word Representations in Vector Space
矢量空间中单词表示的高效估算

Many current NLP systems and techniques treat words as atomic units - there is no notion of similarity between words, as these are represented as indices in a vocabulary. This choice has several good reasons - simplicity, robustness and the observation that simple models trained on huge amounts of data outperform complex systems trained on less data. An example is the popular -gram model used for statistical language modeling - today, it is possible to train -grams on virtually all available data (trillions of words [3]).
当前的许多 NLP 系统和技术都将单词视为原子单位，没有单词之间相似性的概念，因为这些单词在词汇表中表示为索引。这种选择有几个很好的理由--简单、稳健，以及观察到在海量数据上训练出来的简单模型优于在较少数据上训练出来的复杂系统。统计语言建模中常用的 -gram 模型就是一个很好的例子--如今，在几乎所有可用数据（数万亿单词[3]）上训练 -grams是可能的。

However, the simple techniques are at their limits in many tasks. For example, the amount of relevant in-domain data for automatic speech recognition is limited - the performance is usually dominated by the size of high quality transcribed speech data (often just millions of words). In machine translation, the existing corpora for many languages contain only a few billions of words or less. Thus, there are situations where simple scaling up of the basic techniques will not result in any significant progress, and we have to focus on more advanced techniques.
然而，在许多任务中，简单的技术已经达到了极限。例如，用于自动语音识别的相关域内数据量是有限的--性能通常取决于高质量转录语音数据的大小（通常只有数百万字）。在机器翻译中，许多语言的现有语料库仅包含几十亿字或更少。因此，在某些情况下，简单地扩大基本技术的规模不会带来任何重大进展，我们必须将重点放在更先进的技术上。

With progress of machine learning techniques in recent years, it has become possible to train more complex models on much larger data set, and they typically outperform the simple models. Probably the most successful concept is to use distributed representations of words [10]. For example, neural network based language models significantly outperform -gram models [1, 27, 17].
近年来，随着机器学习技术的进步，在更大的数据集上训练更复杂的模型已成为可能，而且这些模型通常优于简单的模型。最成功的概念可能是使用单词的分布式表示[10]。例如，基于神经网络的语言模型明显优于 -gram 模型[1, 27, 17]。

The main goal of this paper is to introduce techniques that can be used for learning high-quality word vectors from huge data sets with billions of words, and with millions of words in the vocabulary. As far as we know, none of the previously proposed architectures has been successfully trained on more
本文的主要目标是介绍可用于从拥有数十亿词汇和数百万词汇的庞大数据集中学习高质量词向量的技术。据我们所知，之前提出的架构都没有成功地在更多的数据集上进行过训练。
than a few hundred of millions of words, with a modest dimensionality of the word vectors between .
数亿词以上，词向量维度适中，介于 之间。

We use recently proposed techniques for measuring the quality of the resulting vector representations, with the expectation that not only will similar words tend to be close to each other, but that words can have multiple degrees of similarity [20]. This has been observed earlier in the context of inflectional languages - for example, nouns can have multiple word endings, and if we search for similar words in a subspace of the original vector space, it is possible to find words that have similar endings [13, [14].
我们使用最近提出的技术来衡量所得到的向量表示的质量，希望不仅相似的词往往彼此接近，而且词可以有多种相似度[20]。早先在词性语言的语境中就观察到了这一点--例如，名词可以有多种词尾，如果我们在原始向量空间的子空间中搜索相似词，就有可能找到词尾相似的词[13, [14]。

Somewhat surprisingly, it was found that similarity of word representations goes beyond simple syntactic regularities. Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that vector("King") - vector("Man") + vector("Woman") results in a vector that is closest to the vector representation of the word Queen [20].
令人惊讶的是，我们发现词语表征的相似性超出了简单的句法规律。例如，利用词偏移技术对词向量进行简单的代数运算，结果表明向量（"King"）-向量（"Man"）+向量（"Woman"）得到的向量最接近 Queen 这个词的向量表示[20]。

In this paper, we try to maximize accuracy of these vector operations by developing new model architectures that preserve the linear regularities among words. We design a new comprehensive test set for measuring both syntactic and semantic regularities and show that many such regularities can be learned with high accuracy. Moreover, we discuss how training time and accuracy depends on the dimensionality of the word vectors and on the amount of the training data.
在本文中，我们试图通过开发新的模型架构来最大限度地提高这些向量运算的准确性，从而保留词语之间的线性规律性。我们设计了一个新的综合测试集，用于测量句法和语义规则性 ，并证明可以高精度地学习到许多此类规则性。此外，我们还讨论了训练时间和准确性如何取决于词向量的维度和训练数据的数量。

Representation of words as continuous vectors has a long history [10, 26, 8]. A very popular model architecture for estimating neural network language model (NNLM) was proposed in [1], where a feedforward neural network with a linear projection layer and a non-linear hidden layer was used to learn jointly the word vector representation and a statistical language model. This work has been followed by many others.
将单词表示为连续向量的方法由来已久[10, 26, 8]。[1]中提出了一种非常流行的用于估算神经网络语言模型（NNLM）的模型结构，即使用带有线性投影层和非线性隐藏层的前馈神经网络来联合学习词向量表示和统计语言模型。这项工作之后又有许多人效仿。

Another interesting architecture of NNLM was presented in [13, 14], where the word vectors are first learned using neural network with a single hidden layer. The word vectors are then used to train the NNLM. Thus, the word vectors are learned even without constructing the full NNLM. In this work, we directly extend this architecture, and focus just on the first step where the word vectors are learned using a simple model.
文献[13, 14]提出了另一种有趣的 NNLM 结构，即首先使用单隐层神经网络学习单词向量。然后使用这些词向量来训练 NNLM。因此，即使不构建完整的 NNLM，也能学习到词向量。在这项工作中，我们直接扩展了这一架构，只关注第一步，即使用简单模型学习词向量。

It was later shown that the word vectors can be used to significantly improve and simplify many NLP applications [4, 5, 29]. Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 29, 23, 19, 9], and some of the resulting word vectors were made available for future research and comparison . However, as far as we know, these architectures were significantly more computationally expensive for training than the one proposed in [13], with the exception of certain version of log-bilinear model where diagonal weight matrices are used [23].
后来的研究表明，词向量可用于显著改善和简化许多 NLP 应用 [4, 5, 29]。词向量的估算本身使用了不同的模型架构，并在不同的语料库上进行了训练[4, 29, 23, 19, 9]，得到的一些词向量可供未来的研究和比较 。然而，据我们所知，除了使用对角权重矩阵的对数-线性模型的某些版本外[23]，这些架构的训练计算成本明显高于[13]中提出的架构。

Many different types of models were proposed for estimating continuous representations of words, including the well-known Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA). In this paper, we focus on distributed representations of words learned by neural networks, as it was previously shown that they perform significantly better than LSA for preserving linear regularities among words [20, 31]; LDA moreover becomes computationally very expensive on large data sets.
人们提出了许多不同类型的模型来估计词语的连续表征，其中包括著名的潜在语义分析（LSA）和潜在德里希特分配（LDA）。在本文中，我们将重点关注通过神经网络学习到的词的分布式表示，因为之前的研究表明，神经网络在保持词之间的线性规律性方面的表现要明显优于 LSA [20, 31]；此外，LDA 在大型数据集上的计算成本也非常高昂。

Similar to [18], to compare different model architectures we define first the computational complexity of a model as the number of parameters that need to be accessed to fully train the model. Next, we will try to maximize the accuracy, while minimizing the computational complexity.
与文献[18]类似，为了比较不同的模型架构，我们首先将模型的计算复杂度定义为完全训练模型所需的参数数量。接下来，我们将尝试在最大限度降低计算复杂度的同时，最大限度地提高准确度。

For all the following models, the training complexity is proportional to
对于以下所有模型，训练复杂度都与

where is number of the training epochs, is the number of the words in the training set and is defined further for each model architecture. Common choice is and up to one billion. All models are trained using stochastic gradient descent and backpropagation [26].
其中， 是训练历元数， 是训练集中的单词数， 是针对每个模型架构的进一步定义。常见的选择是 和 ，最高可达 10 亿。所有模型均采用随机梯度下降法和反向传播法进行训练[26]。

The probabilistic feedforward neural network language model has been proposed in [1]. It consists of input, projection, hidden and output layers. At the input layer, previous words are encoded using 1 -of- coding, where is size of the vocabulary. The input layer is then projected to a projection layer that has dimensionality , using a shared projection matrix. As only inputs are active at any given time, composition of the projection layer is a relatively cheap operation.
概率前馈神经网络语言模型是在 [1] 中提出的。它由输入层、投影层、隐藏层和输出层组成。在输入层， 前一个词使用 1 的 编码，其中 是词汇量的大小。然后，使用共享投影矩阵将输入层投影到投影层 ，投影层的维度为 。由于在任何给定时间内只有 输入处于活动状态，因此组成投影层的操作成本相对较低。

The NNLM architecture becomes complex for computation between the projection and the hidden layer, as values in the projection layer are dense. For a common choice of , the size of the projection layer might be 500 to 2000 , while the hidden layer size is typically 500 to 1000 units. Moreover, the hidden layer is used to compute probability distribution over all the words in the vocabulary, resulting in an output layer with dimensionality . Thus, the computational complexity per each training example is
NNLM 架构在投影层和隐藏层之间的计算变得复杂，因为投影层中的值是密集的。对于 的常见选择，投影层的大小 可能是 500 到 2000，而隐藏层的大小 通常是 500 到 1000 个单位。此外，隐层用于计算词汇表中所有单词的概率分布，因此输出层的维度为 。因此，每个训练实例的计算复杂度为

where the dominating term is . However, several practical solutions were proposed for avoiding it; either using hierarchical versions of the softmax [25, 23, 18], or avoiding normalized models completely by using models that are not normalized during training [4, 9]. With binary tree representations of the vocabulary, the number of output units that need to be evaluated can go down to around . Thus, most of the complexity is caused by the term .
其中主导项为 。然而，为了避免这种情况，人们提出了几种实用的解决方案：或者使用分层版本的 softmax [25, 23, 18]，或者通过使用在训练过程中未归一化的模型来完全避免归一化模型 [4, 9]。使用二叉树表示词汇时，需要评估的输出单元数量可以减少到 左右。因此，大部分复杂性都是由术语 引起的。

In our models, we use hierarchical softmax where the vocabulary is represented as a Huffman binary tree. This follows previous observations that the frequency of words works well for obtaining classes in neural net language models [16]. Huffman trees assign short binary codes to frequent words, and this further reduces the number of output units that need to be evaluated: while balanced binary tree would require outputs to be evaluated, the Huffman tree based hierarchical softmax requires only about Unigram_perplexity . For example when the vocabulary size is one million words, this results in about two times speedup in evaluation. While this is not crucial speedup for neural network LMs as the computational bottleneck is in the term, we will later propose architectures that do not have hidden layers and thus depend heavily on the efficiency of the softmax normalization.
在我们的模型中，我们使用分层软最大值（hierarchical softmax），将词汇表表示为哈夫曼二叉树（Huffman binary tree）。根据之前的观察，在神经网络语言模型中，词频可以很好地用于获取类别[16]。哈夫曼树为频繁出现的单词分配了简短的二进制代码，这进一步减少了需要评估的输出单元数量：平衡二叉树需要评估 个输出，而基于哈夫曼树的分层软最大值只需要约 Unigram_perplexity 。例如，当词汇量为一百万词时，评估速度将提高两倍。由于计算瓶颈在于 项，因此这对神经网络 LM 的速度提升并不重要，但我们稍后将提出没有隐藏层的架构，因此这在很大程度上取决于 softmax 归一化的效率。

Recurrent neural network based language model has been proposed to overcome certain limitations of the feedforward NNLM, such as the need to specify the context length (the order of the model ), and because theoretically RNNs can efficiently represent more complex patterns than the shallow neural networks [15, 2]. The RNN model does not have a projection layer; only input, hidden and output layer. What is special for this type of model is the recurrent matrix that connects hidden layer to itself, using time-delayed connections. This allows the recurrent model to form some kind of short term memory, as information from the past can be represented by the hidden layer state that gets updated based on the current input and the state of the hidden layer in the previous time step.
提出基于递归神经网络的语言模型是为了克服前馈 NNLM 的某些局限性，例如需要指定上下文长度（模型的阶数 ），而且从理论上讲，RNN 比浅层神经网络能有效地表示更复杂的模式[15, 2]。RNN 模型没有投影层，只有输入层、隐藏层和输出层。这种模型的特殊之处在于使用延时连接将隐藏层与自身连接起来的递归矩阵。这使得递归模型能够形成某种短期记忆，因为过去的信息可以通过隐层状态来表示，而隐层状态则根据当前输入和上一时间步的隐层状态进行更新。

The complexity per training example of the RNN model is
RNN 模型每个训练实例的复杂度为

where the word representations have the same dimensionality as the hidden layer . Again, the term can be efficiently reduced to by using hierarchical softmax. Most of the complexity then comes from .
其中，词表示 与隐藏层 的维度相同。同样，通过使用分层软最大值（hierarchical softmax），可以将 有效地缩减为 。这样，大部分复杂性就来自于 。

To train models on huge data sets, we have implemented several models on top of a large-scale distributed framework called DistBelief [6], including the feedforward NNLM and the new models proposed in this paper. The framework allows us to run multiple replicas of the same model in parallel, and each replica synchronizes its gradient updates through a centralized server that keeps all the parameters. For this parallel training, we use mini-batch asynchronous gradient descent with an adaptive learning rate procedure called Adagrad [7]. Under this framework, it is common to use one hundred or more model replicas, each using many CPU cores at different machines in a data center.
为了在海量数据集上训练模型，我们在一个名为 DistBelief [6] 的大规模分布式框架上实现了几个模型，包括前馈 NNLM 和本文提出的新模型。该框架允许我们并行运行同一模型的多个副本，每个副本通过一个保存所有参数的集中式服务器同步梯度更新。在这种并行训练中，我们使用迷你批次异步梯度下降法，并采用名为 Adagrad 的自适应学习率程序 [7]。在此框架下，通常会使用 100 个或更多模型副本，每个副本使用数据中心不同机器上的多个 CPU 内核。

In this section, we propose two new model architectures for learning distributed representations of words that try to minimize computational complexity. The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model. While this is what makes neural networks so attractive, we decided to explore simpler models that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently.
在本节中，我们提出了两种用于学习单词分布式表征的新模型架构，以尽量降低计算复杂度。上一节的主要观点是，大部分复杂性是由模型中的非线性隐藏层造成的。虽然这正是神经网络的魅力所在，但我们决定探索更简单的模型，这些模型可能无法像神经网络那样精确地表示数据，但却可以在更多的数据上高效地进行训练。

The new architectures directly follow those proposed in our earlier work [13, 14], where it was found that neural network language model can be successfully trained in two steps: first, continuous word vectors are learned using simple model, and then the N-gram NNLM is trained on top of these distributed representations of words. While there has been later substantial amount of work that focuses on learning word vectors, we consider the approach proposed in [13] to be the simplest one. Note that related models have been proposed also much earlier [26, 8].
新架构直接沿用了我们早期工作[13, 14]中提出的架构，我们发现神经网络语言模型可以通过两个步骤训练成功：首先，使用简单模型学习连续词向量，然后在这些分布式词表征的基础上训练 N-gram NNLM。虽然后来有大量的工作都集中在学习单词向量上，但我们认为 [13] 中提出的方法是最简单的方法。需要注意的是，相关模型的提出也要早得多 [26, 8]。

The first proposed architecture is similar to the feedforward NNLM, where the non-linear hidden layer is removed and the projection layer is shared for all words (not just the projection matrix); thus, all words get projected into the same position (their vectors are averaged). We call this architecture a bag-of-words model as the order of words in the history does not influence the projection. Furthermore, we also use words from the future; we have obtained the best performance on the task introduced in the next section by building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word. Training complexity is then
第一个提议的架构类似于前馈 NNLM，其中的非线性隐藏层被移除，所有单词共享投影层（而不仅仅是投影矩阵）；因此，所有单词都会被投影到相同的位置（它们的向量被平均化）。我们称这种结构为词袋模型，因为词在历史记录中的顺序不会影响投影。此外，我们还使用了来自未来的单词；在下一节介绍的任务中，我们通过建立一个对数线性分类器，输入四个未来单词和四个历史单词，训练标准是正确分类当前（中间）单词，从而获得了最佳性能。训练复杂度为

We denote this model further as CBOW, as unlike standard bag-of-words model, it uses continuous distributed representation of the context. The model architecture is shown at Figure 1. Note that the weight matrix between the input and the projection layer is shared for all word positions in the same way as in the NNLM.
我们将该模型进一步命名为 CBOW，因为与标准的词袋模型不同，它使用连续的分布式上下文表示法。模型架构如图 1 所示。请注意，输入层和投影层之间的权重矩阵与 NNLM 中的方式相同，都是共享所有词位的权重矩阵。

The second architecture is similar to CBOW, but instead of predicting the current word based on the context, it tries to maximize classification of a word based on another word in the same sentence. More precisely, we use each current word as an input to a log-linear classifier with continuous projection layer, and predict words within a certain range before and after the current word. We found that increasing the range improves quality of the resulting word vectors, but it also increases the computational complexity. Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples.
第二种结构与 CBOW 类似，但它不是根据上下文预测当前单词，而是试图根据同一句子中的另一个单词最大化单词分类。更确切地说，我们将每个当前单词作为带有连续投影层的对数线性分类器的输入，并预测当前单词前后一定范围内的单词。我们发现，扩大范围可以提高得到的词向量的质量，但同时也会增加计算复杂度。由于距离较远的词与当前词的相关性通常低于距离较近的词，因此我们在训练示例中减少了对距离较远的词的采样，从而降低了这些词的权重。

The training complexity of this architecture is proportional to
这种结构的训练复杂度与

where is the maximum distance of the words. Thus, if we choose , for each training word we will select randomly a number in range , and then use words from history and
其中 是单词的最大距离。因此，如果我们选择 ，那么对于每个训练词，我们将在 的范围内随机选择一个数字 ，然后使用历史中的 词，并将其作为训练词。

Figure 1: New model architectures. The CBOW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word.
图 1：新的模型架构。CBOW 架构根据上下文预测当前单词，Skip-gram 则根据当前单词预测周边单词。

words from the future of the current word as correct labels. This will require us to do word classifications, with the current word as input, and each of the words as output. In the following experiments, we use .
当前单词的未来单词作为正确标签。这就要求我们以当前单词为输入，以 的每个单词为输出，进行 单词分类。在下面的实验中，我们使用 。

To compare the quality of different versions of word vectors, previous papers typically use a table showing example words and their most similar words, and understand them intuitively. Although it is easy to show that word France is similar to Italy and perhaps some other countries, it is much more challenging when subjecting those vectors in a more complex similarity task, as follows. We follow previous observation that there can be many different types of similarities between words, for example, word big is similar to bigger in the same sense that small is similar to smaller. Example of another type of relationship can be word pairs big - biggest and small - smallest [20]. We further denote two pairs of words with the same relationship as a question, as we can ask: "What is the word that is similar to small in the same sense as biggest is similar to big?"
为了比较不同版本单词向量的质量，以往的论文通常使用一个表格来显示示例单词及其最相似的单词，并直观地理解它们。虽然很容易看出单词 "法国"（France）与 "意大利"（Italy）以及其他一些国家的单词相似，但要将这些向量用于更复杂的相似性任务，难度就大得多了。根据我们之前的观察，词与词之间可能存在许多不同类型的相似性，例如，词 big 与 bigger 的相似性与词 small 与 smaller 的相似性相同。另一种关系的例子是词对 big - biggest 和 small - smallest [20]。我们还可以将具有相同关系的两对词表示为一个问题，如我们可以问："在 biggest 与 big 相似的意义上，与 small 相似的词是什么？

Somewhat surprisingly, these questions can be answered by performing simple algebraic operations with the vector representation of words. To find a word that is similar to small in the same sense as biggest is similar to big, we can simply compute vector vector("biggest") - vector("big") + vector ("small"). Then, we search in the vector space for the word closest to measured by cosine distance, and use it as the answer to the question (we discard the input question words during this search). When the word vectors are well trained, it is possible to find the correct answer (word smallest) using this method.
令人惊讶的是，这些问题都可以通过对单词的向量表示进行简单的代数运算来解答。要找到一个与 "小 "相似的词，就像 "大 "与 "小 "相似一样，我们只需计算向量 向量（"最大"）- 向量（"大"）+ 向量（"小"）。然后，我们在向量空间中搜索最接近 的单词，并将其作为问题的答案（在搜索过程中，我们会丢弃输入的问题单词）。当单词向量训练有素时，就有可能用这种方法找到正确答案（单词最小）。

Finally, we found that when we train high dimensional word vectors on a large amount of data, the resulting vectors can be used to answer very subtle semantic relationships between words, such as a city and the country it belongs to, e.g. France is to Paris as Germany is to Berlin. Word vectors with such semantic relationships could be used to improve many existing NLP applications, such as machine translation, information retrieval and question answering systems, and may enable other future applications yet to be invented.
最后，我们发现，当我们在大量数据上训练高维词向量时，得到的向量可以用来回答词与词之间非常微妙的语义关系，例如一个城市和它所属的国家，如法国之于巴黎，德国之于柏林。具有这种语义关系的词向量可用于改进许多现有的 NLP 应用，如机器翻译、信息检索和问题解答系统，并可用于其他尚未发明的未来应用。

Table 1: Examples of five types of semantic and nine types of syntactic questions in the SemanticSyntactic Word Relationship test set.
表 1：语义句法词语关系测试集中的五类语义问题和九类句法问题示例。

To measure quality of the word vectors, we define a comprehensive test set that contains five types of semantic questions, and nine types of syntactic questions. Two examples from each category are shown in Table 1. Overall, there are 8869 semantic and 10675 syntactic questions. The questions in each category were created in two steps: first, a list of similar word pairs was created manually. Then, a large list of questions is formed by connecting two word pairs. For example, we made a list of 68 large American cities and the states they belong to, and formed about questions by picking two word pairs at random. We have included in our test set only single token words, thus multi-word entities are not present (such as New York).
为了衡量词向量的质量，我们定义了一个综合测试集，其中包含五类语义问题和九类句法问题。表 1 显示了每类问题的两个示例。总共有 8869 个语义问题和 10675 个句法问题。每个类别的问题都是分两步创建的：首先，人工创建相似词对列表。然后，通过连接两个词对形成一个大的问题列表。例如，我们制作了一个包含 68 个美国大城市及其所属州的列表，并通过随机挑选两个词对形成了约 个问题。我们的测试集只包含单标记词，因此不包含多词实体（如纽约）。

We evaluate the overall accuracy for all question types, and for each question type separately (semantic, syntactic). Question is assumed to be correctly answered only if the closest word to the vector computed using the above method is exactly the same as the correct word in the question; synonyms are thus counted as mistakes. This also means that reaching accuracy is likely to be impossible, as the current models do not have any input information about word morphology. However, we believe that usefulness of the word vectors for certain applications should be positively correlated with this accuracy metric. Further progress can be achieved by incorporating information about structure of words, especially for the syntactic questions.
我们对所有问题类型以及每种问题类型（语义、句法）分别进行总体准确率评估。只有当使用上述方法计算出的向量中最接近的单词与问题中的正确单词完全相同时，问题才会被认为是正确回答；因此，同义词也被算作错误。这也意味着，要达到 的准确率很可能是不可能的，因为目前的模型没有任何关于词形的输入信息。不过，我们相信，词向量在某些应用中的实用性应该与这一准确度指标呈正相关。通过加入词的结构信息，尤其是句法问题的结构信息，我们可以取得更大的进步。

We have used a Google News corpus for training the word vectors. This corpus contains about 6B tokens. We have restricted the vocabulary size to 1 million most frequent words. Clearly, we are facing time constrained optimization problem, as it can be expected that both using more data and higher dimensional word vectors will improve the accuracy. To estimate the best choice of model architecture for obtaining as good as possible results quickly, we have first evaluated models trained on subsets of the training data, with vocabulary restricted to the most frequent words. The results using the CBOW architecture with different choice of word vector dimensionality and increasing amount of the training data are shown in Table 2.
我们使用谷歌新闻语料库来训练词向量。该语料库包含约 6B 个词库。我们将词汇量限制在 100 万个最常出现的词。显然，我们面临的是时间有限的优化问题，因为可以预期，使用更多数据和更高维度的词向量都会提高准确率。为了估算模型架构的最佳选择，以便尽快获得尽可能好的结果，我们首先评估了在训练数据子集上训练的模型，并将词汇量限制在最常见的 单词上。表 2 列出了使用 CBOW 架构、不同的词向量维度选择和不断增加的训练数据量所得到的结果。

It can be seen that after some point, adding more dimensions or adding more training data provides diminishing improvements. So, we have to increase both vector dimensionality and the amount of the training data together. While this observation might seem trivial, it must be noted that it is currently popular to train word vectors on relatively large amounts of data, but with insufficient size
可以看出，在某些情况下，增加维度或增加训练数据所带来的改进是递减的。因此，我们必须同时增加向量维度和训练数据量。虽然这一观点看似微不足道，但必须指出的是，目前流行的做法是在相对较大的数据量上训练词向量，但在数据量不足的情况下

Table 2: Accuracy on subset of the Semantic-Syntactic Word Relationship test set, using word vectors from the CBOW architecture with limited vocabulary. Only questions containing words from the most frequent words are used.
表 2：在语义-句法单词关系测试集子集上的准确率，使用来自 CBOW 架构的单词向量，词汇量有限。只使用包含最常见 词的问题。

Table 3: Comparison of architectures using models trained on the same data, with 640-dimensional word vectors. The accuracies are reported on our Semantic-Syntactic Word Relationship test set, and on the syntactic relationship test set of [20]
表 3：使用在相同数据（640 维词向量）上训练的模型的架构比较。准确率是在我们的语义-句法词关系测试集和 [20] 的句法关系测试集上报告的。

(such as 50 - 100). Given Equation 4, increasing amount of training data twice results in about the same increase of computational complexity as increasing vector size twice.
(例如 50 - 100）。考虑到等式 4，将训练数据量增加两倍与将向量大小增加两倍的计算复杂度增幅大致相同。

For the experiments reported in Tables 2 and 4 , we used three training epochs with stochastic gradient descent and backpropagation. We chose starting learning rate 0.025 and decreased it linearly, so that it approaches zero at the end of the last training epoch.
在表 2 和表 4 中的实验中，我们使用了随机梯度下降和反向传播的三个训练历元。我们选择了 0.025 的起始学习率，并将其线性递减，使其在最后一个训练历元结束时接近于零。

First we compare different model architectures for deriving the word vectors using the same training data and using the same dimensionality of 640 of the word vectors. In the further experiments, we use full set of questions in the new Semantic-Syntactic Word Relationship test set, i.e. unrestricted to the 30k vocabulary. We also include results on a test set introduced in [20] that focuses on syntactic similarity between word
首先，我们使用相同的训练数据和640个词向量的相同维度，比较了推导词向量的不同模型架构。在进一步的实验中，我们使用了新的语义-句法词关系测试集中的全部问题集，即不局限于30k词汇量。我们还使用了[20]中引入的测试集的结果，该测试集主要关注词与词之间的句法相似性。

The training data consists of several LDC corpora and is described in detail in [18] (320M words, vocabulary). We used these data to provide a comparison to a previously trained recurrent neural network language model that took about 8 weeks to train on a single CPU. We trained a feedforward NNLM with the same number of 640 hidden units using the DistBelief parallel training [6], using a history of 8 previous words (thus, the NNLM has more parameters than the RNNLM, as the projection layer has size ).
训练数据由多个 LDC 语料库组成，详细描述见 [18]（3.2 亿字， 词汇量）。我们使用这些数据与之前训练的递归神经网络语言模型进行了比较，后者在单 CPU 上的训练时间约为 8 周。我们使用 DistBelief 并行训练[6]训练了一个前馈 NNLM，它具有相同数量的 640 个隐藏单元，并使用了 8 个先前单词的历史记录（因此，NNLM 比 RNNLM 具有更多参数，因为投影层的大小为 ）。

In Table 3, it can be seen that the word vectors from the RNN (as used in [20]) perform well mostly on the syntactic questions. The NNLM vectors perform significantly better than the RNN - this is not surprising, as the word vectors in the RNNLM are directly connected to a non-linear hidden layer. The CBOW architecture works better than the NNLM on the syntactic tasks, and about the same on the semantic one. Finally, the Skip-gram architecture works slightly worse on the syntactic task than the CBOW model (but still better than the NNLM), and much better on the semantic part of the test than all the other models.
从表 3 中可以看出，来自 RNN 的词向量（如 [20] 中所用）在大部分句法问题上表现良好。NNLM 向量的表现明显优于 RNN--这并不奇怪，因为 RNNLM 中的单词向量是直接连接到非线性隐藏层的。CBOW 架构在句法任务上的表现优于 NNLM，在语义任务上的表现大致相同。最后，Skip-gram 架构在句法任务上的表现略逊于 CBOW 模型（但仍优于 NNLM），而在测试的语义部分则远远优于所有其他模型。

Next, we evaluated our models trained using one CPU only and compared the results against publicly available word vectors. The comparison is given in Table 4 The CBOW model was trained on subset
接下来，我们对仅使用一个 CPU 训练的模型进行了评估，并将结果与公开的词向量进行了比较。比较结果见表 4。

Table 4: Comparison of publicly available word vectors on the Semantic-Syntactic Word Relationship test set, and word vectors from our models. Full vocabularies are used.
表 4：语义-句法单词关系测试集上的公开单词向量与我们的模型中的单词向量的比较。使用的是完整词汇表。

Table 5: Comparison of models trained for three epochs on the same data and models trained for one epoch. Accuracy is reported on the full Semantic-Syntactic data set.
表 5：在相同数据上经过三个历时训练的模型与经过一个历时训练的模型的比较。准确率是在完整的语义-句法数据集上报告的。

of the Google News data in about a day, while training time for the Skip-gram model was about three days.
谷歌新闻数据的训练时间约为一天，而 Skip-gram 模型的训练时间约为三天。

For experiments reported further, we used just one training epoch (again, we decrease the learning rate linearly so that it approaches zero at the end of training). Training a model on twice as much data using one epoch gives comparable or better results than iterating over the same data for three epochs, as is shown in Table 5, and provides additional small speedup.
在进一步报告的实验中，我们只使用了一个训练历元（我们再次线性降低学习率，使其在训练结束时趋近于零）。如表 5 所示，使用一个历元在两倍的数据上训练模型的结果与在相同数据上迭代三个历元的结果相当，甚至更好，而且还能带来额外的小幅提速。

As mentioned earlier, we have implemented various models in a distributed framework called DistBelief. Below we report the results of several models trained on the Google News 6B data set, with mini-batch asynchronous gradient descent and the adaptive learning rate procedure called Adagrad [7]. We used 50 to 100 model replicas during the training. The number of CPU cores is an
如前所述，我们在名为 DistBelief 的分布式框架中实现了各种模型。下面我们将报告在谷歌新闻 6B 数据集上使用迷你批次异步梯度下降和名为 Adagrad 的自适应学习率程序[7]训练的几个模型的结果。在训练过程中，我们使用了 50 到 100 个模型副本。中央处理器内核数

Table 6: Comparison of models trained using the DistBelief distributed framework. Note that training of NNLM with 1000-dimensional vectors would take too long to complete.
表 6：使用 DistBelief 分布式框架训练的模型比较。请注意，使用 1000 维向量训练 NNLM 需要很长时间才能完成。

Table 7: Comparison and combination of models on the Microsoft Sentence Completion Challenge.
表 7：微软 "句子完成挑战 "的模型比较与组合。

estimate since the data center machines are shared with other production tasks, and the usage can fluctuate quite a bit. Note that due to the overhead of the distributed framework, the CPU usage of the CBOW model and the Skip-gram model are much closer to each other than their single-machine implementations. The result are reported in Table 6.
由于数据中心的机器是与其他生产任务共享的，因此使用率会有很大波动。需要注意的是，由于分布式框架的开销，CBOW 模型和 Skip-gram 模型的 CPU 占用率比它们的单机实现更为接近。结果见表 6。

The Microsoft Sentence Completion Challenge has been recently introduced as a task for advancing language modeling and other NLP techniques [32]. This task consists of 1040 sentences, where one word is missing in each sentence and the goal is to select word that is the most coherent with the rest of the sentence, given a list of five reasonable choices. Performance of several techniques has been already reported on this set, including -gram models, LSA-based model [32], log-bilinear model [24] and a combination of recurrent neural networks that currently holds the state of the art performance of accuracy on this benchmark [19].
微软句子补全挑战赛（Microsoft Sentence Completion Challenge）是最近推出的一项推进语言建模和其他 NLP 技术的任务[32]。该任务由 1040 个句子组成，每个句子中缺少一个单词，目标是在给出的五个合理选项中选出与句子其余部分最连贯的单词。在这一组句子中，有几种技术的性能已经得到了报道，包括 -gram 模型、基于 LSA 的模型 [32]、对数-双线性模型 [24] 以及循环神经网络的组合，该组合目前在这一基准上的准确率为 [19]。

We have explored the performance of Skip-gram architecture on this task. First, we train the 640dimensional model on 50M words provided in [32]. Then, we compute score of each sentence in the test set by using the unknown word at the input, and predict all surrounding words in a sentence. The final sentence score is then the sum of these individual predictions. Using the sentence scores, we choose the most likely sentence.
我们探索了 Skip-gram 架构在这项任务中的表现。首先，我们在 [32] 提供的 5000 万个单词上训练 640 维模型。然后，我们以未知单词为输入，计算测试集中每个句子的得分，并预测句子中的所有周边单词。最终的句子得分就是这些单个预测的总和。通过句子得分，我们选出最有可能的句子。

A short summary of some previous results together with the new results is presented in Table 77 While the Skip-gram model itself does not perform on this task better than LSA similarity, the scores from this model are complementary to scores obtained with RNNLMs, and a weighted combination leads to a new state of the art result accuracy ( on the development part of the set and on the test part of the set).
表 77 简要概括了之前的一些结果和新的结果。虽然跳越语法模型本身在这项任务中的表现并不优于 LSA 相似性，但该模型的得分与 RNNLM 的得分相辅相成，通过加权组合得出了新的先进结果 准确率（ ，用于开发集的开发部分， ，用于测试集的测试部分）。

Table 8 shows words that follow various relationships. We follow the approach described above: the relationship is defined by subtracting two word vectors, and the result is added to another word. Thus for example, Paris - France + Italy = Rome. As it can be seen, accuracy is quite good, although there is clearly a lot of room for further improvements (note that using our accuracy metric that
表 8 显示了遵循各种关系的词语。我们采用了上述方法：通过减去两个词向量来定义关系，然后将结果加到另一个词上。例如，巴黎 - 法国 + 意大利 = 罗马。可以看出，准确率相当不错，但显然还有很大的改进空间（请注意，使用我们的准确率度量标准，即"......"）。

Table 8: Examples of the word pair relationships, using the best word vectors from Table 4 (Skipgram model trained on words with 300 dimensionality).
表 8：使用表 4 中最佳词向量的词对关系示例（Skipgram 模型以 词为训练对象，维度为 300）。

assumes exact match, the results in Table 8 would score only about . We believe that word vectors trained on even larger data sets with larger dimensionality will perform significantly better, and will enable the development of new innovative applications. Another way to improve accuracy is to provide more than one example of the relationship. By using ten examples instead of one to form the relationship vector (we average the individual vectors together), we have observed improvement of accuracy of our best models by about absolutely on the semantic-syntactic test.
假设完全匹配，表 8 中的结果得分仅为 左右。我们相信，在维度更大的数据集上训练的词向量性能会好得多，并能开发出新的创新应用。提高准确率的另一种方法是提供不止一个关系示例。通过使用十个而不是一个例子来形成关系向量（我们将单个向量平均到一起），我们观察到我们的最佳模型在语义-句法测试中的准确率绝对提高了约 。

It is also possible to apply the vector operations to solve different tasks. For example, we have observed good accuracy for selecting out-of-the-list words, by computing average vector for a list of words, and finding the most distant word vector. This is a popular type of problems in certain human intelligence tests. Clearly, there is still a lot of discoveries to be made using these techniques.
我们还可以应用向量运算来解决不同的任务。例如，我们观察到，通过计算单词列表的平均向量和查找最远的单词向量，选择列表外单词的准确率很高。这是某些人类智力测验中的常见问题类型。显然，利用这些技术仍有许多新发现。

In this paper we studied the quality of vector representations of words derived by various models on a collection of syntactic and semantic language tasks. We observed that it is possible to train high quality word vectors using very simple model architectures, compared to the popular neural network models (both feedforward and recurrent). Because of the much lower computational complexity, it is possible to compute very accurate high dimensional word vectors from a much larger data set. Using the DistBelief distributed framework, it should be possible to train the CBOW and Skip-gram models even on corpora with one trillion words, for basically unlimited size of the vocabulary. That is several orders of magnitude larger than the best previously published results for similar models.
在本文中，我们研究了各种模型在一系列句法和语义语言任务中得出的词向量表示的质量。我们发现，与流行的神经网络模型（包括前馈和递归模型）相比，使用非常简单的模型架构就可以训练出高质量的词向量。由于计算复杂度低得多，因此可以从更大的数据集中计算出非常精确的高维词向量。使用 DistBelief 分布式框架，即使在词汇量基本不受限制的情况下，也可以在拥有一万亿个单词的语料库中训练 CBOW 和 Skip-gram 模型。这比之前发布的类似模型的最佳结果要大几个数量级。

An interesting task where the word vectors have recently been shown to significantly outperform the previous state of the art is the SemEval-2012 Task 2 [11]. The publicly available RNN vectors were used together with other techniques to achieve over increase in Spearman's rank correlation over the previous best result [31]. The neural network based word vectors were previously applied to many other NLP tasks, for example sentiment analysis [12] and paraphrase detection [28]. It can be expected that these applications can benefit from the model architectures described in this paper.
SemEval-2012 任务 2 [11]是一项有趣的任务，在这项任务中，词向量的性能最近被证明大大超过了以前的技术水平。公开的 RNN 向量与其他技术一起使用，使 Spearman's rank correlation（斯皮尔曼等级相关性）比之前的最佳结果提高了 [31]。基于神经网络的词向量曾被应用于许多其他 NLP 任务，例如情感分析 [12] 和转述检测 [28]。可以预见，这些应用将受益于本文所述的模型架构。

Our ongoing work shows that the word vectors can be successfully applied to automatic extension of facts in Knowledge Bases, and also for verification of correctness of existing facts. Results from machine translation experiments also look very promising. In the future, it would be also interesting to compare our techniques to Latent Relational Analysis [30] and others. We believe that our comprehensive test set will help the research community to improve the existing techniques for estimating the word vectors. We also expect that high quality word vectors will become an important building block for future NLP applications.
我们正在进行的工作表明，词向量可以成功地应用于知识库中事实的自动扩展，也可以用于验证现有事实的正确性。机器翻译实验的结果也很有希望。未来，将我们的技术与潜在关系分析（Latent Relational Analysis）[30]和其他技术进行比较也会很有趣。我们相信，我们的综合测试集将有助于研究界改进现有的词向量估算技术。我们还希望高质量的词向量将成为未来 NLP 应用的重要组成部分。

After the initial version of this paper was written, we published single-machine multi-threaded C++ code for computing the word vectors, using both the continuous bag-of-words and skip-gram architecture The training speed is significantly higher than reported earlier in this paper, i.e. it is in the order of billions of words per hour for typical hyperparameter choices. We also published more than 1.4 million vectors that represent named entities, trained on more than 100 billion words. Some of our follow-up work will be published in an upcoming NIPS 2013 paper [21].
在本文最初版本撰写完成后，我们发布了计算词向量的单机多线程 C++ 代码，同时使用了连续词袋和跳格架构 ，训练速度明显高于本文之前的报告，即在典型的超参数选择下，训练速度达到了每小时数十亿词的数量级。我们还发布了代表命名实体的 140 多万个向量，这些向量是在超过 1000 亿字的基础上训练出来的。我们的一些后续工作将发表在即将出版的 NIPS 2013 论文[21]中。

[1] Y. Bengio, R. Ducharme, P. Vincent. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.
[1] Y. Bengio, R. Ducharme, P. Vincent.神经概率语言模型。Journal of Machine Learning Research, 3:1137-1155, 2003.

[2] Y. Bengio, Y. LeCun. Scaling learning algorithms towards AI. In: Large-Scale Kernel Machines, MIT Press, 2007.
[2] Y. Bengio, Y. LeCun.面向人工智能的扩展学习算法。In：大规模内核机器》，麻省理工学院出版社，2007 年。

[3] T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. Large language models in machine translation. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, 2007.
[3] T. Brants、A. C. Popat、P. Xu、F. J. Och 和 J. Dean。机器翻译中的大型语言模型。In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, 2007.

[4] R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In International Conference on Machine Learning, ICML, 2008.
[4] R. Collobert 和 J. Weston.自然语言处理的统一架构：深度神经网络与多任务学习》。国际机器学习会议，ICML，2008 年。

[5] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12:2493 .
[5] R. Collobert、J. Weston、L. Bottou、M. Karlen、K. Kavukcuoglu 和 P. Kuksa。自然语言处理（几乎）从零开始》。机器学习研究期刊》，12:2493 。

[6] J. Dean, G.S. Corrado, R. Monga, K. Chen, M. Devin, Q.V. Le, M.Z. Mao, M.A. Ranzato, A. Senior, P. Tucker, K. Yang, A. Y. Ng., Large Scale Distributed Deep Networks, NIPS, 2012.
[6] J. Dean、G.S. Corrado、R. Monga、K. Chen、M. Devin、Q.V. Le、M.Z. Mao、M.A. Ranzato、A. Senior、P. Tucker、K. Yang、A. Y. Ng，《大规模分布式深度网络》，NIPS，2012。

[7] J.C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011.
[7] J.C. Duchi, E. Hazan, and Y. Singer.在线学习和随机优化的自适应子梯度方法。机器学习研究》期刊，2011 年。

[8] J. Elman. Finding Structure in Time. Cognitive Science, 14, 179-211, 1990.
[8] J. Elman.Finding Structure in Time.Cognitive Science, 14, 179-211, 1990.

[9] Eric H. Huang, R. Socher, C. D. Manning and Andrew Y. Ng. Improving Word Representations via Global Context and Multiple Word Prototypes. In: Proc. Association for Computational Linguistics, 2012.
[9] Eric H. Huang、R. Socher、C. D. Manning 和 Andrew Y. Ng。通过全局上下文和多个单词原型改进单词表征。In：Proc.计算语言学协会，2012 年。

[10] G.E. Hinton, J.L. McClelland, D.E. Rumelhart. Distributed representations. In: Parallel distributed processing: Explorations in the microstructure of cognition. Volume 1: Foundations, MIT Press, 1986.
[10] G.E. Hinton, J.L. McClelland, D.E. Rumelhart.分布式表示法。In：Parallel distributed processing：Explorations in the microstructure of cognition.第 1 卷：基础》，麻省理工学院出版社，1986 年。

[11] D.A. Jurgens, S.M. Mohammad, P.D. Turney, K.J. Holyoak. Semeval-2012 task 2: Measuring degrees of relational similarity. In: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), 2012.
[11] D.A. Jurgens、S.M. Mohammad、P.D. Turney、K.J. Holyoak。Semeval-2012 任务 2：测量关系相似度。In：第六届语义评估（SemEval 2012）国际研讨会论文集，2012 年。

[12] A.L. Maas, R.E. Daly, P.T. Pham, D. Huang, A.Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of ACL, 2011.
[12] A.L. Maas、R.E. Daly、P.T. Pham、D. Huang、A.Y. Ng 和 C. Potts。为情感分析学习词向量。In Proceedings of ACL, 2011.

[13] T. Mikolov. Language Modeling for Speech Recognition in Czech, Masters thesis, Brno University of Technology, 2007.
[13] T. Mikolov.捷克语语音识别的语言建模》，布尔诺理工大学硕士论文，2007 年。

[14] T. Mikolov, J. Kopecký, L. Burget, O. Glembek and J. Černocký. Neural network based language models for higly inflective languages, In: Proc. ICASSP 2009.
[14] T. Mikolov、J. Kopecký、L. Burget、O. Glembek 和 J. Černocký.基于神经网络的高屈折语语言模型，In：Proc.

[15] T. Mikolov, M. Karafiát, L. Burget, J. Černocký, S. Khudanpur. Recurrent neural network based language model, In: Proceedings of Interspeech, 2010.
[15] T. Mikolov、M. Karafiát、L. Burget、J. Černocký 和 S. Khudanpur。基于循环神经网络的语言模型，In：Interspeech, 2010.

[16] T. Mikolov, S. Kombrink, L. Burget, J. Černocký, S. Khudanpur. Extensions of recurrent neural network language model, In: Proceedings of ICASSP 2011.
[16] T. Mikolov、S. Kombrink、L. Burget、J. Černocký 和 S. Khudanpur。循环神经网络语言模型的扩展，In：Proceedings of ICASSP 2011.

[17] T. Mikolov, A. Deoras, S. Kombrink, L. Burget, J. Černocký. Empirical Evaluation and Combination of Advanced Language Modeling Techniques, In: Proceedings of Interspeech, 2011.
[17] T. Mikolov、A. Deoras、S. Kombrink、L. Burget、J. Černocký.高级语言建模技术的经验评估与组合，In：Interspeech, 2011.

[18] T. Mikolov, A. Deoras, D. Povey, L. Burget, J. Černocký. Strategies for Training Large Scale Neural Network Language Models, In: Proc. Automatic Speech Recognition and Understanding, 2011.
[18] T. Mikolov、A. Deoras、D. Povey、L. Burget、J. Černocký.大规模神经网络语言模型的训练策略，In：Proc.自动语音识别与理解，2011 年。

[19] T. Mikolov. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012.
[19] T. Mikolov.基于神经网络的统计语言模型。博士论文，布尔诺理工大学，2012 年。

[20] T. Mikolov, W.T. Yih, G. Zweig. Linguistic Regularities in Continuous Space Word Representations. NAACL HLT 2013.
[20] T. Mikolov, W.T. Yih, G. Zweig.连续空间词语表征中的语言规律性。naacl hlt 2013.

[21] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed Representations of Words and Phrases and their Compositionality. Accepted to NIPS 2013.
[21] T. Mikolov、I. Sutskever、K. Chen、G. Corrado 和 J. Dean。词和短语的分布式表示及其组合性。已被 NIPS 2013 接收。

[22] A. Mnih, G. Hinton. Three new graphical models for statistical language modelling. ICML, 2007.
[22] A. Mnih, G. Hinton.统计语言建模的三种新图形模型。ICML, 2007.

[23] A. Mnih, G. Hinton. A Scalable Hierarchical Distributed Language Model. Advances in Neural Information Processing Systems 21, MIT Press, 2009.
[23] A. Mnih, G. Hinton.A Scalable Hierarchical Distributed Language Model.神经信息处理系统进展 21》，麻省理工学院出版社，2009 年。

[24] A. Mnih, Y.W. Teh. A fast and simple algorithm for training neural probabilistic language models. ICML, 2012.
[24] A. Mnih, Y.W. Teh.训练神经概率语言模型的快速简单算法。ICML, 2012.

[25] F. Morin, Y. Bengio. Hierarchical Probabilistic Neural Network Language Model. AISTATS, 2005.
[25] F. Morin, Y. Bengio.Hierarchical Probabilistic Neural Network Language Model.AISTATS, 2005.

[26] D. E. Rumelhart, G. E. Hinton, R. J. Williams. Learning internal representations by backpropagating errors. Nature, 323:533.536, 1986.
[26] D. E. Rumelhart, G. E. Hinton, R. J. Williams.通过反向传播错误学习内部表征。自然》，323:533.536，1986.

[27] H. Schwenk. Continuous space language models. Computer Speech and Language, vol. 21, 2007.
[27] H. Schwenk.连续空间语言模型。计算机语音与语言》，第 21 卷，2007 年。

[28] R. Socher, E.H. Huang, J. Pennington, A.Y. Ng, and C.D. Manning. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS, 2011.
[28] R. Socher、E.H. Huang、J. Pennington、A.Y. Ng 和 C.D. Manning。用于转述检测的动态池化和展开递归自动编码器。In NIPS, 2011.

[29] J. Turian, L. Ratinov, Y. Bengio. Word Representations: A Simple and General Method for Semi-Supervised Learning. In: Proc. Association for Computational Linguistics, 2010.
[29] J. Turian, L. Ratinov, Y. Bengio.词语表征：半监督学习的简单而通用的方法。In：Proc.计算语言学协会，2010 年。

[30] P. D. Turney. Measuring Semantic Similarity by Latent Relational Analysis. In: Proc. International Joint Conference on Artificial Intelligence, 2005.
[30] P. D. Turney.用潜在关系分析法测量语义相似性。In. Proc：Proc. International Joint Conference on Artificial Intelligence, 2005.

[31] A. Zhila, W.T. Yih, C. Meek, G. Zweig, T. Mikolov. Combining Heterogeneous Models for Measuring Relational Similarity. NAACL HLT 2013.
[31] A. Zhila, W.T. Yih, C. Meek, G. Zweig, T. Mikolov.结合异构模型测量关系相似性》。naacl hlt 2013.

[32] G. Zweig, C.J.C. Burges. The Microsoft Research Sentence Completion Challenge, Microsoft Research Technical Report MSR-TR-2011-129, 2011.
[32] G. Zweig, C.J.C. Burges.微软研究院句子完成挑战赛》，微软研究院技术报告 MSR-TR-2011-129，2011 年。