Language Modeling Is Compression
语言建模即压缩

We advocate for using (lossless) compression to study foundation models. To that end, we conduct an extensive empirical investigation of the offline (in-context) compression capabilities of large language models, with the rationale that they have recently become readily available (Hoffmann et al., 2022; Touvron et al., 2023) and can thus be used for compression without the training overhead. We empirically demonstrate that these models, while (meta-)trained primarily on text, also achieve state-of-the-art compression rates across different data modalities, using their context to condition a general-purpose compressor to excel at a particular task. Moreover, we shed new light on scaling laws (Kaplan et al., 2020), showing that they also hold true for compression but that measuring the compression rates instead of the log loss adds a twist: Scaling beyond a certain point will deteriorate the compression performance since the model parameters need to be accounted for in the compressed output. Finally, we advocate for framing (self-supervised) prediction through the lens of compression as it encompasses generalization: a model that compresses well generalizes well (Hutter, 2006).
我们提倡使用（无损）压缩来研究基础模型。为此，我们对大型语言模型的离线（上下文）压缩能力进行了广泛的实证研究，理由是这些模型最近变得易于获取（Hoffmann 等，2022；Touvron 等，2023），因此可以在不需要训练开销的情况下用于压缩。我们通过实验证明，这些模型虽然主要在文本上进行（元）训练，但也在不同的数据模态上实现了最先进的压缩率，利用其上下文来调节通用压缩器以在特定任务中表现出色。此外，我们对扩展定律（Kaplan 等，2020）提供了新的见解，表明这些定律也适用于压缩，但测量压缩率而不是对数损失会带来一个转折点：扩展到某个点之后，压缩性能会恶化，因为模型参数需要在压缩输出中加以考虑。最后，我们提倡通过压缩的视角来框定（自监督）预测，因为它包含了泛化：一个压缩效果好的模型泛化效果也好（Hutter，2006）。

We make the following contributions:
我们做出以下贡献：

• •

  We empirically investigate the lossless compression capabilities of foundation models. To that end, we review how to compress with predictive models via arithmetic coding and call attention to the connection between current language modeling research and compression.

  
  • 我们实证研究了基础模型的无损压缩能力。为此，我们回顾了如何通过算术编码使用预测模型进行压缩，并关注当前语言建模研究与压缩之间的联系。
• •

  We show that foundation models, trained primarily on text, are general-purpose compressors due to their in-context learning abilities. For example, Chinchilla 70B achieves compression rates of 43.4% on ImageNet patches and 16.4% on LibriSpeech samples, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively.

  
  • 我们表明，主要在文本上训练的基础模型由于其上下文学习能力，是通用的压缩器。例如，Chinchilla 70B 在 ImageNet 补丁上实现了 43.4% 的压缩率，在 LibriSpeech 样本上实现了 16.4% 的压缩率，分别超过了领域特定的压缩器如 PNG (58.5%) 或 FLAC (30.3%)。
• •

  We provide a novel view on scaling laws, showing that the dataset size provides a hard limit on model size in terms of compression performance and that scaling is not a silver bullet.

  
  • 我们提供了一个关于缩放定律的新视角，表明数据集的大小在压缩性能方面对模型大小提供了一个硬性限制，并且缩放并不是万能的。
• •

  We leverage the compression-prediction equivalence to employ compressors as generative models and visually illustrate the performance of the underlying compressor.

  
  • 我们利用压缩-预测等价性将压缩器用作生成模型，并直观地展示底层压缩器的性能。
• •

  We demonstrate that tokenization, which can be viewed as a pre-compression, does, in general, not improve compression performance, but allows models to increase the information content in their context and is thus generally employed to improve prediction performance.

  
  • 我们证明了标记化，可以看作是一种预压缩，通常不会提高压缩性能，但允许模型增加其上下文中的信息内容，因此通常用于提高预测性能。

A coding distribution $\rho$ is a sequence of probability mass functions $\rho_{n}:\mathcal{X}^{n}\mapsto(0,1]$, which for all $n\in\mathbb{N}$ satisfy the constraint that $\rho_{n}(x_{1:n})=\sum_{y\in\mathcal{X}}\rho_{n+1}(x_{1:n}y)$ for all $x_{1:n}\in\mathcal{X}^{n}$, with the base case $\rho_{0}(\epsilon):=1$. From here on out, whenever the meaning is clear from the argument to $\rho$, we drop the subscript on $\rho$. Under this definition, the conditional probability of a symbol $x_{n}$ given previous data $x_{<n}$ is defined as $\rho(x_{n}\mid x_{<n}):=\rho(x_{1:n})/\rho(x_{<n})$, with the familiar chain rules $\rho(x_{1:n})=\prod_{i=1}^{n}\rho(x_{i}\mid x_{<i})$ and $\rho(x_{j:k}\mid x_{<j})=\prod_{i=j}^{k}\rho(x_{i}\mid x_{<i})$ following.
编码分布 $\rho$ 是概率质量函数序列 $\rho_{n}:\mathcal{X}^{n}\mapsto(0,1]$ ，对于所有 $n\in\mathbb{N}$ 满足约束条件 $\rho_{n}(x_{1:n})=\sum_{y\in\mathcal{X}}\rho_{n+1}(x_{1:n}y)$ 对于所有 $x_{1:n}\in\mathcal{X}^{n}$ ，基准情况为 $\rho_{0}(\epsilon):=1$ 。从此以后，只要从 $\rho$ 的论点中可以清楚地理解其含义，我们就省略 $\rho$ 的下标。在此定义下，给定先前数据 $x_{<n}$ 的符号 $x_{n}$ 的条件概率定义为 $\rho(x_{n}\mid x_{<n}):=\rho(x_{1:n})/\rho(x_{<n})$ ，并遵循熟悉的链式规则 $\rho(x_{1:n})=\prod_{i=1}^{n}\rho(x_{i}\mid x_{<i})$ 和 $\rho(x_{j:k}\mid x_{<j})=\prod_{i=j}^{k}\rho(x_{i}\mid x_{<i})$ 。

The goal of lossless compression is to encode a stream of symbols $x_{1:n}$ sampled from a coding distribution $\rho$ into a bitstream of minimal (expected) length, while ensuring that the original data sequence is recoverable from the bitstream. To that end, we use a binary source code $c:\mathcal{X}^{*}\mapsto\{0,1\}^{*}$, which assigns to each possible data sequence $x_{1:n}$ a binary code word $c(x_{1:n})$ of length $\ell_{c}(x_{1:n})$ (in bits). Thus, the aim is to minimize the expected bits per sequence $L:=\mathrm{\mathbb{missing}}{E}_{x\sim\rho}[\ell_{c}(x)]$, i.e., encoding rare sequences with more bits and frequent sequences with fewer bits. Shannon’s source coding theorem establishes the limit on possible data compression as $L\geq H(\rho)$ for any possible code, where $H(\rho):=\mathbb{E}_{x\sim\rho}[-\log_{2}\rho(x)]$ is the Shannon entropy (Shannon, 1948).
无损压缩的目标是将从编码分布中采样的符号流编码成最小（期望）长度的比特流，同时确保可以从比特流中恢复原始数据序列。为此，我们使用二进制源代码，它为每个可能的数据序列分配一个长度为（以比特为单位）的二进制代码字。因此，目标是最小化每个序列的期望比特数，即用更多的比特编码罕见序列，用更少的比特编码频繁序列。香农的源编码定理确立了任何可能的代码的数据压缩极限，其中是香农熵（Shannon, 1948）。

Given a coding distribution $\rho$ and a sequence $x_{1:n}$, arithmetic coding (Pasco, 1977; Rissanen, 1976) constructs a code with almost optimal length. It directly connects coding and compression with prediction and modeling: compressing well means modeling well in a log-loss sense and vice-versa. Assuming infinite precision for the arithmetic operations involved, the arithmetic code has length $-\lceil\log\rho(x_{1:n})\rceil+1$ bits, whereas the optimal code length is $-\log\rho(x_{1:n})$ bits. A practical implementation that is subject to $B$ bit precision adds further $O(n2^{-B})$ bits (Howard & Vitter, 1991), which is negligible for 32- or 64-bit arithmetic. In the following we consider infinite precision arithmetic coders and refer to Witten et al. (1987) for the finite-precision implementation.
给定一个编码分布 $\rho$ 和一个序列 $x_{1:n}$ ，算术编码（Pasco, 1977; Rissanen, 1976）构建了一个几乎最优长度的编码。它直接将编码和压缩与预测和建模联系起来：良好的压缩意味着在对数损失意义上良好的建模，反之亦然。假设涉及的算术运算具有无限精度，算术编码的长度为 $-\lceil\log\rho(x_{1:n})\rceil+1$ 比特，而最优编码长度为 $-\log\rho(x_{1:n})$ 比特。受限于 $B$ 比特精度的实际实现会增加 $O(n2^{-B})$ 比特（Howard & Vitter, 1991），对于 32 位或 64 位算术来说是可以忽略的。以下我们考虑无限精度的算术编码器，并参考 Witten 等人（1987）的有限精度实现。

The arithmetic code of a sequence $x_{1:n}$ is the binary representation of a number $\lambda\in[0,1)$. We identify $\lambda$ by narrowing down an interval that encloses $\lambda$ step by step (maintaining a growing prefix of the binary representation of $\lambda$ throughout the process). Initially, this interval is $I_{0}=[0,1)$. In step $k>0$ (i.e., encoding $x_{k}$), we first partition the previous interval $I_{k-1}=[l_{k-1},u_{k-1})$ into $N$ sub-intervals $\tilde{I}_{k}(x_{1}),\tilde{I}_{k}(x_{2}),\dots$, one for each letter from $\mathcal{X}=\{x_{1},x_{2},\dots,x_{N}\}$. The size of sub-interval $\tilde{I}_{k}(y)$ that represents letter $y$ is $(u_{k-1}-l_{k-1})\cdot\rho(y\mid x_{<k})$. Formally, we define
一个序列的算术编码是一个数字的二进制表示。我们通过逐步缩小包含该数字的区间来识别该数字（在整个过程中保持该数字二进制表示的前缀不断增长）。最初，这个区间是。在第步（即编码），我们首先将前一个区间划分为个子区间，每个字母对应一个子区间。表示字母的子区间的大小是。正式地，我们定义

assuming a strict order on $\mathcal{X}$. To encode $x_{k}$ we proceed with its corresponding interval, i.e., $I_{k}=\tilde{I}_{k}(x_{k})$. Finally, we choose $\lambda\in I_{n}$ with the shortest binary representation in the terminating interval $I_{n}$ and use that binary representation to encode $x_{1:n}$. Fig. 1 illustrates this process.
假设在 $\mathcal{X}$ 上有一个严格的顺序。为了编码 $x_{k}$ ，我们继续处理其对应的区间，即 $I_{k}=\tilde{I}_{k}(x_{k})$ 。最后，我们选择在终止区间 $I_{n}$ 中具有最短二进制表示的 $\lambda\in I_{n}$ ，并使用该二进制表示来编码 $x_{1:n}$ 。图 1 说明了这个过程。

Given $\lambda$ and $\rho$ decoding the $k$-th letter is easy: Starting with $I_{0}=[0,1)$, find $y$ such that $\lambda\in\tilde{I}_{k}(y)$ to decode $x_{k}=y$, then set $I_{k}=\tilde{I}_{k}(x_{k})$ and proceed with the $k\!+\!1$-st letter.
给定 $\lambda$ 和 $\rho$ ，解码第 $k$ 个字母很容易：从 $I_{0}=[0,1)$ 开始，找到 $y$ 使得 $\lambda\in\tilde{I}_{k}(y)$ 解码 $x_{k}=y$ ，然后设置 $I_{k}=\tilde{I}_{k}(x_{k})$ 并继续处理第 $k\!+\!1$ 个字母。

In practice, the source distribution $\rho$ is usually unknown and is instead estimated with a parametric probabilistic model $\hat{\rho}$. Thus, instead of achieving code length $-\sum_{i=1}^{n}\log_{2}\rho(x_{i}\mid x_{<i})$ for the sequence $x_{1:n}$, we obtain the suboptimal length $-\sum_{i=1}^{n}\log_{2}\hat{\rho}(x_{i}\mid x_{<i})$. As a result, the expected (suboptimal) number of bits is the cross-entropy:
在实践中，源分布 $\rho$ 通常是未知的，而是用参数概率模型 $\hat{\rho}$ 来估计。因此，我们不是为序列 $x_{1:n}$ 实现码长 $-\sum_{i=1}^{n}\log_{2}\rho(x_{i}\mid x_{<i})$ ，而是得到次优码长 $-\sum_{i=1}^{n}\log_{2}\hat{\rho}(x_{i}\mid x_{<i})$ 。结果，期望的（次优）比特数是交叉熵：

Thus, we can minimize the expected length of the encoded data stream with symbols distributed according to $\rho$ by minimizing the cross-entropy with respect to some $\hat{\rho}$, which is equivalent to likelihood maximization (MacKay, 2003). However, Eq. 2 is exactly the same objective used to train current foundation models, i.e., the $\log$-loss. Thus, minimizing the $\log$-loss is equivalent to minimizing the compression rate of that model used as a lossless compressor with arithmetic coding, i.e., current language model training protocols use a maximum-compression objective.
因此，我们可以通过最小化相对于某些 $\hat{\rho}$ 的交叉熵来最小化根据 $\rho$ 分布的符号编码数据流的期望长度，这等同于似然最大化（MacKay, 2003）。然而，方程 2 正是用于训练当前基础模型的目标，即 $\log$ -损失。因此，最小化 $\log$ -损失等同于最小化该模型作为无损压缩器使用算术编码时的压缩率，即当前语言模型训练协议使用最大压缩目标。

Analogous to how a predictive distribution can be used for lossless compression via arithmetic coding (described above), any compressor can be employed for sequence prediction (Frank et al., 2000). The main idea is to define $\rho(x_{1:n})$ as the coding distribution $2^{-\ell_{c}(\cdot)}$, where $\ell_{c}(x_{1:n})$ is the length of sequence $x_{1:n}$ when encoded with compressor $c$ (e.g., gzip). We thus recover the conditional distribution $\rho(x_{i}\mid x_{<i})$ by computing $2^{\ell_{c}(x_{<i})-\ell_{c}(x_{<i}x_{i})}$, for all $x_{i}$.
类似于如何通过算术编码（如上所述）使用预测分布进行无损压缩，任何压缩器都可以用于序列预测（Frank 等，2000）。主要思想是将 $\rho(x_{1:n})$ 定义为编码分布 $2^{-\ell_{c}(\cdot)}$ ，其中 $\ell_{c}(x_{1:n})$ 是使用压缩器 $c$ （例如 gzip）对序列 $x_{1:n}$ 进行编码时的长度。我们因此通过计算 $2^{\ell_{c}(x_{<i})-\ell_{c}(x_{<i}x_{i})}$ 来恢复条件分布 $\rho(x_{i}\mid x_{<i})$ ，对于所有 $x_{i}$ 。

Above we discussed optimal (arithmetic) coding with respect to data sampled from a fixed distribution $\rho$. In contrast, universal (optimal) source coding with respect to all computable sampling distributions can, in theory, be achieved by choosing $\ell_{c}(x_{1:n})$ as the Kolmogorov complexity of $x_{1:n}$ (Kolmogorov, 1998; Li & Vitányi, 2019). For this choice, the conditional distribution described above is universally optimal over $x_{<i}$, recovering the Solomonoff predictor (Solomonoff, 1964a, b; Rathmanner & Hutter, 2011). The Solomonoff predictor is a Bayesian mixture of all predictors that can be programmed in a chosen Turing-complete programming language. More precisely, for a predictor $q$ of program-length $\ell_{c}(q)$ bits, the Solomonoff predictor assigns a prior weight of $2^{-\ell_{c}(q)}$ to predictor $q$. That is, if $\mathcal{Q}$ is the set of all predictors that can be programmed and computed, the Solomonoff predictor assigns probability $S(x_{1:n})=\sum_{q\in{\mathcal{Q}}}2^{-\ell_{c}(q)}q(x_{1:n})$ to a sequence $x_{1:n}$, if every predictor $q$ assigns that sequence probability $q(x_{1:n})$. Therefore, $S(x_{1:n})\geq 2^{-\ell_{c}(q)}q(x_{1:n})$ for all $q\in\mathcal{Q}$, and thus $-\log_{2}S(x_{1:n})\leq-\log_{2}q(x_{1:n})+\ell_{c}(q)$. Observe that $\ell_{c}(q)$ is a constant of $q$ that is independent of the sequence length. Therefore, compressing optimally is equivalent to predicting optimally and vice versa (Hutter, 2005).
上文我们讨论了相对于从固定分布 $\rho$ 采样的数据的最优（算术）编码。相比之下，相对于所有可计算采样分布的通用（最优）源编码理论上可以通过选择 $\ell_{c}(x_{1:n})$ 作为 $x_{1:n}$ 的 Kolmogorov 复杂度来实现（Kolmogorov, 1998; Li & Vitányi, 2019）。对于这种选择，上述条件分布在 $x_{<i}$ 上是普遍最优的，恢复了 Solomonoff 预测器（Solomonoff, 1964a, b; Rathmanner & Hutter, 2011）。Solomonoff 预测器是所有可以在选定的图灵完备编程语言中编程的预测器的贝叶斯混合。更准确地说，对于程序长度为 $\ell_{c}(q)$ 位的预测器 $q$ ，Solomonoff 预测器为预测器 $q$ 分配了先验权重 $2^{-\ell_{c}(q)}$ 。也就是说，如果 $\mathcal{Q}$ 是所有可以编程和计算的预测器的集合，Solomonoff 预测器为序列 $x_{1:n}$ 分配概率 $S(x_{1:n})=\sum_{q\in{\mathcal{Q}}}2^{-\ell_{c}(q)}q(x_{1:n})$ ，如果每个预测器 $q$ 为该序列分配概率 $q(x_{1:n})$ 。因此， $S(x_{1:n})\geq 2^{-\ell_{c}(q)}q(x_{1:n})$ 对于所有 $q\in\mathcal{Q}$ ，因此 $-\log_{2}S(x_{1:n})\leq-\log_{2}q(x_{1:n})+\ell_{c}(q)$ 。注意， $\ell_{c}(q)$ 是 $q$ 的常数，与序列长度无关。 因此，最优压缩等同于最优预测，反之亦然（Hutter, 2005）。

We compare our arithmetic coding-based language model compressors to two competitive general-purpose lossless compressors: gzip (Deutsch, 1996) and its improvement LZMA2 (Pavlov, 2019), used by the 7zip software. Both are based on Huffman coding (Huffman, 1952) and the Lempel-Ziv-Welch algorithm (Welch, 1984). We also consider specialized lossless compressors for image and audio data, i.e., PNG (Boutell, 1997) and FLAC (Coalson, 2008), respectively. Finally, we evaluate two types of language models (of different sizes) with arithmetic coding: vanilla decoder-only Transformers (Vaswani et al., 2017), which we pretrain on the enwik8 dataset, and pretrained Chinchilla-like foundation models (Hoffmann et al., 2022).
我们将基于算术编码的语言模型压缩器与两种具有竞争力的通用无损压缩器进行比较：gzip（Deutsch，1996）及其改进版 LZMA2（Pavlov，2019），后者由 7zip 软件使用。两者都基于霍夫曼编码（Huffman，1952）和 Lempel-Ziv-Welch 算法（Welch，1984）。我们还考虑了用于图像和音频数据的专用无损压缩器，即 PNG（Boutell，1997）和 FLAC（Coalson，2008）。最后，我们评估了两种类型的语言模型（不同大小）与算术编码：仅解码器的基础 Transformer（Vaswani 等，2017），我们在 enwik8 数据集上进行预训练，以及预训练的类似 Chinchilla 的基础模型（Hoffmann 等，2022）。

Leveraging Shannon’s source coding theorem (Shannon, 1948), a plethora of approaches exploit the connection between prediction and compression. For example, context-tree weighting (CTW) (Willems et al., 1995) mixes the predictions of many underlying Markov models to achieve lossless compression via arithmetic coding (Pasco, 1977; Rissanen, 1976). Similarly, prediction by partial matching (PPM) (Cleary & Witten, 1984) also leverages arithmetic coding, but uses a contiguous context matching method to create probability distributions based on the history of characters in a sequence. Likewise, PAQ8 (Knoll & de Freitas, 2012) uses a weighted combination of predictions from a large number of models (most of them based on context matching, but unlike PPM also noncontiguous context matches). In a different setting, Veness et al. (2015) demonstrated how to employ compression to obtain value estimates of a policy in an environment. Frank et al. (2000) and later Teahan & Harper (2003) introduced the idea of classification with compressors. Recently, Jiang et al. (2023) applied this technique with NLP tasks, paired with a k-nearest-neighbour algorithm. The results are surprisingly good for simple general purpose compressors like gzip. Jiang et al. (2022) exploit the same idea but train the compressor on a vast amount of unlabeled data first. Finally, van den Oord & Schrauwen (2014) apply arithmetic coding to image compression using Student distribution mixtures and Gaussian processes as predictors.
利用香农的源编码定理（Shannon, 1948），大量方法利用预测和压缩之间的联系。例如，上下文树加权（CTW）（Willems 等，1995）通过算术编码（Pasco, 1977; Rissanen, 1976）混合许多底层马尔可夫模型的预测来实现无损压缩。同样，部分匹配预测（PPM）（Cleary & Witten, 1984）也利用算术编码，但使用连续的上下文匹配方法根据序列中字符的历史创建概率分布。同样，PAQ8（Knoll & de Freitas, 2012）使用大量模型的预测的加权组合（其中大多数基于上下文匹配，但与 PPM 不同的是也包括非连续上下文匹配）。在不同的设置中，Veness 等（2015）展示了如何利用压缩来获得环境中策略的价值估计。Frank 等（2000）以及后来的 Teahan & Harper（2003）引入了使用压缩器进行分类的想法。最近，Jiang 等（2023）将这一技术应用于 NLP 任务，并与 k 近邻算法配对。 对于像 gzip 这样的简单通用压缩器，结果出奇地好。Jiang 等人（2022）利用了相同的想法，但首先在大量未标记的数据上训练压缩器。最后，van den Oord 和 Schrauwen（2014）将算术编码应用于图像压缩，使用学生分布混合和高斯过程作为预测器。

Prior work demonstrated that neural predictive distributions can be employed to perform lossless compression via arithmetic coding (Schmidhuber & Heil, 1996; Mahoney, 2000; Knoll, 2014; Cox, 2016; Schiopu et al., 2018; Goyal et al., 2019; Liu et al., 2019; Mentzer et al., 2019, 2020; Schiopu & Munteanu, 2020; Rhee et al., 2022; Mikolov, 2012). Similarly, neural networks were also shown to achieve strong lossless compression rates when replacing arithmetic coding with asymmetric numeral systems (Hoogeboom et al., 2019; Kingma et al., 2019; Townsend et al., 2019; Barzen et al., 2022). While these approaches assume the existence of a separate training set, a different line of work investigated arithmetic coding-based neural compression in a purely online fashion, i.e., training the model only on the data stream that is to be compressed (Bellard, 2019; Goyal et al., 2020; Bellard, 2021; Mao et al., 2022). Finally, concurrent work (Valmeekam et al., 2023) also investigated lossless offline compression with foundation models, using arithmetic coding with LLaMA-7B (Touvron et al., 2023).
先前的工作表明，神经预测分布可以通过算术编码实现无损压缩（Schmidhuber & Heil, 1996; Mahoney, 2000; Knoll, 2014; Cox, 2016; Schiopu et al., 2018; Goyal et al., 2019; Liu et al., 2019; Mentzer et al., 2019, 2020; Schiopu & Munteanu, 2020; Rhee et al., 2022; Mikolov, 2012）。同样，当用非对称数字系统替代算术编码时，神经网络也被证明能够实现强大的无损压缩率（Hoogeboom et al., 2019; Kingma et al., 2019; Townsend et al., 2019; Barzen et al., 2022）。虽然这些方法假设存在一个单独的训练集，但另一类工作研究了基于算术编码的神经压缩在纯在线方式下的应用，即仅在要压缩的数据流上训练模型（Bellard, 2019; Goyal et al., 2020; Bellard, 2021; Mao et al., 2022）。最后，同时进行的工作（Valmeekam et al., 2023）也研究了使用基础模型进行无损离线压缩，使用 LLaMA-7B 进行算术编码（Touvron et al., 2023）。

Much effort has been devoted on understanding the inductive biases of neural networks. Here, we are mostly interested in the biases of Natural Language Processing (NLP) and Transformers. Kudo & Richardson (2018) defined a tokenizer for NLP-related research, an improvement of well-known techniques like byte-pair encoding (BPE) (Sennrich et al., 2016), BPE dropout (Provilkov et al., 2020), and subword regularization (Kudo, 2018). In this paper, we show how these tokenization techniques act as pre-compressors for the data, and can significantly affect the final compression rates when paired with a neural model. More general studies have been performed on generalization (Neyshabur et al., 2017), which, we argue, is equivalent to the model’s compressive power when accounting parameters code-length. Finally, some work has been done on compressing the neural models’ parameters themselves (Cheng et al., 2017).
大量努力已投入到理解神经网络的归纳偏差上。在这里，我们主要关注自然语言处理（NLP）和 Transformer 的偏差。Kudo 和 Richardson（2018）为 NLP 相关研究定义了一个分词器，这是对字节对编码（BPE）（Sennrich 等，2016）、BPE dropout（Provilkov 等，2020）和子词正则化（Kudo，2018）等著名技术的改进。在本文中，我们展示了这些分词技术如何作为数据的预压缩器，并且当与神经模型配对时，可以显著影响最终的压缩率。更一般的研究已经在泛化上进行（Neyshabur 等，2017），我们认为，当考虑参数代码长度时，这相当于模型的压缩能力。最后，一些工作已经在压缩神经模型的参数本身上进行（Cheng 等，2017）。