Denoising Diffusion Implicit Models
去噪扩散隐式模型

Deep generative models have demonstrated the ability to produce high quality samples in many domains (Karras et al., 2020; van den Oord et al., 2016a). In terms of image generation, generative adversarial networks (GANs, Goodfellow et al. (2014)) currently exhibits higher sample quality than likelihood-based methods such as variational autoencoders (Kingma & Welling, 2013), autoregressive models (van den Oord et al., 2016b) and normalizing flows (Rezende & Mohamed, 2015; Dinh et al., 2016). However, GANs require very specific choices in optimization and architectures in order to stabilize training (Arjovsky et al., 2017; Gulrajani et al., 2017; Karras et al., 2018; Brock et al., 2018), and could fail to cover modes of the data distribution (Zhao et al., 2018).
深度生成模型在许多领域展示了生成高质量样本的能力（Karras et al., 2020; van den Oord et al., 2016a）。在图像生成方面，生成对抗网络（GANs，Goodfellow et al. (2014)）目前展现出比基于似然的方法（如变分自编码器（Kingma & Welling, 2013）、自回归模型（van den Oord et al., 2016b）和归一化流（Rezende & Mohamed, 2015; Dinh et al., 2016））更高的样本质量。然而，GANs 在优化和架构上需要非常特定的选择以稳定训练（Arjovsky et al., 2017; Gulrajani et al., 2017; Karras et al., 2018; Brock et al., 2018），并可能无法覆盖数据分布的模式（Zhao et al., 2018）。

Recent works on iterative generative models (Bengio et al., 2014), such as denoising diffusion probabilistic models (DDPM, Ho et al. (2020)) and noise conditional score networks (NCSN, Song & Ermon (2019)) have demonstrated the ability to produce samples comparable to that of GANs, without having to perform adversarial training. To achieve this, many denoising autoencoding models are trained to denoise samples corrupted by various levels of Gaussian noise. Samples are then produced by a Markov chain which, starting from white noise, progressively denoises it into an image. This generative Markov Chain process is either based on Langevin dynamics (Song & Ermon, 2019) or obtained by reversing a forward diffusion process that progressively turns an image into noise (Sohl-Dickstein et al., 2015).
最近关于迭代生成模型的研究（Bengio 等，2014），例如去噪扩散概率模型（DDPM，Ho 等（2020））和噪声条件评分网络（NCSN，Song & Ermon（2019））已展示出生成与 GANs 相当的样本的能力，而无需进行对抗训练。为了实现这一点，许多去噪自编码模型被训练以去噪受到不同程度高斯噪声污染的样本。然后，通过一个马尔可夫链生成样本，该链从白噪声开始，逐步将其去噪成图像。这个生成的马尔可夫链过程要么基于朗之万动力学（Song & Ermon，2019），要么通过逆转一个逐步将图像转变为噪声的前向扩散过程获得（Sohl-Dickstein 等，2015）。

A critical drawback of these models is that they require many iterations to produce a high quality sample. For DDPMs, this is because that the generative process (from noise to data) approximates the reverse of the forward diffusion process (from data to noise), which could have thousands of steps; iterating over all the steps is required to produce a single sample, which is much slower compared to GANs, which only needs one pass through a network. For example, it takes around 20 hours to sample 50k images of size $32\times 32$ from a DDPM, but less than a minute to do so from a GAN on a Nvidia 2080 Ti GPU. This becomes more problematic for larger images as sampling 50k images of size $256\times 256$ could take nearly $1000$ hours on the same GPU.
这些模型的一个关键缺陷是它们需要多次迭代才能生成高质量的样本。对于 DDPM 而言，这是因为生成过程（从噪声到数据）近似于正向扩散过程（从数据到噪声）的反向过程，这可能需要数千个步骤；为了生成一个样本，必须遍历所有步骤，这与只需通过网络一次的 GAN 相比，速度要慢得多。例如，从 DDPM 中采样 50,000 张大小为 $32\times 32$ 的图像大约需要 20 小时，而在 Nvidia 2080 Ti GPU 上，从 GAN 中采样则不到一分钟。对于更大的图像，这个问题变得更加严重，因为在同一 GPU 上，采样 50,000 张大小为 $256\times 256$ 的图像可能需要近 $1000$ 小时。

To close this efficiency gap between DDPMs and GANs, we present denoising diffusion implicit models (DDIMs). DDIMs are implicit probabilistic models (Mohamed & Lakshminarayanan, 2016) and are closely related to DDPMs, in the sense that they are trained with the same objective function. In Section 3, we generalize the forward diffusion process used by DDPMs, which is Markovian, to non-Markovian ones, for which we are still able to design suitable reverse generative Markov chains. We show that the resulting variational training objectives have a shared surrogate objective, which is exactly the objective used to train DDPM. Therefore, we can freely choose from a large family of generative models using the same neural network simply by choosing a different, non-Markovian diffusion process (Section 4.1) and the corresponding reverse generative Markov Chain. In particular, we are able to use non-Markovian diffusion processes which lead to "short" generative Markov chains (Section 4.2) that can be simulated in a small number of steps. This can massively increase sample efficiency only at a minor cost in sample quality.
为了缩小 DDPM 和 GAN 之间的效率差距，我们提出了去噪扩散隐式模型（DDIM）。DDIM 是隐式概率模型（Mohamed & Lakshminarayanan, 2016），与 DDPM 密切相关，因为它们使用相同的目标函数进行训练。在第 3 节中，我们将 DDPM 使用的前向扩散过程（这是马尔可夫过程）推广到非马尔可夫过程，对于这些过程，我们仍然能够设计合适的反向生成马尔可夫链。我们展示了所得到的变分训练目标具有共享的替代目标，这正是用于训练 DDPM 的目标。因此，我们可以通过选择不同的非马尔可夫扩散过程（第 4.1 节）和相应的反向生成马尔可夫链，随意选择来自大量生成模型的选项，使用相同的神经网络。特别是，我们能够使用导致“短”生成马尔可夫链的非马尔可夫扩散过程（第 4.2 节），这些链可以在少量步骤中进行模拟。这可以在样本质量仅略有损失的情况下大幅提高样本效率。

In Section 5, we demonstrate several empirical benefits of DDIMs over DDPMs. First, DDIMs have superior sample generation quality compared to DDPMs, when we accelerate sampling by $10\times$ to $100\times$ using our proposed method. Second, DDIM samples have the following ``consistency'' property, which does not hold for DDPMs: if we start with the same initial latent variable and generate several samples with Markov chains of various lengths, these samples would have similar high-level features. Third, because of ``consistency'' in DDIMs, we can perform semantically meaningful image interpolation by manipulating the initial latent variable in DDIMs, unlike DDPMs which interpolates near the image space due to the stochastic generative process.
在第 5 节中，我们展示了 DDIM 相对于 DDPM 的几个实证优势。首先，当我们使用所提出的方法将采样加速从 $10\times$ 到 $100\times$ 时，DDIM 的样本生成质量优于 DDPM。其次，DDIM 样本具有以下“一致性”特性，而 DDPM 则不具备：如果我们从相同的初始潜变量开始，并使用不同长度的马尔可夫链生成多个样本，这些样本将具有相似的高层特征。第三，由于 DDIM 中的“一致性”，我们可以通过操控 DDIM 中的初始潜变量进行语义上有意义的图像插值，而 DDPM 由于随机生成过程则在图像空间附近进行插值。

Given samples from a data distribution $q({\bm{x}}_{0})$, we are interested in learning a model distribution $p_{\theta}({\bm{x}}_{0})$ that approximates $q({\bm{x}}_{0})$ and is easy to sample from. Denoising diffusion probabilistic models (DDPMs, Sohl-Dickstein et al. (2015); Ho et al. (2020)) are latent variable models of the form

where ${\bm{x}}_{1},\ldots,{\bm{x}}_{T}$ are latent variables in the same sample space as ${\bm{x}}_{0}$ (denoted as ${\mathcal{X}}$). The parameters $\theta$ are learned to fit the data distribution $q({\bm{x}}_{0})$ by maximizing a variational lower bound:

where $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$ is some inference distribution over the latent variables. Unlike typical latent variable models (such as the variational autoencoder (Rezende et al., 2014)), DDPMs are learned with a fixed (rather than trainable) inference procedure $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$, and latent variables are relatively high dimensional. For example, Ho et al. (2020) considered the following Markov chain with Gaussian transitions parameterized by a decreasing sequence $\alpha_{1:T}\in(0,1]^{T}$:

where the covariance matrix is ensured to have positive terms on its diagonal. This is called the forward process due to the autoregressive nature of the sampling procedure (from ${\bm{x}}_{0}$ to ${\bm{x}}_{T}$). We call the latent variable model $p_{\theta}({\bm{x}}_{0:T})$, which is a Markov chain that samples from ${\bm{x}}_{T}$ to ${\bm{x}}_{0}$, the generative process, since it approximates the intractable reverse process $q({\bm{x}}_{t-1}|{\bm{x}}_{t})$. Intuitively, the forward process progressively adds noise to the observation ${\bm{x}}_{0}$, whereas the generative process progressively denoises a noisy observation (Figure 1, left).

A special property of the forward process is that

so we can express ${\bm{x}}_{t}$ as a linear combination of ${\bm{x}}_{0}$ and a noise variable $\epsilon$:

When we set $\alpha_{T}$ sufficiently close to $0$, $q({\bm{x}}_{T}|{\bm{x}}_{0})$ converges to a standard Gaussian for all ${\bm{x}}_{0}$, so it is natural to set $p_{\theta}({\bm{x}}_{T}):={\mathcal{N}}({\bm{0}},{\bm{I}})$. If all the conditionals are modeled as Gaussians with trainable mean functions and fixed variances, the objective in Eq. (2) can be simplified to¹¹1Please refer to Appendix C.2 for details.:

where $\epsilon_{\theta}:=\{\epsilon_{\theta}^{(t)}\}_{t=1}^{T}$ is a set of $T$ functions, each $\epsilon_{\theta}^{(t)}:{\mathcal{X}}\to{\mathcal{X}}$ (indexed by $t$) is a function with trainable parameters $\theta^{(t)}$, and $\gamma:=[\gamma_{1},\ldots,\gamma_{T}]$ is a vector of positive coefficients in the objective that depends on $\alpha_{1:T}$. In Ho et al. (2020), the objective with $\gamma={\bm{1}}$ is optimized instead to maximize generation performance of the trained model; this is also the same objective used in noise conditional score networks (Song & Ermon, 2019) based on score matching (Hyvärinen, 2005; Vincent, 2011). From a trained model, ${\bm{x}}_{0}$ is sampled by first sampling ${\bm{x}}_{T}$ from the prior $p_{\theta}({\bm{x}}_{T})$, and then sampling ${\bm{x}}_{t-1}$ from the generative processes iteratively.

The length $T$ of the forward process is an important hyperparameter in DDPMs. From a variational perspective, a large $T$ allows the reverse process to be close to a Gaussian (Sohl-Dickstein et al., 2015), so that the generative process modeled with Gaussian conditional distributions becomes a good approximation; this motivates the choice of large $T$ values, such as $T=1000$ in Ho et al. (2020). However, as all $T$ iterations have to be performed sequentially, instead of in parallel, to obtain a sample ${\bm{x}}_{0}$, sampling from DDPMs is much slower than sampling from other deep generative models, which makes them impractical for tasks where compute is limited and latency is critical.

Because the generative model approximates the reverse of the inference process, we need to rethink the inference process in order to reduce the number of iterations required by the generative model. Our key observation is that the DDPM objective in the form of $L_{\gamma}$ only depends on the marginals²²2We slightly abuse this term (as well as joints) when only conditioned on ${\bm{x}}_{0}$. $q({\bm{x}}_{t}|{\bm{x}}_{0})$, but not directly on the joint $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$. Since there are many inference distributions (joints) with the same marginals, we explore alternative inference processes that are non-Markovian, which leads to new generative processes (Figure 1, right). These non-Markovian inference process lead to the same surrogate objective function as DDPM, as we will show below. In Appendix A, we show that the non-Markovian perspective also applies beyond the Gaussian case.

With $L_{\bm{1}}$ as the objective, we are not only learning a generative process for the Markovian inference process considered in Sohl-Dickstein et al. (2015) and Ho et al. (2020), but also generative processes for many non-Markovian forward processes parametrized by $\sigma$ that we have described. Therefore, we can essentially use pretrained DDPM models as the solutions to the new objectives, and focus on finding a generative process that is better at producing samples subject to our needs by changing $\sigma$.

In this section, we show that DDIMs outperform DDPMs in terms of image generation when fewer iterations are considered, giving speed ups of $10\times$ to $100\times$ over the original DDPM generation process. Moreover, unlike DDPMs, once the initial latent variables ${\bm{x}}_{T}$ are fixed, DDIMs retain high-level image features regardless of the generation trajectory, so they are able to perform interpolation directly from the latent space. DDIMs can also be used to encode samples that reconstruct them from the latent code, which DDPMs cannot do due to the stochastic sampling process.

For each dataset, we use the same trained model with $T=1000$ and the objective being $L_{\gamma}$ from Eq. (5) with $\gamma={\bm{1}}$; as we argued in Section 3, no changes are needed with regards to the training procedure. The only changes that we make is how we produce samples from the model; we achieve this by controlling $\tau$ (which controls how fast the samples are obtained) and $\sigma$ (which interpolates between the deterministic DDIM and the stochastic DDPM).

We consider different sub-sequences $\tau$ of $[1,\ldots,T]$ and different variance hyperparameters $\sigma$ indexed by elements of $\tau$. To simplify comparisons, we consider $\sigma$ with the form:

where $\eta\in\mathbb{R}_{\geq 0}$ is a hyperparameter that we can directly control. This includes an original DDPM generative process when $\eta=1$ and DDIM when $\eta=0$. We also consider DDPM where the random noise has a larger standard deviation than $\sigma(1)$, which we denote as $\hat{\sigma}$: $\hat{\sigma}_{\tau_{i}}=\sqrt{1-{\alpha_{\tau_{i}}}/{\alpha_{\tau_{i-1}}}}$ . This is used by the implementation in Ho et al. (2020) only to obtain the CIFAR10 samples, but not samples of the other datasets. We include more details in Appendix D.

Our work is based on a large family of existing methods on learning generative models as transition operators of Markov chains (Sohl-Dickstein et al., 2015; Bengio et al., 2014; Salimans et al., 2014; Song et al., 2017; Goyal et al., 2017; Levy et al., 2017). Among them, denoising diffusion probabilistic models (DDPMs, Ho et al. (2020)) and noise conditional score networks (NCSN, Song & Ermon (2019; 2020)) have recently achieved high sample quality comparable to GANs (Brock et al., 2018; Karras et al., 2018). DDPMs optimize a variational lower bound to the log-likelihood, whereas NCSNs optimize the score matching objective (Hyvärinen, 2005) over a nonparametric Parzen density estimator of the data (Vincent, 2011; Raphan & Simoncelli, 2011).

Despite their different motivations, DDPMs and NCSNs are closely related. Both use a denoising autoencoder objective for many noise levels, and both use a procedure similar to Langevin dynamics to produce samples (Neal et al., 2011). Since Langevin dynamics is a discretization of a gradient flow (Jordan et al., 1998), both DDPM and NCSN require many steps to achieve good sample quality. This aligns with the observation that DDPM and existing NCSN methods have trouble generating high-quality samples in a few iterations.

DDIM, on the other hand, is an implicit generative model (Mohamed & Lakshminarayanan, 2016) where samples are uniquely determined from the latent variables. Hence, DDIM has certain properties that resemble GANs (Goodfellow et al., 2014) and invertible flows (Dinh et al., 2016), such as the ability to produce semantically meaningful interpolations. We derive DDIM from a purely variational perspective, where the restrictions of Langevin dynamics are not relevant; this could partially explain why we are able to observe superior sample quality compared to DDPM under fewer iterations. The sampling procedure of DDIM is also reminiscent of neural networks with continuous depth (Chen et al., 2018; Grathwohl et al., 2018), since the samples it produces from the same latent variable have similar high-level visual features, regardless of the specific sample trajectory.

We have presented DDIMs – an implicit generative model trained with denoising auto-encoding / score matching objectives – from a purely variational perspective. DDIM is able to generate high-quality samples much more efficiently than existing DDPMs and NCSNs, with the ability to perform meaningful interpolations from the latent space. The non-Markovian forward process presented here seems to suggest continuous forward processes other than Gaussian (which cannot be done in the original diffusion framework, since Gaussian is the only stable distribution with finite variance). We also demonstrated a discrete case with a multinomial forward process in Appendix A, and it would be interesting to investigate similar alternatives for other combinatorial structures.

Moreover, since the sampling procedure of DDIMs is similar to that of an neural ODE, it would be interesting to see if methods that decrease the discretization error in ODEs, including multi-step methods such as Adams-Bashforth (Butcher & Goodwin, 2008), could be helpful for further improving sample quality in fewer steps (Queiruga et al., 2020). It is also relevant to investigate whether DDIMs exhibit other properties of existing implicit models (Bau et al., 2019).

The authors would like to thank Yang Song and Shengjia Zhao for helpful discussions over the ideas, Kuno Kim for reviewing an earlier draft of the paper, and Sharvil Nanavati and Sophie Liu for identifying typos. This research was supported by NSF (#1651565, #1522054, #1733686), ONR (N00014-19-1-2145), AFOSR (FA9550-19-1-0024), and Amazon AWS.