Deep Q-learning algorithms (Mnih et al., 2015) have been labeled as off-policy due to their connection to off-policy Q-learning (Watkins, 1989). However, these algorithms tend to use near-on-policy exploratory policies, such as $\epsilon$-greedy, in conjunction with a replay buffer (Lin, 1992). As a result, the generated dataset tends to be heavily correlated to the current policy. In this section, we examine how these off-policy algorithms perform when learning with uncorrelated datasets. Our results demonstrate that the performance of a state of the art deep actor-critic algorithm, DDPG (Lillicrap et al., 2015), deteriorates rapidly when the data is uncorrelated and the value estimate produced by the deep Q-network diverges. These results suggest that off-policy deep reinforcement learning algorithms are ineffective when learning truly off-policy.
深度 Q 学习算法（Mnih 等，2015 年）被标记为离策略，因为它们与离策略 Q 学习（Watkins，1989 年）有关。然而，这些算法往往使用接近策略的探索策略，如ε-贪心，结合重放缓冲区（Lin，1992 年）。因此，生成的数据集往往与当前策略高度相关。在本节中，我们将研究这些离策略算法在学习与不相关数据集时的表现。我们的结果表明，一种最先进的深度 actor-critic 算法 DDPG（Lillicrap 等，2015 年）在数据不相关且由深度 Q 网络产生的值估计发散时，性能会迅速下降。这些结果表明，当真正离策略学习时，离策略深度强化学习算法是无效的。

Our practical experiments examine three different batch settings in OpenAI gym’s Hopper-v1 environment (Todorov et al., 2012; Brockman et al., 2016), which we use to train an off-policy DDPG agent with no interaction with the environment. Experiments with additional environments and specific details can be found in the Supplementary Material.
我们的实验在 OpenAI gym 的 Hopper-v1 环境中检查了三种不同的批处理设置（Todorov 等，2012 年；Brockman 等，2016 年），我们使用这些设置来训练一个无需与环境交互的离线 DDPG 代理。有关其他环境和具体细节的实验可以在补充材料中找到。

Batch 1 (Final buffer). We train a DDPG agent for 1 million time steps, adding $\mathcal{N}(0,0.5)$ Gaussian noise to actions for high exploration, and store all experienced transitions. This collection procedure creates a dataset with a diverse set of states and actions, with the aim of sufficient coverage.
批次 1（最终缓冲区）。我们训练一个 DDPG 代理器进行 100 万个时间步，对动作添加高探索性的高斯噪声，并存储所有经验转换。这个收集过程创建了一个包含各种状态和动作的数据集，旨在实现充分覆盖。

Batch 2 (Concurrent). We concurrently train the off-policy and behavioral DDPG agents, for 1 million time steps. To ensure sufficient exploration, a standard $\mathcal{N}(0,0.1)$ Gaussian noise is added to actions taken by the behavioral policy. Each transition experienced by the behavioral policy is stored in a buffer replay, which both agents learn from. As a result, both agents are trained with the identical dataset.
第二批（并发）。我们同时训练离策略和行为 DDPG 代理，进行 100 万个时间步。为了确保足够的探索，标准的高斯噪声被添加到行为策略采取的动作中。行为策略经历的每个转换都存储在一个缓冲重放中，两个代理都从中学习。因此，两个代理都使用相同的数据集进行训练。

Batch 3 (Imitation). A trained DDPG agent acts as an expert, and is used to collect a dataset of 1 million transitions.
第三批（模拟）。经过训练的 DDPG 代理充当专家，并用于收集 100 万个转换的数据集。

In Figure 1, we graph the performance of the agents as they train with each batch, as well as their value estimates. Straight lines represent the average return of episodes contained in the batch. Additionally, we graph the learning performance of the behavioral agent for the relevant tasks.
在图 1 中，我们绘制了代理随着每个批次训练的表现，以及它们的价值估计。直线代表批次中包含的剧集的平均回报。此外，我们还绘制了行为代理在相关任务中的学习表现。

Our experiments demonstrate several surprising facts about off-policy deep reinforcement learning agents. In each task, the off-policy agent performances significantly worse than the behavioral agent. Even in the concurrent experiment, where both agents are trained with the same dataset, there is a large gap in performance in every single trial. This result suggests that differences in the state distribution under the initial policies is enough for extrapolation error to drastically offset the performance of the off-policy agent. Additionally, the corresponding value estimate exhibits divergent behavior, while the value estimate of the behavioral agent is highly stable. In the final buffer experiment, the off-policy agent is provided with a large and diverse dataset, with the aim of providing sufficient coverage of the initial policy. Even in this instance, the value estimate is highly unstable, and the performance suffers. In the imitation setting, the agent is provided with expert data. However, the agent quickly learns to take non-expert actions, under the guise of optimistic extrapolation. As a result, the value estimates rapidly diverge and the agent fails to learn.
我们的实验展示了关于离策略深度强化学习代理的几个令人惊讶的事实。在每个任务中，离策略代理的表现明显比行为代理差。即使在并发实验中，两个代理都使用相同的数据集进行训练，每次试验的表现差距仍然很大。这一结果表明，初始策略下状态分布的差异足以使外策略代理的性能大幅下降。此外，相应的价值估计表现出分歧行为，而行为代理的价值估计非常稳定。在最终的缓冲实验中，离策略代理提供了一个大而多样的数据集，旨在提供足够覆盖初始策略的数据。即使在这种情况下，价值估计也非常不稳定，性能受到影响。在模仿设置中，代理被提供专家数据。然而，代理很快学会采取非专家行动，伪装成乐观的外推。结果，价值估计迅速分歧，代理无法学习。

Although extrapolation error is not necessarily positively biased, when combined with maximization in reinforcement learning algorithms, extrapolation error provides a source of noise that can induce a persistent overestimation bias (Thrun & Schwartz, 1993; Van Hasselt et al., 2016; Fujimoto et al., 2018). In an on-policy setting, extrapolation error may be a source of beneficial exploration through an implicit “optimism in the face of uncertainty” strategy (Lai & Robbins, 1985; Jaksch et al., 2010). In this case, if the value function overestimates an unknown state-action pair, the policy will collect data in the region of uncertainty, and the value estimate will be corrected. However, when learning off-policy, or in a batch setting, extrapolation error will never be corrected due to the inability to collect new data.
尽管外推误差不一定是正向偏差，但当与强化学习算法中的最大化结合时，外推误差会产生一种噪音源，可能导致持续的高估偏差。在一个在线策略设置中，外推误差可能是一种有益的探索源，通过一种隐含的“面对不确定性的乐观”策略。在这种情况下，如果价值函数高估了一个未知的状态-动作对，策略将在不确定性区域收集数据，并且价值估计将被修正。然而，在离线学习或批处理设置中，外推误差永远不会被纠正，因为无法收集新数据。

These experiments show extrapolation error can be highly detrimental to learning off-policy in a batch reinforcement learning setting. While the continuous state space and multi-dimensional action space in MuJoCo environments are contributing factors to extrapolation error, the scale of these tasks is small compared to real world settings. As a result, even with a sufficient amount of data collection, extrapolation error may still occur due to the concern of catastrophic forgetting (McCloskey & Cohen, 1989; Goodfellow et al., 2013). Consequently, off-policy reinforcement learning algorithms used in the real-world will require practical guarantees without exhaustive amounts of data.
这些实验表明，在批量强化学习设置中，外推误差可能对离线学习产生严重不利影响。虽然 MuJoCo 环境中的连续状态空间和多维动作空间是外推误差的 contributing factors，但与真实世界环境相比，这些任务的规模较小。因此，即使有足够的数据收集，由于对灾难性遗忘的担忧，外推误差仍可能发生。因此，在真实世界中使用的离线强化学习算法将需要在不需要大量数据的情况下提供实际保证。

In the finite MDP setting, extrapolation error can be described by the bias from the mismatch between the transitions contained in the buffer and the true MDP. We find that by inducing a data distribution that is contained entirely within the batch, batch-constrained policies can eliminate extrapolation error entirely for deterministic MDPs. In addition, we show that the batch-constrained variant of Q-learning converges to the optimal policy under the same conditions as the standard form of Q-learning. Moreover, we prove that for a deterministic MDP, batch-constrained Q-learning is guaranteed to match, or outperform, the behavioral policy when starting from any state contained in the batch. All of the proofs for this section can be found in the Supplementary Material.
在有限 MDP 设置中，外推误差可以被描述为缓冲区中包含的转换与真实 MDP 之间的不匹配所导致的偏差。我们发现，通过引入一个完全包含在批处理中的数据分布，批处理约束策略可以完全消除确定性 MDP 的外推误差。此外，我们证明了 Q-learning 的批处理约束变体在与标准形式的 Q-learning 相同的条件下收敛到最优策略。此外，我们证明了对于确定性 MDP，批处理约束 Q-learning 在从批处理中的任何状态开始时保证与或优于行为策略相匹配。本节的所有证明都可以在补充材料中找到。

A value estimate $Q$ can be learned using an experience replay buffer $\mathcal{B}$. This involves sampling transition tuples $(s,a,r,s^{\prime})$ with uniform probability, and applying the temporal difference update (Sutton, 1988; Watkins, 1989):
可以使用经验重放缓冲区学习价值估计。这涉及使用均匀概率对转换元组进行抽样，并应用时间差异更新（Sutton, 1988; Watkins, 1989）：

If $\pi(s^{\prime})=\operatorname*{argmax}_{a^{\prime}}Q(s^{\prime},a^{\prime})$, this is known as Q-learning. Assuming a non-zero probability of sampling any possible transition tuple from the buffer and infinite updates, Q-learning converges to the optimal value function.
如果是这样，这就被称为 Q 学习。假设从缓冲区中采样任何可能的转移元组的概率不为零，并且更新无限次，Q 学习将收敛到最优值函数。

We begin by showing that the value function $Q$ learned with the batch $\mathcal{B}$ corresponds to the value function for an alternate MDP $M_{\mathcal{B}}$. From the true MDP $M$ and initial values $Q(s,a)$, we define the new MDP $M_{\mathcal{B}}$ with the same action and state space as $M$, along with an additional terminal state $s_{\text{init}}$. $M_{\mathcal{B}}$ has transition probabilities $p_{\mathcal{B}}(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$, where $N(s,a,s^{\prime})$ is the number of times the tuple $(s,a,s^{\prime})$ is observed in $\mathcal{B}$. If $\sum_{\tilde{s}}N(s,a,\tilde{s})=0$, then $p_{\mathcal{B}}(s_{\text{init}}|s,a)=1$, where $r(s,a,s_{\text{init}})$ is set to the initialized value of $Q(s,a)$.
我们首先展示，使用批处理学习的值函数 $Q$ 对应于备用 MDP $M_{\mathcal{B}}$ 的值函数。从真实的 MDP $M$ 和初始值 $Q(s,a)$ 开始，我们定义具有与 $M$ 相同动作和状态空间的新 MDP $M_{\mathcal{B}}$ ，以及一个额外的终止状态 $s_{\text{init}}$ 。 $M_{\mathcal{B}}$ 具有转移概率 $p_{\mathcal{B}}(s^{\prime}|s,a)=\frac{N(s,a,s^{\prime})}{\sum_{\tilde{s}}N(s,a,\tilde{s})}$ ，其中 $N(s,a,s^{\prime})$ 是元组 $(s,a,s^{\prime})$ 在 $\mathcal{B}$ 中观察到的次数。如果 $\sum_{\tilde{s}}N(s,a,\tilde{s})=0$ ，则 $p_{\mathcal{B}}(s_{\text{init}}|s,a)=1$ ，其中 $r(s,a,s_{\text{init}})$ 设置为 $Q(s,a)$ 的初始化值。

Theorem 1. Performing Q-learning by sampling from a batch $\mathcal{B}$ converges to the optimal value function under the MDP $M_{\mathcal{B}}$.
定理 1. 通过从批次 $\mathcal{B}$ 中抽样执行 Q-learning 收敛到 MDP $M_{\mathcal{B}}$ 下的最优值函数。

We define $\epsilon_{\text{MDP}}$ as the tabular extrapolation error, which accounts for the discrepancy between the value function $Q^{\pi}_{\mathcal{B}}$ computed with the batch $\mathcal{B}$ and the value function $Q^{\pi}$ computed with the true MDP $M$:
我们将 $\epsilon_{\text{MDP}}$ 定义为表格外推误差，它考虑了使用批处理 $\mathcal{B}$ 计算的值函数 $Q^{\pi}_{\mathcal{B}}$ 与使用真实 MDP $M$ 计算的值函数 $Q^{\pi}$ 之间的差异：

For any policy $\pi$, the exact form of $\epsilon_{\text{MDP}}(s,a)$ can be computed through a Bellman-like equation:
对于任何政策 $\pi$ ，可以通过类似贝尔曼方程的方式计算出 $\epsilon_{\text{MDP}}(s,a)$ 的确切形式：

This means extrapolation error is a function of divergence in the transition distributions, weighted by value, along with the error at succeeding states. If the policy is chosen carefully, the error between value functions can be minimized by visiting regions where the transition distributions are similar. For simplicity, we denote
这意味着外推误差是转移分布差异的函数，由价值加权，以及后续状态的误差。如果策略选择得当，通过访问转移分布相似的区域，可以最小化值函数之间的误差。为简单起见，我们表示

To evaluate a policy $\pi$ exactly at relevant state-action pairs, only $\epsilon^{\pi}_{\text{MDP}}=0$ is required. We can then determine the condition required to evaluate the exact expected return of a policy without extrapolation error.
要在相关的状态-动作对上精确评估一项政策，只需要 $\epsilon^{\pi}_{\text{MDP}}=0$ 。然后我们可以确定评估一项政策的精确预期回报所需的条件，而不会出现外推误差。

Lemma 1. For all reward functions, $\epsilon^{\pi}_{\text{MDP}}=0$ if and only if $p_{\mathcal{B}}(s^{\prime}|s,a)=p_{M}(s^{\prime}|s,a)$ for all $s^{\prime}\in\mathcal{S}$ and $(s,a)$ such that $\mu_{\pi}(s)>0$ and $\pi(a|s)>0$.
引理 1. 对于所有的奖励函数， $\epsilon^{\pi}_{\text{MDP}}=0$ 当且仅当对于所有的 $s^{\prime}\in\mathcal{S}$ 和 $(s,a)$ ，满足 $\mu_{\pi}(s)>0$ 和 $\pi(a|s)>0$ 。

Lemma 1 states that if $M_{\mathcal{B}}$ and $M$ exhibit the same transition probabilities in regions of relevance, the policy can be accurately evaluated. For a stochastic MDP this may require an infinite number of samples to converge to the true distribution, however, for a deterministic MDP this requires only a single transition. This means a policy which only traverses transitions contained in the batch, can be evaluated without error. More formally, we denote a policy $\pi\in\Pi_{\mathcal{B}}$ as batch-constrained if for all $(s,a)$ where $\mu_{\pi}(s)>0$ and $\pi(a|s)>0$ then $(s,a)\in\mathcal{B}$. Additionally, we define a batch $\mathcal{B}$ as coherent if for all $(s,a,s^{\prime})\in\mathcal{B}$ then $s^{\prime}\in\mathcal{B}$ unless $s^{\prime}$ is a terminal state. This condition is trivially satisfied if the data is collected in trajectories, or if all possible states are contained in the batch. With a coherent batch, we can guarantee the existence of a batch-constrained policy.
引理 1 指出，如果 $M_{\mathcal{B}}$ 和 $M$ 在相关区域展示相同的转移概率，那么可以准确评估策略。对于随机 MDP，这可能需要无限数量的样本才能收敛到真实分布，然而对于确定性 MDP，只需要一个转移。这意味着只遍历批次中包含的转移的策略可以无误地进行评估。更正式地，我们将一个策略 $\pi\in\Pi_{\mathcal{B}}$ 称为批次受限的，如果对于所有 $(s,a)$ ，其中 $\mu_{\pi}(s)>0$ 和 $\pi(a|s)>0$ ，则 $(s,a)\in\mathcal{B}$ 。此外，我们定义一个批次 $\mathcal{B}$ 为一致的，如果对于所有 $(s,a,s^{\prime})\in\mathcal{B}$ ，则 $s^{\prime}\in\mathcal{B}$ ，除非 $s^{\prime}$ 是一个终止状态。如果数据是按轨迹收集的，或者所有可能的状态都包含在批次中，则这个条件显然是满足的。有了一致的批次，我们可以保证存在一个受批次限制的策略的存在。

Theorem 2. For a deterministic MDP and all reward functions, $\epsilon^{\pi}_{\text{MDP}}=0$ if and only if the policy $\pi$ is batch-constrained. Furthermore, if $\mathcal{B}$ is coherent, then such a policy must exist if the start state $s_{0}\in\mathcal{B}$.
定理 2. 对于一个确定性 MDP 和所有奖励函数， $\epsilon^{\pi}_{\text{MDP}}=0$ 当且仅当策略 $\pi$ 是批次受限的。此外，如果 $\mathcal{B}$ 是连贯的，那么这样的策略必须存在，如果起始状态 $s_{0}\in\mathcal{B}$ 。

Batch-constrained policies can be used in conjunction with Q-learning to form batch-constrained Q-learning (BCQL), which follows the standard tabular Q-learning update while constraining the possible actions with respect to the batch:
批量约束策略可以与 Q 学习结合使用，形成批量约束 Q 学习（BCQL），其遵循标准的表格 Q 学习更新，同时限制了与批量相关的可能动作：

BCQL converges under the same conditions as the standard form of Q-learning, noting the batch-constraint is nonrestrictive given infinite state-action visitation.
BCQL 在相同条件下收敛，注意到批量约束在状态-动作访问无限时是非限制性的。

Theorem 3. Given the Robbins-Monro stochastic convergence conditions on the learning rate $\alpha$, and standard sampling requirements from the environment, BCQL converges to the optimal value function $Q^{*}$.
定理 3. 在学习率 $\alpha$

The more interesting property of BCQL is that for a deterministic MDP and any coherent batch $\mathcal{B}$, BCQL converges to the optimal batch-constrained policy $\pi^{*}\in\Pi_{\mathcal{B}}$ such that $Q^{\pi^{*}}(s,a)\geq Q^{\pi}(s,a)$ for all $\pi\in\Pi_{\mathcal{B}}$ and $(s,a)\in\mathcal{B}$.
BCQL 的更有趣的特性是，对于一个确定性 MDP 和任何一致的批次 $\mathcal{B}$

Theorem 4. Given a deterministic MDP and coherent batch $\mathcal{B}$, along with the Robbins-Monro stochastic convergence conditions on the learning rate $\alpha$ and standard sampling requirements on the batch $\mathcal{B}$, BCQL converges to $Q^{\pi}_{\mathcal{B}}(s,a)$ where $\pi^{*}(s)=\operatorname*{argmax}_{a\text{ s.t.}(s,a)\in\mathcal{B}}Q^{\pi}_{\mathcal{B}}(s,a)$ is the optimal batch-constrained policy.
定理 4. 给定确定性 MDP 和一致的批次 $\mathcal{B}$

This means that BCQL is guaranteed to outperform any behavioral policy when starting from any state contained in the batch, effectively outperforming imitation learning. Unlike standard Q-learning, there is no condition on state-action visitation, other than coherency in the batch.
这意味着 BCQL 保证在从批处理中的任何状态开始时优于任何行为策略，有效地超越了模仿学习。与标准 Q 学习不同，除了批处理中的一致性外，对状态-动作访问没有条件。

We introduce our approach to off-policy batch reinforcement learning, Batch-Constrained deep Q-learning (BCQ). BCQ approaches the notion of batch-constrained through a generative model. For a given state, BCQ generates plausible candidate actions with high similarity to the batch, and then selects the highest valued action through a learned Q-network. Furthermore, we bias this value estimate to penalize rare, or unseen, states through a modification to Clipped Double Q-learning (Fujimoto et al., 2018). As a result, BCQ learns a policy with a similar state-action visitation to the data in the batch, as inspired by the theoretical benefits of its tabular counterpart.
我们介绍了我们的离线批量强化学习方法，批量约束深度 Q 学习（BCQ）。BCQ 通过生成模型来处理批量约束的概念。对于给定的状态，BCQ 生成与批量高度相似的可信候选动作，然后通过学习的 Q 网络选择价值最高的动作。此外，我们通过对 Clipped Double Q 学习（Fujimoto 等人，2018）进行修改，对罕见或未见状态进行惩罚以偏置这个价值估计。因此，BCQ 学习了一个与批量数据中的状态-动作访问类似的策略，受到其表格对应物理论优势的启发。

To maintain the notion of batch-constraint, we define a similarity metric by making the assumption that for a given state $s$, the similarity between $(s,a)$ and the state-action pairs in the batch $\mathcal{B}$ can be modelled using a learned state-conditioned marginal likelihood $P_{\mathcal{B}}^{G}(a|s)$. In this case, it follows that the policy maximizing $P_{\mathcal{B}}^{G}(a|s)$ would minimize the error induced by extrapolation from distant, or unseen, state-action pairs, by only selecting the most likely actions in the batch with respect to a given state. Given the difficulty of estimating $P_{\mathcal{B}}^{G}(a|s)$ in high-dimensional continuous spaces, we instead train a parametric generative model of the batch $G_{\omega}(s)$, which we can sample actions from, as a reasonable approximation to $\operatorname*{argmax}_{a}P_{\mathcal{B}}^{G}(a|s)$.
为了保持批次约束的概念，我们通过假设对于给定状态 $s$ ，定义了一个相似度度量，该度量可以用学习的状态条件边际似然 $P_{\mathcal{B}}^{G}(a|s)$ 来建模 $(s,a)$ 和批次 $\mathcal{B}$ 中的状态-动作对之间的相似度。在这种情况下，遵循这样一个策略，最大化 $P_{\mathcal{B}}^{G}(a|s)$ 的策略将通过仅选择相对于给定状态在批次中最可能的动作来最小化由远距离或未见的状态-动作对引起的误差。鉴于在高维连续空间中估计 $P_{\mathcal{B}}^{G}(a|s)$ 的困难，我们改为训练批次 $G_{\omega}(s)$ 的参数生成模型，我们可以从中采样动作，作为 $\operatorname*{argmax}_{a}P_{\mathcal{B}}^{G}(a|s)$ 的合理近似。

For our generative model we use a conditional variational auto-encoder (VAE) (Kingma & Welling, 2013; Sohn et al., 2015), which models the distribution by transforming an underlying latent space¹¹1See the Supplementary Material for an introduction to VAEs.. The generative model $G_{\omega}$, alongside the value function $Q_{\theta}$, can be used as a policy by sampling $n$ actions from $G_{\omega}$ and selecting the highest valued action according to the value estimate $Q_{\theta}$. To increase the diversity of seen actions, we introduce a perturbation model $\xi_{\phi}(s,a,\Phi)$, which outputs an adjustment to an action $a$ in the range $[-\Phi,\Phi]$. This enables access to actions in a constrained region, without having to sample from the generative model a prohibitive number of times. This results in the policy $\pi$:
对于我们的生成模型，我们使用了一个条件变分自动编码器（VAE）（Kingma＆Welling，2013 年; Sohn 等，2015 年），它通过转换潜在的潜在空间来建模分布。生成模型，连同价值函数，可以作为策略使用，通过从中采样动作并根据价值估计选择价值最高的动作。为了增加所见动作的多样性，我们引入了一个扰动模型，它输出一个在范围内的动作调整。这使得可以访问受限区域内的动作，而无需从生成模型中采样过多次。这导致了策略：

The choice of $n$ and $\Phi$ creates a trade-off between an imitation learning and reinforcement learning algorithm. If $\Phi=0$, and the number of sampled actions $n=1$, then the policy resembles behavioral cloning and as $\Phi\rightarrow a_{\text{max}}-a_{\text{min}}$ and $n\rightarrow\infty$, then the algorithm approaches Q-learning, as the policy begins to greedily maximize the value function over the entire action space.
选择 $n$ 和 $\Phi$ 之间存在着模仿学习和强化学习算法之间的权衡。如果 $\Phi=0$ ，以及采样动作的数量 $n=1$ ，那么策略类似于行为克隆，当 $\Phi\rightarrow a_{\text{max}}-a_{\text{min}}$ 和 $n\rightarrow\infty$ 时，该算法接近 Q 学习，因为策略开始贪婪地最大化整个动作空间的值函数。

The perturbation model $\xi_{\phi}$ can be trained to maximize $Q_{\theta}(s,a)$ through the deterministic policy gradient algorithm (Silver et al., 2014) by sampling $a\sim G_{\omega}(s)$:
扰动模型 $\xi_{\phi}$ 可以通过采样 $a\sim G_{\omega}(s)$ ，通过确定性策略梯度算法（Silver 等人，2014 年）进行训练以最大化 $Q_{\theta}(s,a)$ ：