持续深度强化学习中的可塑性损失

强化学习 （RL） 方法已成功用于多个实际应用，从发现视频压缩（Mandhane 等人，2022 年）和矩阵乘法（Fawzi 等人，2022 年）的卓越算法到为 Google 的张量处理单元设计卓越的平面图（Mirhoseini 等人，2021 年）;从有效控制冷却系统（Luo 等人，2022 年）和火力发电机（Zhan 等人，2022 年）和管理库存（Madeka 等人，2022 年）到在平流层中导航气球（Bellemare 等人，2020 年）和自动生产托卡马克核聚变反应堆的等离子体配置（Degrave 等人，2022 年）。

在很大程度上，这些系统可以被描述为对固定策略的搜索，一旦部署就不再更改。搜索通常使用一批历史数据离线完成，或者更频繁地在模拟器上完成（例如，Madeka 等人，2022 年;Zhan 等人，2022 年;Mandhane 等人，2022 年;Fawzi 等人，2022 年;Degrave 等人，2022 年;Bellemare 等人，2020 年）。事实上，在某些情况下，追求稳态策略是完全合适的，因为该策略一旦部署就不需要更改（例如，将矩阵乘以所需计算复杂度的策略）。但是，在许多应用程序中，一旦系统部署并获得更多数据，固定策略就会很快失效。例如，在冷却系统控制器中，天气模式和传感器漂移的变化需要不断学习以产生有效的策略（Luo 等人，2022 年）。更一般地说，在学习系统周围的世界不断变化的情况下，持续的适应是不可避免的（Sutton et al.， 2007）。

如今，强化学习在现实世界中最著名的应用是神经网络函数逼近（即深度强化学习）。用于函数逼近的大部分神经网络工具箱都针对具有固定数据集（即监督学习）的稳态设置进行了完善，这些数据集允许独立和相同分布 （IID） 采样（Hadsell 等人，2020 年）。值得注意的是，最近的研究表明，当数据逐渐可用时，神经网络的学习效果很差（Ash&Adams，2020）。更关键的是，当面临随时间推移的分布变化时，神经网络完全失去了学习能力——这种现象的特征是可塑性丧失（Dohare et al.， 2021）。

也许不足为奇的是，当神经网络首次成功应用于通过强化学习玩Atari 2600游戏时——这是一个固有的非平稳问题，因为数据分布会随着策略的变化而变化——大型重播缓冲区等工具被用来帮助近似稳态（Mnih et al.， 2015）。但是，嵌入复杂世界中的长期强化学习代理必须处理在时间尺度上表现出来的非平稳性，这些非平稳性比可行存储在重放缓冲区中的时间要长得多（Lesort 等人，2020 年;Luo et al.， 2022），强调需要可以跨时间尺度工作的持续学习解决方案。

Despite impressive benchmark performance of RL agents on simulated domains, the inherent non-stationarity of the reinforcement learning problem can still highlight limitations in the prevalent solution methods; limitations that are clearly visible on careful inspection. For instance, neural networks lose capacity over time as they estimate continually changing value functions resulting from changing policies  (Kumar et al., 2021; Lyle et al., 2022). Related, deep reinforcement learning systems overfit to experiences gathered early in training, being unable to adjust as more experience becomes available—a phenomenon known as the primacy bias (Nikishin et al., 2022). We hypothesize that these are all instances of the same underlying problem. Learning systems exhaust their learning capital over time and ultimately fail to meet the demands of a changing world, a phenomenon hereinafter referred to as catastrophic loss of plasticity (Dohare et al., 2021).

In this paper, we contribute to the growing literature on catastrophic loss of plasticity in several ways. First, we demonstrate catastrophic loss of plasticity in a performant deep reinforcement learning system, Rainbow (Hessel et al., 2018), on a variation of the Arcade Learning Environment (ALE) (Bellemare et al., 2013; Machado et al., 2018) that involves non-stationarities from changing Atari 2600 games and game modes over time. Some of the experiments presented in this paper are run up to two billion environment interactions spanning 50 days of wall clock time. The result depicted in Figure 1 provides an example of the larger set of performance results presented in this paper. Next, we perform a careful analysis of weights, gradients, and activations sampled over the lifetime of the experiment. The analysis augments the existing findings in the literature and points to a remarkably simple mitigation. Finally, we evaluate one mitigation—the concatenated ReLU activation—and show its effectiveness.

In this paper we study agents that continually interact with their environment and learn to achieve a goal. In particular, the agent’s objective is to select actions based on some state, $S_{t}\in\mathcal{S}$, to maximize a scalar reward signal, $R_{t}\in\mathbb{R}$, over time. The agent’s action, $A_{t}$, is determined by its policy, $\pi(A_{t}|S_{t})$, which is adapted towards the goal of maximizing total discounted future reward: $G_{t}\doteq\sum_{i=0}^{\infty}\gamma^{i}R_{t+1+i}$, where $\gamma\in[0,1)$.

In this paper we focus exclusively on methods that learn value functions to obtain the policy: Deep Q-learning (DQN) and related variants. Put simply, a DQN agent continually adapts a parameteric estimate of future reward called the value function: $\hat{q}_{\bf w}(s,a)\approx\mathbb{E}[G_{t}|S_{t}=s,A_{t}=a]$, where ${\bf w}$ are the parameters of a neural network. On each timestep, DQN (1) selects an action according to some exploratory policy (e.g., $\epsilon$-greedy), (2) stores the most recent experience in replay buffer, and (3) samples a mini-batch from the replay buffer and updates the weights with gradients computed with the backpropagation algorithm: $\nabla_{\bf w}\mathbb{E}[(R_{t+1}-\gamma\max_{a\in\mathcal{A}}\hat{q}_{\bar{\bf w}}(S_{t+1},a))-\hat{q}_{{\bf w}}(S_{t},A_{t}))^{2}]$, where $\bar{\bf w}$ are the parameters of the target network that are periodically set equal to ${\bf w}$. Often (e.g., in Atari 2600 games) the agent does not observe $S_{t}$, but instead an incomplete summary of the state such as the four most recent game frames.

The majority of our experiments focus on the Rainbow agent: an implementation based on DQN (Mnih et al., 2015), which is highly tuned to play Atari 2600 games (Hessel et al., 2018). In particular, Rainbow combines several algorithmic components including distributional RL (Bellemare et al., 2017), noisy networks for exploration (Fortunato et al., 2018), prioritized experience replay (Schaul et al., 2016), dueling networks (Wang et al., 2016), n-step returns (Sutton & Barto, 2018), and a few other ideas. The performance of Rainbow is representative of the best performing model-free, synchronous (single environment) RL implementations available. Because of dueling networks, Rainbow’s neural network has two streams, one that estimates the value function, $\hat{v}_{{\bf w_{1}}}(s)\approx\mathbb{E}[G_{t}|S_{t}=s]$, and one that estimates the advantage, $\hat{d}_{{\bf w_{2}}}(s,a)$, such that $\hat{v}_{{\bf w_{1}}}(s)+\hat{d}_{{\bf w_{2}}}(s,a)\approx\mathbb{E}[G_{t}|S_{t}=s,A_{t}=a]$. In this paper, we term the shared network layers, before the split for the two streams, convolution network, the set of layers used in $\hat{v}_{{\bf w_{1}}}$’s stream value network, and those used in $\hat{d}_{{\bf w_{2}}}$’s stream advantage network.

In this paper we focus our empirical analysis on a subset of Atari 2600 games using the Arcade Learning Environment (ALE) (Bellemare et al., 2013; Machado et al., 2018). The ALE is a standard domain for evaluating deep reinforcement learning algorithms and the two specific algorithms we study, Rainbow and DQN, were originally developed for Atari 2600 games. Traditionally, the evaluation protocol within ALE consists of having an agent interact with a specific Atari 2600 game for 200 million frames, reporting the final performance achieved by the agent. We abide by Machado et al. (2018)’s recommendations of evaluation protocol, including the use of stochasticity, ignoring the lives signal, and reporting the average performance during training. We diverge from the standard practice only by deploying a single agent, with a single neural network, to play a sequence of games and to do well in all of them (which we describe in detail in the next section). In most prior work agents are forced to follow a specific schedule to ensure samples of all games are seen often during training (e.g., Reed et al., 2022). The setup explored in this paper is designed to mimic non-stationarity in continual learning tasks.

In this section we describe S-ALE, a continual variant of the ALE, and we use it to evaluate the performance of Rainbow under varying degrees of non-stationarity, demonstrating how the Rainbow agent loses plasticity over time.

In this section we take a closer look at how the internal components of Rainbow evolve over time. Before we start, it is helpful to summarize our findings. As shown earlier, the performance progressively worsens each time the Rainbow agent revisits a game and, in some cases, it collapses altogether. We observe the first hint of reduced plasticity by inspecting how much the network weights change each time the agent visits a game—in the game Alien, for example, the weight change consistently shrinks with additional visits (c.f. Figure 4 a). More critically, the weight change continues to diminish even though the (distributional) loss remains large (Figure 4 b). One would expect a large loss to result in large gradients, but we find that the magnitude of gradient also diminishes on every successive visit (Figure 4 c). Finally, looking at the activations over time, we see fewer and fewer units are active (Figure 4 d). This makes sense. The Rainbow agent uses Rectified Linear Units (ReLUs). If a ReLU is not active (i.e. zero output value), then it does not contribute to the change in its incoming weights (via the chain rule). A learning system that is made up of unchanging weights, irrespective of a large global loss, can be well thought of as having lost its ability to learn new things—the network is losing plasticity.

Next we discuss how we computed each of the statistics discussed above and we present more detailed comparisons.

In this section we explore a simple approach to increase and maintain network activations and thus mitigate the loss of plasticity. Recall that the internal activations of Rainbow’s network decayed to the point where only a small fraction of the units produced non-zero values (less than 1% in fact). This collapse in activation resulted in diminishing gradients and prevented further change to the weights, thus, impeding learning. Here we consider a simple change to the network architecture that can prevent activation collapse: Concatenated Rectified Linear Units (CReLUs). Originally proposed to understand and improve convolutional architectures for object recognition (Shang et al., 2016), CReLU concatenates the input with its negation and applies ReLU to the result: $\text{CReLU}(x)\doteq[ReLU(x),ReLU(-x)]$. CReLU outputs twice the number of input signals and ensures that, for every input signal, one of the two output signal is nonzero, except when the input signal is exactly zero.

Evaluating CReLUs requires us to pay attention to the effective capacity of the network. In a given layer, replacing ReLU with CReLU doubles the number of activations while keeping the number of parameters the same. This doubling can potentially double the number of parameters in the following layer. To isolate the contribution of CReLU in mitigating the loss of plasticity, we experiment with two approaches. The first approach controls for the number of input signals before the application of the activation function (invariant input dimension). For the CReLU variant of continual Rainbow, this doubles the number of parameters in every layer (except the first one). The second approach controls for the number of output signals after the application of the activation function (invariant output dimension). This halves the number of parameters in every layer (except the last one) in the CReLU variant. We evaluate both architectures in our S-ALE setup with no change in hyper-parameters.

The performance of the Rainbow agent with CReLU and invariant input dimension is depicted in Figure 5. The results show that Rainbow-CReLU—unlike Rainbow-ReLU—continues to maintain or improve its performance on successive visits in all five games. On every repeated visit to a game, having forgotten what was learned on the previous visit, Rainbow-CReLU (like Rainbow with ReLUs) must learn to play the game again. The Rainbow-CReLU agent demonstrates improved plasticity by matching (or exceeding, in Breakout and Centipede, for example) the performance of the reset agent and its own performance in prior visits. The results for the invariant output dimension can be seen in Figure 17 in the Appendix. This variant of Rainbow-CReLU with less parameters also clearly demonstrates maintenance of plasticity.

It would be reasonable to suspect that the improvement in plasticity is simply due to the general superiority of CReLU over ReLU, but this does not seem to be the case. Figure 6 shows that ReLU-Rainbow (i.e. ReLU Scratch) performs comparably in the conventional (non-continual) ALE, that is, when only learning from scratch in each game. For completeness, we also compared continual Rainbow-CReLU against the scratch Rainbow-CReLU agent (as opposed to the scratch Rainbow-ReLU in the plots above) to quantify the impact in performance that cycling through multiple games instead of learning only from scratch has when using CReLU activations. The result can be found in Figure 18 in the Appendix.

We see a similar pattern when we evaluate Rainbow-CReLU for sequential training on Breakout game modes. As depicted in Figure 7, Rainbow-CReLU continues to maintain or improve its performance on successive visits in all 10 game modes. Rainbow-CReLU shows no significant improvement over Rainbow across the game modes of Space Invaders, presumably because Rainbow did not experience significant loss of plasticity (see Figure 19 in the Appendix for the evaluation of Rainbow-CReLU in the different game modes of Space Invaders).

Finally, we inspect the internal statistics of the Rainbow-CReLU agent to validate whether they are consistent to what we would expect based on the discussion in Section 4. They constitute clear evidence of improvement in plasticity. As before we focus on the game Alien. First, Figure 8d shows that about half of the activations continue to produce non-zero values; CReLU is preventing activation collapse (the green curves). It does so by construction: for every input signal, one of the two output signals is nonzero, except when the input signal is exactly zero (unlikely because of the 32-bit floating point representation of real numbers). Figure 8c shows that the relative norm of the gradients stays large (as we would expect if activations did not collapse). The $\ell_{0}$ norm does not diminish relative to the first visit. Relative $\ell_{1}$ norm does reduce, however, it does not collapse like with ReLUs, reducing much more slowly than Scratch; and the rate of decay becomes smaller over visits. Figure 8a shows that weights in Rainbow-CReLU continue to change significantly relative to the first visit all the way through ten visits: as we would expect if gradients did not collapse. The relative weight change for Rainbow-CReLU is larger than not only Rainbow-ReLU but also the Scratch baseline, which makes sense given that the loss function observed with CReLUs is larger than the loss function observed with ReLU Scratch. Figure 8b shows the sampled loss of Rainbow-CReLU. It does not grow to be as large as Rainbow-ReLU, likely due to its improved plasticity that allows it to continue to learn. Note, as aforementioned, that the training loss of continual Rainbow with CReLUs is still higher than Scratch; this is the topic of the next section.

Up to this point we have discussed a large number of results, and thus it is useful to summarize what we have learned. We have focused on catastrophic loss of plasticity as the core phenomenon, demonstrating it in the S-ALE set up. We analyzed the learning dynamics over time and evaluated a simple mitigation strategy. In this section, we discuss how the same S-ALE experiments relate to the well-established phenomenon of catastrophic forgetting  (French, 1999).

In all the results discussed in this paper, every time the Rainbow agent revisits a game (or mode), it shows no retention of what was previously learned: the agent forgets. The CReLU mitigation does not address this issue and we see no reason to expect as such. Consider the results in Figure 9. The Rainbow agent with CReLU activations, when cycling through five games, can improve the performance over successive visits in some cases (green curve), but the improvement appears modest when we compare with an agent that learns uninterrupted on a single game with the same amount of experience (red curve). Figure 20 in the Appendix shows similar results for a ten-game sequence and Figure 18, also in the Appendix, presents similar results when using CReLU Scratch as a baseline.

It is unclear if it is realistic to expect agents such as Rainbow—designed for non-continual ALE—to remember what they learned over 100M frames ago and transfer learning between visits. Nonetheless, there is a widening gulf between the idealized agent that learn uninterrupted (the red curve in Figure 9) and continual Rainbow. The path forward seems clear: synthesizing continual learning approaches that address stability-plasticity dilemma and address both catastrophic forgetting and the loss of plasticity.

Continual adaptation is essential as no amount of prior learning is sufficient in complex and continually changing environments. Most of the solution methods in deep RL have been developed and evaluated in mostly stationary settings, and it is often assumed that these solution methods are naturally applicable to continual learning. In this paper we showed that canonical value-based deep RL methods in DQN and Rainbow are in fact not able to perform well in continual learning problems.

The experiments were performed in a non-stationary setting where the agent is tasked to play a sequence of Atari 2600 games without any resetting in between. The results demonstrated that deep RL algorithms can lose their ability to learn (i.e. update the weights of the neural network), a phenomenon known as the loss of plasticity. We inspected the evolution of weights, gradients, and activations over the course of learning and observed activation collapse: in the continual setup with a non-stationary environment, a tiny fraction of network units produce non-zero activation value, inhibiting adaptation. Finally, we demonstrated that swapping ReLUs with CReLUs mitigate the loss of plasticity.

There are several promising directions for future work. Further unifying the observed phenomenon in S-ALE to related observations–e.g., implicit under-parameterization, primacy bias, and capacity loss—can lead to better tools for instrumenting deep RL methods. Of course, CReLUs are not the only way of tackling the issues identified in this paper. For example, re-initializing parts of the network randomly, or according to some notion of utility (e.g., Dohare et al., 2021) are promising solutions that should be further investigated. More importantly, while CReLUs do mitigate the loss of plasticity, they are not sufficient for the agent to effectively reuse past experience—they do not resolve catastrophic interference. This paper represents a small step toward the goal continual deep reinforcement learning: developing agents that can maintain plasticity in the face of an ever changing world and efficiently leverage prior learning.

This work was developed while the authors were at DeepMind. The authors would like to thank Thomas Degris for his thorough feedback on an earlier draft, and Joshua Davidson, Finbarr Timbers, and Doina Precup for useful discussions.