Perpetual Humanoid Control for Real-time Simulated Avatars
实时模拟化身的永久人形控制

The simulation state ${\bm{s}_{t}}\triangleq({\bm{s}^{\text{p}}_{t}},\bm{s}^{\text{g}}_{t})$ consists of humanoid proprioception ${\bm{s}^{\text{p}}_{t}}$ and the goal state ${\bm{s}^{\text{g}}_{t}}$. Proprioception ${\bm{s}^{\text{p}}_{t}}\triangleq({\bm{{q}}_{t}},{\bm{\dot{q}}_{t}},\bm{\beta})$ contains the 3D body pose ${\bm{{q}}_{t}}$, velocity ${\bm{\dot{q}}_{t}}$, and (optionally) body shapes $\bm{\beta}$. When trained with different body shapes, $\bm{\beta}$ contains information about the length of the limb of each body link [24]. For rotation-based motion imitation, the goal state ${\bm{s}^{\text{g}}_{t}}$ is defined as the difference between the next time step reference quantitives and their simulated counterpart:
模拟状态 ${\bm{s}_{t}}\triangleq({\bm{s}^{\text{p}}_{t}},\bm{s}^{\text{g}}_{t})$ 由人形本体感受 ${\bm{s}^{\text{p}}_{t}}$ 和目标状态 ${\bm{s}^{\text{g}}_{t}}$ 组成。本体感觉 ${\bm{s}^{\text{p}}_{t}}\triangleq({\bm{{q}}_{t}},{\bm{\dot{q}}_{t}},\bm{\beta})$ 包含 3D 身体姿势 ${\bm{{q}}_{t}}$ 、速度 ${\bm{\dot{q}}_{t}}$ 和（可选）身体形状 $\bm{\beta}$ 。当使用不同的身体形状进行训练时， $\bm{\beta}$ 包含有关每个身体链接的肢体长度的信息[24]。对于基于旋转的运动模仿，目标状态 ${\bm{s}^{\text{g}}_{t}}$ 定义为下一时间步参考量与其模拟对应量之间的差异：

where $\ominus$ calculates the rotation difference. For keypoint-only imtiation, the goal state becomes
其中 $\ominus$ 计算旋转差。对于仅关键点模仿，目标状态变为

All of the above quantities in ${\bm{s}^{\text{g}}_{t}}$ and ${\bm{s}^{\text{p}}_{t}}$ are normalized with respect to the humanoid’s current facing direction and root position [49, 22].
${\bm{s}^{\text{g}}_{t}}$ 和 ${\bm{s}^{\text{p}}_{t}}$ 中的所有上述量均根据人形机器人当前的面向方向和根位置进行归一化 [49, 22]。

Unlike prior motion tracking policies that only use a motion imitation reward, we use the recently proposed Adversarial Motion Prior [35] and include a discriminator reward term throughout our framework. Including the discriminator term helps our controller produce stable and natural motion and is especially crucial in learning natural fail-state recovery behaviors. Specifically, our reward is defined as the sum of a task reward $r^{\text{g}}_{t}$, a style reward $r^{\text{amp}}_{t}$, and an additional energy penalty $r^{\text{energy}}_{t}$ [31]:
与仅使用运动模仿奖励的先前运动跟踪策略不同，我们使用最近提出的对抗性运动先验[35]，并在整个框架中包含判别器奖励项。包含鉴别器项有助于我们的控制器产生稳定和自然的运动，对于学习自然故障状态恢复行为尤其重要。具体来说，我们的奖励被定义为任务奖励 $r^{\text{g}}_{t}$ 、风格奖励 $r^{\text{amp}}_{t}$ 和额外能量惩罚 $r^{\text{energy}}_{t}$ 的总和[31]：

For the discriminator, we use the same observations, loss formulation, and gradient penalty as AMP [35]. The energy penalty is expressed as $-0.0005\cdot\sum_{j\in\text{ joints }}\left|\bm{\mu_{j}}\bm{\omega_{j}}\right|^{2}$ where $\bm{\mu_{j}}$ and $\bm{\omega_{j}}$ correspond to the joint torque and the joint angular velocity, respectively. The energy penalty [10] regulates the policy and prevents high-frequency jitter of the foot that can manifest in a policy trained without external force (see Sec.4.1). The task reward is defined based on the current training objective, which can be chosen by switching the reward function for motion imitation $\bm{\mathcal{R}}^{\text{imitation}}$ and fail-state recovery $\bm{\mathcal{R}}^{\text{recover}}$. For motion tracking, we use:
对于判别器，我们使用与 AMP [35] 相同的观察结果、损失公式和梯度惩罚。能量损失表示为 $-0.0005\cdot\sum_{j\in\text{ joints }}\left|\bm{\mu_{j}}\bm{\omega_{j}}\right|^{2}$ ，其中 $\bm{\mu_{j}}$ 和 $\bm{\omega_{j}}$ 分别对应于关节扭矩和关节角速度。能量惩罚[10]调节策略并防止脚的高频抖动，这种抖动可以在没有外力训练的策略中体现出来（参见第 4.1 节）。任务奖励是根据当前训练目标定义的，可以通过切换运动模仿 $\bm{\mathcal{R}}^{\text{imitation}}$ 和失败状态恢复 $\bm{\mathcal{R}}^{\text{recover}}$ 的奖励函数来选择。对于运动跟踪，我们使用：

where we measure the difference between the translation, rotation, linear velocity, and angular velocity of the rigid body for all links in the humanoid. For fail-state recovery, we define the reward $r^{\text{g-recover}}_{t}$ in Eq.3.
我们测量人形机器人中所有连杆刚体的平移、旋转、线速度和角速度之间的差异。对于失败状态恢复，我们在等式3中定义奖励 $r^{\text{g-recover}}_{t}$ 。

We use a proportional derivative (PD) controller at each DoF of the humanoid and the action $\bm{a}_{t}$ specifies the PD target. With the target joint set as $\bm{q}_{t}^{d}={\bm{a}_{t}}$, the torque applied at each joint is $\bm{\tau}^{i}=\bm{k}^{p}\circ\left({\bm{a}_{t}}-\bm{q}_{t}\right)-\bm{k}^{d}\circ\dot{\bm{q}}_{t}$. Notice that this is different from the residual action representation [52, 22, 30] used in prior motion imitation methods, where the action is added to the reference pose: $\bm{q}^{d}_{t}=\hat{\bm{q}}_{t}+\bm{a}_{t}$ to speed up training. As our PHC needs to remain robust to noisy and ill-posed reference motion, we remove such a dependency on reference motion in our action space. We do not use any external forces [52] or meta-PD control[54].
我们在人形机器人的每个 DoF 处使用比例微分 (PD) 控制器，并且动作 $\bm{a}_{t}$ 指定 PD 目标。将目标关节设置为 $\bm{q}_{t}^{d}={\bm{a}_{t}}$ ，每个关节施加的扭矩为 $\bm{\tau}^{i}=\bm{k}^{p}\circ\left({\bm{a}_{t}}-\bm{q}_{t}\right)-\bm{k}^{d}\circ\dot{\bm{q}}_{t}$ 。请注意，这与先前运动模仿方法中使用的残留动作表示[52,22,30]不同，其中动作被添加到参考姿势： $\bm{q}^{d}_{t}=\hat{\bm{q}}_{t}+\bm{a}_{t}$ 以加速训练。由于我们的 PHC 需要对噪声和不适定的参考运动保持鲁棒性，因此我们消除了对动作空间中参考运动的依赖。我们不使用任何外力[52]或元PD控制[54]。

Our control policy ${\pi_{\text{PHC}}}$$(\bm{a}_{t}|{\bm{s}_{t}})=\mathcal{N}(\mu({\bm{s}_{t}}),\sigma)$ represents a Gaussian distribution with fixed diagonal covariance. The AMP discriminator $\bm{\mathcal{D}}(\bm{s}^{\text{p}}_{t-10:t})$ computes a real and fake value based on the current prioproception of the humanoid. All of our networks (discriminator, primitive, value function, and discriminator) are two-layer multilayer perceptrons (MLP) with dimensions [1024, 512].
我们的控制策略 ${\pi_{\text{PHC}}}$ $(\bm{a}_{t}|{\bm{s}_{t}})=\mathcal{N}(\mu({\bm{s}_{t}}),\sigma)$ 表示具有固定对角协方差的高斯分布。 AMP 判别器 $\bm{\mathcal{D}}(\bm{s}^{\text{p}}_{t-10:t})$ 根据人形机器人当前的先觉来计算真值和假值。我们所有的网络（鉴别器、原语、值函数和鉴别器）都是具有维度 [1024, 512] 的两层多层感知器 (MLP)。

Our humanoid controller can support any human kinematic structure, and we use the SMPL [21] kinematic structure following prior arts [54, 22, 23]. The SMPL body contains 24 rigid bodies, of which 23 are actuated, resulting in an action space of $\bm{a}_{t}\in\mathbb{R}^{23\times 3}$. The body proportion can vary based on a body shape parameter $\beta\in\mathbb{R}^{10}$.
我们的人形控制器可以支持任何人体运动结构，并且我们使用遵循现有技术[54,22,23]的SMPL[21]运动结构。 SMPL 主体包含 24 个刚体，其中 23 个被驱动，导致动作空间为 $\bm{a}_{t}\in\mathbb{R}^{23\times 3}$ 。身体比例可以根据身体形状参数 $\beta\in\mathbb{R}^{10}$ 而变化。

We use reference state initialization (RSI) [31] during training and randomly select a starting point for a motion clip for imitation. For early termination, we follow UHC [22] and terminate the episode when the joints are more than 0.5 meters globally on average from the reference motion. Unlike UHC, we remove the ankle and toe joints from the termination condition. As observed by RFC [52], there exists a dynamics mismatch between simulated humanoids and real humans, especially since the real human foot is multisegment [29]. Thus, it is not possible for the simulated humanoid to have the exact same foot movement as MoCap, and blindly following the reference foot movement may lead to the humanoid losing balance. Thus, we propose Relaxed Early Termination (RET), which allows the humanoid’s ankle and toes to slightly deviate from the MoCap motion to remain balanced. Notice that the humanoid still receives imitation and discriminator rewards for these body parts, which prevents these joints from moving in a nonhuman manner. We show that though this is a small detail, it is conducive to achieving a good motion imitation success rate.
我们在训练期间使用参考状态初始化（RSI）[31]，并随机选择运动剪辑的起点进行模仿。为了提前终止，我们遵循 UHC [22]，并在关节全局平均距参考运动超过 0.5 米时终止该事件。与 UHC 不同，我们将踝关节和脚趾关节从终止条件中移除。正如 RFC [52] 所观察到的，模拟人形机器人和真实人类之间存在动态不匹配，特别是因为真实的人足是多节段的 [29]。因此，模拟人形机器人不可能具有与 MoCap 完全相同的足部运动，盲目跟随参考足部运动可能会导致人形动物失去平衡。因此，我们提出了放松提前终止（RET），它允许人形机器人的脚踝和脚趾稍微偏离MoCap运动以保持平衡。请注意，类人生物仍然会收到这些身体部位的模仿和鉴别器奖励，这会阻止这些关节以非人类的方式移动。我们表明，虽然这是一个小细节，但它有利于实现良好的动作模仿成功率。

When learning from a large motion dataset, it is essential to train on harder sequences in the later stages of training to gather more informative experiences. We use a similar hard negative mining procedure as in UHC [22] and define hard sequences by whether or not our controller can successfully imitate this sequence. From a motion dataset $\bm{\hat{Q}}$, we find hard sequences $\bm{\hat{Q}}_{\text{hard}}\subseteq\bm{\hat{Q}}$ by evaluating our model over the entire dataset and choosing sequences that our policy fails to imitate.
当从大型运动数据集学习时，有必要在训练的后期阶段对更难的序列进行训练，以收集更多信息丰富的经验。我们使用与 UHC [22] 中类似的硬负挖掘程序，并根据我们的控制器是否可以成功模仿该序列来定义硬序列。从运动数据集 $\bm{\hat{Q}}$ 中，我们通过在整个数据集上评估我们的模型并选择我们的策略无法模仿的序列来找到硬序列 $\bm{\hat{Q}}_{\text{hard}}\subseteq\bm{\hat{Q}}$ 。

A PNN [36] starts with a single primitive network $\bm{\mathcal{P}}^{(1)}$ trained on the full dataset $\bm{\hat{Q}}$. Once $\bm{\mathcal{P}}^{(1)}$ is trained to convergence on the entire motion dataset $\bm{\hat{Q}}$ using the imitation task, we create a subset of hard motions by evaluating $\bm{\mathcal{P}}^{(1)}$ on $\bm{\hat{Q}}$. We define convergence as the success rate on $\bm{\hat{Q}}_{\text{hard}}^{(k)}$ no longer increases. The sequences that $\bm{\mathcal{P}}^{(1)}$ fails on is formed as $\bm{\hat{Q}}_{\text{hard}}^{(1)}$. We then freeze the parameters of $\bm{\mathcal{P}}^{(1)}$ and create a new primitive $\bm{\mathcal{P}}^{(2)}$ (randomly initialized) along with lateral connections that connect each layer of $\bm{\mathcal{P}}^{(1)}$ to $\bm{\mathcal{P}}^{(2)}$. For more information about PNN, please refer to our supplementary material. During training, we construct each $\bm{\hat{Q}}_{\text{hard}}^{(k)}$ by selecting the failed sequences from the previous step $\bm{\hat{Q}}_{\text{hard}}^{(k-1)}$, resulting in a smaller and smaller hard subset: $\bm{\hat{Q}}_{\text{hard}}^{(k)}\subseteq\bm{\hat{Q}}_{\text{hard}}^{(k-1)}$. In this way, we ensure that each newly initiated primitive $\bm{\mathcal{P}}^{(k)}$ is responsible for learning a new and harder subset of motion sequences, as can be seen in Fig.2. Notice that this is different from hard-negative mining in UHC [22], as we initialize a new primitive $\bm{\mathcal{P}}^{(k+1)}$ to train. Since the original PNN is proposed to solve completely new tasks (such as different Atari games), a lateral connection mechanism is proposed to allow later tasks to choose between reuse, modify, or discard prior experiences. However, mimicking human motion is highly correlated, where fitting to harder sequences $\bm{\hat{Q}}_{\text{hard}}^{(k)}$ can effectively draw experiences from previous motor control experiences. Thus, we also consider a variant of PNN where there are no lateral connections, but the new primitives are initialized from the weights of the prior layer. This weight sharing scheme is similar to fine-tuning on the harder motion sequences using a new primitive $\bm{\mathcal{P}}^{(k+1)}$ and preserve $\bm{\mathcal{P}}^{(k)}$’s ability to imitate learned sequences.
PNN [36] 从在完整数据集 $\bm{\hat{Q}}$ 上训练的单个原始网络 $\bm{\mathcal{P}}^{(1)}$ 开始。使用模仿任务训练 $\bm{\mathcal{P}}^{(1)}$ 在整个运动数据集 $\bm{\hat{Q}}$ 上收敛后，我们通过在 $\bm{\hat{Q}}$ 来创建硬运动子集/b5> .我们将收敛定义为 $\bm{\hat{Q}}_{\text{hard}}^{(k)}$ 的成功率不再增加。 $\bm{\mathcal{P}}^{(1)}$ 失败的序列形成为 $\bm{\hat{Q}}_{\text{hard}}^{(1)}$ 。然后，我们冻结 $\bm{\mathcal{P}}^{(1)}$ 的参数并创建一个新的原语 $\bm{\mathcal{P}}^{(2)}$ （随机初始化）以及将 $\bm{\mathcal{P}}^{(1)}$ 的每一层连接到 $\bm{\mathcal{P}}^{(2)}$ 来构建每个 $\bm{\hat{Q}}_{\text{hard}}^{(k)}$ ，从而产生越来越小的硬子集： $\bm{\hat{Q}}_{\text{hard}}^{(k)}\subseteq\bm{\hat{Q}}_{\text{hard}}^{(k-1)}$ 。通过这种方式，我们确保每个新启动的基元 $\bm{\mathcal{P}}^{(k)}$ 负责学习新的、更难的运动序列子集，如图 2 所示。请注意，这与 UHC [22] 中的硬负挖掘不同，因为我们初始化一个新的原语 $\bm{\mathcal{P}}^{(k+1)}$ 进行训练。由于最初的 PNN 是为了解决全新的任务（例如不同的 Atari 游戏）而提出的，因此提出了横向连接机制，以允许后续任务在重用、修改或丢弃先前的经验之间进行选择。然而，模仿人类运动是高度相关的，其中拟合更难的序列 $\bm{\hat{Q}}_{\text{hard}}^{(k)}$ 可以有效地从以前的运动控制经验中汲取经验。因此，我们还考虑 PNN 的一种变体，其中没有横向连接，但新基元是根据前一层的权重初始化的。 这种权重共享方案类似于使用新原语 $\bm{\mathcal{P}}^{(k+1)}$ 对较难的运动序列进行微调，并保留 $\bm{\mathcal{P}}^{(k)}$ 模仿学习序列的能力。

In addition to learning harder sequences, we also learn new tasks, such as recovering from fail-state. We define three types of fail-state: 1) fallen on the ground; 2) faraway from the reference motion ($>0.5m$); 3) their combination: fallen and faraway. In these situations, the humanoid should get up from the ground, approach the reference motion in a natural way, and resume motion imitation. For this new task, we initialize a primitive $\bm{\mathcal{P}}^{(F)}$ at the end of the primitive stack. $\bm{\mathcal{P}}^{(F)}$ shares the same input and output space as $\bm{\mathcal{P}}^{(1)}\cdots\bm{\mathcal{P}}^{(k)}$, but since the reference motion does not provide useful information about fail-state recovery (the humanoid should not attempt to imitate the reference motion when lying on the ground), we modify the state space during fail-state recovery to remove all information about the reference motion except the root. For the reference joint rotation ${\bm{\hat{\theta}}_{t}}=[\bm{\hat{\theta}}_{t}^{0},\bm{\hat{\theta}}_{t}^{1},\cdots\bm{\hat{\theta}}_{t}^{J}]$ where $\bm{\hat{\theta}}_{t}^{i}$ corresponds to the $i^{\text{th}}$ joint, we construct ${\bm{\hat{\theta}}_{t}}^{\prime}=[\bm{\hat{\theta}}_{t}^{0},\bm{{\theta}}_{t}^{1},\cdots\bm{{\theta}}_{t}^{j}]$ where all joint rotations except the root are replaced with simulated values (without $\widehat{\cdot}$). This amounts to setting the non-root joint goals to be identity when computing the goal states: $\bm{s}^{\text{g-Fail}}_{t}\triangleq(\bm{\hat{\theta}}_{t}^{\prime}\ominus\bm{\theta}_{t},\bm{\hat{p}}_{t}^{\prime}-\bm{p}_{t},\bm{\hat{v}}_{t}^{\prime}-\bm{v}_{t},\bm{\hat{\omega}}_{t}^{\prime}-\bm{\omega}_{t},\bm{\hat{\theta}}_{t}^{\prime},\bm{\hat{p}}_{t}^{\prime})$. $\bm{s}^{\text{g-Fail}}_{t}$ thus collapse from an imitation objective to a point-goal [49] objective where the only information provided is the relative position and orientation of the target root. When the reference root is too far ($>5m$), we normalize $\bm{\hat{p}}_{t}^{\prime}-\bm{p}_{t}$ as $\frac{5\times(\bm{\hat{p}}_{t}^{\prime}-\bm{p}_{t})}{\|\bm{\hat{p}}_{t}^{\prime}-\bm{p}_{t}\|_{2}}$ to clamp the goal position. Once the humanoid is close enough (e.g. $<0.5m$ ), the goal will switch back to full-motion imitation:
除了学习更难的序列之外，我们还学习新任务，例如从失败状态中恢复。我们定义了三种类型的失败状态：1）跌倒在地上； 2）远离参考运动（ $>0.5m$ ）； 3）他们的组合：堕落和遥远。在这些情况下，人形机器人应该从地面站起来，以自然的方式接近参考运动，然后恢复运动模仿。对于这个新任务，我们在基元堆栈末尾初始化一个基元 $\bm{\mathcal{P}}^{(F)}$ 。 $\bm{\mathcal{P}}^{(F)}$ 与 $\bm{\mathcal{P}}^{(1)}\cdots\bm{\mathcal{P}}^{(k)}$ 共享相同的输入和输出空间，但由于参考运动不提供有关失败状态恢复的有用信息（人形机器人不应在以下情况下尝试模仿参考运动）：躺在地上），我们在故障状态恢复期间修改状态空间，以删除除根之外的有关参考运动的所有信息。对于参考关节旋转 ${\bm{\hat{\theta}}_{t}}=[\bm{\hat{\theta}}_{t}^{0},\bm{\hat{\theta}}_{t}^{1},\cdots\bm{\hat{\theta}}_{t}^{J}]$ ，其中 $\bm{\hat{\theta}}_{t}^{i}$ 对应于 $i^{\text{th}}$ 关节，我们构造 ${\bm{\hat{\theta}}_{t}}^{\prime}=[\bm{\hat{\theta}}_{t}^{0},\bm{{\theta}}_{t}^{1},\cdots\bm{{\theta}}_{t}^{j}]$ ，其中除根之外的所有关节旋转都被替换具有模拟值（没有 $\widehat{\cdot}$ ）。这相当于在计算目标状态时将非根联合目标设置为同一： $\bm{s}^{\text{g-Fail}}_{t}\triangleq(\bm{\hat{\theta}}_{t}^{\prime}\ominus\bm{\theta}_{t},\bm{\hat{p}}_{t}^{\prime}-\bm{p}_{t},\bm{\hat{v}}_{t}^{\prime}-\bm{v}_{t},\bm{\hat{\omega}}_{t}^{\prime}-\bm{\omega}_{t},\bm{\hat{\theta}}_{t}^{\prime},\bm{\hat{p}}_{t}^{\prime})$ 。 $\bm{s}^{\text{g-Fail}}_{t}$ 因此从模仿目标崩溃为点目标[49]目标，其中提供的唯一信息是目标根的相对位置和方向。当参考根太远（ $>5m$ ）时，我们将 $\bm{\hat{p}}_{t}^{\prime}-\bm{p}_{t}$ 标准化为 $\frac{5\times(\bm{\hat{p}}_{t}^{\prime}-\bm{p}_{t})}{\|\bm{\hat{p}}_{t}^{\prime}-\bm{p}_{t}\|_{2}}$ 以限制目标位置。一旦人形物体足够接近（例如 $<0.5m$ ），目标将切换回全动作模仿：