One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer
利用组件感知变压器进行单级三维全身网格复原

Whole-body mesh recovery targets to localize mesh vertices of all human components, including body, hands, and face from monocular images. Most previous works focus only on individual hand [4, 22, 10], face [2, 60, 12, 15], and body [32, 67, 61, 30, 31] reconstruction. In contrast, the joint whole-body estimation methods are less addressed. Some optimization-based works reconstruct 3D bodies by fitting the detected 2D keypoints from images with additional constraints, but they are slow and prone to local optima [49, 63]. Thanks to the whole-body parametric model (e.g., SMPL-X [49]), learning-based models [48, 56, 74, 16, 59] emerge to train networks to predict expressive body pose, shape, hand gesture, and facial expression. Due to the low resolution of hands and face, these whole-body methods crop and resize the hands and face images to higher resolutions and feed them into separate expert networks to conduct the corresponding parameter regression. Specifically, ExPose [48] introduces body-driven attention for higher-resolution crops of the face and hand estimation, a dedicated refinement module, and part-specific knowledge from existing hand-only and face-only datasets. FrankMocap [56] presents a regression-and-integration method to build a fast and accurate system. PIXIE [16] produces animatable whole body with realistic facial details via a moderator to fuse body part features adaptively. Recently, Hand4Whole [39] utilizes both body and hand joint features for accurate 3D wrist rotation and smooth connection between body and hands.
全身网格复原的目标是从单目图像中定位人体所有组成部分的网格顶点，包括身体、手和脸。之前的大多数研究都只关注单个手部[4, 22, 10]、面部[2, 60, 12, 15]和身体[32, 67, 61, 30, 31]的重建。相比之下，对全身联合估计方法的研究较少。一些基于优化的方法通过对图像中检测到的二维关键点进行拟合，再加上额外的约束条件来重建三维人体，但这些方法速度较慢，而且容易出现局部最优[49, 63]。得益于全身参数模型（如 SMPL-X [ 49]），基于学习的模型 [ 48, 56, 74, 16, 59] 应运而生，用于训练网络以预测富有表现力的身体姿势、形状、手势和面部表情。由于手部和面部的分辨率较低，这些全身方法将手部和面部图像裁剪并调整到更高分辨率，然后将其输入到单独的专家网络中，进行相应的参数回归。具体来说，ExPose [ 48] 引入了身体驱动的注意力，用于对脸部和手部进行更高分辨率的裁剪和估算，还引入了专门的细化模块，并从现有的纯手部和纯脸部数据集中引入了特定部位的知识。FrankMocap [ 56] 提出了一种回归与整合方法，以建立一个快速、准确的系统。PIXIE [ 16] 通过调节器自适应地融合身体部位特征，制作出具有逼真面部细节的全身动画。最近，Hand4Whole [ 39] 利用身体和手部关节特征实现了精确的三维手腕旋转和身体与手部之间的平滑连接。

Nevertheless, these methods aim at high performance by using separate networks in a divide-and-conquer fashion for different components and a specific fusion module to paste them together. The multi-stage pipelines lead to high complexity and inevitably cause inconsistent and unnatural articulation of the mesh and implausible 3D wrist rotations, especially in occluded, truncated, and blurry contexts. Until now, one-stage methods in this task are unexplored.
尽管如此，这些方法仍以高性能为目标，对不同组件采用分而治之的独立网络，并使用特定的融合模块将它们粘贴在一起。多阶段流水线会导致很高的复杂性，不可避免地会造成网格衔接不一致、不自然，以及三维手腕旋转不真实，尤其是在遮挡、截断和模糊的情况下。到目前为止，在这项任务中还没有探索过单级方法。

Some datasets with parametric model annotations [47, 49, 62, 6, 23, 40, 26] have been developed to advance the field. Table 1 summarizes these datasets from the annotation type, size, scene diversity, etc. To be specific, EHF [49] is the first evaluation dataset for SMPL-X-based models, which is built by capturing 3D body shapes with a scanning system and then fitting the SMPL-X model to the scans. AGORA [47] is a synthetic dataset with high realism and accurate ground truth, which is by far the most commonly used test data due to the diversity of subjects, environments, clothes, and occlusions. Notably, people in AGORA are often far from the camera, and their hands and face are obscured and have small resolutions, making existing methods focus more on body rather than hand and face estimation.
为了推动这一领域的发展，已经开发了一些带有参数模型注释的数据集[ 47, 49, 62, 6, 23, 40, 26]。表 1 从注释类型、规模、场景多样性等方面总结了这些数据集。具体来说，EHF [ 49] 是第一个基于 SMPL-X 模型的评估数据集，它是通过扫描系统捕捉三维人体形状，然后将 SMPL-X 模型与扫描结果拟合而建立的。AGORA [ 47] 是一个具有高真实度和精确地面实况的合成数据集，由于主体、环境、服装和遮挡物的多样性，它是迄今为止最常用的测试数据。值得注意的是，AGORA 中的人物通常离摄像机较远，手和脸部被遮挡且分辨率较小，这使得现有方法更侧重于身体而非手和脸部的估计。

Since marker-based 3D mocap labels are hard to obtain, there are a few annotation methods [40, 49, 16, 43, 55, 50] for high-precision labeling for both monocular indoor and outdoor scenes. FBA [55] emphasizes the severe failure cases of existing body recovery methods on consumer video data due to unusual camera viewpoints and aggressive truncations. They annotate pseudo 2D body keypoints and SMPL annotations via HMR [28] on 13k frames across four action recognition datasets. Multi-shot-AVA [50] also argues that data from edited media, like movies with rich appearances, interactions between humans, and various temporal contexts, is valuable. They apply the proposed multi-shot optimization on AVA [18] to get pseudo 3D ground truth. Interestingly, a body recovery benchmark [46] finds that simply using the 2D COCO dataset with pseudo-3D labels can surprisingly achieve a better performance and generalization ability. To complement these prior datasets and focus on expressive body recovery, we construct a new benchmark with high-quality 2D and 3D whole-body annotations.
由于基于标记的 3D mocap 标签难以获得，因此有一些标注方法[40, 49, 16, 43, 55, 50]可用于单目室内和室外场景的高精度标注。FBA [ 55] 强调了现有的人体恢复方法在消费类视频数据上的严重失效情况，原因是相机视角不寻常和截断过度。他们通过 HMR [ 28] 在四个动作识别数据集的 13k 帧上标注了伪 2D 身体关键点和 SMPL 注释。多镜头-AVA[ 50]也认为，来自经过编辑的媒体（如具有丰富外观、人与人之间的互动和各种时间背景的电影）的数据很有价值。他们在 AVA [ 18] 上应用了所提出的多镜头优化技术，以获得伪三维地面实况。有趣的是，一项人体复原基准研究[46] 发现，只需使用带有伪三维标签的二维 COCO 数据集，就能出人意料地获得更好的性能和泛化能力。为了补充这些先前的数据集，并专注于富有表现力的人体复原，我们构建了一个具有高质量二维和三维全身注释的新基准。

A one-stage framework is vital to simplify the cumbersome processes without hand-craft and complex integration designs. However, translating from multi-stage methods directly to a one-stage method is nontrivial. We take the present state-of-the-art method Hand4Whole [39] as an example to perform some preliminary studies on bringing the gap between the multi-stage method and one-stage approach. On the one hand, we replace its separate backbones with a shared backbone for all human components. On the other hand, we explore different crop-and-resize image resolutions for the hands and face, as they usually have small image resolutions.
单阶段框架对于简化繁琐的流程，避免手工制作和复杂的集成设计至关重要。然而，将多阶段方法直接转换为单阶段方法并非易事。我们以目前最先进的方法 Hand4Whole [ 39] 为例，对缩小多阶段方法和单阶段方法之间的差距进行了初步研究。一方面，我们用所有人类组件的共享主干取代了其独立的主干。另一方面，由于手部和面部的图像分辨率通常较小，我们对其进行了不同的裁剪和调整。

Table 2 shows that, when we transition from the original setup (Ori.) to a shared backbone (Share Backbone), all recovery errors are severely deteriorated on two datasets. Specifically, MPVPE increases from $183.8$mm to $202.3$mm (a 10.1% drop) on AGORA [47], and from $77.5$mm to $84.7$mm (a 9.3% drop) on EHF for all components (All). These results indicate that extracting the multi-component whole-body features with a shared backbone is difficult. Notably, the hand estimation performance deteriorates by 30.0% on EHF. Based on the results of different resolutions, we summarize some interesting observations as follows: (i) Overall, changing the resolution of the hand results in a larger performance drop than the face on EHF; (ii) When not sharing a backbone, the results are generally worse with smaller input resolutions of the hands and face.
表 2 显示，当我们从原始设置 (Ori.) 过渡到共享主干 (Share Backbone) 时，两个数据集上的所有恢复误差都严重恶化。具体来说，在 AGORA[47]上，MPVPE 从 $183.8$ mm 增加到 $202.3$ mm（下降 10.1%）；在 EHF 上，所有组件（All）的 MPVPE 从 $77.5$ mm 增加到 $84.7$ mm（下降 9.3%）。这些结果表明，提取具有共享骨干的多分量全身特征非常困难。值得注意的是，EHF 的手部估计性能下降了 30.0%。根据不同分辨率的结果，我们总结了以下一些有趣的观察结果：(i) 总体而言，在 EHF 上，改变手的分辨率比改变脸的分辨率导致的性能下降幅度更大；(ii) 在不共享主干的情况下，手和脸的输入分辨率越小，结果越差。

As an attempt to break the above status quo, we propose a one-stage framework with a vision transformer encoder and decoder for expressive full-body mesh recovery, named OSX. It is simple in design and effective in full-body mesh prediction, as we will demonstrate later. We hope it can serve as a baseline for future one-stage methods. Given a human image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$, our component-aware Transformer (CAT) estimates the corresponding body, hand, and face parameters $\hat{\mathcal{P}}=\{\hat{\mathbf{P}}_{body},\hat{\mathbf{P}}_{lhand},\hat{\mathbf{P}}_{rhand},\hat{\mathbf{P}}_{face}\}$ and then feed them into a SMPL-X layer [49] to obtain the final 3D whole-body human mesh. Specifically, $\hat{\mathbf{P}}_{body}$ contains 3D body joint rotation $\theta_{body}\in\mathbb{R}^{22\times 3}$, body shape $\beta\in\mathbb{R}^{10}$, and 3D global translation $t\in\mathbb{R}^{3}$. For $\hat{\mathbf{P}}_{lhand}$ and $\hat{\mathbf{P}}_{rhand}$, they have 3D left and right hand joint rotation $\theta_{lhand}\in\mathbb{R}^{15\times 3}$ and $\theta_{rhand}\in\mathbb{R}^{15\times 3}$, respectively. $\hat{\mathbf{P}}_{face}$ consists of 3D jaw rotation $\theta_{face}\in\mathbb{R}^{3}$ and facial expression $\phi\in\mathbb{R}^{10}$. Our training target is to minimize the distance between the recovered parameters $\hat{\mathcal{P}}$ and the ground-truth parameters $\mathcal{P}$. As shown in Figure 3, the proposed CAT consists of a component-aware encoder to capture the global correlation and extract high-quality multi-scale feature, and a component-aware decoder to strengthen the hand and face regression via an up-sampling strategy to obtain higher-resolution feature maps.
为了打破上述现状，我们提出了一个带有视觉变换器编码器和解码器的单级框架，用于进行富有表现力的全身网格恢复，命名为 OSX。正如我们稍后将演示的那样，它设计简单，在全身网格预测方面效果显著。我们希望它能成为未来单阶段方法的基线。给定人体图像 $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$ 后，我们的组件感知变换器（CAT）会估算出相应的身体、手部和面部参数 $\hat{\mathcal{P}}=\{\hat{\mathbf{P}}_{body},\hat{\mathbf{P}}_{lhand},\hat{\mathbf{P}}_{rhand},\hat{\mathbf{P}}_{face}\}$ ，然后将其输入 SMPL-X 层[49]，以获得最终的三维全身人体网格。具体来说， $\hat{\mathbf{P}}_{body}$ 包含三维身体关节旋转 $\theta_{body}\in\mathbb{R}^{22\times 3}$ 、身体形状 $\beta\in\mathbb{R}^{10}$ 和三维全局平移 $t\in\mathbb{R}^{3}$ 。对于 $\hat{\mathbf{P}}_{lhand}$ 和 $\hat{\mathbf{P}}_{rhand}$ ，它们分别包含三维左右手关节旋转 $\theta_{lhand}\in\mathbb{R}^{15\times 3}$ 和 $\theta_{rhand}\in\mathbb{R}^{15\times 3}$ 。 $\hat{\mathbf{P}}_{face}$ 包括三维下颌旋转 $\theta_{face}\in\mathbb{R}^{3}$ 和面部表情 $\phi\in\mathbb{R}^{10}$ 。我们的训练目标是使恢复的参数 $\hat{\mathcal{P}}$ 与地面实况参数 $\mathcal{P}$ 之间的距离最小。如图 3 所示，拟议的 CAT 包括一个组件感知编码器，用于捕捉全局相关性并提取高质量的多尺度特征；以及一个组件感知解码器，用于通过上采样策略加强手部和面部回归，从而获得更高分辨率的特征图。

In the component-aware encoder, the human image $\mathbf{I}$ is split into fixed-size image patches $\mathbf{P}\in\mathbb{R}^{\frac{HW}{M^{2}}\times(M^{2}\times 3)}$, where $M$ is the patch size. The patches $\mathbf{P}$ are then linearly projected by a convolution layer and added with position embeddings $\mathbf{P_{e}}\in\mathbb{R}^{\frac{HW}{M^{2}}\times C}$ to obtain a sequence of feature tokens $\mathbf{T_{f}}\in\mathbb{R}^{\frac{HW}{M^{2}}\times C}$. To explicitly leverage the body prior and learn the body information in the encoder, we concatenate the feature token $\mathbf{T_{f}}$ with the body tokens $\mathbf{T_{b}}\in\mathbb{R}^{B\times C}$, which are learnable parameters. The concatenated tokens are then fed into a standard Transformer encoder with multiple Transformer blocks [13]. Each block consists of a multi-head self-attention, a feed-forward network (FFN), and two layer normalization. After the global feature fusion, the body tokens and image feature tokens are updated into $\mathbf{T_{b}}^{\prime}\in\mathbb{R}^{B\times C}$ and $\mathbf{T_{f}}^{\prime}\in\mathbb{R}^{\frac{HW}{M^{2}}\times C}$. Finally, we use several fully connected layers to regress the body parameters $\hat{\mathbf{P}}_{body}=\{\theta_{body},\beta,t\}$ based on $\mathbf{T_{b}}^{\prime}$.
在分量感知编码器中，人类图像 $\mathbf{I}$ 被分割成固定大小的图像补丁 $\mathbf{P}\in\mathbb{R}^{\frac{HW}{M^{2}}\times(M^{2}\times 3)}$ ，其中 $M$ 是补丁大小。然后，通过卷积层对这些斑块 $\mathbf{P}$ 进行线性投影，并添加位置嵌入 $\mathbf{P_{e}}\in\mathbb{R}^{\frac{HW}{M^{2}}\times C}$ 以获得特征标记序列 $\mathbf{T_{f}}\in\mathbb{R}^{\frac{HW}{M^{2}}\times C}$ 。为了在编码器中明确利用身体先验和学习身体信息，我们将特征标记 $\mathbf{T_{f}}$ 与身体标记 $\mathbf{T_{b}}\in\mathbb{R}^{B\times C}$ 连接起来，后者是可学习的参数。然后，将并集的标记送入带有多个变换器块的标准变换器编码器[13]。每个区块由多头自注意、前馈网络（FFN）和两层归一化组成。全局特征融合后，人体标记和图像特征标记被更新为 $\mathbf{T_{b}}^{\prime}\in\mathbb{R}^{B\times C}$ 和 $\mathbf{T_{f}}^{\prime}\in\mathbb{R}^{\frac{HW}{M^{2}}\times C}$ 。最后，我们使用多个全连接层来回归基于 $\mathbf{T_{b}}^{\prime}$ 的人体参数 $\hat{\mathbf{P}}_{body}=\{\theta_{body},\beta,t\}$ 。

Our annotation pipeline produces far better 3D pseudo-GT fits with a shorter running time than the previous optimization-based and learning-based methods [26, 16, 49, 40, 50]. Figure 5(a) compares our 2D annotation results with the two wildly used annotation methods (OpenPose [8] and MediaPipe [69]). The quality of our 2D annotations is much more accurate, especially in terms of hand details and the robustness of occlusion and blur. Figure 5(b) compares the 3D annotation of ours with the SOTA NeuralAnnot method [40] on COCO. The quality of our approach is also better for the naked eye in terms of the fit of the body shape and the whole-body poses.
与之前基于优化和学习的方法[26, 16, 49, 40, 50]相比，我们的注释管道能以更短的运行时间生成更好的三维伪 GT 拟合结果。图 5(a) 比较了我们的二维注释结果和两种常用注释方法（OpenPose [ 8] 和 MediaPipe [ 69]）。我们的二维标注质量要准确得多，尤其是在手的细节以及遮挡和模糊的稳健性方面。图 5(b) 比较了我们的三维注释与 COCO 上的 SOTA NeuralAnnot 方法[40]。在体形和全身姿势的拟合方面，我们的方法的肉眼质量也更好。

Compared to the popular datasets illustrated in Figure 2 (a) to (e) and the related human-centric datasets listed in Table 1, UBody possesses unique features that present new challenges for future research. Many videos are from edited media with highly diverse scenes and rich human actions and gestures. They have abrupt shot changes and dynamic camera viewpoints, leading to discontinuities between the frames. Close-up shots of humans cause severe truncation, making existing methods tend to fail. Meanwhile, they have varying degrees of interaction with objects and body components, subtitles, and special effects as occluded scenes. Also, there are high variations in background and light. Those conditions have not appeared in previous datasets. All scenes in UBody have rich hand gestures and facial expressions, making the recognition models pay more attention to these important body components. Lastly, all of these real-life videos provide audio as additional information to serve future multi-modality methods. We also provide statistical comparisons between the key features of UBody and the wildly used dataset AGORA [47] in Figure 6. AGORA’s hand/face bounding box area is generally small, while UBody pays more attention to diverse hand and face scales as evidenced by its more dispersed area distribution. Meanwhile, UBody has more visible face/hand keypoints, underscoring the importance of recognizing hand gestures and facial expressions. Lastly, UBody’s inclusion of real-life videos provides new possibilities for subsequent spatio-temporal modeling that are not available in AGORA, which is an image-based dataset.
与图 2 (a) 至 (e) 所示的流行数据集和表 1 所列的以人为中心的相关数据集相比，UBody 具有独特的特征，为未来研究带来了新的挑战。许多视频来自编辑过的媒体，具有高度多样化的场景和丰富的人类动作和姿态。它们有突然的镜头变化和动态的摄像机视角，导致帧与帧之间的不连续性。人类的特写镜头会造成严重的截断，使现有方法容易失效。同时，它们与物体和身体部件、字幕以及作为遮挡场景的特效之间存在不同程度的交互。此外，背景和光线的变化也很大。这些情况在以前的数据集中都没有出现过。UBody 中的所有场景都有丰富的手势和面部表情，这使得识别模型更加关注这些重要的身体组件。最后，所有这些真实视频都提供了音频作为附加信息，以服务于未来的多模态方法。我们还在图 6 中提供了 UBody 的关键特征与广泛使用的数据集 AGORA [ 47] 之间的统计比较。AGORA 的手部/脸部边界框面积普遍较小，而 UBody 则更加关注不同的手部和脸部尺度，其面积分布更加分散就是明证。同时，UBody 有更多可见的面部/手部关键点，强调了识别手势和面部表情的重要性。最后，UBody 包含的真实生活视频为后续的时空建模提供了新的可能性，这是 AGORA（基于图像的数据集）所不具备的。

Due to the page limit, we leave the detailed experiment setup, implementation, annotation visualization, qualitative comparison with SOTA methods, and more benchmark results and analyses in the appendix.
由于篇幅有限，我们将详细的实验设置、实施、注释可视化、与 SOTA 方法的定性比较以及更多基准结果和分析放在附录中。

Table 3 provides a comprehensive comparison of OSX and existing whole-body mesh recovery methods. As the first one-stage method, OSX surpasses existing multi-stage models with complex designs in most cases. Notably, OSX has not been trained on hand-only and face-only datasets [76, 29, 41]. Our All MPVPEs show a $9.5$% improvement on AGORA test set and $7.8$% improvement on EHF than SOTA [39]. Since AGORA is a more complex and natural dataset than EHF, previous works [39, 47] claim it is more convincing and representative of real-world scenarios. We also visualize the misleading high-error cases on EHF in Figure 7. Besides, we obtain a SOTA performance on the body-only dataset, 3DPW, with a $13.4$% error reduction compared to these whole-body methods. More qualitative results are available in the appendix.
表 3 对 OSX 和现有的全身网格复原方法进行了全面比较。作为第一种单阶段方法，OSX 在大多数情况下都超越了现有的复杂设计的多阶段模型。值得注意的是，OSX 还没有在纯手和纯脸数据集上进行过训练[76, 29, 41]。与 SOTA 相比，我们的所有 MPVPE 在 AGORA 测试集上的改进幅度为 0%，在 EHF 上的改进幅度为 1%[39]。由于 AGORA 是一个比 EHF 更复杂、更自然的数据集，因此之前的研究[39, 47]认为它更有说服力，更能代表真实世界的场景。我们还在图 7 中展示了 EHF 上误导性的高错误案例。此外，我们还在纯身体数据集 3DPW 上获得了 SOTA 性能，与这些全身方法相比，误差减少了 @2 %。更多定性结果见附录。

As a new dataset, we provide both quantitative and qualitative results on UBody. Table 5 presents the performance comparisons of existing 3D whole-body methods. The general result ranking is similar to AGORA. Since the upper body is closer to the camera, their errors will be smaller than AGORA. However, the hand and face will play a more important role than previous data. Besides, we finetune Hand4Whole on AGORA and test again, and we find all errors are significantly enlarged. This observation can be attributed to the data distribution gap between AGORA and UBody, as shown in Figure 6. Moreover, we train OSX on our train set and find a 16.1% improvement compared to the original pretrained model, indicating that UBody can serve to improve the performance on downstream real-life scenes.
作为一个新的数据集，我们提供了 UBody 的定量和定性结果。表 5 列出了现有三维全身方法的性能比较。总体结果排名与 AGORA 相似。由于上半身离摄像机更近，其误差会小于 AGORA。但是，手部和脸部的作用会比之前的数据更重要。此外，我们在 AGORA 上对 Hand4Whole 进行了微调，并再次进行测试，结果发现所有误差都明显增大。如图 6 所示，这一现象可以归因于 AGORA 和 UBody 在数据分布上的差距。此外，我们在训练集上训练 OSX，发现与原始预训练模型相比，OSX 提高了 16.1%，这表明 UBody 可以提高下游真实场景的性能。

To bridge the gap between the basic human mesh recovery task and its downstream applications, we design UBody with two rules. First, we research a wide range of human-related downstream tasks with upper-body scenes, including gesture recognition [65, 19, 38], sign language recognition, and translation [14, 54, 58, 27, 7, 73], person clustering [5], emotion analysis, speaker verification [44], micro-gesture understanding [36], audio-visual generation and separation [53], human action recognition, and localization [50, 18, 55, 57, 17], and human video segmentation [33]. We select the corresponding high-quality datasets from these existing tasks as a part of our data for the corresponding scenarios. In order to ensure a balanced amount of data for each scene, for datasets with many videos (e.g., lasting 20k minutes), we manually selected the videos in which the upper body appeared more frequently.
为了缩小基本人体网格复原任务与其下游应用之间的差距，我们在设计 UBody 时遵循了两条原则。首先，我们研究了大量与人体上半身场景相关的下游任务，包括手势识别[65, 19, 38]、手语识别和翻译[14, 54, 58, 27, 7, 73]、人物聚类[5]、情感分析、说话者验证[44]、微手势理解[36]、视听生成和分离[53]、人体动作识别和定位[50, 18, 55, 57, 17]以及人体视频分割[33]。我们从这些现有任务中选择相应的高质量数据集作为相应场景的数据。为了确保每个场景的数据量均衡，对于视频数量较多（如持续 20k 分钟）的数据集，我们手动选择了上半身出现频率较高的视频。

Second, with all kinds of athletic competitions, entertainment shows, we media, online conferences, and online classes being more and more indispensable, we carefully selected a large number of rich videos from YouTube to provide new opportunities and challenges for potential applications.
其次，随着各种体育比赛、娱乐节目、微媒体、在线会议和在线课堂越来越不可或缺，我们从 YouTube 上精心挑选了大量丰富的视频，为潜在应用提供了新的机遇和挑战。

Since some untrimmed videos may have missing main characters, extraneous images such as opening and closing credits, and repetitive actions, we manually fine-cut the long videos. Each edited video is 10 seconds long, which ensures the high quality of the video.
由于一些未经剪辑的视频可能会出现主角缺失、片头片尾等无关图像以及重复动作等问题，因此我们对长视频进行了人工精剪。每个经过剪辑的视频时长为 10 秒，从而确保了视频的高质量。

In order to prevent infringement of ownership rights, we only provide download links to the corresponding videos and our labels without any personal information.
为了防止侵犯所有权，我们只提供相应视频的下载链接和我们的标签，不提供任何个人信息。

In summary, we collect fifteen real-life scenarios with more than 105,1k frames. We split the train/test sets from two protocols as follows.
总之，我们收集了 15 个真实场景，帧数超过 105,1k 帧。我们将两个协议的训练/测试集拆分如下。

• •

  Intra-scene: in each scene, the former 70% of the videos are the training set, and the last 30% are the test set. The benchmark was provided in the main paper.

  
  - 场景内：在每个场景中，前 70% 的视频是训练集，后 30% 的视频是测试集。主要论文中提供了基准。
• •

  Inter-scene: we use ten scenes of the videos as the training set and the other five scenes as the test set. Due to the page limit, we present the benchmark in Table S-4.

  
  - 场景间：我们使用视频中的 10 个场景作为训练集，另外 5 个场景作为测试集。由于篇幅限制，我们在表 S-4 中列出了基准值。

As shown in Figure S-1, we design a thorough whole-body annotation pipeline with high precision. It is divided into two stages: 2D whole-body keypoint annotation and 3D SMPLX annotations fitting. Since UBody scenes have a number of unpredictable transitions and cutscenes that make it difficult to use the temporal smoothing approaches [66, 52, 68], the annotation is conducted on a single frame.
如图 S-1 所示，我们设计了一个全面、高精度的全身标注流水线。它分为两个阶段：二维全身关键点标注和三维 SMPLX 标注拟合。由于 UBody 场景有许多不可预知的转场和剪辑，很难使用时间平滑方法[66, 52, 68]，因此注释是在单帧上进行的。