Powerful pre-trained models have become essential in the field of computer vision, particularly in object detection (OD) tasks $[29,30]$$[29,30]$[29,30][29,30]$[29,30]$. These OD models are typically pretrained on extensive natural-image RGB datasets, such as COCO [26]. Moreover, the knowledge encoded by these models can be leveraged for various tasks in a zero-shot way or with additional fine-tuning for downstream tasks [39]. However, adding new modalities to these models, such as infrared (IR), without losing the intrinsic knowledge of the detector remains a challenge [27].
强大的预训练模型已经成为计算机视觉领域的必备工具，特别是在目标检测（OD）任务中。这些 OD 模型通常是在广泛的自然图像 RGB 数据集上预训练的，比如 COCO。此外，这些模型所编码的知识可以以零-shot 方式或通过额外的微调用于下游任务。然而，向这些模型添加新的模态，比如红外（IR），而不丢失探测器的内在知识仍然是一个挑战。

These additional modalities, though not as common as RGB images, are still important in various tasks, like surveillance [5], autonomous driving [37], and robotics [35], which strive to achieve robust performance across environmental changes such as different illumination conditions [2]. The dominant way to adapt pre-trained detectors to
这些额外的模态虽然不像 RGB 图像那样常见，但在各种任务中仍然很重要，比如监控[5]、自动驾驶[37]和机器人技术[35]，这些任务旨在实现在不同照明条件[2]等环境变化下的稳健性能。调整预训练检测器的主要方法是

Fig. 1: Bounding box predictions over different OD methods for infrared images on two benchmarks: LLVIP and FLIR. Yellow and red boxes show the ground truth and predicted detections, respectively. FastCUT is an unsupervised image translation approach that takes as input infrared images (IR) and produces pseudo-RGB images. It does not focus on detection and requires both modalities for training. Fine-tuning is the standard approach to adapting the detector to the new modality. It requires only IR data but forgets the original knowledge of the pre-trained detector. Finally, ModTr, our approach focuses the translation on detection, requires only IR data, and does not forget the original knowledge so that it can be reused for other tasks.
图 1：在 LLVIP 和 FLIR 两个基准上，不同目标检测方法对红外图像的边界框预测。黄色和红色框分别显示了地面真实和预测检测结果。FastCUT 是一种无监督图像翻译方法，以红外图像（IR）作为输入并生成伪 RGB 图像。它不专注于检测，需要两种模态进行训练。微调是将检测器适应新模态的标准方法。它只需要 IR 数据，但会忘记预训练检测器的原始知识。最后，我们的方法 ModTr 专注于检测翻译，只需要 IR 数据，并不会忘记原始知识，因此可以用于其他任务。

these novel modalities is by fine-tuning the model. However, fine-tuning often results in catastrophic forgetting and can destroy the intrinsic knowledge of the detector [22]. Ideally, we would like to adapt the detector to new modalities without changing the original model. This is most useful for server-side applications, where a single model runs uninterrupted and different inputs, ideally on different modalities, can query it. The main challenge here is the significant distribution shift between the new modalities, such as the difference between the visual information in RGB images and the thermal data in IR images. This shift can degrade the performance of models when applied directly to new input, as the features learned from one modality may not be relevant or present in another. This can ultimately impact the final performance of the detector [40].
这些新颖的模态是通过微调模型来实现的。然而，微调经常会导致灾难性遗忘，并可能破坏探测器的内在知识[22]。理想情况下，我们希望能够适应新的模态，而不改变原始模型。这对于服务器端应用非常有用，其中单个模型可以连续运行，并且可以查询不同输入，最好是来自不同的模态。这里的主要挑战是新模态之间存在显著的分布偏移，比如 RGB 图像中的视觉信息与 IR 图像中的热数据之间的差异。这种偏移会降低模型在直接应用于新输入时的性能，因为从一个模态学习到的特征可能在另一个模态中不相关或不存在。这最终可能会影响探测器的最终性能[40]。

Image translation methods have emerged as a powerful tool to overcome the downsides of fine-tuning and narrowing the gap between source and target modalities [16]. These methods do not directly work on the weight space of the original detector but rather adapt the input values to reduce the discrepancy between the source and target modalities. However, such methods often require access to source data or some statistics about it during training. Furthermore, their primary focus is on image reconstruction quality rather than the final detection task, which can cause a significant drop in performance. For instance, in Figure 1, some detections (in red) using these generated images, in b) and d), and the fine-tuning in c) are illustrated. Additionally, results for different methods are provided in supplementary material.
图像翻译方法已经成为一种强大的工具，用于克服微调和缩小源模态和目标模态之间差距的缺点。这些方法不直接作用于原始检测器的权重空间，而是调整输入值以减少源模态和目标模态之间的差异。然而，这些方法通常在训练期间需要访问源数据或一些关于源数据的统计信息。此外，它们的主要重点是图像重建质量，而不是最终的检测任务，这可能会导致性能显著下降。例如，在图 1 中，使用这些生成的图像的一些检测结果（红色）在 b)和 d)中，以及 c)中的微调被说明。此外，不同方法的结果在补充材料中提供。

Our work aims to improve the image translation paradigm while addressing its limitations. Our proposed approach, Modality Translation for OD (ModTr), incorporates the detector's knowledge into the translation module by training directly for the final de-
我们的工作旨在改进图像翻译范式，同时解决其局限性。我们提出的方法，OD 的模态翻译（ModTr），通过直接为最终的检测器知识进行训练，将检测器的知识纳入翻译模块中。

Fig. 2: Different approaches to deal with multiple modalities and/or domains. (a) The simplest approach is to use a different detector adapted to each modality. This can lead to a high level of accuracy but requires storing models in memory multiple times. (b) Our proposed solution is based on using a single pre-trained model normally trained on the more abundant data (RGB) and then adapting the input through our ModTr model. (c) A single detector is trained on all modalities jointly. This allows using of a single model but requires access to all modalities jointly, which is often not possible, especially when dealing with large pre-trained models.
图 2：处理多模态和/或领域的不同方法。(a) 最简单的方法是使用适应于每种模态的不同检测器。这可能会导致高水平的准确性，但需要多次在内存中存储模型。(b) 我们提出的解决方案是基于使用单个预训练模型，通常是在更丰富数据（RGB）上训练，然后通过我们的 ModTr 模型调整输入。(c) 一个单一的检测器同时训练所有模态。这允许使用单一模型，但需要同时访问所有模态，这通常是不可能的，特别是在处理大型预训练模型时。

tection task. Unlike traditional image translation methods, ModTr does not require any source data. It is a simple approach that can be easily integrated with any detector, be it a one-stage or two-stage detector. This approach also has applications in many settings. By incorporating new modalities for a pre-trained detector, it can act as a server that receives diverse images translated from each ModTr block trained for each modality. The detector then generates the desired output with comparable performance to fully fine-tuning but requiring much less computational power on the entire system. In Figure 2, we present several options to integrate new modalities into our system. Figure 2a illustrates the $\mathrm{N}$$\mathrm{N}$N\mathrm{N}$\mathrm{N}$-Detectors approach, where each detector is trained on a specific modality. While this method is effective, it requires a significant amount of computational resources. Alternatively, we have the N-ModTr-1-Detector approach in Figure $2\mathrm{b}$$2\mathrm{b}$2b2 \mathrm{~b}$2\mathrm{b}$, which is less resource-intensive as it involves training a specialized model for each modality. Finally, Figure 2c shows a single detector trained on joint modalities, which is not memory-intensive. However, it may not be as optimal as the other methods due to the challenges in multimodal learning. In this work, we focus on the effectiveness of our approach for IR modality, which is commonly used in surveillance and robotics, and the incremental modality detector server-based application, which are important for many settings that require non-interrupt detection predictions.
ModTr 不需要任何源数据，与传统的图像翻译方法不同。这是一种简单的方法，可以轻松集成到任何检测器中，无论是单阶段还是双阶段检测器。这种方法在许多场景中也有应用。通过为预训练检测器引入新的模态，它可以充当一个服务器，接收从每个为每种模态训练的 ModTr 块翻译的多样化图像。然后，检测器生成所需的输出，性能可与完全微调相媲美，但在整个系统上需要更少的计算资源。在图 2 中，我们提供了几种将新模态集成到我们系统中的选项。图 2a 展示了 $\mathrm{N}$$\mathrm{N}$N\mathrm{N}$\mathrm{N}$ -检测器方法，其中每个检测器都是针对特定模态进行训练的。虽然这种方法有效，但需要大量的计算资源。另外，我们在图 $2\mathrm{b}$$2\mathrm{b}$2b2 \mathrm{~b}$2\mathrm{b}$ 中有 N-ModTr-1-检测器方法，这种方法的资源消耗较少，因为它涉及为每种模态训练专门的模型。最后，图 2c 展示了一个训练在联合模态上的单个检测器，这不占用大量内存。 然而，由于多模态学习中的挑战，它可能不像其他方法那样优化。在这项工作中，我们专注于我们的 IR 模态方法的有效性，该方法通常用于监控和机器人技术，以及基于服务器的增量模态检测器应用程序，这对许多需要非中断检测预测的设置至关重要。

(1) We present ModTr, a method for adapting pre-trained ODs from large RGB datasets to new scarce modalities like IR, without requiring access to any source dataset, by translating the input signal.
我们提出了 ModTr，一种方法，可以将来自大型 RGB 数据集的预训练 ODs 适应新的稀缺模态，如 IR，而无需访问任何源数据集，通过翻译输入信号。

(2) In contrast to standard fine-tuning, our approach does not modify the original detector weights. This allows the detector to retain the knowledge of the source data while adapting to a new modality. As a result, a single model can be used to handle multiple modalities across various translators. For instance, the same model can be used to process RGB during the daytime and IR at nighttime.
(2) 与标准微调相比，我们的方法不会修改原始检测器的权重。这使得检测器能够保留源数据的知识，同时适应新的模态。因此，一个模型可以用来处理各种翻译器之间的多种模态。例如，同一个模型可以用来处理白天的 RGB 和夜晚的 IR。

(3) Our proposed approach, ModTr, is evaluated in several scenarios, showcasing its
(3) 我们提出的方法 ModTr 在几种场景中进行了评估，展示了其
advantages and flexibility. In particular, ModTr achieves competitive OD accuracy compared with image translation methods on two challenging visible/infrared datasets (LLVIP and FLIR).
优势和灵活性。特别是，在两个具有挑战性的可见/红外数据集（LLVIP 和 FLIR）上，ModTr 在 OD 准确性方面与图像翻译方法具有竞争力。

Object detection. OD is a computer vision task that has as its objective to provide labels and localization for the objects in the image [43]. The two categories of OD are twostage and one-stage detectors. Two-stage detectors, exemplified by Faster R-CNN [36], first generate regions of interest and then use a second classifier to confirm object presence within those regions. On the other hand, one-stage detectors streamline the detection process by eliminating the proposal generation stage, aiming for end-to-end training and real-time inference speeds. RetinaNet [25] is a one-stage OD model that utilizes a focal loss function to address class imbalance during training. Also, models like FCOS [38] have emerged in this category, eliminating predefined anchor boxes to potentially enhance inference efficiency. The proposed work investigates these three traditional and powerful detectors: the two-stage Faster R-CNN detector, which is highly used on the benchmarks, RetinaNet, and FCOS. The choice of such detectors was due to the simplicity in implementation and integration among other methods, as well as a different range of pre-trained backbone weights, such as ResNet [12] and MobileNet [14]. Image Translation. Image translation is a pivotal task in computer vision, aiming to map images from a source domain to a target domain while preserving inherent content [32]. The goal is to discover a transformation function such that the distribution of images in the translated domain is aligned with the distribution of images in the target domain. The commonly used approaches for image translation are based on variational autoencoders (VAEs) [21] and generative adversarial network (GANs) [10, 32]. Isola et al. developed the Pix2Pix [19], a method that consists of a generator (based on UNet) and a discriminator (based on GANs architecture) that work together to generate images based on input data and labels. Then, Zhu et al. proposed a method called CycleGAN [46], which is based on GANs, with the objective of unsupervised domain translation. Even though CycleGAN can produce quite visual results, it's hard to optimize due to the adversarial mechanism and memory footprint needed. In contrast, VAEs are easier to train than GANs but require more constraints in the optimization to produce images of good quality than GAN-based approaches. Recent advancements include diffusion models known for their high-quality image generation, although they may not inherently suit domain translation tasks. To enhance models such as CycleGAN, novel methods like Contrastive Unpaired Translation (CUT) [33] and FastCUT [33] have been introduced. CUT, in particular, accelerates the image translation process by maximizing mutual information between image patches, achieving competitive results quickly. In the context of RGB/IR modality, InfraGAN presents an image-level adaptation for RGB to IR conversion, prioritizing image quality [31]. This approach is distinct in its focus on optimizing image quality losses. Moreover, Herrmann et al. have explored OD in RGB/IR modality by adapting IR images to RGB using traditional image preprocessing techniques, allowing the use of RGB object detectors without parameter modification [13]. Recently, there have been many advances in image translation, but
目标检测。OD 是一项计算机视觉任务，其目标是为图像中的对象提供标签和定位[43]。OD 的两个类别是两阶段和一阶段检测器。两阶段检测器，如 Faster R-CNN[36]，首先生成感兴趣区域，然后使用第二个分类器来确认这些区域内是否存在对象。另一方面，一阶段检测器通过消除提议生成阶段来简化检测过程，旨在进行端到端训练和实时推理速度。RetinaNet[25]是一种一阶段 OD 模型，利用焦点损失函数来解决训练过程中的类别不平衡问题。此外，像 FCOS[38]这样的模型已经出现在这个类别中，消除了预定义的锚定框，可能提高推理效率。所提出的工作调查了这三种传统和强大的检测器：两阶段 Faster R-CNN 检测器，在基准测试中被广泛使用的 RetinaNet 和 FCOS。 这些探测器的选择是由于在实施和集成方面的简单性，以及其他方法之间的不同预训练骨干权重范围，如 ResNet [12]和 MobileNet [14]。图像翻译。图像翻译是计算机视觉中的一个关键任务，旨在将图像从源域映射到目标域，同时保留固有内容[32]。目标是发现一个转换函数，使得翻译域中的图像分布与目标域中的图像分布对齐。图像翻译的常用方法基于变分自动编码器（VAEs）[21]和生成对抗网络（GANs）[10, 32]。Isola 等人开发了 Pix2Pix [19]，这是一种基于 UNet 的生成器和基于 GANs 架构的鉴别器共同工作以根据输入数据和标签生成图像的方法。然后，Zhu 等人提出了一种名为 CycleGAN [46]的方法，它基于 GANs，旨在进行无监督域翻译。 尽管 CycleGAN 能够产生相当视觉效果，但由于需要对抗机制和内存占用量较大，优化起来很困难。相比之下，VAEs 比 GANs 更容易训练，但需要更多的约束条件来优化生成质量较好的图像，而不是基于 GAN 的方法。最近的进展包括以高质量图像生成而闻名的扩散模型，尽管它们可能并不天然适用于领域转换任务。为了增强诸如 CycleGAN 之类的模型，引入了像对比无配对翻译（CUT）[33]和 FastCUT [33]这样的新方法。特别是 CUT 通过最大化图像块之间的互信息来加速图像翻译过程，快速实现竞争性结果。在 RGB/IR 模态的背景下，InfraGAN 提出了一种 RGB 到 IR 转换的图像级适应方法，优先考虑图像质量[31]。这种方法在其专注于优化图像质量损失方面具有独特性。此外，Herrmann 等人通过使用传统图像预处理技术，将 IR 图像调整为 RGB，从而探索了 RGB/IR 模态中的目标检测，使得可以在不修改参数的情况下使用 RGB 目标检测器[13]。最近，图像翻译方面取得了许多进展，但
they do not target OD tasks. Therefore, Medeiros et al. proposed the HalluciDet [27], which uses an image translation mechanism for OD, but in their work, they have as a prior the access to the source data to pre-train the detector on the same RGB domain of the OD modality that the model needs to adapt.
他们不针对 OD 任务。因此，Medeiros 等人提出了 HalluciDet [27]，它使用图像翻译机制进行 OD，但在他们的工作中，他们将源数据作为先验条件，以在与模型需要适应的 OD 模态的相同 RGB 域上对检测器进行预训练。

Adapting without forgetting. Catastrophic forgetting (CF) is the idea that a neural network tends to forget knowledge when sequentially trained on a different task and replaces it with knowledge tailored to the new objective [41]. CF can be harmful or beneficial. Researchers identified harmful learning as situations where retaining the original knowledge while adapting to a different task is necessary. In that case, it is imperative to mitigate the risk of CF. However, some CF can also be beneficial, for instance, to prevent privacy leakage from large pre-trained models, to enhance the generalization, or to remove noisy information from the originally, acquired knowledge that is negatively affecting the new tasks. In our case, knowledge-forgetting is harmful. There are different ways to address this issue including simple techniques like decreasing the learning rate [15], use weight decay [4,45] or mixout regularization [23] during fine-tuning or more complex approaches like Recall and learn [6], Robust Information Fine-tuning [42] or CoSDA [9]. Some adaptation methods use techniques based on replay of the source data or even using the weights of the initial model to keep some prior information [28]. However, these methods may still produce a loss of knowledge since the original parameters are not frozen. Furthermore, in adapting without forgetting, an Adapter, which adopts a frozen pre-trained backbone to generate a representation followed by a different classifier for each downstream task [41], can be seen as a powerful method for keeping knowledge. Even though our ModTr shares some similarities, we work in the input space to adapt to the new modalities, and we address this incremental modality adaptation, optimizing the translation directly for the final OD task.
适应而不忘记。灾难性遗忘（CF）是指神经网络在顺序训练不同任务时往往会忘记知识，并用适合新目标的知识替代[41]。CF 可能有害或有益。研究人员确定有害学习是指在适应不同任务的同时保留原始知识是必要的情况。在这种情况下，必须减轻 CF 的风险。然而，一些 CF 也可能是有益的，例如，防止大型预训练模型泄露隐私，增强泛化能力，或者从最初获得的知识中去除负面影响新任务的嘈杂信息。在我们的情况下，遗忘知识是有害的。有不同的方法来解决这个问题，包括简单的技术，如降低学习率[15]，使用权重衰减[4,45]或在微调期间使用 mixout 正则化[23]，或者更复杂的方法，如 Recall and learn[6]，Robust Information Fine-tuning[42]或 CoSDA[9]。 一些适应方法使用基于源数据重播甚至使用初始模型的权重来保留一些先前信息[28]的技术。然而，由于原始参数未被冻结，这些方法仍可能产生知识损失。此外，在适应而不遗忘中，一个适配器采用冻结的预训练骨干生成表示，然后为每个下游任务使用不同的分类器[41]，可以被视为保留知识的强大方法。尽管我们的 ModTr 有一些相似之处，我们在输入空间中工作以适应新的模态，并解决这种增量模态适应，直接为最终 OD 任务优化翻译。

Preliminary Definitions. We denote the training set for the OD task as $\mathcal{D}=\{(x,\mathcal{Y})\}$$\mathcal{D}=\{(x,\mathcal{Y})\}$D={(x,Y)}\mathcal{D}=\{(x, \mathcal{Y})\}$\mathcal{D}=\{(x,\mathcal{Y})\}$, where $x\in {\mathbb{R}}^{W\times H\times C}$$x\in {\mathbb{R}}^{W\times H\times C}$x inR^(W xx H xx C)x \in \mathbb{R}^{W \times H \times C}$x\in {\mathbb{R}}^{W\times H\times C}$ represents an image in the dataset, with dimensions $W\times H$$W\times H$W xx HW \times H$W\times H$ and $C$$C$CC$C$ channels. Subsequently, the object detection model aims to identify $N$$N$NN$N$ regions of interest within these images, denoted as $\mathcal{Y}={\left\{(b_i,c_i)\right\}}_{i=1}^N$$\mathcal{Y}={\left\{\left(b_i,c_i\right)\right\}}_{i=1}^N$Y={(b_(i),c_(i))}_(i=1)^(N)\mathcal{Y}=\left\{\left(b_{i}, c_{i}\right)\right\}_{i=1}^{N}$\mathcal{Y}={\left\{(b_i,c_i)\right\}}_{i=1}^N$. Each region of interest $b_i$$b_i$b_(i)b_{i}$b_i$ is defined by the top-left corner coordinates and the width and height of the object. Additionally, a classification label $c_i$$c_i$c_(i)c_{i}$c_i$ is assigned to each detected object, indicating its corresponding class within the dataset. In this study, the number of input channels for the detector is fixed at three, corresponding to RGB-like inputs. In terms of optimization, the primary goal of this task is to maximize detection accuracy, often measured using the average precision (AP) metric across all classes. An OD is formally represented as the mapping $f_{\theta }:{\mathbb{R}}^{W\times H\times C}\to \mathcal{Y}$$f_{\theta }:{\mathbb{R}}^{W\times H\times C}\to \mathcal{Y}$f_(theta):R^(W xx H xx C)rarrYf_{\theta}: \mathbb{R}^{W \times H \times C} \rightarrow \mathcal{Y}$f_{\theta }:{\mathbb{R}}^{W\times H\times C}\to \mathcal{Y}$, where $\theta$$\theta$theta\theta$\theta$ denotes the parameter vector. To train such a detector effectively, a differentiable surrogate for the AP metric, referred to as the detection cost function, ${\mathcal{C}}_{\text{det}}(\theta )$${\mathcal{C}}_{\text{det }}(\theta )$C_("det ")(theta)\mathcal{C}_{\text {det }}(\theta)${\mathcal{C}}_{\text{det }}(\theta )$, is employed. The typical structure of such a cost function involves computing the average detection loss over dataset $\mathcal{D}$$\mathcal{D}$D\mathcal{D}$\mathcal{D}$, denoted as ${\mathcal{L}}_{\text{det}}$${\mathcal{L}}_{\text{det }}$L_("det ")\mathcal{L}_{\text {det }}${\mathcal{L}}_{\text{det }}$ :
初步定义。我们将 OD 任务的训练集表示为 $\mathcal{D}=\{(x,\mathcal{Y})\}$$\mathcal{D}=\{(x,\mathcal{Y})\}$D={(x,Y)}\mathcal{D}=\{(x, \mathcal{Y})\}$\mathcal{D}=\{(x,\mathcal{Y})\}$ ，其中 $x\in {\mathbb{R}}^{W\times H\times C}$$x\in {\mathbb{R}}^{W\times H\times C}$x inR^(W xx H xx C)x \in \mathbb{R}^{W \times H \times C}$x\in {\mathbb{R}}^{W\times H\times C}$ 代表数据集中的一幅图像，具有 $W\times H$$W\times H$W xx HW \times H$W\times H$ 维和 $C$$C$CC$C$ 通道。随后，目标检测模型旨在识别这些图像中的 $N$$N$NN$N$ 感兴趣区域，表示为 $\mathcal{Y}={\left\{(b_i,c_i)\right\}}_{i=1}^N$$\mathcal{Y}={\left\{\left(b_i,c_i\right)\right\}}_{i=1}^N$Y={(b_(i),c_(i))}_(i=1)^(N)\mathcal{Y}=\left\{\left(b_{i}, c_{i}\right)\right\}_{i=1}^{N}$\mathcal{Y}={\left\{(b_i,c_i)\right\}}_{i=1}^N$ 。每个感兴趣区域 $b_i$$b_i$b_(i)b_{i}$b_i$ 由对象的左上角坐标、宽度和高度定义。此外，为每个检测到的对象分配一个分类标签 $c_i$$c_i$c_(i)c_{i}$c_i$ ，表示其在数据集中对应的类别。在本研究中，检测器的输入通道数固定为三，对应于类似 RGB 的输入。在优化方面，该任务的主要目标是最大化检测准确性，通常使用所有类别的平均精度（AP）指标进行衡量。OD 被形式化表示为映射 $f_{\theta }:{\mathbb{R}}^{W\times H\times C}\to \mathcal{Y}$$f_{\theta }:{\mathbb{R}}^{W\times H\times C}\to \mathcal{Y}$f_(theta):R^(W xx H xx C)rarrYf_{\theta}: \mathbb{R}^{W \times H \times C} \rightarrow \mathcal{Y}$f_{\theta }:{\mathbb{R}}^{W\times H\times C}\to \mathcal{Y}$ ，其中 $\theta$$\theta$theta\theta$\theta$ 表示参数向量。为了有效训练这样的检测器，使用了一个可微的替代 AP 指标的检测成本函数 ${\mathcal{C}}_{\text{det}}(\theta )$${\mathcal{C}}_{\text{det }}(\theta )$C_("det ")(theta)\mathcal{C}_{\text {det }}(\theta)${\mathcal{C}}_{\text{det }}(\theta )$ 。这种成本函数的典型结构涉及计算数据集 $\mathcal{D}$$\mathcal{D}$D\mathcal{D}$\mathcal{D}$ 上的平均检测损失，表示为 ${\mathcal{L}}_{\text{det}}$${\mathcal{L}}_{\text{det }}$L_("det ")\mathcal{L}_{\text {det }}${\mathcal{L}}_{\text{det }}$ ：

Modality Translation Module (ModTr). Our approach primarily consists of an imageto-image translation network responsible for converting the input modality into an RGB-like space intelligible to the detector. These networks typically adopt an encoderdecoder structure to synthesize and reconstruct knowledge in a pixel-wise manner. While we employ U-Net as the translation network in this work, our framework is general and not limited by the translation architecture. In general terms, this mapping is denoted as $h_{\vartheta }^d:{\mathbb{R}}^{W\times H\times C}\to {\mathbb{R}}^{W\times H\times 3}$$h_{\vartheta }^d:{\mathbb{R}}^{W\times H\times C}\to {\mathbb{R}}^{W\times H\times 3}$h_(vartheta)^(d):R^(W xx H xx C)rarrR^(W xx H xx3)h_{\vartheta}^{d}: \mathbb{R}^{W \times H \times C} \rightarrow \mathbb{R}^{W \times H \times 3}$h_{\vartheta }^d:{\mathbb{R}}^{W\times H\times C}\to {\mathbb{R}}^{W\times H\times 3}$, with a translation network assigned to each available input modality $d$$d$dd$d$. Unlike the detection network, the number of input channels varies depending on the modality, for instance, $C=1$$C=1$C=1C=1$C=1$ for IR and depth images. It's important to note that, being a pixel-level architecture, the output of such a network retains the spatial resolution of the input. However, the number of output channels is consistently fixed at three, corresponding to RGB-like images $(C=3)$$(C=3)$(C=3)(C=3)$(C=3)$.
模态翻译模块（ModTr）。我们的方法主要包括一个图像到图像翻译网络，负责将输入模态转换为对检测器可理解的类似 RGB 的空间。这些网络通常采用编码器-解码器结构以像素级方式合成和重建知识。虽然我们在这项工作中采用 U-Net 作为翻译网络，但我们的框架是通用的，不受翻译架构的限制。一般来说，这种映射被表示为 $h_{\vartheta }^d:{\mathbb{R}}^{W\times H\times C}\to {\mathbb{R}}^{W\times H\times 3}$$h_{\vartheta }^d:{\mathbb{R}}^{W\times H\times C}\to {\mathbb{R}}^{W\times H\times 3}$h_(vartheta)^(d):R^(W xx H xx C)rarrR^(W xx H xx3)h_{\vartheta}^{d}: \mathbb{R}^{W \times H \times C} \rightarrow \mathbb{R}^{W \times H \times 3}$h_{\vartheta }^d:{\mathbb{R}}^{W\times H\times C}\to {\mathbb{R}}^{W\times H\times 3}$ ，每个可用输入模态分配一个翻译网络 $d$$d$dd$d$ 。与检测网络不同，输入通道的数量取决于模态，例如， $C=1$$C=1$C=1C=1$C=1$ 适用于红外和深度图像。值得注意的是，作为像素级架构，这种网络的输出保留了输入的空间分辨率。然而，输出通道的数量始终固定为三，对应于类似 RGB 的图像 $(C=3)$$(C=3)$(C=3)(C=3)$(C=3)$ 。

Unlike other image-to-image translation approaches, we drive the translation process using the aforementioned detection cost. Thus, the underlying optimization problem is formulated as ${\vartheta }^{\ast }=\arg min{\mathcal{L}}_{\text{det}}(\vartheta )$${\vartheta }^{\ast }=\arg min{\mathcal{L}}_{\text{det }}(\vartheta )$vartheta^(**)=arg minL_("det ")(vartheta)\vartheta^{*}=\arg \min \mathcal{L}_{\text {det }}(\vartheta)${\vartheta }^{\ast }=\arg min{\mathcal{L}}_{\text{det }}(\vartheta )$, incorporating the output of the composition $(f_{\theta }\circ h_{\vartheta }^d)(x)$$\left(f_{\theta }\circ h_{\vartheta }^d\right)(x)$(f_(theta)@h_(vartheta)^(d))(x)\left(f_{\theta} \circ h_{\vartheta}^{d}\right)(x)$(f_{\theta }\circ h_{\vartheta }^d)(x)$ at the loss function level. To streamline the learning process, we utilize a residual learning strategy in which the function $h_{\vartheta }^i$$h_{\vartheta }^i$h_(vartheta)^(i)h_{\vartheta}^{i}$h_{\vartheta }^i$ focuses on capturing the small variations in the input that are necessary to solve the task. This approach is similar to the one employed on diffusion models, which served as inspiration for our work. For the sake of simplicity, we separate the fusion step from the translation mapping in our notation, as various types of fusion are investigated. Consequently, the proposed image-to-image translation loss function is defined as:
与其他图像到图像翻译方法不同，我们使用上述检测成本来驱动翻译过程。因此，基本的优化问题被制定为 ${\vartheta }^{\ast }=\arg min{\mathcal{L}}_{\text{det}}(\vartheta )$${\vartheta }^{\ast }=\arg min{\mathcal{L}}_{\text{det }}(\vartheta )$vartheta^(**)=arg minL_("det ")(vartheta)\vartheta^{*}=\arg \min \mathcal{L}_{\text {det }}(\vartheta)${\vartheta }^{\ast }=\arg min{\mathcal{L}}_{\text{det }}(\vartheta )$ ，在损失函数级别结合 $(f_{\theta }\circ h_{\vartheta }^d)(x)$$\left(f_{\theta }\circ h_{\vartheta }^d\right)(x)$(f_(theta)@h_(vartheta)^(d))(x)\left(f_{\theta} \circ h_{\vartheta}^{d}\right)(x)$(f_{\theta }\circ h_{\vartheta }^d)(x)$ 的输出。为了简化学习过程，我们利用残差学习策略，其中函数 $h_{\vartheta }^i$$h_{\vartheta }^i$h_(vartheta)^(i)h_{\vartheta}^{i}$h_{\vartheta }^i$ 专注于捕捉输入中必要的小变化以解决任务。这种方法类似于扩散模型上采用的方法，这为我们的工作提供了灵感。为了简单起见，我们在符号中将融合步骤与翻译映射分开，研究了各种类型的融合。因此，提出的图像到图像翻译损失函数定义如下：

where $\mathrm{\Phi }(.,$$\mathrm{\Phi }(.,$Phi(.,\Phi(.,$\mathrm{\Phi }(.,$.$)isanon-parametricfusionfunction.Notethattheoutputof{h}_{\vartheta }^d(x)$$)isanon-parametricfusionfunction.Notethattheoutputof{h}_{\vartheta }^d(x)$)isanon-parametricfusionfunction.Notethattheoutputofh_(vartheta)^(d)(x)) is a non-parametric fusion function. Note that the output of h_{\vartheta}^{d}(x)$)isanon-parametricfusionfunction.Notethattheoutputof{h}_{\vartheta }^d(x)$ is an RGB-like image, whereas $x$$x$xx$x$ may only consist of a single channel, depending on the input modality. We have chosen this definition to simplify the notation, but appropriate reshaping should be performed during implementation to ensure compatibility.
在 $\mathrm{\Phi }(.,$$\mathrm{\Phi }(.,$Phi(.,\Phi(.,$\mathrm{\Phi }(.,$ . $)isanon-parametricfusionfunction.Notethattheoutputof{h}_{\vartheta }^d(x)$$)isanon-parametricfusionfunction.Notethattheoutputof{h}_{\vartheta }^d(x)$)isanon-parametricfusionfunction.Notethattheoutputofh_(vartheta)^(d)(x)) is a non-parametric fusion function. Note that the output of h_{\vartheta}^{d}(x)$)isanon-parametricfusionfunction.Notethattheoutputof{h}_{\vartheta }^d(x)$ 是一幅类似 RGB 的图像，而 $x$$x$xx$x$ 可能只包含一个通道，这取决于输入模态。我们选择了这个定义来简化符号表示，但在实现过程中应进行适当的重塑以确保兼容性。

In addition, note that, while a detection loss is employed to update the translation network, the weight vector $\theta$$\theta$theta\theta$\theta$ remains constant. This constraint is consistent with the premise of this study, where a pre-trained detector is solely available on the server side and remains unaltered. An overview of the proposed approach can be seen in Fig. 2b.
此外，请注意，虽然使用检测损失来更新翻译网络，但权重向量 $\theta$$\theta$theta\theta$\theta$ 保持不变。这个约束与本研究的前提一致，即预训练的检测器仅在服务器端可用且保持不变。所提出方法的概述可在图 2b 中看到。

Fusion strategy. As previously mentioned, we utilize a non-parametric fusion of the intermediate representation $h_{\vartheta }(x)$$h_{\vartheta }(x)$h_(vartheta)(x)h_{\vartheta}(x)$h_{\vartheta }(x)$ and the original input $x$$x$xx$x$ to simplify the learning process of the translation network. In this context, we investigate three straightforward nonparametric fusion functions inspired by previous works. The first approach involves element-wise summation, commonly used in ResNet and diffusion models. The second approach employs an element-wise product, also known as the Hadamard product, which is particularly interesting for attention mechanisms and has been explored previously for re-calibrating feature maps based on their importance [17]. The final approach
融合策略。如前所述，我们利用中间表示 $h_{\vartheta }(x)$$h_{\vartheta }(x)$h_(vartheta)(x)h_{\vartheta}(x)$h_{\vartheta }(x)$ 和原始输入 $x$$x$xx$x$ 的非参数融合来简化翻译网络的学习过程。在这种情况下，我们研究了受先前作品启发的三种直接的非参数融合函数。第一种方法涉及逐元素求和，常用于 ResNet 和扩散模型。第二种方法采用逐元素乘积，也称为 Hadamard 乘积，特别适用于注意机制，并且先前已经探索过根据其重要性重新校准特征图的功能。最后一种方法
is based on DenseFuse [24]; however, instead of applying it at the feature level, we apply it between the input and output of the translation network.
基于 DenseFuse [24]；然而，我们不是在特征级别应用它，而是在翻译网络的输入和输出之间应用它。
a) ${\mathrm{ModTr}}_{+}$${\mathrm{ModTr}}_{+}$ModTr_(+)\operatorname{ModTr}_{+}${\mathrm{ModTr}}_{+}$: The addition mechanism involves forwarding the input modality and summing it with the output of the translation network. This residual connection of the input serves as regularization for the model, aiding the network in learning the missing information necessary for detector operation. This operator learns the new representation by amplifying pixel values when the weights of the translation representation tend toward 1 , or preserving the original information when they tend toward 0 . Such a range of values is due to our modification on the U-Net to generate, in which we changed the last layer to a sigmoid layer so we can better control the generated image to be closer to a real RGB-like image:
a) ${\mathrm{ModTr}}_{+}$${\mathrm{ModTr}}_{+}$ModTr_(+)\operatorname{ModTr}_{+}${\mathrm{ModTr}}_{+}$ ：添加机制涉及将输入模态转发并将其与翻译网络的输出相加。输入的这种残差连接作为模型的正则化，帮助网络学习检测器操作所需的缺失信息。当翻译表示的权重趋向于 1 时，该运算符通过放大像素值来学习新的表示，或者当它们趋向于 0 时保留原始信息。这种值范围是由于我们对 U-Net 进行的修改而产生的，我们将最后一层改为 Sigmoid 层，这样我们可以更好地控制生成的图像更接近真实的 RGB 样式图像。

b) ${\mathrm{ModTr}}_{\odot }$${\mathrm{ModTr}}_{\odot }$ModTr_(o.)\operatorname{ModTr}_{\odot}${\mathrm{ModTr}}_{\odot }$ : The Hadamard product-based fusion serves as a gating mechanism to filter or highlight information from the input image. In this approach, the output of the translation network acts as a weight map for the input, and they are fused using pixel-wise multiplication, $\odot$$\odot$o.\odot$\odot$. Consequently, the translation networks tend to highlight information from the input when the pixel value tends toward 1 or discard it when it approaches 0 . Additionally, the output translation modality can be interpreted as an attention map:
b) ${\mathrm{ModTr}}_{\odot }$${\mathrm{ModTr}}_{\odot }$ModTr_(o.)\operatorname{ModTr}_{\odot}${\mathrm{ModTr}}_{\odot }$ ：基于 Hadamard 乘积的融合作为一个门控机制，用于过滤或突出输入图像中的信息。在这种方法中，翻译网络的输出充当输入的权重图，它们使用逐像素乘法进行融合， $\odot$$\odot$o.\odot$\odot$ 。因此，当像素值趋向于 1 时，翻译网络倾向于突出输入中的信息，或者当像素值接近 0 时丢弃它。此外，输出翻译模态可以被解释为一个注意力图：

c) ${\mathrm{ModTr}}_{\oplus }$${\mathrm{ModTr}}_{\oplus }$ModTr_(o+)\operatorname{ModTr}_{\oplus}${\mathrm{ModTr}}_{\oplus }$ : The subsequent fusion mechanism draws inspiration from DenseFuse [24], which employs the relative importance of pixels as an attention mechanism for both the translation network's output and input. This attention mechanism operates by providing a weighted average of the channels. The implementation details for such an operator can be found on [24]. Then, the proposed loss function is given by:
c) ${\mathrm{ModTr}}_{\oplus }$${\mathrm{ModTr}}_{\oplus }$ModTr_(o+)\operatorname{ModTr}_{\oplus}${\mathrm{ModTr}}_{\oplus }$ ：后续的融合机制灵感来自于 DenseFuse [24]，它利用像素的相对重要性作为翻译网络输出和输入的注意力机制。这种注意力机制通过提供通道的加权平均值来运作。有关这种运算符的实现细节可以在[24]中找到。然后，提出的损失函数如下：

with 用

In our design choices, we opt to utilize these straightforward non-parametric functions to assist in optimization while maintaining low inference costs.
在我们的设计选择中，我们选择利用这些直接的非参数函数来帮助优化，同时保持低推断成本。

(a) Datasets: LLVIP: The LLVIP dataset is a surveillance dataset composed of 30,976 images, in which $24,050(12,025$$24,050(12,025$24,050(12,02524,050(12,025$24,050(12,025$ IR and 12, 025 RGB paired images) are used for training and 6,926 for testing (3,463 IR and 3,463 RGB paired images) with only pedestrians annotated. FLIR ALIGNED: For the FLIR, we used the sanitized and aligned
（a）数据集：LLVIP：LLVIP 数据集是一个监控数据集，由 30,976 张图像组成，其中 $24,050(12,025$$24,050(12,025$24,050(12,02524,050(12,025$24,050(12,025$ 红外和 12,025 RGB 成对图像用于训练，6,926 用于测试（3,463 红外和 3,463 RGB 成对图像），只有行人有注释。FLIR ALIGNED：对于 FLIR，我们使用了经过清理和对齐的数据。
paired sets provided by Zhang et al. [44], which has 10, 284 images, that is 8,258 for training (4,129 IRs and 4, 129 RGBs) and 2,026 (1,013 IRs and 1,013 RGBs) for test. The FLIR images are taken from the perspective of a camera in the front of a car, and the resolution is 640 by 512 . It contains classes of bicycles, dogs, cars, and people. It has been found that for the case of FLIR, the "dog" objects are inadequate for training [3], thus we decided to remove the dogs.
张等人提供的配对数据集[44]，其中包含 10,284 张图像，其中 8,258 张用于训练（4,129 张红外图像和 4,129 张 RGB 图像），2,026 张用于测试（1,013 张红外图像和 1,013 张 RGB 图像）。FLIR 图像是从汽车前方摄像头的视角拍摄的，分辨率为 640x512。其中包含自行车、狗、汽车和人等类别。研究发现，在 FLIR 情况下，“狗”对象不足以进行训练[3]，因此我们决定移除狗。

(b) Implementation details: In our experiments, we used $80\mathrm{\%}$$80\mathrm{\%}$80%80 \%$80\mathrm{\%}$ of the training set for training and the rest for validation. All results reported are on the test set. For the FLIR dataset, we removed the dogs as there are not many annotations. As starting pre-trained weights for the detectors, we used Torchvision models with COCO [26] weights and for the U-Net translation network, we used PyTorch Segmentation Models [18] and we changed the last layer for 3-channel (RGB-like) with sigmoid, to be closer to an image with values between 0 and 1 , to perform translation instead of traditional segmentation. For the translation network backbones, we explored our default ResNet34, and for subsequent studies on reducing parameters, we dive into ResNet and MobileNet-family. All the code is available on GitHub for reproducibility in the experiments. To ensure fairness, we trained the detectors under the library version and the same experimental design, i.e., data order, augmentations, etc. Furthermore, we trained with PyTorch Lightning [8] training framework, evaluated the APs with TorchMetrics [7], and logged all experiments with WandB [1] logging tool. The different measured APs can be found in the supplementary material as additional metrics provided in this work.
（b）实施细节：在我们的实验中，我们使用 $80\mathrm{\%}$$80\mathrm{\%}$80%80 \%$80\mathrm{\%}$ 的训练集进行训练，其余用于验证。所有报告的结果都是在测试集上得出的。对于 FLIR 数据集，我们移除了狗，因为注释不多。作为检测器的初始预训练权重，我们使用了带有 COCO [26]权重的 Torchvision 模型，对于 U-Net 翻译网络，我们使用了 PyTorch 分割模型[18]，并将最后一层更改为 3 通道（类似 RGB）带有 sigmoid 的形式，以更接近值在 0 和 1 之间的图像，以执行翻译而不是传统的分割。对于翻译网络的骨干，我们探索了我们默认的 ResNet34，并在后续的参数减少研究中，我们深入研究了 ResNet 和 MobileNet 系列。所有代码都可以在 GitHub 上找到，以便在实验中进行再现。为了确保公平性，我们在相同的实验设计下训练检测器，即数据顺序，数据增强等。此外，我们使用 PyTorch Lightning [8]训练框架进行训练，使用 TorchMetrics [7]评估 APs，并使用 WandB [1]记录工具记录所有实验。 不同的测量 AP 可以在补充材料中找到，作为本文提供的额外指标。

In this section, we compare the ModTr method with different image-to-image methods employing different learning strategies. These include basic image processing strategies [13], reconstruction strategies such as CycleGAN [47], CUT [34], and FastCUT [34], which employs a contrastive learning approach, as well as HalluciDet [27], which utilizes a detection-based loss. As outlined in Table 1, we evaluated the methods based on their final detection performance across three commonly used detectors: FCOS, RetinaNet, and Faster R-CNN. The reported results are derived from the infrared test set and are averaged over three different seeds, which helps mitigate the impact of randomness across runs and splits of the training and validation datasets.
在本节中，我们将 ModTr 方法与采用不同学习策略的不同图像对图像方法进行比较。这些方法包括基本图像处理策略[13]、重建策略，如 CycleGAN[47]、CUT[34]和 FastCUT[34]，采用对比学习方法，以及 HalluciDet[27]，它利用基于检测的损失。如表 1 所示，我们根据它们在三种常用检测器（FCOS、RetinaNet 和 Faster R-CNN）上的最终检测性能评估这些方法。报告的结果来自红外测试集，并在三个不同种子上进行平均，这有助于减轻训练和验证数据集的运行和拆分中随机性的影响。

For each method, we also consider its dependency on the source data (RGB) and ground truth bounding boxes on the IR images (Box). Methods that rely on reconstruction techniques do not require box annotations on IR images but cannot provide accurate translations for detection purposes. However, HalluciDet and ModTr require box annotations to adjust the input image in a discriminative manner. The main difference between HalluciDet and ModTr is the use of source images. HalluciDet requires RGB images for an initial fine-tuning of the model, while our approach can work without that fine-tuning by reusing the detector's zero-shot knowledge.
对于每种方法，我们还考虑其对源数据（RGB）和红外图像上的地面真实边界框（Box）的依赖性。依赖重建技术的方法不需要在红外图像上进行框注释，但无法为检测目的提供准确的翻译。然而，HalluciDet 和 ModTr 需要框注释以区分性地调整输入图像。HalluciDet 和 ModTr 之间的主要区别在于对源图像的使用。HalluciDet 需要 RGB 图像对模型进行初始微调，而我们的方法可以通过重复使用检测器的零样本知识而无需进行该微调。

The proposed ModTr method demonstrates robustness across the three detectors and consistently exhibits improvement on two different datasets: LLVIP [20] and FLIR
提出的 ModTr 方法展示了对三个检测器的稳健性，并在两个不同数据集 LLVIP [20]和 FLIR 上持续展现改进
aligned [11]. It's worth noting that each algorithm described in Table 1 employs different training supervisions. For instance, CycleGAN employs an adversarial mechanism with both RGB and infrared modalities in an unpaired setting. Similarly, CUT and FastCUT operate with positive and negative patches in an unpaired setting. In contrast, HalluciDet doesn't require the presence of both modalities during training but employs a detection mechanism during training similar to ours. Note that in our approach, we solely require examples from the target modality. In this section, we present the performance of our best approach ModTr ${}_{\odot }$${}_{\odot }$_(o.)_{\odot}${}_{\odot }$. For additional results, refer to the supplementary materials.
值得注意的是，表 1 中描述的每种算法都采用不同的训练监督。例如，CycleGAN 在无配对设置中使用了对抗机制，同时使用 RGB 和红外模态。类似地，CUT 和 FastCUT 在无配对设置中使用正面和负面补丁。相比之下，HalluciDet 在训练过程中不需要同时存在两种模态，但在训练过程中采用了类似于我们的检测机制。请注意，在我们的方法中，我们仅需要目标模态的示例。在本节中，我们展示了我们最佳方法 ModTr 的性能。有关更多结果，请参考补充材料。

Table 1: Comparison of the detection performance (AP) of three different detectors (FCOS, RetinaNet, and Faster R-CNN) over different image-to-image methods to translate the infrared to RGB-like images and our proposed method (ModTr). The methods were evaluated on the infrared test set of LLVIP and FLIR datasets. The column RGB tells whether the method needs access to RGB images during training, and Box refers to the use of ground truth bounding boxes during training.
表 1：比较三种不同检测器（FCOS、RetinaNet 和 Faster R-CNN）在不同图像到图像方法上的检测性能（AP），将红外转换为类似 RGB 的图像以及我们提出的方法（ModTr）。这些方法在 LLVIP 和 FLIR 数据集的红外测试集上进行评估。RGB 列表示该方法在训练期间是否需要访问 RGB 图像，Box 表示在训练期间是否使用真实边界框。

As depicted in Table 1, the detection performance of ModTr over the LLVIP dataset exhibited significant improvements. Specifically, it surpassed HalluciDet, the second best, by more than 29.0 AP with both FCOS and RetinaNet architectures, while obtaining comparable results with Faster R-CNN. Such disparity with the previous technique can be attributed to the loss of previous knowledge inherent in HalluciDet, which necessitates a pre-fine-tuning strategy on the source modality. Although the performance of the FLIR dataset also improved, the dataset's inherent challenges, such as changing the background from a moving car setup, make detection more difficult. Nonetheless,
如表 1 所示，ModTr 在 LLVIP 数据集上的检测性能显示出显著的改进。具体而言，它在 FCOS 和 RetinaNet 架构下的性能超过了第二名的 HalluciDet 超过 29.0 AP，同时与 Faster R-CNN 获得了可比较的结果。这种与先前技术的差异可以归因于 HalluciDet 中固有的先前知识的丢失，这需要在源模态上进行预微调策略。尽管 FLIR 数据集的性能也有所提高，但数据集固有的挑战，如将背景从移动车辆设置更改，使检测变得更加困难。然而，
our proposal consistently enhances results, with improvements of more than $11\mathrm{A}\mathrm{P}$$11\mathrm{A}\mathrm{P}$11AP11 \mathrm{AP}$11\mathrm{A}\mathrm{P}$ for FCOS and RetinaNet, and over 7 AP for Faster R-CNN. We also observed improvements on the ${\mathrm{A}\mathrm{P}}_{50}$${\mathrm{A}\mathrm{P}}_{50}$AP_(50)\mathrm{AP}_{50}${\mathrm{A}\mathrm{P}}_{50}$ and ${\mathrm{A}\mathrm{P}}_{75}$${\mathrm{A}\mathrm{P}}_{75}$AP_(75)\mathrm{AP}_{75}${\mathrm{A}\mathrm{P}}_{75}$. Because of the space constraint, we include these in Supplementary Materials. These promising results indicate that our proposal can effectively translate images from the original IR modality to an RGB-like representation, sufficiently close to the source data to be usable by the detector.
我们的提案持续增强结果，FCOS 和 RetinaNet 的改进超过 $11\mathrm{A}\mathrm{P}$$11\mathrm{A}\mathrm{P}$11AP11 \mathrm{AP}$11\mathrm{A}\mathrm{P}$ ，Faster R-CNN 的 AP 超过 7。我们还观察到 ${\mathrm{A}\mathrm{P}}_{50}$${\mathrm{A}\mathrm{P}}_{50}$AP_(50)\mathrm{AP}_{50}${\mathrm{A}\mathrm{P}}_{50}$ 和 ${\mathrm{A}\mathrm{P}}_{75}$${\mathrm{A}\mathrm{P}}_{75}$AP_(75)\mathrm{AP}_{75}${\mathrm{A}\mathrm{P}}_{75}$ 的改进。由于空间限制，我们将这些内容包含在补充材料中。这些令人期待的结果表明，我们的提案能够有效地将原始 IR 模态的图像转换为类似 RGB 的表示，与源数据足够接近，可供检测器使用。

In this section, we further demonstrate that the proposed approach can be trained jointly with both translation and detector, which preserves the detector's knowledge. Here, we compare the performance of the three proposed fusion functions with that of the standard detector's fine-tuning.
在这一部分中，我们进一步证明了所提出的方法可以与翻译和检测器一起进行联合训练，从而保留了检测器的知识。在这里，我们将三种提出的融合函数的性能与标准检测器的微调进行比较。

As depicted in Tab. 2, we include the three fusion variations for ModTr and the finetuning. The table shows AP for the LLVIP and FLIR datasets, with a consistent trend across all detectors (FCOS, RetinaNet, and Faster R-CNN). Furthermore, in the case of the FLIR dataset, we observed enhancements over the standard detector fine-tuning with the various fusion techniques. As demonstrated, our approach surpasses standard fine-tuning while maintaining the detector's performance in the original modality. It is worth noting that our method also improves performance in terms of localization metrics such as ${\mathrm{A}\mathrm{P}}_{50}$${\mathrm{A}\mathrm{P}}_{50}$AP_(50)\mathrm{AP}_{50}${\mathrm{A}\mathrm{P}}_{50}$ and ${\mathrm{A}\mathrm{P}}_{75}$${\mathrm{A}\mathrm{P}}_{75}$AP_(75)\mathrm{AP}_{75}${\mathrm{A}\mathrm{P}}_{75}$ compared to fine-tuning alone, and we provide detailed results in the Supplementary Materials.
如表 2 所示，我们包括了 ModTr 的三种融合变体和微调。表格显示了 LLVIP 和 FLIR 数据集的 AP，所有检测器（FCOS、RetinaNet 和 Faster R-CNN）都表现出一致的趋势。此外，在 FLIR 数据集的情况下，我们观察到使用各种融合技术相比标准检测器微调有所增强。正如所示，我们的方法在保持检测器在原始模态下性能的同时超越了标准微调。值得注意的是，我们的方法还在定位指标（如 ${\mathrm{A}\mathrm{P}}_{50}$${\mathrm{A}\mathrm{P}}_{50}$AP_(50)\mathrm{AP}_{50}${\mathrm{A}\mathrm{P}}_{50}$ 和 ${\mathrm{A}\mathrm{P}}_{75}$${\mathrm{A}\mathrm{P}}_{75}$AP_(75)\mathrm{AP}_{75}${\mathrm{A}\mathrm{P}}_{75}$ ）方面相比单独微调有所提高，并在补充材料中提供了详细结果。

Table 2: Comparison of the fine-tuning of the detector and the different non-parametric fusion strategies for ModTr. The detection performance of three different detectors (FCOS, RetinaNet, and Faster R-CNN) was evaluated on the infrared test set of LLVIP and FLIR datasets.
表 2：检测器微调和 ModTr 的不同非参数融合策略的比较。在 LLVIP 和 FLIR 数据集的红外测试集上评估了三种不同检测器（FCOS、RetinaNet 和 Faster R-CNN）的检测性能。

In this context, we focused on evaluating the ModTr architecture and examining the trade-off between performance and parameter cost. It is widely recognized that increasing the number of parameters can enhance performance, but this relationship is not strictly linear. We demonstrated that models with fewer parameters can still achieve good performance; for example, MobileNet ${}_{v2}$${}_{v2}$_(v2){ }_{v 2}${}_{v2}$, with fewer parameters than ${\mathrm{ResNet}}_{18}$${\mathrm{ResNet}}_{18}$ResNet_(18)\operatorname{ResNet}_{18}${\mathrm{ResNet}}_{18}$, sometimes outperformed it. This trade-off highlights the versatility of the model, which can be deployed with MobileNet-based architectures and utilized in low-cost devices. In Table 3, we successfully reduced the default number of parameters from $24.4\mathrm{M}$$24.4\mathrm{M}$24.4M24.4 \mathrm{M}$24.4\mathrm{M}$ ( ${\mathrm{ResNet}}_{34}$${\mathrm{ResNet}}_{34}$ResNet_(34)\operatorname{ResNet}_{34}${\mathrm{ResNet}}_{34}$ ) to $6.6\mathrm{M}$$6.6\mathrm{M}$6.6M6.6 \mathrm{M}$6.6\mathrm{M}$ using MobileNet ${}_{v2}$${}_{v2}$_(v2){ }_{v 2}${}_{v2}$ while maintaining similar performance. For instance, on LLVIP, MobileNet ${}_{v2}$${}_{v2}$_(v2){ }_{v 2}${}_{v2}$ achieved a mean AP of 56.15 , comparable to 56.35 ${\mathrm{A}\mathrm{P}}_{50}$${\mathrm{A}\mathrm{P}}_{50}$AP_(50)\mathrm{AP}_{50}${\mathrm{A}\mathrm{P}}_{50}$ from ${\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{N}\mathrm{e}\mathrm{t}}_{34}$${\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{N}\mathrm{e}\mathrm{t}}_{34}$ResNet_(34)\mathrm{ResNet}_{34}${\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{N}\mathrm{e}\mathrm{t}}_{34}$ (others APs and detectors are reported in the supplementary material).
在这种情况下，我们专注于评估 ModTr 架构并检查性能和参数成本之间的权衡。众所周知，增加参数数量可以提高性能，但这种关系并非严格线性。我们证明了参数较少的模型仍然可以实现良好的性能；例如，MobileNet ${}_{v2}$${}_{v2}$_(v2){ }_{v 2}${}_{v2}$ ，比 ${\mathrm{ResNet}}_{18}$${\mathrm{ResNet}}_{18}$ResNet_(18)\operatorname{ResNet}_{18}${\mathrm{ResNet}}_{18}$ 参数更少，有时表现更好。这种权衡突显了模型的多功能性，可以与基于 MobileNet 的架构一起部署，并在低成本设备中使用。在表 3 中，我们成功将默认参数数量从 $24.4\mathrm{M}$$24.4\mathrm{M}$24.4M24.4 \mathrm{M}$24.4\mathrm{M}$ （ ${\mathrm{ResNet}}_{34}$${\mathrm{ResNet}}_{34}$ResNet_(34)\operatorname{ResNet}_{34}${\mathrm{ResNet}}_{34}$ ）减少到 $6.6\mathrm{M}$$6.6\mathrm{M}$6.6M6.6 \mathrm{M}$6.6\mathrm{M}$ ，同时保持类似的性能，使用 MobileNet ${}_{v2}$${}_{v2}$_(v2){ }_{v 2}${}_{v2}$ 。例如，在 LLVIP 上，MobileNet ${}_{v2}$${}_{v2}$_(v2){ }_{v 2}${}_{v2}$ 实现了 56.15 的平均 AP，与 ${\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{N}\mathrm{e}\mathrm{t}}_{34}$${\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{N}\mathrm{e}\mathrm{t}}_{34}$ResNet_(34)\mathrm{ResNet}_{34}${\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{N}\mathrm{e}\mathrm{t}}_{34}$ （其他 AP 和检测器在补充材料中报告）的 56.35 相当。

This approach opens up new possibilities, particularly in scenarios where using one translation network and one detector (e.g., one ModTr and one detector for IR/RGB) proves advantageous. This setup requires a total of $44.9\mathrm{M}$$44.9\mathrm{M}$44.9M44.9 \mathrm{M}$44.9\mathrm{M}$ parameters, compared to 83.6M parameters, when employing two detectors-one for each modality (for example, for Faster R-CNN). Similar reductions in parameter costs were observed for FCOS (from $66.4\mathrm{M}$$66.4\mathrm{M}$66.4M66.4 \mathrm{M}$66.4\mathrm{M}$ to $36.3\mathrm{M}$$36.3\mathrm{M}$36.3M36.3 \mathrm{M}$36.3\mathrm{M}$ ) and RetinaNet (from $68\mathrm{M}$$68\mathrm{M}$68M68 \mathrm{M}$68\mathrm{M}$ to $37.1\mathrm{M}$$37.1\mathrm{M}$37.1M37.1 \mathrm{M}$37.1\mathrm{M}$ ) when using one detector for both modalities while preserving the knowledge of the previous modality and incorporating a new one. These numbers are based on MobileNet ${}_{v3s}$${}_{v3s}$_(v3s){ }_{v 3 s}${}_{v3s}$, which strikes a balance between performance and the number of parameters, making it suitable for memoryrestricted systems. The complete evaluations for FCOS and RetinaNet are included in the supplementary material.
这种方法打开了新的可能性，特别是在使用一个翻译网络和一个检测器（例如，一个用于 IR/RGB 的 ModTr 和一个检测器）的情况下，这种方法被证明是有利的。与使用两个检测器（例如，对于 Faster R-CNN，每个模态一个）时的 83.6M 参数相比，这种设置需要总共 $44.9\mathrm{M}$$44.9\mathrm{M}$44.9M44.9 \mathrm{M}$44.9\mathrm{M}$ 个参数。当使用一个检测器同时保留先前模态的知识并引入一个新模态时，FCOS（从 $66.4\mathrm{M}$$66.4\mathrm{M}$66.4M66.4 \mathrm{M}$66.4\mathrm{M}$ 到 $36.3\mathrm{M}$$36.3\mathrm{M}$36.3M36.3 \mathrm{M}$36.3\mathrm{M}$ ）和 RetinaNet（从 $68\mathrm{M}$$68\mathrm{M}$68M68 \mathrm{M}$68\mathrm{M}$ 到 $37.1\mathrm{M}$$37.1\mathrm{M}$37.1M37.1 \mathrm{M}$37.1\mathrm{M}$ ）的参数成本也有类似的降低。这些数字基于 MobileNet ${}_{v3s}$${}_{v3s}$_(v3s){ }_{v 3 s}${}_{v3s}$ ，它在性能和参数数量之间取得了平衡，使其适用于内存受限系统。FCOS 和 RetinaNet 的完整评估包含在补充材料中。

In this section, we show how various adaptation paradigms, depicted in Figure 2, are capable of solving the final task while preserving intrinsic knowledge. We compare our proposed method, ModTr, to two other fine-tuning baseline methods. The first baseline method consists of $\mathrm{N}$$\mathrm{N}$N\mathrm{N}$\mathrm{N}$-detectors, each of which fine-tunes the target modality individually. The second baseline method is a single detector trained on the joint modality with balanced sampling.
在本节中，我们展示了如何通过图 2 中所示的各种适应范式来解决最终任务，同时保留内在知识。我们将我们提出的方法 ModTr 与另外两种微调基准方法进行了比较。第一种基准方法包括 $\mathrm{N}$$\mathrm{N}$N\mathrm{N}$\mathrm{N}$ -检测器，每个检测器都会单独微调目标模态。第二种基准方法是一个在联合模态上进行平衡采样训练的单个检测器。

Table 4 illustrates the final performance. While both $\mathrm{N}$$\mathrm{N}$N\mathrm{N}$\mathrm{N}$-detectors and single detectors exhibit relatively similar final performance, ModTr stands out as the only method capable of preserving knowledge while excelling at the final task. Specifically, both fine-tuning baselines compromised the detector's zero-shot capability. This consistent trend is observable across various detectors and datasets.
表 4 显示了最终性能。虽然 $\mathrm{N}$$\mathrm{N}$N\mathrm{N}$\mathrm{N}$ -检测器和单个检测器表现出相对相似的最终性能，但 ModTr 凭借其在保留知识的同时在最终任务上表现出色而脱颖而出。具体而言，微调基线都损害了检测器的零样本能力。这种一致的趋势在各种检测器和数据集中都是可观察的。

In Figure 3, we present qualitative results for LLVIP and FLIR, alongside comparison with fine-tuning. Panel a) displays the ground-truth RGB, while b) showcases the results of fine-tuning using IR. Subsequently, in c), we present ModTr with additionbased fusion, in d) ModTr with a Hadamard product-based fusion, and in e) ModTr with
在图 3 中，我们展示了 LLVIP 和 FLIR 的定性结果，并与微调进行了比较。面板 a）显示了地面真实的 RGB，而 b）展示了使用 IR 进行微调的结果。随后，在 c）中，我们展示了使用基于加法的融合的 ModTr，在 d）中是使用基于 Hadamard 乘积的融合的 ModTr，而在 e）中是使用 ModTr。

Table 3: Study on the translation backbone describing the tradeoff between the number of parameters for the translation network and the detection performance for each detector. The results comprise the three default detectors used (FCOS, RetinaNet, and Faster R-CNN) in AP under different MobileNet and ResNet backbones for the translation network.
表 3：研究翻译骨干描述翻译网络参数数量与每个检测器的检测性能之间的权衡。结果包括在不同 MobileNet 和 ResNet 骨干下使用的三个默认检测器（FCOS、RetinaNet 和 Faster R-CNN）在翻译网络中的 AP。

attention-based fusion for each dataset over the Faster R-CNN detector. Due to space constraints, we provide additional visualizations for the other detectors and competitive methods in the supplementary material. Notably, the IR exhibits some false positives, especially when there is an overlap of the detected objects, which is mitigated by our method. It's worth noting that, as described, the product-based fusion yields a darker image, contrasting with the additive characteristics of other fusion strategies. Further insights into the visualization are also provided in supplementary material, revealing how our method effectively blurs or removes objects that do not have elements of the target classes, thereby facilitating detection for the OD. Although the obtained intermediate representations are not visually pleasant, they prove more efficient for incorporating the knowledge necessary for the OD. Additionally, we conducted experiments with loss function terms aimed at enhancing the visual effects of the image, but they were not conclusive in terms of helping the detection performance.
基于注意力的融合对 Faster R-CNN 检测器上的每个数据集进行了处理。由于空间限制，我们在补充材料中为其他检测器和竞争方法提供了额外的可视化。值得注意的是，IR 在检测到的对象重叠时会出现一些误报，我们的方法可以缓解这种情况。值得注意的是，如描述的那样，基于产品的融合会产生更暗的图像，与其他融合策略的加法特性形成对比。在补充材料中还提供了对可视化的进一步见解，揭示了我们的方法如何有效地模糊或移除没有目标类别元素的对象，从而有助于 OD 的检测。虽然获得的中间表示在视觉上并不令人愉悦，但它们证明更有效地融入了 OD 所需的知识。此外，我们进行了旨在增强图像视觉效果的损失函数项的实验，但在帮助检测性能方面并不具有定论。

The main reason to use ModTr is to avoid fine-tuning the detector for a specific task so that it can preserve its knowledge and be used for multiple modalities. However, in this section, we consider what would happen if we learn jointly ModTr and the detector weights. Results are reported in Figure 4. We see that fine-tuning the detector can further boost performance. Thus, another application of ModTr could be to use it to improve the fine-tuning of a detector with a reduced additional computational cost.
使用 ModTr 的主要原因是避免为特定任务微调检测器，以便它可以保留知识并用于多种模态。然而，在本节中，我们考虑如果同时学习 ModTr 和检测器权重会发生什么。结果报告在图 4 中。我们看到微调检测器可以进一步提升性能。因此，ModTr 的另一个应用可能是使用它来改善检测器的微调，而附加计算成本较低。

Fig. 3: Illustration of a sequence of 8 images of LLVIP and FLIR dataset for Faster R-CNN. For each dataset, the first row is the RGB modality, followed by the IR modality, followed by FastCUT, and different representations created by ModTr and their variations. For visualizations for other detectors, see Suppl. Material.
图 3：LLVIP 和 FLIR 数据集的 Faster R-CNN 的 8 个图像序列示例。对于每个数据集，第一行是 RGB 模态，接着是 IR 模态，然后是 FastCUT，以及由 ModTr 及其变体创建的不同表示。有关其他检测器的可视化，请参阅补充资料。

In this work, we present ModTr, a novel method for adapting ODs without changing their parameters. The proposed approach performs well in different settings and results better than powerful image-to-image models and previous competitors. The proposed ModTr method was evaluated for different tasks, such as detection based on image
在这项工作中，我们提出了 ModTr，一种新颖的方法，用于在不改变其参数的情况下适应 ODs。所提出的方法在不同设置中表现良好，并且结果优于强大的图像到图像模型和先前的竞争对手。所提出的 ModTr 方法已针对不同任务进行评估，例如基于图像的检测。

Table 4: Average Precision (AP) comparison of knowledge preserving techniques N-Detectors, 1-Detector, and $\mathrm{N}$$\mathrm{N}$N\mathrm{N}$\mathrm{N}$-ModTr-1-Detector. The results comprise three detectors (FCOS, RetinaNet, and Faster R-CNN) across diverse datasets such as LLVIP, FLIR, and COCO.
表 4：知识保留技术 N-Detectors、1-Detector 和 $\mathrm{N}$$\mathrm{N}$N\mathrm{N}$\mathrm{N}$ -ModTr-1-Detector 的平均精度（AP）比较。结果涵盖了 FCOS、RetinaNet 和 Faster R-CNN 等三个检测器在 LLVIP、FLIR 和 COCO 等多样化数据集上的表现。