Abstract 摘要
This study summarizes current trends and future directions in projection mapping technologies. Projection mapping seamlessly merges the virtual and real worlds through projected imagery onto physical surfaces, creating an augmented reality environment. Beyond traditional applications in advertising, art, and entertainment, various fields, including medical surgery, product design, and telecommunications, have embraced projection mapping. This study categorizes recent techniques that address technical challenges in accurately replicating desired appearances on physical surfaces through projected imagery into four groups: geometric registration, radiometric compensation, defocus compensation, and shadow removal. It subsequently introduces unconventional projectors developed to resolve specific technical issues and discusses two approaches for overcoming the inherent limitations of projector hardware, such as the inability to display images floating above physical surfaces. Finally, this study concludes the discussion with possible future directions for projection mapping technologies.
本研究总结了投影映射技术当前趋势和未来发展方向。投影映射通过将投影图像投射到物理表面上，无缝融合虚拟和现实世界，创造一个增强现实环境。除了在广告、艺术和娱乐等传统应用之外，包括医疗手术、产品设计、电信等各个领域都采用了投影映射。本研究将解决通过投影图像在物理表面上准确复制所需外观的技术挑战的最新技术分为四类：几何配准、辐射补偿、失焦补偿和阴影消除。随后，介绍了为解决特定技术问题而开发的非传统投影仪，并讨论了两种克服投影仪硬件固有局限性的方法，例如无法显示漂浮在物理表面上的图像。最后，本研究总结了投影映射技术可能的未来发展方向。
Keywords: projection mapping, augmented reality, projector-camera systems
关键词：投影映射、增强现实、投影相机系统
1. Introduction 1. 简介
Projection mapping (PM) overlays computer-generated imagery onto physical surfaces using projectors, creating an augmented reality (AR) environment where the virtual and real worlds seamlessly merge. These surfaces encompass not only flat and uniform white screens but also general, non-planar, textured surfaces in our surroundings. The PM-based AR provides additional information, such as annotations directly on the target physical surface.1) Furthermore, it can visually alter the material of the physical surface, thus transforming a plaster statue into metallic, transparent, or furry material.2) The PM-based AR is often referred to as spatial AR,3) offering several advantages over other AR display technologies, such as video and optical see-through displays. For instance, PM does not require users to wear or hold displays, such as head-mounted displays or smartphones, thus not restricting the user’s field of view (FOV). Furthermore, it enables multiple users to simultaneously share in-situ AR experiences.
投影映射（PM）通过投影仪将计算机生成的图像叠加到物理表面上，创建一个虚拟与现实世界无缝融合的增强现实（AR）环境。这些表面不仅包括平坦均匀的白色屏幕，还包括我们周围的一般非平面、纹理表面。基于 PM 的 AR 提供额外的信息，如直接在目标物理表面上的注释。此外，它还可以从视觉上改变物理表面的材料，将石膏雕像变成金属、透明或毛茸茸的材料。基于 PM 的 AR 通常被称为空间 AR，提供比其他 AR 显示技术（如视频和光学透视显示）更多的优势。例如，PM 不需要用户佩戴或手持显示设备，如头戴式显示器或智能手机，因此不会限制用户的视野（FOV）。此外，它还使多个用户能够同时共享现场 AR 体验。
Thanks to these advantages, various application fields have been explored beyond typical ones, such as advertising, art, and entertainment.4–6) For example, PM is used to navigate users to target locations by superimposing arrows onto the physical environment.7) Similarly, PM is useful in supporting object searches in physical space by highlighting the searched object.8–12) The highlighting technique is also beneficial in medical surgery, where an invisible emission signal indicating the resection area in a human organ is visualized by projected imagery.13) It can also be used for artwork creation in such a way that projected patterns indicate where to paint on a canvas or where to dig in clay.14–16) Rich graphical information, such as a navigator’s avatar, is projected onto the artwork or its surrounding surfaces for museum guides.17,18) Projected avatars of distant people are also used for tele-communication.19–21) Similarly, human body silhouette projection extends the reaching distance of our body.22–24) The PM on a human face supports makeup.25,26) Apparent material transformation is useful in product design.27–30)
得益于这些优势，各种应用领域已经超越了典型的领域，例如广告、艺术和娱乐。例如，PM 通过在物理环境中叠加箭头来引导用户到目标位置。同样，PM 在支持物理空间中的物体搜索时很有用，通过突出显示搜索到的物体。这种突出显示技术在医疗手术中也很有益，通过投影图像将人体器官中的切除区域的无形发射信号可视化。它还可以用于艺术作品的创作，以这种方式，投影图案指示在画布上绘画或在哪里挖粘土。丰富的图形信息，如导航员的头像，被投影到艺术品或其周围表面上，用于博物馆导游。远程通信中也使用了遥远的人的投影头像。类似地，人体轮廓投影扩展了我们身体的触及距离。人脸上的 PM 支持化妆。明显的材料转换在产品设计中有用。
More conceptually, PM can make any real-world surface, such as a table, wall, and even our skin, visually programmable. When combined with proper sensing technologies, these surfaces become responsive to user actions. Artificial intelligence (AI) technologies further enhance their interactive capability. This enables a paradigm shift in human-computer interaction from inorganic, tangible-based input and output schemas, requiring typical mice, keyboards, touch panels, and 2D monitors, to organic ones. In this paradigm, real-world objects, including our bodies, are covered by so-called smart skins through which we interact with AI in an intimate way. More specifically, we might perceive that AI is symbiotically embedded in our body or exists anywhere surrounding us, which could fundamentally change our perceptual and cognitive model of AI. Such an ultimate application would be achieved when PM becomes ubiquitous, and typical room lights are substituted with projectors.
更概念化地说，PM 可以使任何现实世界的表面，如桌子、墙壁，甚至我们的皮肤，变得视觉可编程。当与适当的传感技术结合时，这些表面对用户动作做出响应。人工智能（AI）技术进一步增强了它们的交互能力。这使得人机交互从无机、基于物理的输入和输出模式，需要典型的鼠标、键盘、触摸屏和二维显示器，转变为有机模式。在这个范式下，现实世界的物体，包括我们的身体，被所谓的智能皮肤所覆盖，通过这些皮肤我们以亲密的方式与 AI 互动。更具体地说，我们可能会感觉到 AI 与我们的身体共生地嵌入，或者存在于我们周围的任何地方，这可能会从根本上改变我们对 AI 的感知和认知模型。当 PM 变得无处不在，典型的室内灯光被投影仪取代时，这种终极应用就会实现。
Technically speaking, these applications work properly only when we can accurately replicate desired appearances on real-world surfaces through projected imagery. However, this is not straightforward, as our projection targets are unconstrained and arbitrary surfaces (e.g., non-planar and textured), often unsuitable for projection. Without careful considerations, the projected image gets severely degraded.
从技术上讲，这些应用程序只有在我们可以通过投影图像准确复制所需外观在现实世界表面时才能正常工作。然而，这并不简单，因为我们的投影目标是未受限制和任意的表面（例如，非平面和有纹理的），通常不适合投影。如果不加仔细考虑，投影图像会严重退化。
First, when a target surface is non-planar, the projected image gets deformed. Thus, geometric registration of a projector to the surface is required to allow a PM system to determine which projector pixel illuminates which surface point. Second, even when pixel correspondence is established, achieving the desired color reproduction on the surface, especially when it is textured, is rarely straightforward. Thus, radiometric compensation is necessary to correct distorted colors. In cases where the surface is non-planar, parts of projected result are defocused and, consequently, appear blurred. Defocus blur should be compensated; otherwise, it significantly reduces high spatial frequency components (or details) of a projected image. Finally, shadows, which occur when a user occludes projected light, significantly reduce the sense of immersion in the AR experience. Therefore, shadow removal is also an important technical issue in PM.
首先，当目标表面非平面时，投影图像会变形。因此，需要对投影仪进行几何配准，以便 PM 系统确定哪个投影仪像素照亮了哪个表面点。其次，即使建立了像素对应关系，在表面上实现所需的颜色再现，尤其是在表面有纹理的情况下，通常并不简单。因此，需要辐射补偿来校正失真的颜色。在表面非平面的情况下，投影结果的部分会失焦，因此看起来模糊。应该补偿失焦模糊；否则，它将显著降低投影图像的高空间频率成分（或细节）。最后，当用户遮挡投影光线时产生的阴影，会显著降低 AR 体验的沉浸感。因此，阴影消除也是 PM 中的一个重要技术问题。
Over the past 25 years, researchers have dedicated their efforts to addressing these technical challenges. They mathematically model the image degradation processes and solve their inverse problems to generate compensation images, allowing them to reproduce desired appearances on target surfaces through projection. However, owing to the technical limitations of projector hardware, such as a limited dynamic range (displayable luminance range) and shallow depth-of-field, compensation images are not always readily displayable. Researchers have successfully overcome these limitations beyond the capabilities of the original projector hardware. Recent trends to achieve this include combining near-eye optics in PM and applying perceptual tricks.
过去 25 年中，研究人员致力于解决这些技术挑战。他们通过数学模型模拟图像退化过程，并解决其逆问题以生成补偿图像，从而通过投影在目标表面上重现所需的视觉效果。然而，由于投影仪硬件的技术限制，如有限的动态范围（可显示亮度范围）和浅景深，补偿图像并不总是容易显示。研究人员已经成功克服了这些超越原始投影仪硬件能力的限制。实现这一目标的最新趋势包括在 PM 中结合近眼光学和运用感知技巧。
This review briefly introduces technical solutions for each challenge and discusses future research directions in PM technologies. Notably, comprehensive surveys on this topic were previously published in 200831) and 2018.32) Therefore, this review focuses on summarizing recent works, specifically those not discussed in the aforementioned literature. In addition, this review concentrates on the technological aspects of PM and does not provide an overview of the trends in recent PM applications.
本综述简要介绍了每个挑战的技术解决方案，并讨论了项目管理技术未来的研究方向。值得注意的是，关于这个主题的全面调查分别在 2008 年和 2018 年发表。因此，本综述侧重于总结近期作品，特别是那些在上文文献中没有讨论的作品。此外，本综述专注于 PM 的技术方面，并不提供近期 PM 应用趋势的概述。
2. Geometric registration
2. 几何配准
The PM system needs to determine which projector pixel incidents on which surface point to display a desired appearance on the surface. Conventional keystone correction addresses this issue only when the surface is flat. However, non-planar surfaces are frequently used in PM, thereby making it necessary to establish proper geometric registration of a projector with the target surface.
投影仪系统需要确定哪个投影仪像素映射到哪个表面点以在表面上显示所需的外观。传统的梯形校正仅当表面平坦时才解决这个问题。然而，在 PM 中，非平面表面经常被使用，因此有必要建立投影仪与目标表面的适当几何注册。
2.1. Projector calibration.
2.1. 投影仪校准。
The geometric relationship between two-dimensional (2D) coordinate value of a projector pixel (x, y) and three-dimensional (3D) coordinate value of corresponding surface point (X, Y, Z) is mathematically described using a pinhole camera model as [x, y, 1]t = PM[X, Y, Z, 1]t, where P and M are 3 × 4 and 4 × 4 matrices, respectively. This model comprises the projector’s intrinsic parameters such as its focal length in P, and the extrinsic parameters that determine the pose of the projector relative to the target surface in M.
二维（2D）投影器像素坐标值（x，y）与对应表面点三维（3D）坐标值（X，Y，Z）之间的几何关系，使用针孔相机模型从数学上描述为[x，y，1] t = PM[X，Y，Z，1] t ，其中 P 和 M 分别是 3×4 和 4×4 矩阵。此模型包括投影器的内在参数，如 P 中的焦距，以及 M 中确定投影器相对于目标表面姿态的外在参数。
Efficient and accurate calibration of these parameters has been successfully established for static PM setups based on a camera calibration framework.33) The standard method involves using a camera to capture projected calibration patterns through which the parameters are then estimated. Recent technologies have also used projector-camera (ProCam) systems for unique setups. For example, Xie et al. introduced a user-friendly calibration technique, allowing a user to use a handheld mobile phone camera in the calibration process.34) Sugimoto et al. proposed an attachment-type system for calibrating intrinsic parameters of a projector in limited space.35) Another group attempted to achieve precise calibration by incorporating a LiDAR (light detection and ranging) sensor with a ProCam system.36) The calibration of multiple projectors using various camera setups, including multiple cameras and a depth camera, is also currently an active area of research.37–39)
高效且精确地校准这些参数已成功建立，适用于基于相机校准框架的静态 PM 设置。 33) 标准方法涉及使用相机捕获投影校准图案，然后通过这些图案估计参数。最近的技术也使用了投影仪-相机（ProCam）系统进行独特设置。例如，谢等人介绍了一种用户友好的校准技术，允许用户在校准过程中使用手持移动电话相机。 34) 菅本等人提出了一种附件式系统，用于在有限空间内校准投影仪的内参。 35) 另一组尝试通过结合 LiDAR（光探测与测距）传感器和 ProCam 系统来实现精确校准。 36) 使用各种相机设置（包括多个相机和深度相机）对多个投影仪进行校准也是当前研究的一个活跃领域。 37–39)
2.2. Dynamic projection mapping.
2.2. 动态投影映射。
The trend in PM research has undergone a dramatic shift from static PM to dynamic PM (DPM) in recent years. In DPM, aligning the projected image with the surface of a moving object is imperative. As a projector’s intrinsic parameters remain constant as long as the lens settings are unchanged, they can be pre-calibrated. Therefore, the primary challenge in achieving DPM is the rapid estimation of extrinsic parameters, specifically, fast tracking of the moving surface. Researchers have tackled this challenge by capturing distinctive visual markers attached to surfaces. The 3D positions of these markers on the surfaces were predetermined. Subsequently, extrinsic parameters could be computed by establishing a relationship between the 3D positions of the markers and their corresponding 2D positions in the captured images.
近年来，项目管理（PM）研究趋势经历了从静态 PM 到动态 PM（DPM）的巨大转变。在 DPM 中，将投影图像与移动物体的表面对齐至关重要。由于投影仪的内在参数在镜头设置不变的情况下保持不变，因此它们可以预先校准。因此，实现 DPM 的主要挑战是快速估计外参数，特别是快速跟踪移动表面。研究人员通过捕捉附着在表面上的独特视觉标记来应对这一挑战。这些标记在表面上的 3D 位置是预先确定的。随后，可以通过建立标记的 3D 位置和它们在捕获图像中相应 2D 位置之间的关系来计算外参数。
Interestingly, multiple research groups have focused on aligning projected images onto deformable surfaces such as cloth by tracking dot array markers captured by an RGB camera.40–43) Typical projectors inevitably introduce noticeable delays in projected images onto a target surface in DPM, even when fast marker tracking is available. Maeda and Koike addressed this problem using deep neural networks (DNNs) for object pose prediction.44) Another issue in marker-based DPM is the visibility of markers. Visible markers under projection significantly reduce a user’s immersion in DPM experience. Notably, this is also a critical concern in low-latency DPM (see Sec. 2.3). Researchers have attempted to reduce the visibility of markers using imperceptible materials (e.g., infrared (IR) ink only detectable using an IR camera)45) or IR LEDs46) as markers (Fig. 1) and have further reduced visibility by projecting complementary colors onto the marker area.47) Edible markers have been developed for projecting images onto foods, and these are also designed to reduce their visibility using transparent materials48) or by embedding markers in internal structures of foods.49)
有趣的是，多个研究小组已经专注于通过跟踪由 RGB 相机捕获的点阵标记，将投影图像对齐到可变形表面，如布料。在 DPM 中，典型的投影仪不可避免地会在将投影图像投射到目标表面时引入明显的延迟，即使有快速标记跟踪。Maeda 和 Koike 使用深度神经网络（DNN）进行物体姿态预测来解决此问题。在基于标记的 DPM 中，另一个问题是标记的可见性。在投影下可见的标记会显著减少用户在 DPM 体验中的沉浸感。值得注意的是，这也是低延迟 DPM 中的一个关键问题（见第 2.3 节）。研究人员已经尝试使用不可察觉的材料（例如，仅使用红外（IR）相机可检测的红外墨水）或 IR LED 作为标记（图 1），并通过在标记区域投射互补颜色来进一步减少可见性。为了在食物上投影图像，已经开发了可食用标记，并且这些标记也设计为使用透明材料或通过将标记嵌入食物的内部结构来减少其可见性。
Figure 1. 图 1
(Color online) Invisible markers for the geometric registration in DPM.46) (a) Projection target. (b) Multiple holes on the bottom of projection target, into which IR LEDs are inserted. (c) Internal structure of the projection target, created using a multi-material 3D printer with embedded optical fibers. The IR light from the LEDs is routed to the surface by the fibers. (d) Captured IR image of the target surface. (e, f) DPM results. (IEEE Trans. Vis. Comput. Graph. 2020, 26, 2030–2040)
(在线彩色) DPM 中几何配准的无形标记。 46) (a) 投影目标。 (b) 投影目标底部多个孔，其中插入红外 LED。 (c) 投影目标的内部结构，使用带有嵌入式光纤的多材料 3D 打印机创建。LED 的红外光通过光纤导向表面。 (d) 目标表面的捕获红外图像。 (e, f) DPM 结果。（IEEE Trans. Vis. Comput. Graph. 2020，26，2030-2040）
Marker-less tracking has also been explored. For example, the pose of a rigid target object was robustly tracked based on silhouette information, even in the presence of occlusions caused by the user’s hands, using multiple cameras.50) However, the marker-less approach is generally error-prone. In cases where multiple projectors are used in DPM, tracking errors can result in noticeable misalignments of projected images from different projectors. To address this challenge, Kurth et al. proposed a scalable online solution for their depth camera-based marker-less DPM. Their approach optimizes overlapped projection images by reducing the pixel values from projectors other than the one projecting the finest and brightest pixels, particularly in areas with discontinuities in the depth of the surface point from the projector and in the color of the projected image.51)
无标记跟踪也得到了探索。例如，基于轮廓信息，即使在使用用户手部遮挡的情况下，也能稳健地跟踪刚性目标对象的姿态，使用多个相机。然而，无标记方法通常容易出错。在 DPM 中使用多个投影仪的情况下，跟踪错误可能导致来自不同投影仪的投影图像出现明显的错位。为了应对这一挑战，Kurth 等人提出了一种基于深度相机无标记 DPM 的可扩展在线解决方案。他们的方法通过减少除投影最精细和最亮的像素之外的其他投影仪的像素值，优化了重叠的投影图像，尤其是在投影仪与表面点深度和投影图像颜色不连续的区域。
2.3. Low-latency dynamic projection mapping.
2.3. 低延迟动态投影映射。
Typical projectors with a 60 Hz refresh rate are ill-suited for DPM because the human visual system detects misalignment when the delay from motion to projection exceeds 6–7 ms.52) A promising game-changer overcoming this limitation is a recently developed high-speed projector capable of achieving almost 1,000 frames per second (fps) full-color video projection.53) Alongside the high-speed projector, researchers have used high-speed cameras, nearly 1,000 fps, for tracking 3D pose of a target surface. Marker-based tracking techniques have demonstrated effectiveness in handling rigid surfaces54–56) and non-rigid surfaces.57) In addition, researchers have sought to enhance the quality of projected images while meeting low-latency demands. Nomoto et al. introduced a distributed cooperative approach in multi-projection DPM to ensure that projected images cover the entire surfaces of target objects.58) The same research group elevated the realism of the projected results using a ray tracing technique.59)
典型的 60Hz 刷新率的投影仪不适合 DPM，因为当运动到投影的延迟超过 6-7 毫秒时，人眼视觉系统会检测到错位。 52) 一种有希望的颠覆性技术是最近开发的高速投影仪，能够实现近 1000 帧每秒的全彩视频投影。 53) 与高速投影仪并行，研究人员使用了几乎 1000 帧每秒的高速摄像机来跟踪目标表面的 3D 姿态。基于标记的跟踪技术在处理刚性表面 54–56) 和非刚性表面方面已证明其有效性。 57) 此外，研究人员还寻求在满足低延迟需求的同时提高投影图像的质量。Nomoto 等人引入了一种分布式协作方法，用于多投影 DPM，以确保投影图像覆盖目标对象的整个表面。 58) 同一研究小组使用光线追踪技术提高了投影结果的真实感。 59)
Marker-less tracking is a key element in making DPM more applicable. The most successful field for marker-less DPM is makeup, mainly due to the robust and fast face tracking technologies that have already been established in computer vision research.60) However, addressing fast enough marker-less tracking in other application fields remains a technical challenge, as it necessitates the projection of calibration patterns onto surfaces for estimating the extrinsic parameters. Researchers have proposed projecting calibration patterns and their complementary patterns at high speeds to meet low-latency demands and to make the calibration patterns imperceptible to human observers.61) Another team uses a high-speed IR projector.62) An alternative approach to avoid the pattern projection requirement involves the use of a co-axial high-speed ProCam system where the projector and camera share their optical axes.63)
无标记跟踪是使 DPM 更具应用性的关键要素。无标记 DPM 最成功的领域是化妆，这主要归功于计算机视觉研究中已经建立的强大且快速的人脸跟踪技术。然而，在其它应用领域中实现足够快的无标记跟踪仍然是一个技术挑战，因为它需要将校准图案投射到表面上以估计外参数。研究人员提出了以高速投射校准图案及其互补图案，以满足低延迟需求并使校准图案对人类观察者不可察觉。另一组使用高速红外投影仪。另一种避免图案投射要求的替代方法涉及使用同轴高速 ProCam 系统，其中投影仪和相机共享其光学轴线。
Low-latency DPM can be achieved without the need for high-speed projectors. A promising alternative involves combining a typical projector with a dual-axis galvanometer for rapid redirection of the projector’s illumination direction. Although this approach has the drawback of not being able to quickly adjust the projected image for local pose changes of the target surface, it allows the projected image to smoothly follow the target without noticeable latency, even when the target moves over a large area. For instance, researchers have demonstrated a DPM on a screen mounted on a flying drone.64) A downsized version of the galvanometer-based system can even be worn and used for PM on a moving hand.65) Previous studies have shown that the mentioned drawback can be overcome by substituting the high-speed projector for the typical projector.54,55)
低延迟 DPM 可以在不需要高速投影仪的情况下实现。一种有希望的替代方案是将典型的投影仪与双轴哥尔丹纳结合，以快速改变投影仪的照明方向。尽管这种方法存在无法快速调整投影图像以适应目标表面局部姿态变化的缺点，但它允许投影图像平滑地跟随目标，即使目标在大范围内移动，也不会出现明显的延迟。例如，研究人员已经在安装在飞行无人机上的屏幕上展示了 DPM。 64) 基于哥尔丹纳的缩小版系统甚至可以佩戴并用于移动手部的 PM。 65) 先前的研究表明，通过用典型投影仪代替高速投影仪，可以克服上述缺点。 54,55)
3. Radiometric compensation
3. 放射性补偿
Radiometric or photometric compensation is another essential technique in PM that calculates projector pixel values to display a desired color even on a textured surface, thus making it appear as if it were a uniformly white surface. The typical forward model used in radiometric compensation is described by ci = fi (pj) + ei, where the RGB color vectors of
辐射或光度补偿是 PM 中另一种基本技术，它计算投影仪像素值以在纹理表面上显示所需颜色，从而使它看起来像是一个均匀的白色表面。在辐射补偿中使用的典型正向模型由 c = f(p) + e 描述，其中
To reproduce a desired appearance
为了在 PM 表面重现所需的外观
3.1. Compensation techniques based on hand-crafted color transformation models.
3.1. 基于手工调色模型的对色补偿技术。
Two decades ago, pioneering studies applied linear transformation models as f.31) However, due to the nonlinear nature of color processing in projector hardware, these models suffered from limited compensation accuracy. Thereafter, Grundhöfer demonstrated that a nonlinear model outperforms the linear ones.67,68) Specifically, they used a thin-plate spline (TPS) to approximate nonlinear transformation, which, however, required projecting hundreds of uniformly colored images onto the target surface in advance to calibrate the model parameters.
二十年前，开创性研究将线性变换模型应用于 f。然而，由于投影仪硬件中颜色处理的非线性特性，这些模型在补偿精度方面受到限制。此后，Grundhöfer 证明非线性模型优于线性模型。具体来说，他们使用薄板样条（TPS）来近似非线性变换，但这需要事先将数百张均匀着色的图像投影到目标表面上以校准模型参数。
A recent study has simplified the model complexity using a second-order polynomial.69) It continuously updates the model parameters in a real-time projection-and-capturing feedback loop and adjusts the projected colors accordingly, enabling it to handle changing lighting conditions. Li et al. approximated the nonlinear transformation using a piecewise linear function and significantly reduced the number of projecting calibration patterns by embedding multiple colors into a single pattern, assuming that the spectral reflectance of most real-world materials is smooth.70)
最近的一项研究通过使用二阶多项式简化了模型复杂性。 69) 它在实时投影-捕获反馈循环中持续更新模型参数，并相应地调整投影颜色，使其能够处理变化的照明条件。Li 等人使用分段线性函数近似非线性变换，并通过将多种颜色嵌入到单个图案中，显著减少了投影校准图案的数量，假设大多数真实世界材料的光谱反射率是平滑的。 70)
In addition to efforts solely focused on improving compensation accuracy, other research groups have explored various extension possibilities of radiometric compensation framework. Researchers have concentrated on estimating reflectance properties of target surfaces by decomposing the captured images under different color projections.71,72) The estimated reflectance maps were subsequently used to create novel target appearances, such as reducing color saturation. Amano and their group applied distributed multiple ProCam systems to control the appearance of a surface with view-dependent reflectance properties.73–76) As other extensions, Hashimoto and Yoshimura adapted a radiometric compensation technique for a moving fabric, supporting DPM.77) Pjanic et al. achieved seamless multi-projection displays using TPS-based color transformation model,78) ensuring a seamless transition in the overlapping area of images projected by different projectors.
除了专注于提高补偿准确性的努力之外，其他研究小组还探索了辐射补偿框架的各种扩展可能性。研究人员集中通过在不同颜色投影下分解捕获的图像来估计目标表面的反射率特性。随后，估计的反射率图被用来创建新的目标外观，例如降低色彩饱和度。阿马诺及其团队应用分布式多 ProCam 系统来控制具有视点相关反射率特性的表面外观。作为其他扩展，hashimoto 和 yoshimura 将辐射补偿技术应用于移动织物，支持 DPM。Pjanic 等人使用基于 TPS 的色彩变换模型实现了无缝多投影显示，确保了由不同投影仪投射的图像重叠区域的无缝过渡。
3.2. DNN-based end-to-end compensation techniques.
3.2. 基于深度神经网络的端到端补偿技术。
Very recently, Huang et al. found that DNNs can approximate the nonlinear transformation more accurately than hand-crafted models. They initially demonstrated that DNNs comprising a UNet-like backbone network and an autoencoder subnet, outperformed the classical TPS-based technique79) (Fig. 2). Subsequently, they extended their DNNs to enable geometric registration and radiometric compensation for PM on non-planar surfaces.80) They further improved compensation accuracy by introducing a siamese architecture into their network.81) Other researchers used a differentiable rendering framework in radiometric compensation.82) Handling high-resolution images typically requires long training times and involves high memory costs. Wang et al. mitigated this issue by incorporating a sampling scheme into the network and introducing attention blocks.83) Li et al., in their latest work, reduced the network size by using a network solely for the color transformation of the projector.84) Interestingly, they also demonstrated that a hand-crafted, precise physics-based model of the PM process with limited reliance on neural networks outperformed the end-to-end compensation techniques described above.
最近，黄等人发现，DNNs 可以比手工模型更准确地逼近非线性变换。他们最初证明了由类似 UNet 的骨干网络和自动编码器子网络组成的 DNNs 在经典基于 TPS 的技术 79) （图 2）上表现更优。随后，他们将他们的 DNNs 扩展到能够对非平面表面的 PM 进行几何配准和辐射补偿。 80) 他们通过将 Siamese 架构引入他们的网络，进一步提高了补偿精度。 81) 其他研究人员在辐射补偿中使用了可微渲染框架。 82) 处理高分辨率图像通常需要较长的训练时间和涉及较高的内存成本。王等人通过将采样方案纳入网络并引入注意力块来缓解了这个问题。 83) 李等人在其最新工作中，通过仅使用用于投影仪颜色变换的网络来减小网络大小。 有趣的是，他们还证明了一个手工制作的、基于物理的 PM 过程模型，该模型对神经网络的依赖有限，其性能优于上述所述的端到端补偿技术。
Figure 2. 图 2
(Color online) Radiometric compensation using DNNs.79) (a) Projection target under uniformly white projection. (b) Target appearance. (c) PM result of the target appearance without any compensation. (d) PM result using a classical TPS-based technique. (e) PM result using the DNN-based technique. (2019 IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) 2019, 6803–6812)
(在线彩色) 使用深度神经网络进行辐射补偿。 79) (a) 均匀白光投影下的投影目标。 (b) 目标外观。 (c) 未进行任何补偿的目标外观 PM 结果。 (d) 使用基于经典 TPS 技术的 PM 结果。 (e) 使用基于 DNN 技术的 PM 结果。（2019 IEEE/CVF 计算机视觉与模式识别会议（CVPR）2019，6803–6812）
DNNs can be applied to various tasks in addition to radiometric compensation. Huang and Ling demonstrated that their networks could reconstruct the shape of a projected scene and simulate the scene’s appearance under a novel image projection.85) The latter is particularly useful for testing or debugging PM without the requirement for actual PM operations. Erel et al. successfully decomposed scene geometry and view-dependent reflectance properties and estimated the projector’s intrinsic and extrinsic parameters by training neural representations of the scene from multi-view captures under PM with different color patterns.86) They showcased that their DNNs can handle geometric registration and radiometric compensation for novel viewpoints.
深度神经网络可以应用于除了辐射校正以外的各种任务。黄和凌展示了他们的网络能够重建投影场景的形状并模拟场景在新型图像投影下的外观。后者在无需实际 PM 操作的情况下，特别适用于测试或调试 PM。Erel 等人成功分解了场景几何和视点相关的反射特性，并通过在具有不同颜色模式的 PM 下从多视角捕获中训练场景的神经表示来估计投影仪的内在和外在参数。他们展示了他们的 DNN 可以处理新型视点的几何配准和辐射校正。
4. Defocus compensation 4. 虚化补偿
As projectors are designed to emit maximum brightness through their lenses, they have a large aperture size, resulting in a shallow depth-of-field (DOF). The typical forward model of defocus blur is described by I′ = K * I, where I, I′, and K represent a projected image without suffering from defocus, a defocused result, and a spatially varying 2D defocus kernel, respectively. In this equation, * represents a 2D convolution process. Deblurring the projected result is achieved by computing the inverse of the forward model. However, standard algorithms such as Wiener filter are unsuitable because the dynamic range of a projector device is not infinite (e.g., the maximum luminance is limited, and negative light is physically not displayable).
投影仪设计为通过镜头发射最大亮度，因此具有较大的光圈尺寸，导致景深（DOF）较浅。典型的散焦模糊前向模型由 I′ = K * I 描述，其中 I、I′ 和 K 分别代表未受散焦影响的投影图像、散焦结果和空间变化的二维散焦核。在这个方程中，* 代表二维卷积过程。通过计算前向模型的逆来去模糊投影结果。然而，如维纳滤波器等标准算法不适用，因为投影仪设备的动态范围不是无限的（例如，最大亮度有限，负光在物理上无法显示）。
A classical study solved this problem using an iterative, constrained steepest-descent algorithm.87) A recent work introduced a non-iterative technique that simply enhances the pixel intensities around the edge areas that are lost due to defocus blur, resulting in reduced computational time.88) These techniques require a dot pattern projection to obtain spatially varying blur kernels every time either the projector or the surface moves. Kageyama et al. recently addressed this issue using DNNs89,90) (Fig. 3). Specifically, their DNNs estimated the blur kernels from the PM result of a natural image and generated the projection image compensating for defocus blur.
经典研究使用迭代、约束最速下降算法解决了这个问题。 87) 最近的一项工作介绍了一种非迭代技术，该技术简单地增强了由于失焦模糊而丢失的边缘区域的像素强度，从而减少了计算时间。 88) 这些技术需要点图案投影来获取每次投影仪或表面移动时的空间变化模糊核。Kageyama 等人最近使用 DNNs 解决了这个问题 89,90) （图 3）。具体来说，他们的 DNNs 从自然图像的 PM 结果中估计模糊核，并生成补偿失焦模糊的投影图像。
Figure 3. 图 3
(Color online) Software-based defocus compensation using DNNs.90) (a) Experimental setup: a robotic arm repeatedly moves the target surface along the same path for comparison. (b) Target appearance. (c) PM result without compensation. (d) Compensated PM result. (IEEE Trans. Vis. Comput. Graph. 2022, 28, 2223–2233)
(在线彩色) 基于 DNN 的软件式散焦补偿。 90) (a) 实验装置：机械臂沿相同路径重复移动目标表面以进行比较。 (b) 目标外观。 (c) 未补偿的 PM 结果。 (d) 补偿后的 PM 结果。（IEEE Trans. Vis. Comput. Graph. 2022，28，2223-2233）
The compensation capacity of the software-based solutions mentioned above is restricted by limited dynamic range of projector hardware. Researchers have developed hardware-based solutions to overcome this limitation. Xu et al. proposed a multifocal projector comprising an electrically focus-tunable lens (ETL) and a synchronized high-speed projector.91) They modulate the focal length of the ETL at more than 60 Hz, thus making it imperceptible to human observers, and project images precisely when the focusing distance of the projector corresponds to the target surface. The same setup also achieved a varifocal projector in which ETL’s focal length was constantly adjusted to match the target surface.92) Although a large aperture ETL would be suitable for these systems, the response time of such ETLs is limited. The ETL is made of an optical fluid sealed off by an elastic polymer membrane. An actuator ring exerts pressure on the outer zone of the container, changing the curvature of the lens. The response time limitation is caused by the rippling of the optical fluid after actuation. Researchers demonstrated that input signals computed using sparse optimization can speed up the response time.93)
上述基于软件的解决方案的补偿能力受到投影机硬件动态范围有限的限制。研究人员已经开发了基于硬件的解决方案来克服这一限制。徐等人提出了一种多焦点投影仪，包括一个电聚焦可调镜头（ETL）和一个同步的高速投影仪。他们在超过 60 Hz 的频率下调制 ETL 的焦距，使其对人类观察者来说难以察觉，并在投影仪的聚焦距离与目标表面相对应时精确投影图像。同样的设置还实现了一种变焦投影仪，其中 ETL 的焦距不断调整以匹配目标表面。尽管大孔径 ETL 适合这些系统，但此类 ETL 的响应时间有限。ETL 由一个弹性聚合物膜密封的光学流体制成。一个执行器环对容器的外部区域施加压力，改变镜头的曲率。响应时间限制是由执行后光学流体的波动引起的。 研究人员证明，使用稀疏优化计算输入信号可以加快响应时间。 93)
Other hardware-based solutions control the waveform of the projected light. Li et al. proposed optimizing the diffractive optical element to preserve the high spatial frequency components of the projected image over various distances, thereby extending the DOF of the projector.94) Other researchers have proposed spatially adaptive focal projection, coining the term “focal surface projection” to describe their approach, using a phase-only spatial light modulator. This approach enables focusing on all parts of a non-planar target surface.95)
其他基于硬件的解决方案控制投影光波的波形。李等人提出了优化衍射光学元件，以保留投影图像在不同距离上的高空间频率成分，从而扩展投影机的景深。 94) 其他研究人员提出了空间自适应焦点投影，创造了“焦点表面投影”这一术语来描述他们的方法，使用相位仅空间光调制器。这种方法能够聚焦于非平面目标表面的所有部分。 95)
5. Shadow removal 5. 去阴影
Cast shadows significantly degrade the sense of immersion in PM. Previous studies removed shadows using synthetic aperture approaches. Specifically, they spatially distributed multiple projectors to ensure that users do not simultaneously occlude a projection target from all projectors. Once either an occluder or its shadow is detected by cameras, the system compensates for the shadow by illuminating that area from an unoccluded projector.96) Although they computed the projection images for all projectors on a single central server, the recent research trend has shifted toward applying cooperative distributed algorithms since around 2015.97) Uesaka and Amano proposed a technique in which multiple co-axial ProCam systems cooperatively remove shadows.98) Nomoto et al. demonstrated shadow removal in DPM with multiple high-speed projectors using a cooperative algorithm.58) However, these synthetic aperture approaches suffer from a delay in computational compensation process. In other words, a shadow cannot be perfectly removed while an occluder is moving.
投影产生的阴影显著降低了 PM 的沉浸感。先前的研究通过合成孔径方法去除阴影。具体来说，他们通过空间分布多个投影仪来确保用户不会被所有投影仪同时遮挡投影目标。一旦摄像头检测到遮挡物或其阴影，系统就会通过从未被遮挡的投影仪照亮该区域来补偿阴影。尽管他们计算了所有投影仪在单个中央服务器上的投影图像，但自 2015 年以来，最新的研究趋势已经转向应用协作分布式算法。Uesaka 和 Amano 提出了一种技术，其中多个同轴 ProCam 系统协作去除阴影。Nomoto 等人使用协作算法在 DPM 中展示了使用多个高速投影仪去除阴影。然而，这些合成孔径方法在计算补偿过程中存在延迟。换句话说，当遮挡物移动时，阴影无法被完美去除。
By contrast, optical approaches achieve delay-free shadow removal and have also attracted significant attention from researchers. Hiratani et al. applied a large-format retrotransmissive plate to project images onto a surface from wide viewing angles99,100) (Fig. 4). The retrotransmissive plate collects the light rays emitted from a point in space at a plane-symmetrical position with respect to it. They prepared a white diffuse object (proxy object) with a shape that is plane-symmetrical to the projection target and placed the target and proxy objects in a plane-symmetrical arrangement with respect to retrotransmissive plate. When an image is projected onto the proxy object, the reflected light rays pass through the retrotransmissive plate and converge on the target object. Consequently, the appearance of the proxy object is duplicated on the target object’s surface. When the size of the retrotransmissive plate is sufficiently large relative to an occluder, shadowless PM is achieved without the need for the shadow removal computations used in conventional synthetic aperture approaches.
相比之下，光学方法实现了无延迟的阴影消除，并也引起了研究者的广泛关注。Hiratani 等人应用了一种大尺寸反透射板，从宽视角将图像投射到表面上（图 4）。反透射板收集来自空间中某点的光线，相对于它处于平面对称位置。他们准备了一个白色扩散物体（代理物体），其形状与投影目标平面对称，并将目标和代理物体放置在相对于反透射板的平面对称排列中。当图像投射到代理物体上时，反射的光线穿过反透射板并聚焦在目标物体上。因此，代理物体的外观在目标物体表面上被复制。当反透射板相对于遮挡物足够大时，无需使用传统合成孔径方法中使用的阴影消除计算，即可实现无阴影的 PM。
Figure 4. 图 4
(Color online) Shadow removal using a large-format retrotransmissive plate.99) (a) Schematic illustrating the principle. (b) Typical PM result with an occluder. (c) PM result using the shadow removal system with the same occluder. (IEEE Trans. Vis. Comput. Graph. 2023, 29, 2280–2290)
(在线彩色) 使用大尺寸反透射板进行阴影去除。 99) (a) 原理解示图。 (b) 带遮挡物的典型 PM 结果。 (c) 使用相同遮挡物的阴影去除系统 PM 结果。（IEEE Trans. Vis. Comput. Graph. 2023，29，2280–2290）
The above optical solution is restricted to static DPM because the proxy and target objects must be placed at the plane-symmetrical positions. Other researchers have overcome this limitation by moving the proxy object using a robotic arm to match its pose with the target object.101) The same research group also proposed using a volumetric display102) and light field display103) instead of placing a physical proxy object to generate light rays as if they were emitted from the surface of a proxy object whose pose matches that of the target object.
上述光学解决方案仅限于静态 DPM，因为代理对象和目标对象必须放置在平面对称的位置。其他研究人员通过使用机械臂移动代理对象来克服这一限制，使其姿态与目标对象匹配。该研究小组还提出了使用体积显示①和光场显示②，而不是放置物理代理对象来生成光线，就像它们是从与目标对象姿态匹配的代理对象表面发出的。
6. Unconventional projectors
6. 非传统投影仪
As discussed in the previous sections, unconventional projectors such as those with ETLs and high-speed ones can fundamentally resolve the specific technical issues that could not be addressed using standard projectors. This section introduces three types of unconventional projectors each of which has been currently explored by multiple research groups.
如前几节所述，非传统投影仪，如带有 ETLs 和高速的投影仪，可以从根本上解决标准投影仪无法解决的具体技术问题。本节介绍了三种非传统投影仪，每种都已被多个研究小组探索过。
6.1. Wearable projectors.
6.1. 便携式投影仪。
Following pioneering work,104) several researchers have explored PM using wearable projectors.105,106) The recent trend of downsizing projector hardware, coupled with bright light sources such as LEDs and lasers, has driven the research in this direction. Wearable projectors are valuable for PM onto nearby surfaces or the user’s body. For example, a tiny projector was used as a display component of a smartwatch, enabling a user to interact with the overlaid smartwatch image contents on their arm.107) Another study combined a wearable projector with a pan-tilt mirror and high-speed camera to make projected images follow a moving target surface without perceivable latency.65) Head-mounted setups were also explored, wherein the distance between a projector and user’s eye is reduced, enabling nearly occlusion-free PM.108)
在开创性工作的基础上，多位研究人员已经探索了使用可穿戴投影仪进行投影映射（PM）。近期投影仪硬件小型化的趋势，以及 LED 和激光等明亮光源的配合，推动了这一方向的研究。可穿戴投影仪对于在附近表面或用户身体上进行投影映射非常有价值。例如，一个小型投影仪被用作智能手表的显示组件，使用户能够与其手臂上叠加的智能手表图像内容进行交互。另一项研究将可穿戴投影仪与全景倾斜镜和高速摄像机结合，使投影图像能够无感知延迟地跟随移动的目标表面。还探索了头戴式设置，其中投影仪与用户眼睛之间的距离减小，实现了几乎无遮挡的投影映射。
A current research trend involves combining actuated head-mounted projectors with head-mounted displays. Wang et al. proposed attaching a projector to a virtual reality (VR) headset.109) Their system projected VR scenes that the headset user is watching onto the floor around them, enabling them to share their VR experiences with others. Hartmann et al. suggested using a head-mounted projector with an optical see-through AR headset and demonstrated the sharing of augmented image contents displayed in the headset with people in the vicinity through projected imagery.110) They also proposed displaying auxiliary information and user interface widgets with the head-mounted projector to support interaction with the image contents displayed in the headset.
当前研究趋势涉及将驱动式头戴投影仪与头戴式显示器相结合。王等人提出将投影仪附着在虚拟现实（VR）头盔上。 109) 他们的系统将头盔用户正在观看的 VR 场景投射到他们周围的地面，使他们能够与他人分享他们的 VR 体验。哈特曼等人建议使用带有光学透视增强现实（AR）头盔的头戴式投影仪，并通过投影图像展示了头盔中显示的增强图像内容与附近的人的共享。 110) 他们还提出使用头戴式投影仪显示辅助信息和用户界面小部件，以支持与头盔中显示的图像内容的交互。
6.2. Omnidirectional projectors.
6.2. 全向投影仪。
The FOV of a typical projector is limited, necessitating the use of multiple projectors to achieve large-area PM. An omnidirectional projector, using a fisheye lens with almost a 180-degree FOV, presents a potential solution for this issue. A research group proposed an omnidirectional projector and demonstrated various PM applications using it.111,112) The geometric registration of an omnidirectional projector is non-trivial because the pinhole camera model is no longer valid. The researchers addressed this problem using a co-axial approach in which a projector and camera share their optical axes using a beam-splitter before the fisheye lens. Their co-axial omnidirectional ProCam system can project images onto physical surfaces without distortion, on which visual markers are attached.
典型投影仪的视场角有限，需要使用多个投影仪来实现大面积的 PM。一个全向投影仪，使用几乎 180 度视场角的鱼眼镜头，为解决这个问题提供了一个潜在方案。一个研究小组提出了一种全向投影仪，并展示了使用它进行各种 PM 应用。全向投影仪的几何配准非同寻常，因为针孔相机模型不再有效。研究人员使用了一种共轴方法来解决这一问题，其中投影仪和相机在鱼眼镜头之前使用分束器共享它们的视轴。他们的共轴全向 ProCam 系统可以将图像投射到物理表面上，不产生扭曲，并在其上附加视觉标记。
Yamamoto et al. implemented an omnidirectional ProCam system using another unique approach.113) They proposed a monocular ProCam system in which the projector and camera share the same objective lens. Using relay optics, they optically transferred the image panels of the camera and projector to the focal point of the objective lens, resulting in overlaid image panels. The overlaid pixels have sensing and displaying capabilities. They realized an omnidirectional ProCam system using a fisheye lens as the objective within this framework. Furthermore, they showcased the high scalability of their approach by implementing a high dynamic range ProCam system using a traditional double modulation framework.114,115)
山本等人采用一种独特的方法实现了全向 ProCam 系统。他们提出了一种单目 ProCam 系统，其中投影仪和相机共享同一个物镜。通过使用中继光学，他们将相机和投影仪的图像面板光学地转移到物镜的焦点，从而实现了叠加的图像面板。叠加的像素具有感知和显示功能。他们在这个框架内使用鱼眼镜头作为物镜实现了全向 ProCam 系统。此外，他们通过实现一个使用传统双调制框架的高动态范围 ProCam 系统，展示了他们方法的高可扩展性。
6.3. Visible light communication projectors.
6.3. 可见光通信投影仪。
Embedding invisible code independently in each projected pixel enables the control of electronic devices with photo sensors within the projector’s FOV while simultaneously presenting meaningful images to human observers who remain unaware of the embedded information. In other words, the projector has the capability for visible light communication at the pixel level. This can be achieved by modulating the projected light intensity at a very high speed such as 1 MHz, which is much higher than critical flicker fusion frequency of the human visual system. While pioneering work was published in 2007,116) where fixed information was embedded in grayscale images, this topic is still actively explored by multiple research groups.
将不可见代码独立嵌入到每个投影像素中，可以在投影仪的视场内控制带有光电传感器的电子设备，同时向人类观察者展示有意义的图像，而观察者对嵌入的信息浑然不觉。换句话说，投影仪具有在像素级别进行可见光通信的能力。这可以通过以非常高的速度（如 1 MHz）调制投影光强度来实现，这比人类视觉系统的临界闪烁融合频率要高得多。虽然开创性的工作在 2007 年就已经发表， 116) 其中固定信息被嵌入到灰度图像中，但这个主题仍然被多个研究小组积极研究。
A recent study achieves embedding information in full-color images that can be updated interactively.117) It was demonstrated that the embedded information controls multiple robots,118,119) and wearable haptic displays120) (Fig. 5), in cooperation with graphical images. Although these systems read embedded information using photo sensors, Kumar et al. demonstrated that a high-speed camera can simultaneously read the information embedded in different pixels.121) Researchers have developed a projector emitting RGB as well as IR light, embedding information in the IR channel.122)
最近的一项研究实现了在彩色图像中嵌入可交互更新的信息。 117) 已证明嵌入的信息可以控制多个机器人， 118,119) 以及可穿戴触觉显示器 120) （图 5），并与图形图像合作。尽管这些系统使用光电传感器读取嵌入的信息，但 Kumar 等人证明了高速相机可以同时读取不同像素中嵌入的信息。 121) 研究人员开发了一种发射 RGB 以及红外光的投影仪，在红外通道中嵌入信息。 122)
Figure 5. 图 5
(Color online) Pixel-level visible light communication.120) (Top) Schematic illustrating the embedding of unique information into each projector pixel while presenting an image to human observers. Note that fPVLC is higher than the critical flicker fusion frequency. (Bottom) A user wearing a haptic device experiences different vibration patterns based on the touched position in a projected image. (IEEE Trans. Vis. Comput. Graph. 2023, 29, 2005–2019)
(在线彩色) 像素级可见光通信。 120) （顶部）展示给人类观察者图像的同时将独特信息嵌入到每个投影像素中的示意图。注意，f PVLC 高于临界闪烁融合频率。（底部）用户佩戴触觉设备，根据在投影图像中触摸的位置体验不同的振动模式。（IEEE Trans. Vis. Comput. Graph. 2023，29，2005–2019）
7. Overcoming technical limitations
7. 克服技术限制
Projector hardware has inherent limitations that cannot be addressed by projector devices alone. This section summarizes two approaches to tackle these limitations; one combines near-eye optics in PM, and the other uses perceptual tricks.
投影仪硬件存在无法仅通过投影仪设备解决的固有局限性。本节总结了两种应对这些局限性的方法；一种结合了 PM 中的近眼光学，另一种使用感知技巧。
7.1. Combining near-eye optics.
7.1. 近眼光学组合。
In typical PM, projectors alter the appearance of target surfaces, although displaying images floating above physical surfaces is not feasible. Stereoscopic PM technology overcomes this limitation, allowing users to perceive 3D objects that appear to float above physical surfaces with arbitrary shapes. These effects are achieved through the tracking of an observer’s viewpoint, rendering perspectively correct images with appropriate disparity for each eye, and projecting these two images in a time-sequential manner within each frame. The projected images are viewed through active-shutter glasses equipped with liquid crystal shutters, which prevent image interference between the two eyes. Researchers have recognized the potential of stereoscopic PM in various fields, including museum guides,18) product design,30) architecture planning,123) and teleconferencing.124)
在典型的项目管理中，投影仪改变目标表面的外观，尽管在物理表面上方显示图像是不可行的。立体项目管理技术克服了这一限制，使用户能够感知似乎漂浮在任意形状物理表面上的 3D 对象。这些效果是通过跟踪观察者的视角，渲染具有适当视差且视角正确的图像，并在每个帧中以时间顺序投影这两幅图像来实现的。投影的图像通过配备液晶快门的主动式快门眼镜观看，这可以防止两只眼睛之间的图像干扰。研究人员已经认识到立体项目管理在博物馆导游、产品设计、建筑规划、以及远程会议等各个领域的潜力。
A recent work applied the principle of stereoscopic PM to alter the appearance of a mirror surface.125) This technique does not directly project images onto a mirror surface; instead, it projects images onto diffuse surfaces that are visible to an observer through the mirror. With stereoscopic PM, the distance of the projected diffuse surfaces matches that of the mirror surface.
最近的一项工作应用了立体 PM 原理来改变镜面外观。 125) 这种技术并非直接将图像投影到镜面上；相反，它将图像投影到观察者通过镜子可见的漫反射表面上。使用立体 PM，投影的漫反射表面距离与镜面表面相匹配。
Typical stereoscopic PM technology only addresses binocular cues and cannot provide accurate focus cues, leading to a vergence-accommodation conflict (VAC) that causes significant discomfort, fatigue, and distorted 3D perception for the observer. Recent studies have tried to mitigate VAC. Fender et al. optimized the placement of the displayed 3D objects such that the depth difference becomes small between the projected physical surface and displayed objects.126) Kimura et al. proposed a multifocal stereoscopic PM to address VAC.127) They attached ETLs to active-shutter glasses and applied fast and periodical focal sweeps to ETLs, causing the “virtual image” (as an optical term) of every part of the real scene seen through ETLs to move back and forth during each sweep period. In each frame, the 3D objects were projected from a synchronized high-speed projector at the exact moment that the virtual image of the projected imagery on a real surface is located at a desired distance from ETLs.
典型立体 PM 技术仅处理双眼线索，无法提供准确的聚焦线索，导致视觉辐辏调节冲突（VAC），这会引起观察者显著的不适、疲劳和扭曲的 3D 感知。最近的研究试图减轻 VAC。Fender 等人优化了显示 3D 物体的位置，使得投影的物理表面和显示物体之间的深度差异变得很小。 126) Kimura 等人提出了一种多焦点立体 PM 来解决 VAC。 127) 他们将 ETLs 附着在主动快门眼镜上，并对 ETLs 进行快速和周期性的聚焦扫描，导致通过 ETLs 看到的真实场景的每个部分的“虚拟图像”（作为一个光学术语）在每个扫描周期内来回移动。在每一帧中，3D 物体由同步的高速投影仪在虚拟图像位于 ETLs 所需距离处时投影出来。
Using ETLs as eyeglasses in conjunction with a synchronized high-speed projector creates other novel vision experiences that cannot be achieved using projectors alone. Ueda et al. proposed using a high-speed projector to illuminate a real scene rather than overlaying images onto it. This approach allows for spatially non-uniformly defocused real-world appearances, irrespective of the distance from the user’s eyes to observed real objects128) (Fig. 6). They achieved this by periodically modulating the focal lengths of the glasses at a rate exceeding 60 Hz. During a specific phase when optical power of ETLs is too high for a user to adjust their vision to focus on the scene, one part of the scene intended to appear blurred is illuminated by the projector, whereas another part intended to appear focused is illuminated during a different phase. This process realizes the spatial defocusing effect that can be used for gaze navigation.129) Based on a similar principle, Ueda et al. used two ETLs for each eye for spatial zooming, where a part of a scene is zoomed in.130)
使用 ETLs 作为眼镜与同步的高速投影仪结合，创造出仅使用投影仪无法实现的全新视觉体验。上田等人提出使用高速投影仪照亮真实场景，而不是将其图像叠加到场景上。这种方法可以实现空间非均匀的模糊真实世界外观，无论用户眼睛与观察到的真实物体之间的距离如何（图 6）。他们通过以超过 60Hz 的速率周期性地调制眼镜的焦距来实现这一点。在 ETLs 的光学功率过高，用户无法调整视力以聚焦于场景的特定阶段，投影仪照亮场景中预定模糊的部分，而在不同的阶段照亮预定聚焦的部分。这个过程实现了可用于视线导航的空间模糊效果。基于类似原理，上田等人使用每只眼睛两个 ETLs 进行空间缩放，其中场景的一部分被放大。
Figure 6. 图 6
(Color online) Combining near-eye optics wit