Research Overview／ 3D Reconstruction
Introduction
Three-dimensional reconstruction in computer vision is a technique for recovering the three-dimensional shape of objects. Although three-dimensional shapes can be measured with high accuracy by illuminating objects with lasers and similar devices, being able to reconstruct three-dimensional structure from photographs taken with digital cameras alone makes three-dimensional data much more accessible. Here, we introduce image-based three-dimensional reconstruction.
Triangulation
Have you heard the term “triangulation”? You may have encountered it in a mathematics class as an example of measuring the distance to a tree on the opposite bank, as shown in Figure 1. Let points on the same side be $A$ and $B$, and let the tree on the opposite bank be $C$ . If the distance $l$ between $A$ and $B$, $\alpha = \angle CAB$, and $\beta = \angle CBA$ are known, the distance $d$ to the tree on the opposite bank can be computed. Using trigonometric ratios, the distance $d$ is given by the following equation. \[ d = \frac{l}{\frac{1}{\tan\alpha}+\frac{1}{\tan\beta}}=\frac{\sin\alpha\sin\beta}{\sin(\alpha+\beta)}l \] Considering $\tan\alpha$ for the right triangle on the left and $\tan\beta$ for the right triangle on the right yields the expression for $d$ . The technique of recovering three-dimensional shape from images using the principle of triangulation is called stereo vision.
Figure 1: An example of measuring the distance to a tree on the opposite bank using triangulation
Stereo Vision
Stereo vision is a technique for recovering the three-dimensional shape of an object using two cameras, as shown in Figure 2. Compared with triangulation, the number of variables increases. This is because stereo vision uses a projection model that describes how an object appears in camera images. What becomes important here are the camera parameters, which characterize the camera. There are two types of camera parameters. The first is the intrinsic parameters. These describe the characteristics of the camera itself and consist of quantities such as the image center $c$, the lens center $C$, and the focal length $f$. The second is the extrinsic parameters. Because two cameras are used, the relative pose between them (translation and rotation, etc.) is required. In Figure 2, this is represented by the baseline length $b$ between the cameras. When point $m$ in the left image and point $m'$ in the right image correspond to the same location on the object, the three-dimensional position $M$ of the object can be computed using the relationship shown in Figure 2. By performing similar computations for every pixel in the images, the detailed three-dimensional shape of the object can be recovered. It may seem somewhat complicated, but the underlying principle is triangulation.
Figure 2: Basic principle of stereo vision
3D Reconstruction System Using Digital Cameras
We introduce a three-dimensional reconstruction system using digital cameras developed in our laboratory. By taking two or more photographs with a digital camera, the three-dimensional shape of the target object can be recovered. For details, please refer to references [1] and [2]. Video 1 is a demonstration of this system. After several photographs are taken and transferred to a notebook computer, the three-dimensional shape is reconstructed. As additional photographs are captured, the entire object is gradually reconstructed.
Multi-View Stereo
Whereas stereo vision recovers three-dimensional shape from two images, the technique of recovering three-dimensional shape from multiple images is called multi-view stereo. If the entire object fits within a single image, regions that are not visible cannot be reconstructed. By using images captured from multiple viewpoints around the object, the complete three-dimensional shape of the object can be recovered. Video 2 shows 202 multi-view images and the three-dimensional shape reconstructed from them. By using a large number of images as input, the detailed shape of the object can be recovered.
Neural Radiance Fields (NeRF)
An image is obtained by projecting a three-dimensional scene onto a two-dimensional plane. In this process, each pixel in the image can be associated with a point in three-dimensional space along a straight line. This straight line is called a “ray.” A three-dimensional scene and an image are connected by many such rays, and the image is formed by them. Neural Radiance Fields (NeRF) is a method that estimates these rays using deep learning. Once NeRF has been estimated, images can be synthesized as if the scene were photographed from arbitrary viewpoints. Because depth (the distance from the camera to the object) is estimated in the process of inferring rays, NeRF can be used to obtain a depth map, which represents the distance from the camera to the object. Multi-view stereo can recover detailed surface shape, whereas NeRF can recover object boundaries accurately. We are currently investigating new methods that combine multi-view stereo with NeRF, as described in references [5] and [6].
Various 3D Reconstruction Results
We present some results obtained from three-dimensional reconstruction research conducted in our laboratory.
- Video 3 shows obstacle detection using a stereo camera mounted on the windshield of a vehicle. The road surface is shown in blue, and other obstacles are displayed in red for nearer objects and yellow for farther ones. Although this is not three-dimensional reconstruction per se, measuring the distance to vehicles ahead can be applied to driver assistance. For details, please refer to reference [3].
- Figure 3 shows an application of the multi-view stereo method proposed in our laboratory (reference [4]) to the digital archiving of cultural properties. Approximately 80 photographs of a transom wood carving near the entrance of the main hall of Zuiganji Temple in Matsushima were taken with a digital camera. From these 80 images, we reconstructed the shape model of the transom wood carving. Digital preservation of cultural properties is advancing; however, lasers cannot always be applied to cultural properties, and objects often cannot be moved, which limits measurement conditions. Methods that can conveniently recover three-dimensional shape with a digital camera are therefore valuable.
- Figures 4 and 5 show three-dimensional shapes reconstructed from multiple casually taken photographs with a digital camera. Figure 4 shows the t’Serclaes monument in the Grand Place in Brussels, Belgium. Figure 5 shows a courtyard wall of the Doge’s Palace in Venice, Italy. In this way, by taking multiple photographs while traveling, one can preserve the three-dimensional shape as a memento. Figures 4 and 5 were reconstructed using the methods proposed in references [7] and [8].
Figure 3: Shape model of the transom wood carving near the entrance of the main hall of Zuiganji Temple (Matsushima), reconstructed from 80 images (joint research with 凸版印刷株式会社).
Figure 3: Shape model of the transom wood carving near the entrance of the main hall of Zuiganji Temple (Matsushima), reconstructed from 80 images (joint research with 凸版印刷株式会社).
Summary
We have briefly introduced research on three-dimensional reconstruction. Three-dimensional reconstruction is an inverse problem, as illustrated in Figure 6. We live in three-dimensional space, so objects have three-dimensional shape. For example, recording an object as a two-dimensional image is what photography does. Because a three-dimensional object is mapped to a two-dimensional image, this is a forward problem and is easy to solve. Video games work similarly: a three-dimensional model is displayed on a screen, which is again a mapping from three dimensions to two. Three-dimensional reconstruction corresponds to solving the inverse of this process. Because three-dimensional structure must be inferred from two-dimensional images, although mathematical models exist, various techniques are needed to obtain practical solutions. As we have described, if three-dimensional shape can be recovered from images, many applications become possible. For this reason, many researchers in computer vision work on this challenging problem.
References
- 三浦衛ほか,"カメラの移動撮影に基づく2視点からの3次元形状計測とその性能評価," 映像情報メディア学会誌, Vol. 68, No. 4, pp. J135--J143, April 2014. [PDF]
- S. Yamao et al., "A sequential online 3D reconstruction system using dense stereo matching," Proc. IEEE Winter Conf. Applications of Computer Vision, pp. 341--348, January 2015. [PDF]
- 和泉圭祐ほか, "基線長が短い車載ステレオカメラのための障害物検出手法,"電子情報通信学会論文誌 A, Vol. J98-A, No. 1, pp. 165--175, January 2015. [PDF]
- S. Sakai et al., "Phase-based window matching with geometric correction for multi-view stereo," IEICE Trans. Information and Systems, Vol. E98-D, No. 10, pp. 1818--1828, October 2015. [PDF, Project page]
- S. Ito et al., "Depth map estimation from multi-view images with NeRF-based refinement," Proc. Int'l Conf. Image Processing, pp. 2955--2959, October 2023. [PDF]
- S. Ito et al., "Neural radiance field-inspired depth map refinement for accurate multi-view stereo," J. Imaging, vol. 10, no. 3, pp. 68-1--68-21, March 2024 (Open access).
- K. Yodokawa et al., "Outlier and artifact removal filters for multi-view stereo," Proc. IEEE Int'l Conf. Image Processing, pp. 3638-3642, October 2018. [PDF]
- K. Ito et al., "PM-MVS: PatchMatch multi-view stereo," Machine Vision and Applications, vol. 34, no. 32, pp. 1--16, March 2023 (Open access).