A Panoramic 3D Reconstruction System Based on the Projection of Patterns

Abstract

This work presents the implementation of a 3D reconstruction system capable of reconstructing a 360-degree scene with a single acquisition using a projection of patterns. The system is formed by two modules: the first module is a CCD camera with a parabolic mirror that allows the acquisition of catadioptric images. The second module consists of a light projector and a parabolic mirror that is used to generate the pattern projections over the object that will be reconstructed. The projection system has a 360-degree field of view and both modules were calibrated to obtain the extrinsic parameters. To validate the functionality of the system, we performed 3D reconstructions of three objects, and show the reconstruction error analysis.

Keywords

Panoramic 3D Reconstruction Calibration Catadioptric Camera Catadioptric Projector

1. Introduction

In recent times, 3D reconstruction systems have become fundamental for object modelling applications such as virtual environments development for archaeology [1, 2], video surveillance [3, 4], quality control processes [5, 6], disease diagnosis [7, 8], and measurement [9, 10].

The 3D reconstruction systems that incorporate lenses and mirrors into acquired images with an omnidirectional field of view are called catadioptric systems. A common approach for 3D reconstruction is the use of structured light projection with a 360-degree laser line combined with omnidirectional stereo cameras that take images with multiple views [11, 12]. The disadvantage of these approaches is that we can only recover 3D information in the area of the laser incidence, limiting the reconstruction area. Another approach is the use of an omnidirectional camera constructed by a traditional perspective camera and a pyramid reflector [13].

In [14], the authors propose a panoramic sensor using a convex mirror capable of producing panoramic depth maps by feature matching from two panoramic images. However, this technique fails with textureless surfaces, being unable to match the texture features. A stereo system with a texture projector can eliminate or reduce error matching. For this reason we propose a new 3D reconstruction system based on a projection system and a single catadioptric camera.

In order to obtain a good reconstruction, it is necessary to calibrate the catadioptric cameras, estimating the geometric model of the camera. In recent decades, several algorithms have been proposed for the calibration of omnidirectional cameras [14 –18].

The 3D reconstruction systems based on conventional cameras have been well-studied [19 –22]. However, for applications where we need to process a 360-degree field of view (for example inside hollow objects), conventional systems implement a piecemeal reconstruction. In this paper, we describe the 360-degree 3D reconstruction of a scene from a single acquisition using a projection of patterns.

2. Geometrical model of the catadioptric vision system

2.1 Experimental setup

Figure 1 shows a schematic overview of the experimental setup for the proposed panoramic 3D reconstruction system, which consists of two parabolic mirrors, a Marlin F-080 CCD camera, a composite lens (L), and a Sony VPL-ES5 3LCD light projector.

Figure 1.

Schematic drawing of the experimental setup for the proposed panoramic 3D reconstruction system.

First, a catadioptric camera was assembled, aligning the mirror (PM1) with the CCD camera in order to capture the full environment that is projected onto the mirror. Using a chessboard calibration pattern, we calibrate the catadioptric camera obtaining the intrinsic and extrinsic parameters. Once the camera is calibrated, we design the optical setup of the pattern projection system, placing the mirrors (PM1, PM2) back-to-back. In front of the projector, a composite lens (L) was inserted in order to reduce the projected image towards the mirror (PM2), as seen in Figure 2.

Figure 2.

Experimental setup for the panoramic 3D reconstruction system. The system consists of a Marlin F-080 CCD camera, two parabolic mirrors (PM1 and PM2) on a back-to-back configuration, and a Sony VPL-ES5 3LCD light Projector with a composite lens (L).

We focused our work on two areas: calibration and 3D reconstruction. For calibration, we used a plane with a printed calibration pattern and a second calibration pattern projected by the PM2-projector system. The printed pattern was used to calibrate the camera-PM1 system, while the projected pattern was used to calibrate the PM2-projector, and to calculate the rotation and translation matrix between the PM1 and PM2 mirrors. For reconstruction, we performed matching between the projected image and the acquired image. For matching, we used the epipolar curves constraints. Finally, we use three objects to evaluate the reconstruction system: two perpendicular planes, a cylinder, and a semi-sphere.

2.2 Geometrical model description for the catadioptric camera calibration

In order to calibrate the catadioptric camera, we use the geometric model proposed in [18]. To solve this model, we use a chessboard pattern to obtain the intrinsic parameters: mirror curvature and image centre, as well as the extrinsic parameters: rotation and translation between the camera reference system and the pattern.

The general model for the catadioptric camera has two reference systems: the camera image plane, represented by ( $u^{'}$ , $v^{'}$ ), and the sensor plane, represented by ( $u^{″}$ , $v^{″}$ ). Let X be a point in the scene, then we assume that $u^{″} = {[u^{″}, v^{″}]}^{T}$ is the projection of X onto the sensor plane and $u^{'} = {[u^{'}, v^{'}]}^{T}$ is the image in the camera plane (see Figure 3). As described in [23], both systems are related by an affine transformation $u^{″} = A u^{'} + t$ $A ∊ ℜ^{2 x 2}$ y $t ∊ ℜ^{2 x 1}$ .

An imaging function g relates a point $u^{″}$ in the sensor plane with the vector p = [ $u^{″}$ , $v^{″}$ , $w^{″}$ ] coming from the viewpoint O toward a scene point X (see Figure 3). The relation between a point in pixels $u^{'}$ and a point in the scene X is given by eq. (1).

λ \cdot p = λ \cdot g (u^{″}) = λ \cdot g (A u^{'} + t) = P X

(1)

Figure 3.

General model for the catadioptric camera. A point X in the scene is projected in the sensor plane ( $u^{″}$ ), in metric coordinates, and in the image plane ( $u^{'}$ ), in pixel coordinates. p is the point X seen from the viewpoint O. R and T are the extrinsic parameters (rotation matrix and translation vector) that relates the camera's coordinate system with the world coordinate system.

where X ∊ ℝ⁴ is represented in homogeneous coordinates and P ∊ ℝ^3x4 is the perspective projection matrix, and λ is an scaling factor. Through camera calibration, we estimate the matrix A, the vector t, and the non-linear function g, such that all the vectors $g (A u^{'} + t)$ satisfy eq. (1). Eq. (2) is proposed for g.

g (u^{″}, v^{″}) = {(u^{″}, v^{″}, f (u^{″}, v^{″}))}^{T}

(2)

The function f depends on $u^{″}$ and $v^{″}$ only and can be described in polynomial form as shown in Eq. (3).

f (u^{″}, v^{″}) = a_{0} + a_{1} ρ^{″} + a_{2} {ρ^{″}}^{2} + \dots + a_{N} {ρ^{″}}^{N}

(3)

where $ρ^{″} = \sqrt{u^{' 2} + v^{' 2}}$ . The coefficients a_i, i = 0, 1, 2, …, N and the polynomial degree N are determined by the calibration process.

Thus, eq. (1) can be rewritten as eq. (4).

\begin{array}{l} λ \cdot [\begin{matrix} u^{″} \\ v^{″} \\ w^{″} \end{matrix}] = λ \cdot g (A u^{'} + t) \\ = λ [\begin{matrix} (A u^{″} + t) \\ f (u^{″}, v^{″}) \end{matrix}] \\ = P \cdot X, λ > 0 \end{array}

(4)

For cameras with parabolic mirrors we use eq. (5) as proposed in [25]. This simplification allows us to assume a₁ = 0, and rewriting eq. (3) we obtain eq. (6).

{\frac{d f}{d ρ} |}_{ρ = 0} = 0

(5)

\begin{array}{l} f (u^{″}, v^{″}) = a_{0} + a_{2} {ρ^{″}}^{2} + \dots + a_{N} {ρ^{″}}^{N} \end{array}

(6)

From the camera's calibration process, the parameters [A, t, a₀, a₂, …, a_N] are estimated with an iterative procedure. Parameters A and t describe an affine transformation that relates the sensor plane with the image plane of the camera. A is the rotation matrix and t is a translation vector between I_c and O_c in the image plane of the camera (see Figure 3).

According to this, we assume that A = I, and t = 0, therefore $u^{″}$ = $u^{'}$ . Replacing this relation in eq. (4) and using eq. (6), we get the projection shown in eq. (7).

\begin{array}{l} λ \cdot [\begin{matrix} u^{″} \\ v^{″} \\ w^{″} \end{matrix}] = λ \cdot g (u') \\ = λ \cdot [\begin{matrix} u^{'} \\ v^{'} \\ f (ρ^{'}) \end{matrix}] \\ = λ \cdot [\begin{matrix} u^{'} \\ v^{'} \\ a_{0} + a_{2} {ρ^{'}}^{2} + \dots + a_{N} {ρ^{'}}^{N} \end{matrix}] \\ = P \cdot X λ > 0 \end{array}

(7)

where $u^{'}$ and $v^{'}$ are the pixel coordinates of an image, λ is the scale factor, and $ρ^{'}$ is the Euclidean distance between $u^{'}$ and $v^{'}$ . Accordingly, we only need to estimate the parameters a₀, a₂, …, a_N.

To perform the calibration, we used a printed chessboard pattern in different positions, which relates the camera's coordinate system with the world W through a rotation matrix R_W→PM1 and a translation vector T_W→PM1, called extrinsic parameters. Let Iⁱ be the calibration pattern image, X_W = [x, y, z] representing the 3D coordinates of a point in the calibration plane reference system, and m_ij = [u_ij, v_ij] are the pixel coordinates in the image plane. Considering a calibration pattern with a flatness shown in Figure 4, we assume that z_ij = 0. Consequently, we obtain eq. (8).

\begin{array}{l} λ \cdot p_{i j} = λ_{i j} \cdot [\begin{matrix} u_{i j} \\ v_{i j} \\ a_{0} + \dots + a_{N} ρ_{i j}^{N} \end{matrix}] = P^{i} \cdot X_{W} \\ = [\begin{matrix} r_{1}^{i} & r_{2}^{i} & r_{3}^{i} & T_{_{W \to PM1}}^{i} \end{matrix}] \cdot [\begin{matrix} x_{i j} \\ y_{i j} \\ 0 \\ 1 \end{matrix}] \\ = [\begin{matrix} r_{1}^{i} & r_{2}^{i} & T_{_{W \to PM1}}^{i} \end{matrix}] \cdot [\begin{matrix} x_{i j} \\ y_{i j} \\ 1 \end{matrix}] \end{array}

(8)

Figure 4.

Calibration pattern flatness (in mm) obtained using the Mitutoyo BJ1015 Coordinate measuring machine. This calibration pattern was used to calibrate the catadioptric camera (CCD-PM1) and the projection system (PM2).

For the calibration process, we obtain the extrinsic parameters for each image that indicate the position of the calibration pattern with respect to the camera reference system.

To eliminate the depth scale dependency λ_ij, both sides of eq. (8) are multiplied vectorially by p_ij resulting in eq. (9).

\begin{array}{l} λ_{i j} \cdot p_{i j} \times p_{i j} = p_{i j} \times [\begin{matrix} r_{1}^{i} & r_{2}^{i} & T_{_{W \to PM1}}^{i} \end{matrix}] \cdot [\begin{matrix} x_{i j} \\ y_{i j} \\ 1 \end{matrix}] = 0 \Rightarrow \\ [\begin{matrix} u_{i j} \\ v_{i j} \\ a_{0} + \dots + a_{N} ρ_{i j}^{N} \end{matrix}] \times [\begin{matrix} r_{1}^{i} & r_{2}^{i} & T_{_{W \to PM1}}^{i} \end{matrix}] \cdot [\begin{matrix} x_{i j} \\ y_{i j} \\ 1 \end{matrix}] = 0 \end{array}

(9)

From eq. (9), for each point p_ij of the calibration pattern, we obtain the three homogeneous equations:

v_{j} \cdot q_{3} - f (ρ_{j}) \cdot q_{2} = 0

(10.1)

f (ρ_{j}) \cdot q_{1} - u_{j} \cdot q_{3} = 0

(10.2)

u_{j} \cdot q_{2} - v_{j} - q_{1} = 0

(10.3)

where x_j, y_j and z_j and u_j, v_j are known, and correspond to the mirror points and image points, respectively. q₁, q₂ and q₃ are given by:

\begin{array}{l} q_{1} = (r_{11} x_{j} + r_{12} y_{j} + t_{1}) \\ q_{2} = (r_{21} x_{j} + r_{22} y_{j} + t_{2}) \\ q_{3} = (r_{31} x_{j} + r_{32} y_{j} + t_{3}) \end{array}

Eq. (10.3) is linear and has the unknown parameters r₁₁, r₁₂, r₂₁, r₂₂, t₁, t₂. Rewriting eq. (10.3) for L points of the calibration pattern as a system of linear equations, we get eq. (11).

M \cdot H = 0

(11)

where:

\begin{array}{c} H = {[r_{11}, r_{12}, r_{21,} r_{22}, t_{1}, t_{2}]}^{T} \\ M = [\begin{matrix} - v_{1} x_{1} & - v_{1} y_{1} & u_{1} x_{1} & u_{1} y_{1} & - v_{1} & u_{1} \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ - v_{L} x_{L} & - v_{L} y_{L} & - u_{L} x_{L} & u_{L} y_{L} & - v_{L} & u_{_{L}} \end{matrix}] \end{array}

(12)

Using singular value decomposition (SVD), we can represent M = UΣV^T, the solution to the system can be found in the singular vector v₆ associated with the least singular vector σ₆.

Once eq. (10.3) is solved to find the camera extrinsic parameters, we substitute the estimated values into the equations (10.1) and (10.2), and solve for the camera intrinsic parameters a₀,a₂, …, a_N that describe the shape of the imaging function g.

Equations (10.1) and (10.2) are rewritten again as a linear system of equations. But now, we incorporate all the k calibration pattern observations to obtain eq. (13).

[\begin{matrix} A_{1} & A_{1} ρ_{1}^{2} & \dots & A_{1} ρ_{1}^{N} & - v_{1} & 0 & \dots & 0 \\ C_{1} & C_{1} ρ_{1}^{2} & \dots & C_{1} ρ_{1}^{N} & - u_{1} & 0 & \dots & 0 \\ ⋮ & ⋮ & \dots & ⋮ & ⋮ & ⋮ & \dots & ⋮ \\ A_{k} & A_{k} ρ_{k}^{2} & \dots & A_{k} ρ_{k}^{N} & 0 & 0 & \dots & - v_{k} \\ C_{k} & C_{k} ρ_{k}^{2} & \dots & C_{k} ρ_{k}^{N} & 0 & 0 & \dots & - u_{k} \end{matrix}] \cdot r = B

(13)

where:

\begin{array}{c} A_{i} = r_{21}^{i} x^{i} + r_{22}^{i} y^{i} + t_{2}^{i}, B_{i} = v_{i} \cdot (r_{31}^{i} x^{i} + r_{32}^{i} y^{i}), C_{i} = r_{11}^{i} x^{i} + r_{12}^{i} y^{i} + t_{1}^{i}, D_{i} = u^{i} \cdot (r_{31}^{i} x^{i} + r_{32}^{i} x^{i}) . \\ r = {[a_{0} a_{2} \dots a_{N} t_{3}^{1} t_{3}^{2} \dots t_{3}^{k}]}^{T} \\ B = {[B_{1} D_{1} \dots B_{k} D_{k}]}^{T} \end{array}

The solution of the system is obtained using the pseudoinverse matrix. Thus, the intrinsic parameters a₀,a₂, …, a_N which describe the model, are now known. In order to compute the polynomial degree N, we actually start from N=2, then we increase N by unitary steps and we compute the average value of the reprojection error of all calibration points. The procedure stops when the minimum error is found.

2.3 Calibration of the catadioptric camera with the light projection system

In order to calibrate the catadioptric camera with the projection system (see Figure 6), we develop the following procedure:

A calibration pattern image was projected (see Figure 5a)) in different positions over the mirror PM2. The mirror PM2 reflects the image until the calibration plane W is found. The calibration plane position is known with respect to PM1, (as seen in Figure 5b)) because the calibration plane contains the printed pattern used to calibrate the camera-PM1 system. Once a catadioptric image is acquired by the camera-PM1, a subpixel algorithm provides the points of the projected calibration pattern. These points are represented as $u_{PM1}^{i}$ .

Figure 5.

a) Projected calibration pattern, b) Image captured by the catadioptric camera showing two calibration patterns in a white plane: the printed pattern is used to calibrate the catadioptric camera (camera CCD-PM1 mirror), and the projected pattern is used to calibrate the projection system (PM2 mirror-projector).

Figure 6.

Schematic diagram of the panoramic 3D reconstruction system. The projections of 3D points X_W on the mirrors (PM1, PM2) are denoted by $x_{PM1}^{i}$ and $x_{PM2}^{i}$ . $u_{PM1}^{i}$ are the image points acquired by the catadioptric camera, and $u_{PM2}^{i}$ are the points of the projected image.

The plane where the projection pattern is located is defined by $\prod_{w}^{i}$ and is calculated with eq. (14) [24].

Π_{w}^{i} = [\begin{matrix} n_{e}^{i} \\ D_{e}^{i} \end{matrix}] = [\begin{matrix} R_{W \to PM1}^{- 1} & 0^{T} \\ - {(R_{W \to PM1}^{- 1} * T_{W \to PM1})}^{T} & 1 \end{matrix}] [\begin{matrix} n^{i} \\ D^{i} \end{matrix}]

(14)

where R_W→PM1 and T_W → PM1 represent the rotation matrix and translation vector respectively between the world and the mirror PM1. 0 denotes the null vector, nⁱ and $n_{e}^{i}$ are unit normal vectors to the plane $\prod_{w}^{i}$ , and finally Dⁱ and $D_{e}^{i}$ denote the distance from the mirror PM1 points to the plane $\prod_{w}^{i}$ .

The points $u_{PM1}^{i}$ are mapped to the plane $\prod_{w}^{i}$ , and are denoted as $X_{PM1}^{i}$ . These points are defined on the mirror PM1. A rigid transformation is required to represent the points $X_{PM1}^{i}$ on the mirror PM2 coordinate system, as described in eq. (15).

X_{PM2}^{i} = R_{PM1 \to PM2} X_{PM1}^{i} + T_{PM1 \to PM2}

(15)

When the points $X_{PM2}^{i}$ are projected onto the mirror PM2, they are represented by $x_{PM2}^{i}$ . Eq. (7) is used to define the points $x_{PM2}^{i}$ in the projector image as $v_{PM2}^{i}$ . However, as the rotation matrix R_PM1→PM2 and the translation vector T_PM1→PM2 are unknown parameters, a function that optimizes the value for these parameters was developed.

This function represented by g2 has known input parameters such as: the points $X_{PM1}^{i}$ , the extracted points $u_{PM2}^{i}$ from the projected pattern (see Figure 5a)), and the unknown parameters: rotation R_PM1→PM2 and translation T_PM1→PM2.

The $v_{PM2}^{i}$ points are used to calculate the error with eq. (16). This error is minimized according to g₂(θ_xθ_yθ_z, t_xt_yt_z) using the Levenberg-Marquardt optimization method in MATLAB®, resulting in the rotation and translation between the camera-mirror (PM1) and the mirror(PM2)-projector.

e = {(u_{PM2}^{i} - v_{PM2}^{i})}^{2}

(16)

2.4 Epipolar geometry for panoramic cameras

Epipolar geometry describes the relationship between positions of corresponding points in a pair of images acquired by central catadioptric cameras [14, 25], as seen in Figure 7. In this section, the alignment parameters (rotation and translation) between the camera-mirror (PM1) and the mirror(PM2)-projector are described.

Figure 7.

Epipolar geometry between two catadioptric cameras with a parabolic mirror. R_PM1→PM2 is the rotation matrix and T_PM1→PM2 the translation vector between both mirrors (PM1–PM2). ∏ is the epipolar plane.

The projections of 3D points X_W on the mirrors (PM1, PM2) are denoted by $x_{PM1}^{i}$ and $x_{PM2}^{i}$ . The coplanarity of vectors $x_{PM1}^{i}$ , $x_{PM2}^{i}$ and T_PM1→PM2 = [tx, ty, tz]^T can be expressed with eq. (17):

{(x_{PM2}^{i})}^{T} R_{PM1 \to PM2} (T_{PM1 \to PM2} \times x_{PM1}^{i}) = 0

(17)

where × denotes the cross product. Simplifying the coplanarity constraint described in eq. (17), we obtain eq. (18):

{(x_{PM2}^{i})}^{T} E x_{PM1}^{i} = 0

(18)

where:

E = R_{PM1 \to PM2} {[T]}_{x PM1 \to PM2}

(19)

E is the essential matrix and [T]_xPM1→PM2 is the antisymmetric matrix given by eq. (20):

{[T]}_{x PM1 \to PM2} = [\begin{matrix} 0 & - t_{z} & t_{y} \\ t_{z} & 0 & - t_{x} \\ - t_{y} & t_{x} & 0 \end{matrix}]

(20)

To each point u₁ in one image, an epipolar conic is uniquely assigned in the other image u₂. To find the corresponding points between both images, we use eq. (21):

u_{2}^{T} A_{2} (E, u_{1}) u_{2} = 0

(21)

In a general case, the matrix A₂(E, u₁) is a nonlinear function of the essential matrix E, the point u₁, and the calibration parameters of the catadioptric camera.

The vector T_PM1→PM2 defines the epipolar plane ∏. Hence, the normal vector of the plane ∏ expressed in the camera's coordinates system is given by eq. (22):

n_{1} = T_{PM1 \to PM2} \times x_{PM1}^{i}

(22)

The normal vector n₁ can be expressed in the projector's coordinates system using the essential matrix resulting in eq. (23):

\begin{array}{l} n_{2} = R_{PM1 \to PM2} n_{1} \\ = R_{PM1 \to PM2} (T_{PM1 \to PM2} \times x_{PM1}^{i}) \\ = R_{PM1 \to PM2} {[T]}_{x PM1 \to PM2} x_{PM1}^{i} \\ = E x_{PM1}^{i} \end{array}

(23)

Thus, eq. (23) is denoted by eq. (24) as:

n_{2} = {[p, q, s]}^{T}

(24)

we can write the equation of the plane ∏ in the projector's coordinate system resulting in eq. (25):

p x + q y + s z = 0

(25)

The derivation of A₂(E, u₁) from eq. (21) based on a parabolic mirror, z is substituted into eq. (25) resulting in eq. (26) that allows the calculation of the epipolar conics for this type of mirror:

s x^{2} + 2 b_{2} p x + s y^{2} + 2 b_{2} q y - s b^{2} = 0

(26)

where b₂ is the mirror parameter and p, q, s are defined by eq. (24).

2.5 3D reconstruction

Knowing the rotation R_PM1→PM2 and translation T_PM1→PM2 between both mirrors, we can find the relationship between the pixel coordinates in the image and the angles defined in the parabolic mirror reference system (see Figure 8). Having this information, 3D point coordinates are calculated from the correspondence of points in the projected image and the acquired image by the catadioptric camera [14].

Figure 8.

ω₁ and ω₂ are the angles between the points $x_{PM1}^{i}$ and $x_{PM2}^{i}$ , |D| is the distance between these points. X are the reconstructed 3D points.

Given two points $x_{PM1}^{i}$ and $x_{PM2}^{i}$ on the surfaces of both mirrors PM1 and PM2, and the coordinates [x, y, z] from a point $x_{PM2}^{i}$ , we have eq. (27):

\begin{array}{l} ϕ_{1} = \arctan (\frac{y}{x}) \\ θ_{1} = \arctan (\frac{\sqrt{x^{2} + y^{2}}}{z}) \\ ω_{1} = \arccos (\frac{{(x_{PM1}^{i})}^{T} (- R_{PM1 \to PM2} T_{PM1 \to PM2})}{| {(x_{PM1}^{i})}^{T} | | T_{PM1 \to PM2}_{} |}) \\ ω_{2} = \arccos (\frac{{(x_{PM2}^{i})}^{T} T_{PM1 \to PM2}}{| {(x_{PM2}^{i})}^{T} | | T_{PM1 \to PM2}_{} |}) \end{array}

(27)

From the distance |D| between two points $x_{PM1}^{i}$ and $x_{PM2}^{i}$ , the coordinates of a point X are determined with eq. (28).

\begin{array}{l} | D | = \frac{| T_{PM1 \to PM2} | \sin ω_{2}}{\sin (ω_{1} + ω_{2})} \\ X = [\begin{matrix} | D | \sin θ_{1} \cos ϕ_{1} \\ | D | \sin θ_{1} \sin ϕ_{1} \\ | D | \cos θ_{1} \end{matrix}] \end{array}

(28)

3. Results

3.1 Calibration

We used 26 images to calibrate the catadioptric camera, some of these are shown in Figure 9. As we can see, the white plane ( $\prod_{w}^{i}$ ) contains two calibration patterns: one printed and the other projected.

Figure 9.

Calibration images. Twenty-six images were used to calibrate the catadioptric camera. Each image contains a white plane with the two calibration patterns in different positions and orientations.

The printed pattern is used to calibrate the catadioptric camera formed by the CCD camera and the PM1 mirror, and to compute the plane $\prod_{w}^{i}$ . Once the camera-PM1 setup is calibrated, the projected pattern is mapped to the mirror and then to plane $\prod_{w}^{i}$ . This pattern is used to compute the rotation and translation between the mirrors PM1 and PM2, as explained in section 2.3, and to calibrate the PM2-projector setup.

The intrinsic parameters from the catadioptric camera are presented in Table 1. The values of the mirror's curvature allow us to confirm that it is a parabola.

Table 1.

Intrinsic parameters of the camera-PM1

Polynomial coefficients f (ρ)					Image centre
a ₀	a ₁	a ₂	a ₃	a ₄	x _c	y _c
−147.8362	0.0000	0.0033	0.0000	0.0000	542.3271	702.6857

The intrinsic parameters obtained for the PM2-projector system are presented in Table 2.

Table 2.

Intrinsic parameters of the PM2-projector

Polynomial coefficients f (ρ)					Image centre
$a_{0}^{'}$	$a_{1}^{'}$	$a_{2}^{'}$	$a_{3}^{'}$	$a_{4}^{'}$	x _c	y _c
55.1000	0	-0.0056	0.0000	0.0000	575.0802	499.7993

The rotation and translation matrix between the mirror PM1 and the mirror PM2 as described in section 2.4, correspond to the values shown in eq. (29).

\begin{array}{l} R = [\begin{matrix} - 0.3924 & 0.9184 & 0.0505 \\ - 0.8897 & - 0.3929 & 0.2324 \\ 0.2333 & 0.0463 & 0.9713 \end{matrix}] \\ {[T]}_{x} = [\begin{matrix} 0 & - 108.2657 & - 0.4395 \\ 108.2657 & 0 & - 44.9961 \\ 0.4395 & 44.9961 & 0 \end{matrix}] \end{array}

(29)

When building the optical setup for the panoramic 3D reconstruction system, we carefully aligned the PM1 and PM2 mirrors. However, as can be seen in the rotation matrix R, a small misalignment exists.

To calculate the calibration error in the camera-mirror PM1 system, we considered the position of the 3D points on the printed calibration pattern in the catadioptric image. With the rotation and translation parameters, these points are projected towards the PM1 mirror, using the intrinsic parameters (shown in Table 1) that are mapped to the image. This way, we compared the points from the printed pattern projected to the catadioptric image with the points extracted from the acquired image by the camera.

The calibration error in the catadioptric projection system (PM2-projector) is obtained as follows: we extract the positions of the 3D points from the projected calibration pattern in the catadioptric image, and then these points are projected towards the mirror PM1 and are represented in the plane $\prod_{w}^{i}$ . With the 3D points in the plane, we calculate the rotation and translation matrix between the PM1 and PM2 mirrors in order to represent the points in the PM2 mirror reference system, using the intrinsic parameters (shown in Table 2) that are mapped towards the projector. Finally, we compared these points from the projected pattern to obtain the error.

The calibration error from the catadioptric camera (camera-PM1) is about 0.4 pixels, and the calibration error from the catadioptric projection system (PM2-proyector) is 1.56 pixels.

3.2 Angle of view

Figure 10 shows the direction of T₁ as function of the camera focus (F₁) and the sensor size (L₁). T₁ is defined by eq. (30):

T_{1} = λ [\begin{matrix} F_{1} \\ L_{1} \end{matrix}] = [\begin{matrix} Z_{1} \\ ρ_{1}^{m} \end{matrix}]

(30)

where $Z_{1} = a_{0} + a_{2} {(ρ_{1}^{m})}^{2}$ , $ρ_{1}^{m}$ corresponds to the largest radius of the mirror that can be seen by the camera. λ is a scale factor, a₀, a₂ are the polynomial coefficients that describe the PM1 mirror curvature.

Figure 10.

Real dimensions of the experimental setup. L₁, F₁, L₂ and F₂ are shown at a larger size for visualization purposes. L₁, F₁, L₂ and F₂ are the sensor size, camera focus, display size and projector focus, respectively.

There are two solutions for $ρ_{1}^{m}$ and λ, the solution associated with our experimental setup is eq. (31):

\begin{matrix} λ = \frac{F_{1} - \sqrt{F_{1}^{2} - 4 a_{0} a_{2} L_{1}^{2}}}{(2 a_{2} L_{1}^{2})} & , & ρ_{1}^{m} = \frac{F_{1} - \sqrt{F_{1}^{2} - 4 a_{0} a_{2} L_{1}^{2}}}{(2 a_{2} L_{1})} \end{matrix}

(31)

The normal unit vector of the mirror in (Z₁, $ρ_{1}^{m}$ ) is given by:

n_{1} = \frac{N_{1}}{\sqrt{N_{1}^{T} N_{1}}}

where:

N_{1} = [\begin{matrix} 2 a_{2} ρ_{1}^{m} \\ - 1 \end{matrix}]

Considering that the angle formed by T₁, N₁ is the same as the one formed by N₁, S₁, we obtain the eq. (32):

S_{1} = 2 (T_{1}^{T} n_{1}) n_{1} - T_{1};

(32)

Approximating R_PM1→PM2 = I, we have the eq. (33):

S_{2} = 2 (T_{2}^{T} n_{2}) n_{2} - T_{2};

(33)

where:

\begin{matrix} n_{2} = \frac{N_{2}}{\sqrt{N_{2}^{T} N_{2}}}, & N_{2} = [\begin{matrix} - 2 {a^{'}}_{2} ρ_{2}^{m} \\ - 1 \end{matrix}] & and T_{2} = [\begin{matrix} - a_{0^{'}} - a_{2^{'}} {(ρ_{2}^{m})}^{2} \\ ρ_{2}^{m} \end{matrix}] \end{matrix}

$ρ_{2}^{m} = \frac{F_{2} - \sqrt{F_{2}^{2} - 4 a_{0}^{'} a_{2}^{'} L_{2}^{2}}}{(2 a_{2}^{'} L_{2})}$ , L₂ and F₂ are the display size and focus of the projector, while $a_{0}^{'}$ and $a_{2}^{'}$ are the polynomial coefficients that model the curvature of the mirror PM2.

The angle of view θ of the reconstruction system is given by:

θ = \cos^{- 1} (\frac{S_{1}^{T} S_{2}}{\sqrt{S_{1}^{T} S_{1}} \sqrt{S_{2}^{T} S_{2}}})

Substituting the values of the camera, projector and the mirrors PM1 and PM2, we obtain θ = 125.4125 degrees.

3.3 Reconstruction

In order to test the functionality of the system, we reconstruct the object shown in Figure 11 containing two perpendicular planes.

Figure 11.

Test object 1: Perpendicular planes. A set of two perpendicular planes was used to test the functionality of the system.

The perpendicularity of the two planes was measured using the Mitutoyo BJ1015 Coordinate-measuring machine and Metrolog XG® software (see Figure 12) obtaining an angle of 89.192 degrees between them.

Figure 12.

Measuring perpendicularity of the two planes using the Mitutoyo BJ1015 Coordinate-measuring machine.

To perform the reconstruction, we projected a pattern of points onto the perpendicular planes (see Figure 13), and then we acquired an image from this projection using the catadioptric camera (see Figure 14). The projected pattern points have random colours. We perform the correspondence between the acquired image and the projected image using the colour of the projected points and the epipolar curves constraints (described in section 2.4).

Figure 13.

Projected pattern. The pattern projected by the PM2-projector system over the perpendicular planes consists of a set of points with random colours.

Figure 14.

Catadioptric image with the points projected on the perpendicular planes, captured by the camera CCD-mirror PM1.

Epipolar geometry from the 3D reconstruction system is shown in Figure 15. Here we can see that the conic lines obtained with eq. (26) intersect in the epipole and passed through the corresponding points. Once the correspondence is found, we build the 3D reconstruction of the object. The reconstructed planes can be seen in Figure 16.

Figure 15.

Epipolar geometry. Coloured projected points with their corresponding conics (epipolar lines).

Figure 16.

3D reconstruction of the perpendicular planes.

Having previous knowledge about the object formed by the two planes, we segmented the reconstructed points in the objects. Each object corresponds to one of the two perpendicular planes ∏₁ and ∏₂. Then, we calculate the error, considering the perpendicular distance d_i between the points and the corresponding plane using eq. (34).

\tilde{e} = \frac{1}{n} \sum_{i = 1, p_{i} ∊ Π_{j}}^{n} d_{i} (p_{i}, Π_{j})

(34)

In Figure 17, the perpendicular distance in millimetres from each point with the corresponding plane is shown. The graph shows a mean error of 2.4 mm on plane 1 and 3.6 mm on plane 2. The angle between the two planes is 89.8488 degrees.

Figure 17.

Reconstruction error in perpendicular planes. Perpendicular distance error in millimetres of the points in plane one (red) and plane two (blue).

We performed two additional reconstructions test using a cylindrical object (see Figure 18) and a semi-sphere (see Figure 19).

Figure 18.

Test object 2: Cylindrical object. A section of a plastic tube was used to test the functionality of the system with curved-shape objects.

Figure 19.

Test object 3: A semi-sphere was used to test the functionality of the system.

A pattern of points with random colours was projected inside the cylinder to obtain the results shown in Figure 20.

Figure 20.

3D reconstruction of the plastic tube.

In order to know the real values of the cylinder, we perform a curve fitting with the 3D reconstructed points using Metrolog XG® software. The results indicate an error of 4.7657 mm (see Table 3) with respect to the real object radius.

Table 3.

Curve fitting results of reconstructed 3D points using Metrolog XG® software

Results
Real radius[mm]	Radius [mm]	Error [mm]
115.79	111.02	4.7657

The reconstruction of the semi-sphere shown in Figure 21.

Figure 21.

3D reconstruction of the semi-sphere.

4. Conclusions and future work

The panoramic 3D reconstruction systems found in the literature are based on two catadioptric cameras. Thus, the 3D reconstruction is based on the matching of corresponding points between two views. However, these methods fail in applications with textureless objects. In this work, we proposed a new reconstruction system based on a single catadioptric camera and a catadioptric projection system. The reconstruction can be done with a single acquisition or with multiple acquisitions moving the position of the projected points. For this, we first calibrate the catadioptric vision system, and then we calibrate the catadioptric projector. The reconstruction error in an object composed by two perpendicular planes was 2.4 mm on the first plane, and 3.6 mm on the second plane, with an angle of 89.8488 degrees between planes. The reconstruction error in the cylindrical object was 4.7 mm. Our future work will focus on obtaining denser 3D reconstructions and characterization of reconstruction uncertainty. Moreover, increasing the reconstruction area by moving the system inside the object.

Footnotes

5.

The authors wish to acknowledge the financial support for this work by the Consejo Nacional de Ciencia y Tecnología (CONACYT) through project SEP-2005-O1-51004/25293 and the financial support scholarship number 256319, and by the Instituto Politécnico Nacional through project SIP-20130165.

References

Pollefeys

Van Gool

Vergauwen

Cornelis

Verbiest

Tops

(2001) Image-based 3D Acquisition of Archaeological Heritage and Applications, Proceedings of VAST, 255–262.

Earl

, (2013) Modeling in Archaeology: Computer Graphic and Other Digital Pasts. Perspectives on Science, vol.21, no.2, 226–244.

Muller

Smolic

Drose

Voigt

Wiegand

(2005) 3-D reconstruction of a dynamic environment with a fully calibrated background for traffic scenes, IEEE Trans. Circuits Syst. Video Technol., vol.15, no.4, 538–549.

Kim

(2006) Stereoscopic video surveillance system for detection of target's 3D location coordinates and moving trajectories. Optics Communications, vol.266, no.1, 67–79.

Bascle

Hampel

Navab

Baxter

Zhang

(2001) An industrial 3D graphical interface for process control. IEEE Symposium on Emerging Technologies and Factory Automation, ETFA, 559–562.

Dogar

, (2010) Integrating 3D quality control function into an automated visual inspection system for manufacturing industry. University Politehnica of Bucharest Scientific Bulletin, Series C: Electrical Engineering, 72(2), 63–76.

Zachow

Zilske

Hege

, (2007) 3D reconstruction of individual anatomy from medical image data: Segmentation and geometry processing. In 25. ANSYS Conference, CADFEM Users, Meeting, Techn. Report: ZR 07–41, Zuse Institute Berlin.

Morgado

Silveira

Marques

(2013) Extending local binary patterns to 3D for the diagnosis of Alzheimer's Disease. Proceedings - International Symposium on Biomedical Imaging, 17–120.

Shang

Lee

Hwang-bo

Lee

Kim

(2012) 3D shape reconstruction system for dielectric material specimens surface measurement. Condition Monitoring and Diagnosis (CMD), International Conference on Digital Object, 332–335.

10.

Furferi

Governi

Volpe

Carfagni

(2013) Design and assessment of a machine vision system for automatic vehicle wheel alignment. International Journal of Advanced Robotic Systems, vol.10, no. 242.

11.

Gledhill

Yun

Taylor

Clarke

(2004) 3D Reconstruction of a Region of Interest Using Structured Light and Stereo Panoramic Images. IEEE Proceedings of the Eighth International Conference on Information Visualisation, 1007–1012.

12.

Sang

(2008) Simultaneous calibration of a structured light-based catadioptric stereo sensor. Ph.D Thesis, Universitat de Girona.

13.

Zhou

Peng

Cui

Wang

Tan

(2013) A novel laser vision sensor for omnidirectional 3D measurement. Optics & Laser Technology, 1–12.

14.

Gonzalez-Barbosa

Lacroix

(2005) Fast dense panoramic stereovision. IEEE International Conference on Robotics and Automation, 1210–1215.

15.

Mei

Rives

(2007) Single View Point Omnidirectional Camera Calibration from Planar Grids. IEEE International Conference on Robotics and Automation, 3945–3950.

16.

Puig

Bastanlar

Sturm

Guerrero

J.J

Barreto

(2011) Calibration of Central Catadioptric Cameras Using a DLT-like Approach. International Journal of Computer Vision, 1–14.

17.

Barreto

Araujo

(2005) Geometric Properties of Central Catadioptric Line Images and Their Application in Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, 1327–1333.

18.

Scaramuzza

Martinelli

Siegwart

(2006) A toolbox for easily calibrating omnidirectional cameras. IEEE International Conference on Intelligent Robots and Systems, 1327–1333.

19.

Furukawa

Kawasaki

(2005) Dense 3D reconstruction with an uncalibrated stereo system using coded structured light. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, 107–107.

20.

Zhang

Huang

(2006) High-resolution, real-time three-dimensional shape measurement. Optical Engineering, vol. 45.

21.

Chen

(2008) Vision Processing for Real-time 3-D Data Acquisition Based on Coded Structured Light. IEEE Transactions on Image Processing, vol.45.

22.

Zhang

(2010) Recent progresses on real-time 3D shape measurement using digital fringe projection techniques. Optics and Lasers in Engineering, vol.48, 149–158.

23.

Micusik

Pajdla

(2004) Para-catadioptric Camera Auto-calibration from Epipolar Geometry. Proceedings of the Asian Conference on Computer Vision.

24.

Garcia-Moreno

Gonzalez-Barbosa

(2012) GPS Precision Time Stamping for the HDL-64E Lidar Sensor and Data Fusion, Electronics, Robotics and Automotive Mechanics Conference (CERMA), IEEE. 48–53.

25.

Svoboda

Pajdla

(2002) Epipolar Geometry for Central Catadioptric Cameras. International Journal of Computer Vision, vol.45, no.1, 23–37.