Three-dimensional simulation method of fish-eye ... - OSA Publishing

Research Article

Vol. 32, No. 7 / July 2015 / Journal of the Optical Society of America A

1337

Three-dimensional simulation method of fish-eye lens distortion for a vehicle backup rear-view camera DAEHEE KIM

AND

JOONKI PAIK*

The Graduate School of Advanced Imaging Science, Multimedia and Film, Chung-Ang University, 84 Heuseok-Ro, Dongjak-Gu, Seoul, South Korea *Corresponding author: [email protected] Received 3 April 2015; revised 20 May 2015; accepted 29 May 2015; posted 29 May 2015 (Doc. ID 237115); published 23 June 2015

Recently, various cameras have been embedded in vehicles for driver safety and convenience. In this context, a backup rear-view camera has attracted increasing attention in helping drivers’ parking convenience. Preinstallation of a rear-view camera requires the calibration of a wide-angle lens, such as a fish-eye lens, and the registration of guidelines to the three-dimensional (3D) scene. The proposed method provides a novel simulation method for the optical distortion of a wide-angle lens in a vehicle rear-view camera. The proposed method consists of three steps: (i) generation of the 3D virtual space, (ii) field number-based viewing angle estimation, and (iii) distorted image generation in the 3D space. The major contribution of this work is the lens specification-based simulation of 3D distortion for accurate and efficient preinstallation of vehicle rear-view cameras. The proposed simulation method can also be used to design a general optical imaging system for intelligent surveillance and medical imaging. © 2015 Optical Society of America OCIS codes: (100.2000) Digital image processing; (330.4595) Optical effects on vision; (330.7326) Visual optics, modeling. http://dx.doi.org/10.1364/JOSAA.32.001337

1. INTRODUCTION For the past few years, vehicles have been equipped with various cameras for driver safety, convenience, and video event recording to provide extended visual information. As the demand of a vehicle camera increases, the related research has become popular, such as lane detection [1,2], video event data recording, and inattentive drive monitoring [3]. Recently, the National Highway Traffic Safety Administration (NHTSA) has announced the condition of testing the installation status of rear-view cameras. In the NHTSA legislation, the rear-view camera should be able to represent an object in the front by 30 cm and side by 150 cm. However, a normal-angle camera cannot satisfy the test conditions because of a limited angle of view. Therefore, it is practically impossible to enforce the legislation without using a very wide-angle or fish-eye lens camera. A vehicle rear-view camera can prevent both minor and fatal backward collisions by displaying all the necessary rear-view information on the built-in monitor. In this context, a wide-angle or fish-eye lens is a natural choice for such rear-view cameras because it can acquire a very wide field of view (FOV). In order to employ a fish-eye lens camera in a vehicle, the position and viewing angle of the camera should be accurately controlled based on the lens specifications. Since the after-installation 1084-7529/15/071337-07$15/0$15.00 © 2015 Optical Society of America

test is time consuming and entails a high cost, an accurate preinstallation simulation is very important to utilize advanced consumer imaging device technology in the automotive industry. However, major concerns of recent researchers include the detection of rear obstacles [4,5], top-view transformation [6], and ground surface detection [7]. For example, optical and lighting simulation (OPTIS) provided digital vision and surveillance (DVS) [8]. However, DVS is expensive and cannot simulate a nonuniformly split lens. In this paper, a three-dimensional (3D) simulation method of fish-eye lens distortion for a vehicle rear-view camera is presented. The proposed method is to analyze the shape of the lens distortion using lens specifications. In addition, the analysis result can be used to estimate the distortion angle of each pixel. The angle is determined in the form of a geometrically distorted image of an object in 3D space. After generating the 3D virtual space using an open software library, the distortion angle of each pixel is estimated by cross-checking the grid and poles in the distorted virtual space using lens specifications including the image size, upper bound of the angle, and the field number. The input image is finally registered into the 3D distorted scene as shown in Fig. 1. This paper is organized as follows. Section 2 presents the 3D virtual space generation and distortion methods. The proposed

1338


Research Article

rd

N X

κn r nu κ 1 r u κ 2 r 2u κ N r N u ;

(1)

n1

Fig. 1. Block diagram of the proposed fish-eye simulation method.

where κ n , n 1; …; N represent the distortion coefficients, and N ≤ 5. This model is independent of the underlying fish-eye mapping function. The perspective model is based on a rectilinear pinhole camera, where the radius of an undistorted pixel is determined by r u f tanθ;

simulator is presented in Section 3. After providing experimental results in Section 4, Section 5 concludes the paper. 2. SIMULATION OF A FISH-EYE LENS IMAGE USING 3D VIRTUAL DATA In order to solve the problems of existing 2D simulation methods, a novel 3D virtual space generation method is presented for accurately simulating fish-eye lens images. Lens specifications are needed to generate the 3D virtual space. In the real simulation, the angle range of the lens view in the range is used, which produces the most similar FOV to the human vision. The generation process starts by detecting the angle of each pixel based on the given lens specification. We then detect intersections of poles in the 3D virtual space and radially distorted infinite lines. As a result, we obtain the 3D virtual space. A. Mathematical Models for Radial Distortion: A Review

A wide-angle lens produces geometric distortion in the radial direction called the barrel distortion since it compresses the peripheral region to contain a wide angle of view in the image plane. Considering this problem, many researchers have proposed various models to correct the barrel distortion of the wide-angle lens. A two-dimensional (2D) approximated barrel distortion model is shown in Fig. 2 [9,10], where an original pixel P u moves toward the center at P d along the radial direction in the image plane. A polynomial model was proposed to approximate various types of wide-angle lenses using the distortion coefficients. The distance of the distorted pixel P u is determined by the polynomial equation

(2)

where f is the distance between the principal point and the image plane, called the focal length of the fish-eye camera, and θ is the incident angle between a light ray and the optical axis. In an equidistant projection of the perspective model, the radial distance is directly proportional to the incident angle in the image plane. The projection function of the perspective model is determined by r d f θ:

(3)

The perspective model of radial distortion is derived from Eqs. (2) and (3) using −1 r u : (4) r d f tan f f is not the same as the actual focal length of the fish-eye lens [11]. The inverse of Eq. (4) can be easily obtained by r (5) r u f tan d ; f These equations describe the conversion between projection image space and fish-eye image space. The fish-eye transform model is based on a logarithmic function [12,13], r d s ln1 λr u ;

(6)

where s is a scaling constant, and λ controls the amount of displacement across the image. The inverse of this model is expressed as expr d ∕s − 1 : (7) λ This model uses the characteristic of a fish-eye image that the central region has higher resolution than peripheral regions. The FOV model is described by [14] 1 ω ; (8) r d tan−1 2r u tan ω 2 ru

where ω is the FOV of the fish-eye lens camera, and its inverse is given as ru

Fig. 2. Radial distortion in the 2D imaging plane: O represents the image center, P u , P d ∈ R 2 are pixel coordinate vectors in the input undistorted and output distorted images, respectively, r u , r d ∈ R are the distances of P u and P d from the center, and θ θu θd is the angle of OP u or, equivalently, OP d .

tanr d ω : 2 tanω∕2

(9)

Since existing models define the relationship between the distorted and undistorted pixels commonly based on the 2D imaging plane [15], they are not suitable for simulation in 3D space. More specifically, in order to solve practical vision problems, such as occlusion and distance-based ordering problems, an accurate 3D simulation model is needed. In addition, there are no existing models that incorporate the lens specifications to the best of the authors’ knowledge. The proposed

Research Article


1339

simulation method can accurately simulate the distortion of any types of lenses by predicting the distortion angle of each pixel based on the lens specifications. B. 3D Virtual Space Generation

We used an open graphic library that projects an object in 3D space into a 2D imaging plane to simulate a photo taken from a camera. In order to generate a virtual space by simulating a normal lens, we insert grids and poles that are identical to the real-world test environment, in the 3D simulation. The location and size of the poles are determined based on the NHTSA’s regulation as shown in Fig. 3. The NHTSA test environment standards do not include any conditions for grids but they have been inserted for convenience when it comes to measuring distances. C. Field Number-Based Angle Estimation

In an image acquisition process, specification of a lens determines the amount of refraction, and a distorted image is finally formed according to the extent of the refraction. However, the distortion simulation cannot be considered the many beams of the real world. So, the proposed simulation generates the distorted image using the refraction angle estimation method of each pixel based on the lens field and the line cross-checking method. The proposed method uses the spherical coordinate system in 3D space and the polar coordinate system to estimate the lens field-based refraction angle in the 2D imaging plane. Since the lens is assumed to be spherical, it can be represented by using a spherical coordinate system in 3D space, and the corresponding image can be represented using a polar coordinate system in the 2D imaging plane as shown in Fig. 4. Because a fish-eye lens can be conserved as a part of a hemisphere, it can be represented on a spherical coordinate system while the image plane can be represented on a 2D rectangular coordinate system. Based on the principle that the distortion factor of the fish-eye lens is determined by the field number of the lens, we can obtain the refraction angles θ and φ from each pixel of an image from the lens specification data.

Fig. 4. Lens in the spherical coordinate (top) and the image plane in the polar coordinate (bottom).

φ is determined using the coordinate of each pixel in the imaging plane by φ arctan

yi : xi

(10)

Next, θ can be computed using the field number of the lens. Typically, a lens is divided into 10 to 20 fields, and the field number, denoted as F n , is defined as the cumulative portion of the concentric region. For example, if the FOV is 180°, the number of the field is 10, and the type of lens is distributed, then F 0 0°, F 1 18°,…, and F n 180°. The pixel x i ; y i belonging to the field n is calculated by n

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x 2i y 2i ∕U ;

(11)

where U represents the width of the image. θ is then computed using Eq. (11) as θ F n F n1 − F n ∕U ∕n∕2;

(12)

where U ∕n represents the number of pixels in each field. The reason for using U is that the lens is defined in the horizontal axis. θ is an estimated value using the lens specification of the distortion angle of each pixel, so we can produce an accurately distorted image. D. Generation of a Simulated Fish-eye Lens Image

The fish-eye lens image is generated by checking the linecrossing points. The line cross-checking method uses the virtual ! ! 3D data B 1 B 2 and the line Ol P . The coordinate of the point PX l ; Y l ; Z l on the surface of the lens is first computed using θ and φ as X l r sin θ cos φ; Y l r sin θ sin φ; Z l r cos θ;

Fig. 3. Test environment proposed by NHTSA: (a) top view of the test region with a rear-view camera and pole locations, (b) pole located at 4, 5, 6, and 7, and (c) pole located at 1, 2, and 3. (All numbers are in centimeters.)

(13)

where r is typically set to 1. Next, we detect the crossing point of a line in the virtual 3D ! ! space and Ol P as shown in Fig. 5. As shown in Fig. 5, B 1 B 2 is ! ! defined as a line in 3D space, and Ol P and B 1 B 2 represent linear equations using

1340


Research Article

Fig. 5. Perspective projection model of a fish-eye lens camera.

8 > > x Ox 1 P x 2 − Ox 1 t; ! < Ol P y Oy1 P y2 − Oy1 t; > > : z O P − O t; z1 z2 z1 8 > > x B x 1 B x 2 − B x 1 s; ! < B 1 B 2 y B y1 B y2 − B y1 s; > > : z B B − B s; z1

z2

Fig. 6. Working space of the proposed simulator.

(14)

z1

where s and t represent the parameters of the line, expressed by s

B y1 − Oy1 P x 2 − Ox 1 − B x 1 − Ox 1 P y2 − Oy1 ; B x 2 − B x 1 P y2 − Oy1 − B y2 − B y1 P x 2 − Ox 1 B −O P −O −B −O P −O

t

B x 1 B x 2 − B x 1 Byx1 −Byx1 Pxy2 −Oxy1 −B xy 1 −B yx1P xy2−Oxy1 − Ox 1 2

1

2

1

P x 2 − Ox 1

2

1

2

1

:

(15) If two lines intersect, the equation should be satisfied using Ox 1 P x 2 − Ox 1 t B x 1 B x 2 − B x 1 s; Oy1 P y2 − Oy1 t B y1 B y2 − B y1 s; Oz 1 P z 2 − Oz 1 t B z 1 B z 2 − B z 1 s:

(16)

We validate the cross-checking of two lines by using and pole are not an inEqs. (15) and (16). However, the grid ! finite straight line. So the 0 ≤ s ≤ jB 1 B 2 j should be satisfied. The image of the 3D distortion is made by inserting the specified color in the intersection pixels. 3. INTRODUCTION OF THE FUNCTIONS OF THE PROPOSED SIMULATOR Four main functions of the proposed simulator are described in this section. Figure 6 shows the initial working space of the proposed simulator. The 3D virtual space is displayed on the upper left side of the window, the fish-eye lens image is registered and overlapped with the distorted virtual space as shown in the bottom left side of the window, and a set of functions of the simulator are displayed in the right side of the window.

The simulator can set the number, color, position, width, height, and depth of any poles. It can also set the horizontal and vertical widths and the color of the grid. It can display a 24-bit RGB color, and all the length units are in centimeters. B. 3D Simulation Coordinates

The camera position and viewing angle depend on the vehicle model and technical specifications. The simulator can move the camera mounting position up, down, left, and right. It can also change the viewing angle by panning and tilting the virtual camera. Figure 7 shows the position coordinate and rotation axis of the mounted camera. C. Lens Specification

The simulator can select the lens specification. More specifically, it determines the number of field. In the same field, the lens has the same distortion factor. The more numbers of fields, the more details of the lens distortion factor are obtained. It can also select the image height, the viewing angle, and the lens size in millimeters as shown in Fig. 8. The default viewing angle is initially set to 180°. D. 3D Distortion and Registration

The simulated 3D distortion result is displayed on the popup window. Results of the registration of the virtual space and the camera input image can be displayed with or without the distortion on the virtual space image.

A. Open Software Library-Based Virtual Space Generation

The fundamental information acquired by a vehicle rear-view camera is (i) the distance of an object from the camera and (ii) the degree of radial distortion.

Fig. 7. Two coordinates of a rear-view camera: (a) the mounting position and (b) the viewing axis.

Research Article


1341

Table 1. Lens Specifications Used in the Experiment Field

Image Height (mm)

Angle (deg)

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

0.000 0.200 0.400 0.600 0.800 1.000 1.200 1.400 1.600 1.800 2.000

0.00 18.20 36.40 54.60 72.80 91.00 109.20 127.40 145.60 163.80 182.00

Fig. 8. Lens specification selection dialog window.

4. EXPERIMENTAL RESULTS In order to evaluate the performance of the proposed system, we used 640 × 480 test images taken by a camera with a fish-eye lens that will be employed in a vehicle rear-view camera. The simulator accurately generates geometrical distortion of each image according to the camera position and viewing angle. In the experiment, centimeter is used for the length value, and degrees is used for the angle value. The 3D virtual grid was generated with the size 30 cm × 30 cm, and the location and size of the pillars were set based on the NHTSA standard. The generated 3D virtual spaces are shown in Fig. 9. In order to simulate the 3D distortion onto the virtual space, we used the lens specifications given in Table 1, which is provided from a lens manufacturer. We applied the proposed 3D geometric transformation to simulate the distortion onto the pre-generated virtual space. The results of the transformation are shown in Fig. 10. The performance of the proposed method is evaluated registering the camera output and simulated images. For the experiment, the FOV of a distributed lens is set to 192 deg. Figure 11

Fig. 9. Results of generating 3D virtual spaces using an open software library (camera height, vertical angle, tilt angle, x coordinate of the camera): (a) H120, L60, L30, 0; (b) H120, 0, 0, 0; (c) H120, R60, R30, 0; (d) H120, L60, L30, 30; (e) H120, 0, 0, 30; and (f ) H120, R60, R30, 30.

Fig. 10. Results of 3D distortion simulation (camera height, vertical angle, tilt angle, x coordinate of the camera): (a) H120, L60, L30, 0; (b) H120, 0, 0, 0; (c) H120, R60, R30, 0; (d) H120, L60, L30, 30; (e) H120, 0, 0, 30; and (f ) H120, R60, R30, 30. See Visualization 1.

shows the camera output image containing a rectangular grid pattern on the floor. Because of practical imperfections of fish-eye lenses, the simulated image does not exactly match the real distorted image. For this reason, the simulation error is measured by comparing a set of corresponding feature points between the distorted virtual space and the input image. Figure 12 shows the registration results of the two images.

Fig. 11. Camera input image (camera height, vertical angle, tilt angle, x coordinate of the camera): (a) H120, L60, L30, 0; (b) H120, 0, 0, 0; (c) H120, R60, R30, 0; (d) H120, L60, L30, 30; (e) H120, 0, 0, 30; and (f ) H120, R60, R30, 30.

1342


Fig. 12. Registration of the distorted virtual space and the camera input images (camera height, vertical angle, tilt angle, x coordinate of the camera): (a) H120, L60, L30, 0; (b) H120, 0, 0, 0; (c) H120, R60, R30, 0; (d) H120, L60, L30, 30; (e) H120, 0, 0, 30; and (f ) H120, R60, R30, 30.

As shown in Fig. 12, the gray region represents the green tape, while the red region represents the simulation results. The proposed method can accurately simulate the real world images acquired by a fish-eye lens camera. The matching error can be measured for the quantitative performance evaluation using error d pixel ∕mpixel × 100;

(17)

where d pixel represents the number of detected pixels and mpixel is the number of mismatching pixels. The proposed simulation method has error rates of about 2.86. Table 2 summarizes the error rate, which becomes high if the camera moves to the right. Because the rotation angle and height of the camera cannot be precisely measured, an error rate less than 3% can be considered to be accurate. The performance of the proposed method is compared with an existing optical simulator, as shown in Fig. 13. The existing simulator, CODE V, provides a 2D image-based simulation result, while the proposed simulator provides the simulation result in virtual 3D space. Figure 13 shows the comparison results for the viewing angle of 170°. In addition to CODE V, POV-Ray [16] and Blender [17] can also simulate fish-eye lens images. However, those simulators cannot avoid the black region that cannot be reconstructed by the 2D simulation, as shown in Fig. 13(b). On the other hand, the proposed method can reconstruct the hidden object in 3D space, as shown in Fig. 13(c). For this reason, existing Table 2.

Error Rate of Proposed Simulation Results

Input Image H120, L60, L30, 0 H120, 0, 0, 0 H120, R60, R30, 0 H120, L60, L30, 30 H120, 0, 0, 30 H120, R60, R30, 30 Average

Detecting Pixels

Missing Pixels

Error Rate (%)

3457

1

0.0289

3561 3490

51 189

1.4322 5.4155

4157

148

3.5603

3979 4013

84 186

2.1111 4.6349

3776.17

109.83

2.8638

Research Article

Fig. 13. Comparison of fish-eye lens distortion simulation: (a) the original 3D virtual image, (b) simulation result using 2D-based CODE V, and (c) the result using the proposed 3D-based method.

2D-based simulators are not suitable for recognizing 3D objects using the vehicle rear-view camera. 5. CONCLUSION In this paper, we presented a novel 3D simulation method for fish-eye lens distortion in a vehicle rear-view camera. The proposed method creates a geometrically distorted image of an object in 3D space according to the lens specifications. The 3D virtual space is first generated using an open software library. Next, the proposed 3D simulation algorithm estimates the distortion angle of each pixel and cross-checks the grid and poles in the distorted virtual space using the lens specifications including the image size, upper bound of the angle, and field number. The input image is then registered to the 3D distorted scene under user-specified conditions. A major contribution of this work is the lens-specific 3D distortion simulation that optimizes the accuracy and efficiency for preinstalling vehicle rear-view cameras. Also, the proposed simulation method was simpler and less expensive than conventional simulators. The proposed simulation method can be applied to designing a general optical imaging system for intelligent surveillance as well as a vehicle rear-view backup camera. Ministry of Science, ICT and Future Planning (B0101-150525); Ministry of Culture, Sports and Tourism (MCST) and Korea Creative Content Agency (KOCCA) in the Culture Technology (CT) Research and Development Program; National Research Foundation of Korea (2013R1A1A2060470). REFERENCES 1. R. Danescu and S. Nedevschi, “Probabilistic lane tracking in difficult road scenarios using stereovision,” IEEE Trans. Intell. Transp. Syst. 10, 272–282 (2009). 2. P. Hsiao, C. Yeh, S. Huang, and L. Fu, “A portable vision-based realtime lane departure warning system: day and night,” IEEE Trans. Veh. Technol. 58, 2089–2094 (2009). 3. Y. Dong, Z. Hu, K. Uchimura, and N. Murayama, “Driver inattention monitoring system for intelligent vehicles: a review,” IEEE Trans. Intell. Transp. Syst. 12, 596–614 (2011). 4. J. Choi, D. Kim, H. Yoo, and K. Sohn, “Rear obstacle detection system based on depth from Kinect,” in 2012 15th International IEEE Conference on Intelligent Transportation Systems, Anchorage, Alaska, 2012, pp. 98–101. 5. S. Silberstein, D. Levi, V. Kogan, and R. Gazit, “Vision-based pedestrian detection for rear-view cameras,” in 2014 IEEE Intelligent Vehicles Symposium Proceedings, Michigan, 2014, pp. 853–860. 6. V. Maik, D. Kim, H. Kim, J. Park, D. Kim, and J. Paik, “Automatic top-view transformation for vehicle backup rear-view camera,” in

Research Article

7.

8. 9. 10.

11.


The 18th IEEE International Symposium on Consumer Electronics, JeJu Island, South Korea, 2014, pp. 1–2. Z. Yu, W. Zhang, and V. Kumar, “Robust rear-view ground surface detection with hidden state conditional random field and confidence propagation,” in IEEE International Conference on Image Processing (IEEE, 2014), pp. 991–995. http://www.optis‑world.com/. J. Park, S. Byun, and B. Lee, “Lens distortion correction using ideal image coordinates,” IEEE Trans. Consum. Electron. 55, 987–991 (2009). C. Hughes, M. Glavin, E. Jones, and P. Denny, “Wide-angle camera technology for automotive applications: a review,” IET Intell. Transp. Syst. 3, 19–31 (2009). C. Ishii, Y. Sudo, and H. Hashimoto, “An image conversion algorithm from fish eye image to perspective image for human eyes,” in

12. 13.

14.

15. 16. 17.

1343

Proceedings of IEEE/ASME International Conference Advanced Intelligent Mechatronics (IEEE, 2003), Vol. 2, pp. 1009–1014. A. Basu and S. Licardie, “Alternative models for fisheye lenses,” Pattern Recogn. Lett. 16, 433–441 (1995). A. Basu and S. Licardie, “Modeling fisheye lenses,” in Proceedings of IEEE/RSJ International Conference Intelligent Robots and Systems (IEEE, 1993), Vol. 3, pp. 1822–1828. F. Devernay and O. Faugeras, “Straight lines have to be straight: automatic calibration and removal of distortion from scenes of structured environments,” Mach. Vision Appl. 13, 14–24 (2001). C. Hughes, P. Denny, E. Jones, and M. Glavin, “Accuracy of fisheye lens models,” Appl. Opt. 49, 3338–3347 (2010). http://www.povray.org/. http://www.blender.org/.