Using Geometric Constraints for Fisheye Camera Calibration

1 downloads 0 Views 566KB Size Report
After combination with the Tamron lens, the horizontal FOV is roughly 130°, which is physically tested by a stepping motor. This “assembled” fisheye lens is.
Using Geometric Constraints for Fisheye Camera Calibration Tzung-Hsien Ho1 Christopher C. Davis1 Stuart D. Milner2 2 Department of Electrical and Computer Department of Civil and Environmental Engineering Engineering University of Maryland, College Park, MD University of Maryland, College Park, MD 1

Abstract This paper presents a technique to linearly calibrate a fisheye camera using the fact that the distortion is in the radial direction (angle preservation on the X-Y plane). First, a linear equation based on this constraint is proposed for estimating intrinsic/extrinsic parameters. The skewness in the intrinsic parameters is assumed to be zero in this paper. Second, a pure rotation for the fisheye camera is applied to recover the depth of the principal plane. Last, distortion parameters are estimated based on least squares polynomial fitting. Our method provides a linear and closed-form solution. Besides, all the parameters can be estimated in the least squares sense, which largely reduces the errors created by the existing noise in the measurement. Our method can be applied for both circular and full-frame imaging with a fisheye lens. In comparison with classical techniques, our method does not require precise scene arrangement or any prior information such as the field-of-view of the lens or the location of the camera center.

1. INTRODUCTION Omnivision methodology provides a wide field-of-view (FOV) for various applications ranging from photography to computer vision. In the computer vision community, a wider field of view can resolve the ambiguity in the elimination of rotation and translation, which allows better recovery of the 3D scene using the structure from motion scheme [1]. We have been investigating ommivision to identify beacons and other properties of communications nodes in free space laser communications [19]. Many omnivision devices have been proposed, using convex mirrors [2,3], lenses [3-9], multi-camera devices [10] and camera self-rotation [11]. Currently, the use of mirrors (with a single viewpoint constraint) in conjunction with a camera has been considered as the most mature technique because the optical properties of the system have been well-defined. Geyer et.al. [2] propose the catadioptric geometry, where they prove the catadioptric projection is equivalent to the spherical projection. Ying [3] expand the idea to a unifying two-step model in which 1) the scene points are first mapped onto a quadric surface and 2) the image points on the surface are further projected through an effective pin-hole, located on the axis containing both the

foci, onto the image plane. Since no evidence suggests that dioptric cameras (using fisheye lenses) have similar optical properties (e.g. rays are projected on a quadric surface) like central catadioptric cameras, the two-step model can lead to a erroneous correction for some fisheye lenses. For a dioptric (fisheye) camera, several lenses are used to reduce the incident angle of rays in order to project onto a CCD array with limited size. Figure 1 shows the layout of a fisheye lens (US Pat 4,412,726 designed by M. Horimoto at Minolta), generated by CodeV software [13]. The first two elements are used to reduce the incident angle and the rest of them function as an imaging lens. The multi-element configuration at the front-end complicates the analyses of incidence/refraction relations. In addition, most of off-theshelf lenses do not even include their lens structure. Consequently, the analysis of dioptric cameras using raytracing as for catadioptric cameras is very difficult because of these two problems, which also exist in calibrating the regular camera. However, the preservation of θ and φ (fig 2a) is sufficient for the calibration without knowing the real lens structure in a regular camera. This motivates us to think how much we can do by only knowing φ (fig 2b) is preserved.

0.77

Fisheye Lens

U.S. Pat. 4,412,726

Scale:

1.30

ORA

IN

20-May-05

Figure 1: Fisheye lens (US patent No. 4,412,726) In general, the calibration of a dioptric camera involves the estimation of an intrinsic matrix [16] along with a projection model. The intrinsic matrix, which maps the camera coordinates to the image coordinates, is parameterized by principal points, focal length, aspect ratio and skewness. Note that focal length and skewness are not included in our fisheye camera. The details are explained in section 2.1. Because of large distortions, most the calibration techniques take additional measurements to estimate the

intrinsic matrix. Most often the principal points are found with the following techniques: Circular imaging fisheye lens: A circular fisheye camera results from the size of the image plane charged coupled device (CCD) being larger than the image produced by the fisheye lens. Since the lens projects the scene with a circular symmetry, a circle containing the scene is produced near the center with a shaded area on the edge of the CCD. The center of the circular image is assumed to be a principal point. [5,6,8]. Laser pointer: A laser pointer produces straight rays like the rays from object points, which makes it useful in finding the principal point. Shah et.al. [4] apply this method and further apply the correspondences near the principal point to estimate the aspect ratio. Image center: Fitzgibbon [7] and SriRam et.al. [8] assume a known distortion center as the image center directly. Compared with a unified projection model [2] for central catadioptric cameras, there exist many projection models for fisheye cameras. The existing projection models can be divided into two categories [3]. 1. Fisheye image radius (P) vs. its corresponding perspective image radius (P΄): P and P΄ are the distances from the distortion center to the distorted image point and the corresponding perspective image point respectively. In both images, the center is the same. The corresponding distances can be transformed as: P ' = P * (1 + K1P 2 + K 2 P 4 + ...) . Fitzgibbon [7] proposes a division model: P' = K1P /(1 + K 2 P 2 ) , which is further generalized as P' = P /(1 + K1P 2 + K2 P 4 + ...) in [8,18]. 2. Fisheye image radius (P) vs. incident angle (θ): The incident angle θ is the angle between the optical axis and the incident ray (Fig. 2). Fleck [14] has introduced several mapping methods, including: stereo-graphic: P = 2 f * tan θ / 2 , equidistance: P = f *θ , sine-law: P= f *sinθ , and equisolid angle: P= 2 f *sinθ / 2, where f is the focal length. Some hybrid models have been suggested such as P = α sin βθ in [15], or P = a sin bθ + c tan dθ in [5]. Kannala et.al [6] suggest the projection model in a general polynomial form: P = ( aθ + bθ 3 + cθ 5 + K) . The techniques to recover the coefficients of the projection model are: Straight line constraint: The straight line constraint states that the straight line remains straight under perspective projection, which is applied in [4,17]. Single view constraint: Hartley, et.al. [16] have mentioned that under perspective projection, a homography (3x3 transformation matrix) exists between two planar scenes, which is the constraint used by [6]. Schwalbe [9] proposes a constraint that the incident angle is proportional to the projection radius, which is not rigid for most fisheye lenses.

Epipolar/Trilinear constraint: Fitzgibbon [7] proposes a technique to simultaneously estimate the fundamental matrix and the distortion coefficients. The whole problem can be modeled as a quadratic-eigenvalue problem (QEP). Miscusik, et.al. [8] apply a similar method but with a more complicated projection model in their calibration. SriRam et.al. [18] propose that since the distortion is in the radial direction, the camera can be modeled as a 1D camera. A radial trifocal tensor can be used to estimate the distortions from seven triplets. Our calibration technique estimates the intrinsic and extrinsic matrices with a well-known constraint (angle preservation on the X-Y plane), which is also adapted by SriRam et.al. [18]. However, this constraint cannot provide sufficient information for the depth of the principal plane, which can be recovered by an additional rotation. Once the extrinsic matrix is recovered, the projection model can be easily modeled by a least squares curve fitting. Both computer simulations and real data have been used to test the proposed technique and satisfactory results have been obtained. Compared with the classic techniques, our technique is considerably more flexible because it requires no additional measurements or any prior knowledge.

2. Geometric Constraints for Fisheye Lens 2.1 Angle preservation on the X-Y plane The regular narrow field of view camera can be described by the pin-hole model, which preserves the angular information between the X-Y, Y-Z and X-Z planes as depicted in figure 2(a). Only two of these three constraints are actually independent. Each 3D point and its corresponding image provide two geometric constraints. The equation (up to a scale) between the 3D points and their correspondences is:  X ' ' u  Fx α Ox  Y ' '  λ v  = K * M *   , with K = 0 Fy Oy M = [r1, r2 , r3 , t ]     Z "  1   0 0 1    1  where Fx : focal length, Fy /Fx : aspect ratio

(1)

α : skewness,[Ox , Oy ] : principalpoint ri : i - th columnin the rotationmatrix

If we multiple by K-1 on both sides, equation (1) becomes α αOy   1 , - Fy , Fy − Ox   X ' '   u  Y ' '  FxOy    λ  Fx * v = M *  0, ,     Z "  Fy Fy Fx   1    Fx    0 , 0 , 1   

(2)

Note that this equation describes the preservation of T tan θ [cos φ , sin ϕ ] , where θ is the angle between the

incident ray and the Z-axis and φ is the angle formed by the ray projected on the image plane and the X-axis (shown in figure 2(a)). The angles are (θ,φ) defined in spherical coordinates. Note that 1/Fx is used to compute tanθ. The fisheye camera model between a 3D object and its image can be described using equation (3). X' ' u  Y' ' λ v = K * f (M *  )   Z"    1  1  where K : Intrinsicmatrix,f(•) : Projection Model

~

u

x 

x

λK v  = f ( y), where y = r1 X "+r2Y "+r3Z"+t   1 

     z   z  1 0 X 0  ~ K = inv( K ) = 0 β Y0     0 0 1  β = Fx / Fy , X0 = −Ox , Y0 = −Oy * β

(4)

(3)

Note that f(•) only changes the length of the projection, which is in the radial direction (Fig 2b). In other words, the incident angle of the incoming light ray will be bent without following the pin-hole model, which results in a change in angle θ. Thus, Fx no longer requires estimation since tan θ is not preserved in the fisheye lens. Also, we make an assumption that the skewness (α) can be neglected, which is true for most cameras. Then, the intrinsic parameters in our calibration are reduced to three [Fx/Fy, Ox, Oy].

(a)

Function f has two effects 1) it computes θ through vector [x y z] and 2) it computes the projected image radius. Thus, equation (4) can be simplified as u + X 0  =  β v + Y0 

λ

g (θ )

x   x + y  y 2

2

(5)

where θ = tan −1 ( x 2 + y 2 / z ), g ( • ) : unknown projection model

g (θ ) / x 2 + y 2 in (5) is a constant, which can be

eliminated with λ to obtain equation (6) (u + X 0 ) y − ( βv + Y0 ) x = 0

(6)

After combining all the terms, we obtain a linear equation for our constraint given in equation (7).  r21  r   22   r23    ty  βr   11  βr12  [uX" , uY" , uZ" , u,-vX" ,-vY" ,-vZ" ,-v, X" , Y" , Z" ,1] =0 βr  13  βt x  - Y r + X r  0 21   0 11 - Y0 r12 + X 0 r22    - Y0 r13 + X 0 r23  - Y0 t x + X 0 t y   

(7)

Using 11 correspondences, the parameters can be solved with the direct linear transform (DLT) described in [16]. Note that the estimated parameters are scaled. The twelve parameters can be rearranged to get equation (8). a 5 a 6 a 7 a 8   β 0 a a a a  = λ  0 1  r11 r12 r13 t x   1 2 3 4    r r r t  a 9 a10 a11 a12  - Y0 X 0   21 22 23 y 

(8)

where ai is the ith element of the vector.

(b) Figure 2: Geometric relations in the (a) pin-hole and (b) fisheye cameras. Using the intrinsic matrix, we can start to derive the equation for the angle preservation constraint. First, we expand equation (3) to get (4)

From equation (8), it can be seen that the last row (a9 to a12) is dependent on the first two rows. We can estimate the principal point [Ox,Oy] by inverting  a 5 a1  a a  - Y0  6 2  β a 7 a 3      X 0 a 8 a 4 

 =  

a 9  a  .  10  a11    a12 

(9)

The estimated eight parameters have seven degrees of freedom and there are five extrinsic parameters (3 for rotation and 2 for translation). Thus, two internal constraints are embedded and can be written as

T  1  0   a5a6 a7   a5a 6 a 7   β  a a a  a a a   1   1 2 3   1 2 3   0

 0 λ 0 =  0 λ  1   

(10)

After eliminating λ, we can estimate β as the following: 1 2 (11) (a + a 2 + a 2 ) = (a 2 + a 2 + a 2 ) β2

5

6

7

1

2

3

After β is solved, the third row of the rotation matrix can be obtained as: [r31, r32 ,r33] = [r11, r12 ,r13] x [r21, r22 ,r23].

2.2 Recover the Depth of the Principal Plane with Multiple Rotations The intrinsic matrix and eight out of the nine elements of the extrinsic matrix have been estimated in the previous section. The only unknown parameter is the depth of the principal plane (tz). To recover the depth information, a pure rotation has to be applied to the camera. After the camera rotation, the new orientation of the calibration object provides a new extrinsic matrix (which excludes the depth of the new principal plane). The relation between the old and new extrinsic matrices is listed in equation (12) ( t zn , t oz are the only two unknown parameters). r11n , r12n , r13n , t xn   r11o , r12o , r13o , t xo   n n n n  o o o o c r21 , r22 , r23 , t y  = λ * R *  r21 , r22 , r23 , t y   n n n n  o o o o r31 , r32 , r33 , tz   r31 , r32 , r33 , t z 

(12)

From equation (12), we can find that t nx , t ny includes o z

n z

o z

and vice versa. Thus, t , t can be information on t estimated by first finding R, given by r , r , r  r , r , r  1  n n n  o o o  R =  r21 , r22 , r23   r21 , r22 , r23  λ  n n n  o o o   r31 , r32 , r33   r31 , r32 , r33  n 11

n 12

n 13

o 11

o 12

o 13

−1

c

(13)

A good projection model has to be: 1) Easy to fit and 2) easy to invert; thus, a polynomial based model is selected for our calibration because it has a closed-form least squares solution. Although there is no inverse closed-form solution for a polynomial model of order above cubic, an fsolve function in Matlab can be used. To test its performance, we investigated eight different fisheye lenses available in CodeV [13] with different kinds of polynomial fitting, which are shown in Figure 3. This simulation is conducted by arranging a checkerboard frontal-parallel to the fisheye lens. Thus, the incident angles can be estimated directly from the corresponding image points. The incident angles vary from 0 to 85 degrees with 1 degree spacing, which ensures that the fitting data covers the whole field of view. The results indicate that the mean error of estimated incident angles is fewer than 0.5 mrads (0.03° for all eight fisheye lenses) under the model ( P = aθ 5 + bθ 3 + cθ ). -3

2.5

x 10

2 Model 1: a*θ3+b*θ2+c*θ+d Model 2: a*θ3+b*θ2+c*θ Model 3: a*θ5+b*θ3+c*θ

1.5

1

0.5

0 1

2

3

4

5

6

7

8

Lens Type

Figure 3: Incident angle error with different projection models

3. Calibration Methods

Then, two linear equations exist for finding t oz 1 n o c o c o  t z = R c (t x − R11t x − R12t y )  13  c o c o  t oz = 1 (t yn − R21 t x − R22 ty ) c R23 

2.3 Projection Model

Incident Angle Error (rad)

 1  β   0

(14)

t zn estimation follows the same procedure. Note that the function of the rotations is not only used to estimate the depth, but also generate more scenes for the least squares estimation of the intrinsic/projection model parameters. After estimating all the parameters, the so-computed R (the first three columns) in each extrinsic matrix does not in general satisfy the property of a rotation matrix because of the noise in data. We apply the techniques described in [12] to estimate the best rotation matrix(R’) (Minimize the Frobenious norm of | R-R’|).

3.1 Estimate Intrinsic Parameters First, we retrieve the correspondences from the calibration object and plug into equation (7) to estimate the 12 parameters. If we have k orientations, take the last 4 parameters estimated from the calibration object to form a vector (4k x 1) and the remaining 8 parameters to form a matrix (4k x 2) like equation (13). We can estimate Y0/β and X0 in the least squares sense.

 a 51 , a 11   a 91   1 1   1  a6 , a2   a 10  1 1 a , a  a1   7 3   11  1  a 81 , a 41   Y 0   a 12     = M  β M       a k , a k   - X 0   a k   5 1  9   a 6k , a 2k   a 10k   k k  k  a7 , a3   a 11  k k a , a  a k   12   8 4

(15)

Each orientation provides one constraint for solving β as described in equation (10). For k orientations, β can be estimated as follows: a11 + a12 + a13  a15 + a16 + a17      M M   *β =    k2  k2 k2 k2  k2 k2  a 5 + a 6 + a 7  a 5 + a 6 + a 7  2

2

2

2

2

(16)

v1 z

v2 z

unit vector, the incident angles are equivalent. Thus, both objects are projected to the same image point. After a pure rotation is applied, the resulting new incident angles are both equal to −1 Rv 1 • z , which implies that both rays are cos

(

)

v1 z

projected to the same spot.

To recover the depth of a single principal plane, a pure rotation is applied to get a new orientation whose extrinsic parameters can again be estimated using equation (7). These two sets of extrinsic matrices create 2 linear equations according to equation (12). Assuming the rotations are applied k-1 times and k orientations are obtained, 2k-2 equations (except for the orientation itself) can be formed for each depth of the single principal plane in equation (14). For example, for the 1st orientation, the 2nd to kth orientations can be used.

i

Proposition 1. Pure rotation does not expose an object that is blocked originally to the fisheye camera. It is a well-known fact that if an object is blocked by other objects, a pure rotation cannot expose the object under projective projection. This fact still exists for the fisheye camera. Intuitively, given two objects with the same ray unit vector (only different in scale) v2= k*v1 with respect to the fisheye camera center, their incident angles are cos −1 ( v1 • z ) and cos −1 ( v 2 • z ) . Since they have the same

2

3.2 Estimate the Depth of the Principal Plane

t1x − R11c1 txi − R12c1 t iy  R13c1  1   c1  c i c i t y − R211 tx − R221 t y  R23    M ti =  M   z  t k − Rck t i − Rck t i  R13ck  x 11 x 12 y    ck  ck i ck i  t yk − R21 t x − R22 ty  R23  

the rotation is applied. Finally, we place an L-shaped calibration object in position and start our process.

(17)

where t z is the depth of the principal plane in the ith orientation. As mentioned earlier, the applied camera motion has to be a pure rotation. How can we obtain a pure rotation movement? In other words, how do we eliminate translation? Translation comes mainly from two sources: 1) imperfect rotation of the motor, and 2) the pivoting point of the motor and camera center are displaced. The first source is minimal for most modern motors. The second source can be eliminated by the following steps. First, we attach the camera to the motors and manually create a scene where at least one object is blocked by another object. The greater the distance between the two objects, the better a pure rotation can be obtained. Secondly, we apply the rotation and observe the blocked objects. Then we adjust the position of the camera until no additional portion of the blocked object can be seen after

3.3 Estimate Parameters for the Projection Model After recovering the extrinsic matrices, the world coordinates can be mapped to the camera coordinates, where the incident angles from each 3D object point can be computed as mentioned in equation (5). Combining the incident angles with the corresponding projection radius, the coefficients of the projection model can be estimated by polynomial least squares fitting. We choose to use P = aθ 5 + bθ 3 + cθ (model 3) mentioned in section 2.  Pi 2 + Pi 2  θi5,θi3,θi1  T x y ,   [a, b, c] =     M  M

(18)

where i means the ith 3D point.

3.4 Minimize Re-Projection Error We are given n images of a plane and there are m points on the calibration object. Assume that the image points are disturbed by independent and identically distributed (IID) noise. The maximum likelihood estimation can be obtained by minimizing the re-projection error function: n m (19) m − mˆ ( β , X , Y , R , t , M )

∑∑ i =1 j =1

ij

0

0

i

i

j

where m ˆ ( β , X 0 , Y0 , R i , t i , M j ) is the projection of point Mj in image I. Ri is the rotation matrix for image i, which can be parameterized by a vector of 3 parameters using Rodrigues formula, ti is the translation vector. This minimization is a nonlinear problem, which can be optimized with the Levenberg-Marquardt algorithm implemented in the lsqnonlin function in Matlab.

3.5 Summary

4.1 Minimum Coverage Ratio

The proposed calibration procedure is as follows: 1. Prepare two planes with a checkerboard pattern attached and orthogonally connect them into an L-shape; 2. Attach the camera to the moving platform and rotate it; 3. Adjust the position of the camera based on section 3.2 to get pure rotation; 4. Apply several rotations and take a few images (>2) of the model object; 5. Detect the feature points in the images; 6. Estimate the intrinsic parameters and eight extrinsic parameters for each orientation using the closed-form solution as described in section 3.1; 7. Estimate the depth of the principal plane (tz) based on section 3.2; 8. Apply the intrinsic/extrinsic matrices to every orientation and compute the incident angle for each 3D point; 9. Each angle is combined with its projection radius to estimate the projection model (a,b,c) as explaining in section 3.3; 10. Refine all the parameters by minimizing the reprojection errors.

It is well known that the image of the calibration object has to be sufficient large to reduce measurement errors. However, if the object occupies too much field of view (FOV), a rotation may take the object out of the FOV. Thus, an indicator, coverage ratio (CR) is defined and a simulation is conducted to find the minimum image size. Area of image covered by the calibration object CR = Area of the image Since our calibration procedure requires several orientations, each orientation has its coverage ratio. We select the minimum CR among all the orientations as our evaluation standard. The initial translation in the Zdirection is set from 0.7 to 1.5 meters in this experiment. For each translation value in Z, the test is executed 100 times with individual X-Y translations between -14 and 14 centimeters. Then, four rotations are applied (±30° in Yaxis and ±30° in X-axis) and the parameters are estimated based on these four rotation orientations. Note that if the distance is too close for a particular rotation, the rotation angle is automatically decreased in intervals of 7.5°. The noise used in this experiment had 0 mean and variance 0.5. For each translation value, the estimated parameters are averaged and then compared with the true values. Figure 4 shows the errors of the intrinsic parameter estimation. Figure 5 shows the errors of the projection model estimation. 25

0.35

Ox Oy

Beta

0.3

Relative error (%)

20

Absolute error (pixels)

15

10

0.25 0.2 0.15 0.1

5

0.05 0 5

10

15

20

25

30

Occupied ratio(%)

35

40

0 5

45

10

15

20

25

30

Occupied ratio(%)

35

40

45

Figure 4: Principal point and aspect ratio errors vs. the coverage ratios 14

300

Fifth order coeff (-5) Third order coeff(-15) First order coeff(515)

250

200

150

100

10 8 6 4 2

50

0 5

Error of the Projection model

12

Absolute error (pixels)

To understand the calibration performance of our algorithm, we conducted three experiments. The first one is to find the minimum size of the image size. The second one is to investigate how the measurement noise affects our results. The third one is to investigate the improvement of our estimation when more rotations are used. Our calibration object is an L-shape board consisting of two orthogonal planes with a checkerboard pattern. Both of the checkerboards are square (77cm x77cm); thus, the total number of corner points is 11x11x2=242. Pure rotations are generated corresponding to the requirement of the experiments. The simulated camera has a known intrinsic matrix with coefficients [β=5/7, Ox=705, Oy=500]. The field of view of the camera is 85° and the coefficients [a,b,c] of the lens projection model are [-5,-15,515]. All the coefficients result in a resolution of 1380 x 985 pixels. In each experiment, we measure the absolute errors for principal point [Ox,Oy] and relative error for β for the intrinsic parameters. For the projection model estimation, the absolute errors of the three estimated coefficients are presented. In order to provide a better understanding of the projection errors, a set of angles 0 to 85° with a 3° interval is plugged into the projection model with both estimated and true coefficients. The absolute errors between the two sets of the projection radius are presented.

Absolute error

4. Simulation Results

10

15

20

25

30

Occupied ratio(%)

35

40

45

0 5

10

15

20

25

30

35

40

45

Occupied ratio(%)

Figure 5: Projection model coefficients error and projection errors vs the coverage ratios.

It can be seen that in general, the smaller the coverage ratio, the larger the errors in the estimated parameters. However, the errors decrease slowly with an increment in the CR. The saddle point provides a balance between a sufficiently large CR and still enough space to rotate. Considering the saddle points among the estimated

60

900

1st plane 2nd plane 3rd plane 4th plane

800 30

700 600

20

Mapping model projection error Error of the Projection Model

8 7

50 40 30

500

10

6 5 4 3 2 1

0 0

0.5

1

Noise Level (pixels)

1.5

0 0

2

0.5

1

Noise Level (pixels)

1.5

2

Figure 8: Projection coefficient error / projection errors v.s the noise level of the image points. 9

0.5 Ox Oy

Beta 0.45

7

Relative error (%)

Absolute error (pixels)

8

6 5 4 3

0.4

0.35

0.3

0.25

0.2

2 1 2

4

6

8

10

12

14

16

Number of Planes

0.15 2

4

6

8

10

12

14

16

Number of Planes

Figure 9: Principal point and aspect ratio errors vs. the number of orientations

10

400 0

25

5 0 -2

0

2

6

4

200

8

100

x

0 200

400

600

800

1000

1200

1400

Figure 6: Calibration object and image points from four orientations

As we can see from figure 7, both errors increase linearly with the noise variance. For the variance=0.5 (typical noise in a practical calibration), the error in β is around 0.3%. The errors in Ox and Oy are approximately 10 and 6 pixels, respectively. The estimation of Oy looks better than the one of Ox for two reasons (1) Oy is shrunk by the ratio of 5/7 because of β. (2) The distortion causes the checkerboard images to bend differently along the X and Y axes. In this case, the Y is less sensitive to the noise because more data points have a larger component in Y rather than in X. 35

0.9 Ox Oy

Beta

0.8 0.7

Relative error (%)

25 20 15 10

0.6 0.5 0.4 0.3 0.2

5 0 0

0.1 0.5

1

Noise Level (pixels)

1.5

2

0 0

0.5

1

Noise Level (pixels)

1.5

2

Figure 7: Principal point and aspect ratio errors vs. the noise level of the image points

Figure 8 shows the results for the estimation of the projection model coefficients and the projection errors resulting from the use of wrong coefficients. It can be seen

2.4

Absolute error (pixels)

z

Error of the Projection model 2.6

20

Absolute error

10

30

2.8 Fifth order coeff (-5) Third order coeff(-15) First order coeff(515)

300

15

Absolute error (pixels)

9

Fifth order coeff (-5) Third order coeff(-15) First order coeff(515)

20

In this experiment, four different orientations are used. The calibration object (shown in figure 6) is translated by [3,-3,15] (CR is 16.5%), and then the same four rotations are applied as in the previous experiment. The image points from the four orientations are shown on the right of figure 6. The applied noise is Gaussian with 0 mean and variance from 0.1 to 2.0 pixels with spacing 0.1. For each noise level, 100 independent trials are performed and the presented results are the average. The estimated results are compared with the true ones.

-y

70

Absolute error (pixels)

4.2 Noise Level

that if the variance =0.5, the estimated mapping model produces absolute errors around 1.8 pixels.

Absolute error

parameters, we choose the coverage ratio to be at least 16%. This number provides a selection rule for our proposed technique. If any of the orientations in our calibration have CR below 16%, we should drop the results estimated from this orientation.

15

10

2.2 2 1.8 1.6 1.4

5

1.2 0 2

4

6

8

10

12

14

Number of Planes

16

1 2

4

6

8

10

12

14

16

Number of Planes

Figure 10: Projection model coefficient errors and projection errors vs. the number of orientations

4.3 Number of Planes This experiment is conducted to see the effects while many orientations are used. The first four orientations are the same as in the previous two experiments. From the fifth image, we randomly choose two rotation angles (one for Xaxis and one for Y-axis) to create a new orientation. Two orientations are the minimum requirement for our proposed calibration technique. Thus, we vary the number of orientations from 2 to 16. For each number, 100 trials of independent orientations (except for the first four) and independent noise with 0 mean and variance 0.5 pixels are conducted. Note that we added two limits for choosing our rotations: 1) none of the new orientations can have a maximum incident angle larger than 85° and, 2) if any of the orientations has a CR less than 16%, that orientation is dropped. Figure 9 shows the errors in the intrinsic parameters. Figure 10 shows the errors of the projection model coefficients and the projection error. It can be observed that all the errors decreased when more orientations are incorporated into our procedure.

5. Real-Test Results 100

An experiment was conducted using a fisheye converter (FC-E8) from Nikon combined with a Tamron lens (23FM25SP). Note that the fisheye converter (FC-E8) is not itself a “Fisheye lens”. It has to be combined with an imaging lens to project the incident rays onto the CCD. Note that an FC-E8 is specified to have a FOV of 183°, which is the diagonal FOV instead of the horizontal or vertical FOV. After combination with the Tamron lens, the horizontal FOV is roughly 130°, which is physically tested by a stepping motor. This “assembled” fisheye lens is cheaper than an all-in-one fisheye lens and the converter itself can be easily combined with other imaging lenses to produce a full-frame or circular-image fisheye. The camera used is a Pulnix (TM-1325) with resolution 1392x1040 pixels. The camera motion is provided by a set of two-axes stepping motors (AS66AA H-50) from Oriental Motors. A pure rotation is obtained by using the procedure described in section 3.2. In this test, four orientations are used, whose configuration is similar to figure 6. The correspondences are retrieved manually with Baker‘s pointclick function [20] for reducing measurement errors. Other corner detector algorithms can be used to retrieve the image points. The average re-projection errors for the four orientations is [1.62, 1.36] pixels after the optimization. Figure 11 shows a corrected image taken from our laboratory using the parameters we estimated. We choose this image to display because it contains many straight lines, which can be examined by using the straight line constraint mentioned in section 1. The recovered scene shown in figure 11 has a 120° FOV.

6. Conclusion In this paper, we have described a simple technique for fisheye camera calibration that has a linear and closed-form solution. All the parameters in our technique can be estimated in the least squares sense, which greatly reduces the errors caused by measurement noise. The technique uses a geometric constraint for the fisheye - the angle is preserved on the X-Y plane. By using an L-shaped calibration pattern and several rotations, the coefficients for the intrinsic/extrinsic matrices and the projection model can be estimated with satisfactory accuracy. Simulations suggest that our calibration technique is robust with respect to regular operational noise and its performance can be further enhanced by using more rotations. We hope that our technique will provide users an easier access to the omnivision world.

200 300 400 500 600 700 800 900 1000 200

400

600

800

1000

1200

200

400

600

800

1000

1200

100 200 300 400 500 600 700 800 900 1000

Figure 11: The original scene vs. the corrected scene in our laboratory

7. Acknowledgement The authors wish to thank Cornelia Fermuller for her advice and suggestions on our paper. This research is supported by AFOSR, MIPS, and NSF.

8. References [1] J. Gluckman and S. Nayar. “Ego-motion and omnidirectional cameras” In Proc. of the International Conference on Computer Vision, pages 999--1005, 1998. [2] C. Geyer and K. Daniilidis, “Catadioptric Projective Geometry,” Int’l J. Computer Vision, vol.45, no. 3, pp.223-243, 2001. [3] X. Ying, Z. Hu, “Can We Consider Central Catadioptric Cameras and Fisheye Cameras within a Unified Imaging Model” ECCV 2004, pp 442-455. [4] Shishir Shah, J. K. Aggarwal ”A Simple Calibration Procedure for Fish-Eye (High-Distortion) Lens Camera” pp. 3422-3427 ICRA 1994. [5] Hynek Bakstein and Tomas Pajdla. “Panoramic mosaicing with a 180◦ field of view lens” In A. D. Williams, editor, Proc. of the IEEE Workshop on Omnidirec-tional Workshop, pages 60–67, Los Alamitos, CA, jun 2002. IEEE Press [6] J. Kannala and S. Brandt “A Generic Camera Calibration Method for Fisheye Lens”, Vol 1, ICPR 2004. [7] A. Fitzgibbon, “Simultaneous linear estimation of multiple view geometry and lens distortion” In Proc. CVPR, vol 1, Page 125-132, 2001.

[8] B. Micusik and T. Pajala “Estimation of omnidirectional camera model from epipolar geometry” Conference on Computer Vision and Pattern Recognition (CVPR), 2003. [9] E. Schwalbe “Geometric modeling and calibration of fisheye lens camera systems”, Proc. ISPRS, Feb 2005. [10] P. Baker, C. Fermuller, Y. Aloimonos and R. Pless. “A Spherical eye from multiple cameras (makes better models of the world).” In A. Jacobs and T. Baldwin, editors, Proc. Of the CVPR 01, vol1, pp.576-583, Los Alomitos, CA. [11] L. McMillian and G. Bishop “Plenoptic Modelling: An Image-Based rendering System.” Computer Graphics: Proc. of SIGGRAPH 95 pp.39-46. [12] Z. Zhang. “A flexible new technique for camera calibration” IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000. [13] CodeV software. Product of Optical Research Associates (ORA). http://www.opticalres.com/products_f.html#codev [14] F.M.M. “Perspective Projection: the wrong imaging model.” Technical report TR95-01, Comp. Sci., U. Iowa 1995. [15] J. Kumler and M. Bauer “Fisheye lens designs and their relative performance”, http://www.coastalopt.com/fisheyep.pdf. [16] R. Hartley and A. Zisserman, Multiple view geometry in computer vision, Cambridge university press, 2003. [17] R. Swaminathan and S.Nayer. “Non-metric calibration of wide-angle lens”, In DARPA Image Understanding Workshop, pp.1079-1084 1998. [18] SriRam Thirthala and Marc Pollefeys “The Radial Trifocal Tensor: A Tool for Calibrating the Radial Distortion of WideAngle Cameras. “ CVPR (1) 2005: 321-328. [19] T.H. Ho, S.D. Milner and C.C.Davis “Pointing, Tracking and Acquisition system with Omnivision” SPIE Photonics and Optics Conference, Aug. 2005. [20] http://www.cfar.umd.edu/~pbaker