Dynamic camera calibration - IEEE Xplore

0 downloads 0 Views 519KB Size Report
calibration block. The dynamic scheme calibrates the ste- reo cameras using camera parameters derived from the static calibration even after unknown motion of ...
Dynamic Camera Calibration C. Tony Huang and 0. Robert Mitchell Intelligent Systems Center University of Missouri Rolla Rolla, MO65401

-

tion process for multiple images and large scale matrix operations.

Abstract

Close-range camera calibration scheme can be divided into a static scheme and a dynamic scheme. The static scheme is used to calibrate single camera by applying bundle adjustment on multiple digitized images of a calibration block. The dynamic scheme calibrates the stereo cameras using camera parameters derived from the static calibration even after unknown motion of the cameras and change of focus. An unknown object can be used as the calibration block. After the cameras are dynamically calibrated, scaled dimensions of the unknown object can be determined as well. As additional conditions, prior knowledge of the object can be integrated into the bundle adjustment of the dynamic scheme to increase the accuracy. 1.

In this paper, we use the results from the full static scheme to recalibrate the effective focal length and the exterior orientations dynamically. Almost any observed object can be used for calibration. Partial knowledge of the object can be integrated into the dynamic scheme as extra conditions to improve the accuracy of calibration results. Based on the manufacturing process of the object, some information can be assumed to be ground truth, e.g. parallel lines, perpendicular lines.

2.

To apply dynamic calibration, the camera must have been previously calibrated intrinsically. Multiple images of the same calibration block (see Figure 1) are captured by each camera. The calibration block is not only shifted but also rotated under the camera to generate a broad range of control points. Only approximately orthogonal views are used because the image centers of the detected ellipse centers will be different from the centers of real circles for highly oblique views.

Introduction

Various close-range calibration techniques for CCD cameras have been introduced. Tsai used the radial constraint to estimate effective focal length and radial distortion factor [5]. If these effects are not considered, calibration accuracy is generally poor (off by many pixels). Weng, Cohen and Hemiou used a two-step approach to calculate the nonlinear distortion factors such as radial, decentering and thin prism factors [a]. Beyer proposed a sophisticated camera model and used the photogrammetry bundle adjustment method to solve the distortion parameters for the model [ 11.

Bundle adjustment (using least squares) is used for our static calibration. The parameters considered are image center, effective focal length, scale factor in the horizontal direction, and the first order radial distortion term. The effects of other parameters are within the subpixel ranges. Those additional parameters proposed by Beyer are shear and decentering parameters [l]. Tsai’s method [5] is used to get the initial condition for bundle adjustment.

Most of the research done is based on a fixed configuration of cameras: e.g., fixed focusing and fixed relative orientation between cameras. However, in most manufacturing and robot vision applications, we often need to reposition the cameras as well as refocus the cameras due to the different sizes and shapes of the observed objects. For example, in order to get more accurate feature locations on the image domain for a smaller object, we need to move the cameras closer to the object and to refocus the cameras so that the image of the object can fill the field of view. Due to the changes, a calibration artifact with similar size may be needed using previous techniques [ 1,561. On the other hand, a full camera calibration, such as Beyer’s, is rather time consuming due to the image feature detec-

2.1 Equation Setup

We use Tsai’s notation [5] to setup the collinearity conditions. Let (Xc, Yo the position of a world point P (X,I: n T i n camera coordinates. Let (cx, be the image center in pixels. Let (x, Y ) be ~ the measured image position of point P in pixels. Let (xd, yJT be the distorted image position of point P converted from (x, Y ) ~Let . (xu, y,JTbe the estimated undistorted image position of point P calculated from (&, yd)? Let R be the rotation matrix between the camera coordinates and world coordinates. 169

0-8186-7190-4/95 $4.00 0 1995 IEEE

Static Calibration

Let T be the translation vector between camera coordinates and world coordinates. Let s, be the scale factor between the camera and digitizer. Let d, and dybe the sensor size per pixel in the x and y directions respectively. Let n, be the number of CCD sensors in x direction. Let klbe the first order radial distortion term. Letfc be the effective focal length. The coordinate transform between the world coordinates and camera coordinates are as follows.

1 =

~

~

pjB+[ +

~

=

(1)

Figure 1. A snapshot of the calibration block

The transform between image pixel space and camera coordinates can be summarized as follows.

cl,x,z

CCDwidth X n,

d, =

width (image)

'6x12 ' 6 x 1 2 '6x12

' 6 x 1 2 c26x12 '6x12 '6x12

d y = CCDheight

' 6 x 1 2 ' 6 x 1 2 c36x12 '6x12

The colinear condition and constraints for least square adjustment can be set up as follows. The adjustment method [4] is briefly reviewed in Appendix A. The rotation matrix is assumed to have 9 variables with 6 constraints instead of using 3 angle variables. This setup simplifies the calculations of the partial derivatives as well reduces the numerical errors.

*

..

...

...

...

...

' 6 x 1 2 ' 6 x 1 2 ' 6 x 1 2 cm6xlz

1.

Figure 2. Equation setup for bundle adjustment (Assume that m images are acquired, and n features for each image.)

B'l...w Cl...mand C'l...mif m images are used (see Figure 2. If we have n image,features f?r each image, then the size of each block Bi, B i, Ciand C are 2n x 12,2nx 5,6 x 12 and 6 x 12, respectively. By applying the calculation on non-zero blocks only, the calculation time can be reduced. Detailed derivations can be found in [2].

r r +r r +r r = O 3, 4,

0, 1,

r r +r r 1, 2,

4,

6, 7 ,

+ r7,rg,= 0 5,

r r + r r +rg,r6,= 0 2, 0,

5, 3,

ri, + r:,

+ ri, = 1

(constraints)

(4)

2.2 Experimental Results

r2 +r2 +r; = 1

4

4,

r2 + rt 2,

I

The initial values of parameters are estimated using Tsai's method [5] by applying nominal values for the calibration. The bundle adjustment are then used for finer adjustment. The RMS residues of calibration is below 0.1 pixel for our 1/2inch format CCD camera. Depending on the combination of the camera and the digitizer used in our experiments, the principal point shifted by 10 pixels from the center of the digitized image.

I

+ ri, = 1 i: index of the image

j : index of the image feature position

The matrix B and C for the least square adjustments (see Appendix A) can be divided into m blocks, e.g. B,...,,,

170

Since the focal length is the only factor that affects the camera intrinsic parameters given our assumptions,the image center and radial distortion can be expressed as functions of the effective focal length. Due to the imperfection of the camera model, the parameter functions may not change according to ideal behavior. For example, the radial distortion of the same camera and lens combination should be inversely proportional to the square of the ratio between the two effective focal lengths under a perfect model. Lookup tables can be used for image centers and radial distortions under those situations. The ratio between the number of conditions and unknowns becomes proportional to the number of control points per image as more images are used. Increasing the number of control points is a desired way to increase the accuracy.

3

X

X

I

/ I

camera 2 Figure 3. Coplanar Property.

Camera 1

Dynamic Calibration

After the camera is statically calibrated, camera intrinsic parameters can be derived from these results. Since every intrinsic parameter is either fixed or is a function of the effective focal length, the effective focal length becomes the only intrinsic parameter that needs to be determined during dynamic calibration. The following derivations assume a constant image center and radial distortion for small variations of focal length. Depending on the camera and the digitizer, lookup tables may need to be used for large variations of the focal length. The derivatives with respect to the effective focal length need to be estimated from the lookup tables under these situations.

For each pass point, [XI, y1, fllT, T, and R I@,~ 2f2IT . will be coplanar (see Figure 3). Let T = [r, rl rz] . The volume generated by those three vectors should be zero, because of coplanarity.

--

Using more images for dynamic calibration increases the accuracy of the results. However, it also increases the complexities for automation, e.g. matching for more images, and increases the response time for calibration, e.g. feature detection, matching. In this paper, we derive the conditions and constraints for two images only. However, it can be extended to the multiple images case if necessary.

I-1

I - 1

3.2 Initial guess A closed form solution for determining the R and T of equation (5) has been proposed by Longuet-Higgens[3]. Although not all the constraints are used, LonguetHiggens’s solution can be used as a good initial guess for finding the nonlinear solution of the R and T. With slight modification, the steps are shown as follows.

Relative orientation with coplanar conditions is used because it considers much fewer parameters than does the absolution Orientation with collinear conditions. For example, the 3-D positions of the matched features are not considered as parameters for relative orientation, and they have to be estimated for absolute orientation. However, adding the extra conditions, see section 3.5, tums out to be more complex for the relative orientation cases than the absolute orientation ones.

(a) Solve for the ratio between q’s by taking the pseudo-inverse on coplanar equation (6). (b) Scale the q’s so that the following equation

holds.

3.1 Coplanar Condition

R

c4;= 1

Let the C, coordinates be related to C , coordinates as follows. R and T stand for the rotation matrix and translation vector for a rigid body transform

i=O

(c) Let 7’ = [I, by bJT. by and bz can be solved as

171

qOrO+43tl +46t2 = Q l f O + q4tl+

q7t2 = 0

}

(9)

42‘0+45‘1 +4sr2 =

Equation (7) is a basic assumption for relative orien-

tation to eliminate the ambiguities of scale factor. Equation (8) can be derived by multiplying Q and Q*(see [31).

Let Tbe the normalized vector of T, then T = T / IT I = [to tl r21T. ~ a s e don the sequence of the way images are taken, we assume the second camera position is at the right hand side of the first camera position. The value of to has to be greater than zero.

Equation (9) can be derived from (6).According to the definition of Q, we have Q i = T x R j , i = 1,2,3 :.Qi

(d) Construct the rotation matrix R as follows.

R = [RI R2 R3]

Q=

[Ql Q2

Q3]

Detailed derivations for the adjustments can be found in [2]. This simplified method solves the relative orientation using the essential matrix Q instead of the rotation matrix R. The six constraints on the orthonormal rotation matrix R are exactly the six constraints for the 4’s and t’s. Experimental results are shown in Figure 4 for a set of simulated data.

40 41 42

= q3 q4 45 [6

47 4~

R,=W1+W2xW3 R, = W2 + W3 x Wl R3=W3+W1xW2

As mentioned in section 3.1, the coplanar condition is actually the volume of span of three vectors: ray from left canera, ray from right camera and translation vector (seeFigure 3). This may cause a problem when updating the effective focal length. Changing the focal length to infinity will cause every ray coming out of the camera to be along the principal axis. Also, changing the rotation axis so that the principal axis from one camera is parallel to the other would make the volume go to zero. This kind of global minimum is not desired. Normalizing the coplanar condition is preferred.

W,=Q,xT Wz= Q2xT W3=Q3xT

Similar to the way we determine ro, we assume our stereo camera system is close to the configuration of our eyes. If r, is less than zero, invert the sign of each element in Q, i.e. 4’s. 3.3 Method I: least square adjustment using essential matrix

3.4 Method XI: least square adjustment using normalized coplanar conditions

After the initial guess is done using the method described in section 3.2, we can apply least square adjustment [4] to improve the calibration results, such as R, T and effective focal length. (a) Parameters: = [qO 41 42 43 44 45 46 47 48 ?l t2f1 f2IT9 wherefl andf2 stand for the effective focal length for camera 1 and camera 2, respectively. (b) Conditions: For each observation pair (XI$ yl$) and (%J y2$, form the coplanar equation as shown in (6). (c) Constraints t;+t:+ti

= 1

T = 0, i = 1,2,3

The actual error should be defined as the disparity between the two rays from two cameras; in other words, the distance between the two rays. h t v l =xi

From Figure 5, the error function can be defined as the coplanar condition divided be the span area of v1 and v2 as follows.

(7)

The basic equation setup for this approach is rather straight forward. The parameters used are the rotation

172

Ideal solution 0.862730 0.172325 0.973493 0.215903 0.500000 -0.150384 (1) Observation accuracy to 1 pixel 0.864878 0.175240 0.975783 0.190114 0.501241 -0.110365 0.855525 0.860159 0.172756 0.973359 0.216839 0.504553 -0.150757 0.850117 (2) Observation accuracy to 1/10 pixel 0.864878 0.175240 0.501241 -0.1 10365 0.860159 0.172756 0.504553 -0.150757 0.850117 Figure 4. Experimental results with different observation accuracies

Experimental results are shown in Figure 6. The observation accuracy is simulated to be 1 pixel. The ideal effective focal length is 1.2 inch. The initial effective focal length estimate is 1.0 inch. The ideal solution of R and Tis the same as shown in Figure 4. The results are better than those from method I (see Figure 4). Normally the size of the observed object is much larger than the CCD sensor size for most close-range vision application. The change of effective focal length is not very much. However, calibration of the effective focal length is possible as demonstrated above. A degenerate case is possible when many input points are coplanar.

ray-left > l is assumed because it applies to the constraint equations. W, reflects the confidence in initial guess.

4. Conclusion A two-step camera calibration scheme is proposed. The first step uses absolute orientation to estimate the camera intrinsic parameters. The intrinsic parameters are functions of effective focal length. The second step uses the intrinsic parameters derived from the first step to estimate the relative orientation of stereo cameras by observing an unknown object. Two methods are presented. Method I is simplest. Method I1 offersmore accuracy. Partial knowledge of the observed object, such as parallel lines and perpendicular lines, can be integrated into the bundle adjustment of relative orientation to increase the accuracy.

References H. A. Beyer, “Accurate Calibration of CCD-Cameras,” IEEE Computer Society Conference on CVPR, 1992. C . T.Huang, “Image Analysis for Pose-Determination in Manufacturing Applications,” PhD dissertation, University of Texas at Arlington, 1995. H. C. Longuet-Higgens, “A Computer Algorithm for Reconstructing a Scene from Two Projections,” Nature, Vol. 293, pp.133-135, 10 Sep. 1981. E. M. Mikhail, Observations and Least Squares, Thomas Y. Crowell Company, University Press of America,1976. R. Y. Tsai, “A Versatile Camera Calibration Technique for High-accuracy 3D Machine Vision Metrology Using Off-the-shelf TV Cameras and Lenses,” IEEE Trans. on Robotics and Automation, Vol. 3, N0.4, pp. 323-344, Aug. 1987. J. Weng, I? Cohen, and M. Hemiou, “Camera Calibration with Distortion Models and Accuracy Evaluation,” IEEE Trans. on PAMI, Vol. 14, NO.10, pp. 965-980, Oct. 1992.

This approach allows us to use a single fixed camera for stereo just by rotating the part. The dynamic scheme

can accurately estimate the new camera - point orientation and produce normalized dimensions of the part. Appendix A. Least square adjustment with conditions and constraints Least square adjustment has been widely used for finding optimal solutions for nonlinear systems involving uncertainties [4]. The adjustment setup for unified condi-

174