Camera Calibration

13 downloads 301 Views 63KB Size Report
What is Camera Calibration? • Primarily, finding the quantities internal to the camera that affect the imaging process. ▫ Position of image center in the image.
Camera Calibration

What is Camera Calibration? • Primarily, finding the quantities internal to the camera that affect the imaging process § Position of image center in the image • It is typically not at (width/2, height/2) of image

§ Focal length § Different scaling factors for row pixels and column pixels § Skew factor § Lens distortion (pin-cushion effect)

Motivation • Good calibration is important when we need to § Reconstruct a world model: Virtual L.A. project § Interact with the world • Robot, hand-eye coordination Image plane

We see a square of known size

Evaluation of position of a square for 2 focal lengths (red and blue projection geometry)

Scaling of Rows and Columns in Image • Camera pixels are not necessarily square • Camera output may be analog (NTSC) • Image may be obtained by digitizing card § A/D converter samples NTSC signal Camera

CCD/ CMOS

NTSC signal

Digitizing

Monitor display

Compound Lens Imaging • Inexpensive single lens systems distort image at its periphery • Compound lenses may be used to reduce chromatic effects and pin-cushion effects Camera Image plane Nodal Point

f

Principal planes

Image plane for equivalent pinhole camera is not camera image plane

f C Center of Projection

Variety of Techniques • VERY large literature on the subject • Work of Roger Tsai influential • Linear algebra method described here § Can be used as initialization for iterative non linear methods.

• Some interesting methods use vanishing points

Camera and Calibration Target

Calibration Procedure • Calibration target: 2 planes at right angle with checkerboard patterns (Tsai grid) § We know positions of pattern corners only with respect to a coordinate system of the target § We position camera in front of target and find images of corners § We obtain equations that describe imaging and contain internal parameters of camera • As a side benefit, we find position and orientation of camera with respect to target (camera pose)

Image Processing of Image of Target • Canny edge detection • Straight line fitting to detected linked edges • Intersecting the lines to obtain the image corners • Matching image corners and 3D target checkerboard corners § By counting if whole target is visible in image

• We get pairs (image point)--(world point) ( xi , yi ) → ( X i , Yi , Z i )

Central Projection If world and image points are represented by homogeneous vectors, central projection is a linear transformation:

xs xi = f zs ys yi = f zs

u   f v  =  0    w  0

0 f 0

 xs  0 0   ys    0 0  zs  1 0   1 y

center of projection

x

xi = u / w, yi = v / w

Image plane Scene point (xs , ys , zs ) Image point (xi , yi , f) z

C f

Transformation From Lengths to Pixels Transformation uses: • image center (x0, y0)

y

ypix y0

Image center

• scaling factors kx and ky

x0

xs xi = f zs ys yi = f zs  u '  α x 0  v'  =  0 α y    w'  0 0

x0 y0 1

 xs  0   ys    0 with  zs  0   1

Image point (xi , yi , f)

x xpix

x pix

xs + zs x0 = k x xi + x0 = f k x zs

y pix

y s + z s y0 = k y yi + y0 = f k y zs

α x = f kx α y = f ky

x pix = u ' / w' then

y pix = v' / w'

Internal Camera Parameters  u '  α x  v'  =  0    w'  0 α x s 0 α y   0 0

 xs  s x0 0    x pix = u ' / w' α x = f kx y s  α y y0 0   with  zs  y pix = v' / w' α = − f k y y 0 1 0   1 x0 0 α x s x0  1 0 0 0 y0 0 =  0 α y y0  0 1 0 0 = K [I 3 | 03 ] 1 0  0 0 1  0 0 1 0

• αx and αy “focal lengths” in pixels • x0 and y0 coordinates of image center in pixels •Added parameter s is skew parameter • K is called calibration matrix. Five degrees of freedom. •K is a 3x3 upper triangular matrix

From Camera Coordinates to World Coordinates y C

x Image plane

Image point (xi , yi , f)

Z

z

Y (xs , ys , zs ) M (Xs , Ys , Z s )

O

X

From Camera Coordinates to World Coordinates 2 x X

i j

k

z

M

C

I

(Xs , Ys , Z s ) in world cordinate system (xs , ys , zs ) in image cordinate system

CO = T translation vector J O

K CM = CO + OM xS i + y S j + zS k = Tx i + Ty j + Tz k + X S I + YS J + Z S K

xS = Tx + X S I.i + YS J.i + Z S K.i  xS  Tx   I.i  y  = T  +  I.j  S   y   z S  Tz  I.k

K.i   X S  J.j K.j   YS  J.k K.k   Z S  J.i

Z

Homogeneous Coordinates  xS  Tx   I.i J.i K.i   X S   y  = T  +  I.j J.j K.j   Y   S  y   S   z S  Tz  I.k J.k K.k   Z S   xS   I.i  y   I.j  S =   z S  I.k    1  0

Tx   X S  Ty   YS  J.k K.k Tz   Z S    0 0 1  1  J.i J.j

K.i K.j

 xS  y   S = R  z S  0T3   1

X S  T  YS  1   Z S     1 

From Camera Coordinates to World Coordinates 3 x X

i j

k

z

C

I

M (Xs , Ys , Z s ) (xs , ys , zs )

J

CM = OM - OC

O

K

Z

xS i + yS j + z S k = ( X S - X C ) I + (YS - YC ) J + ( Z S - Z C ) K

xS = ( X S - X C ) I.i + (YS - YC ) J.i + ( Z S - Z C ) K.i ~ x cam = R (X - C)

~ (T = -R C)

~ C is vector OC expressed in world coordinate system

Homogeneous Coordinates 2 ~ • Here we use - R C instead of T

 xS  y    S = R  z S  0T3   1

XS  ~   - R C  YS   1  ZS     1 

Linear Transformation from World Coordinates to Pixels • Combine camera projection and coordinate transformation matrices into a single matrix P u'  v '  = K [I 3   w'

 xS  y    S = R  z S  0T3   1

 xs  y  | 0 3 ] s   zs    1

XS  ~   - R C  YS   1  ZS     1 

u'  R   ⇒  v'  = K [I 3 | 0 3 ] T 0 3  w' XS  u' Y  ⇒  v'  = P  S  ZS  w'   1 

XS  ~   - R C  YS   1  Z S     1 

x = PX

Properties of Matrix P • Further simplification of P: R | 0 3 ] T 0 3

x = K [I 3

[I 3

R | 0 3 ] T 0 3

[

~ - R C X 1 

~ - R C ~ ~ = R R C = R I | C  3 1 

[

] [

]

]

~ x = K R I3 | -C X

[

~ P = K R I3 | - C

]

• P has 11 degrees of freedom: ~ • 5 from triangular calibration matrix K, 3 from R and 3 from C • P is a fairly general 3 x 4 matrix •left 3x3 submatrix KR is non-singular

Calibration • 1. Estimate matrix P using scene points and their images • 2. Estimate the interior parameters and the exterior parameters

[

~ P = K R I3 | - C

]

§ Left 3x3 submatrix of P is product of uppertriangular matrix and orthogonal matrix

Finding Camera Translation • Find homogeneous coordinates of C in the scene • C is the null vector of matrix P § P C = 0:

[

P = K R I3

~ | -C

]

1 0 0 0 1 0  0 0 1

 X c  0 X c     Yc  0   Yc  =  Z c  0 Z c       1  0

• Find null vector C of P using SVD § C is the unit singular vector of P corresponding to the smallest singular value (the last column of V, where P = U D VT is the SVD of P)

Finding Camera Orientation and Internal Parameters • Left 3x3 submatrix M of P is of form M=K R § K is an upper triangular matrix § R is an orthogonal matrix

• Any non-singular square matrix M can be decomposed into the product of an uppertriangular matrix K and an orthogonal matrix R using the RQ factorization § Similar to QR factorization but order of 2 matrices is reversed

RQ Factorization of M 1 0 0   c' 0 s ' c' ' − s' ' 0 R x = 0 c − s  , R y =  0 1 0  , R z = s ' ' c' ' 0 0 s c  − s ' 0 c'  0 0 1

m33 m32 , s= 2 • Compute c = − 2 2 1/ 2 2 1/ 2 ( m32 + m33 ) (m32 + m33 ) •

Multiply M by Rx. The resulting term at (3,2) is zero because of the values selected for c and s • Multiply the resulting matrix by Ry, after selecting c’ and s’ so that the resulting term at position (3,1) is set to zero • Multiply the resulting matrix by Rz, after selecting c’’ and s’’ so that the resulting term at position (2,1) is set to zero

M R x R y R z = K ⇒ M = K R Tz R Ty R Tx = K R

Computing Matrix P • Use corresponding image and scene points § 3D points Xi in world coordinate system § Images xi of Xi in image

• Write xi = P Xi for all i • Similar problem to finding projectivity matrix H (i.e. homography) in homework

Improved Computation of P • xi = P Xi involves homogeneous coordinates, thus xi and P Xi just have to be proportional: x i × P Xi = 0 • Let p1T, p2T, p3T be the 3 row vectors of P  v'i p T3 Xi − w'i p T2 Xi    T T xi × P X i =  w'i p1 Xi − u 'i p 3 Xi   u'i p T2 X i − v'i p1T X i   

p 1T X i   T  P X i = p 2 X i  p T3 X i   

 0T4  ⇒  w'i XTi − v'i XTi 

− w'i X Ti 0T4 u 'i X Ti

v 'i XTi  p1   T  − u'i X i  p 2  = 0 0T4  p 3 

p1    p2  is a 12 × 1 vector p3   

Improved Computation of P, cont’d • Third row can be obtained from sum of u’i times first row - v’i times second row  0T4  T w ' X  i i  − v'i X Ti 

− w'i X Ti 0T4 u 'i XTi

v'i X Ti  p1    − u 'i X Ti  p 2  = 0 0T4  p 3 

• So we get 2 independent equations in 11 unknowns (ignoring scale) • With 6 point correspondences, we get enough equations to compute matrix P Ap=0

Solving A p = 0 • Linear system A p = 0 • When possible, have at least 5 times as many equations as unknowns (28 points) • Minimize || A p || with the constraint || p || = 1 § P is the unit singular vector of A corresponding to the smallest singular value (the last column of V, where A = U D V T is the SVD of A)

• Called Direct Linear Transformation (DLT)

Improving P Solution with Nonlinear Minimization • Find p using DLT • Use as initialization for nonlinear 2 minimization of ∑ d ( x i , PX i ) i

§ Use Levenberg-Marquardt iterative minimization

Radial Distortion • We have assumed that lines are imaged as lines • Not quite true for real lenses § Significant error for cheap optics and for short focal lengths

Scene

Image

Corrected image

Radial Distortion Modeling • In pixel cordinates the correction is written xc − x0 = L(r ) ( x − x0 ) (x0, y0)

(x, y) (xc, yc)

yc − y0 = L ( r ) ( y − y0 ) with r 2 = ( x − x0 ) 2 + ( y − y0 ) 2 L(r ) = 1 + κ 1 r + κ 2 r 2 + ...

Distorted line And ideal line: (x, y) (x’, y’)

• Minimize f (κ 1 ,κ 2 ) = ∑ ( x'i − xci ) 2 + ( y 'i − yci ) 2 i using lines known to be straight (x’,y’) is radial projection of (x,y) on straight line

References • Multiple View Geometry in Computer Vision, R. Hartley and A. Zisserman, Cambridge University Press, 2000, pp. 138-183 • Three-Dimensional Computer Vision: A Geometric Approach, O. Faugeras, MIT Press, 1996, pp. 33-68 • “A Versatile Camera Calibration Technique for 3D Machine Vision”, R. Y. Tsai, IEEE J. Robotics & Automation, RA-3, No. 4, August 1987, pp. 323-344