What is Camera Calibration? • Primarily, finding the quantities internal to the
camera that affect the imaging process. ▫ Position of image center in the image.
Camera Calibration
What is Camera Calibration? • Primarily, finding the quantities internal to the camera that affect the imaging process § Position of image center in the image • It is typically not at (width/2, height/2) of image
§ Focal length § Different scaling factors for row pixels and column pixels § Skew factor § Lens distortion (pin-cushion effect)
Motivation • Good calibration is important when we need to § Reconstruct a world model: Virtual L.A. project § Interact with the world • Robot, hand-eye coordination Image plane
We see a square of known size
Evaluation of position of a square for 2 focal lengths (red and blue projection geometry)
Scaling of Rows and Columns in Image • Camera pixels are not necessarily square • Camera output may be analog (NTSC) • Image may be obtained by digitizing card § A/D converter samples NTSC signal Camera
CCD/ CMOS
NTSC signal
Digitizing
Monitor display
Compound Lens Imaging • Inexpensive single lens systems distort image at its periphery • Compound lenses may be used to reduce chromatic effects and pin-cushion effects Camera Image plane Nodal Point
f
Principal planes
Image plane for equivalent pinhole camera is not camera image plane
f C Center of Projection
Variety of Techniques • VERY large literature on the subject • Work of Roger Tsai influential • Linear algebra method described here § Can be used as initialization for iterative non linear methods.
• Some interesting methods use vanishing points
Camera and Calibration Target
Calibration Procedure • Calibration target: 2 planes at right angle with checkerboard patterns (Tsai grid) § We know positions of pattern corners only with respect to a coordinate system of the target § We position camera in front of target and find images of corners § We obtain equations that describe imaging and contain internal parameters of camera • As a side benefit, we find position and orientation of camera with respect to target (camera pose)
Image Processing of Image of Target • Canny edge detection • Straight line fitting to detected linked edges • Intersecting the lines to obtain the image corners • Matching image corners and 3D target checkerboard corners § By counting if whole target is visible in image
• We get pairs (image point)--(world point) ( xi , yi ) → ( X i , Yi , Z i )
Central Projection If world and image points are represented by homogeneous vectors, central projection is a linear transformation:
xs xi = f zs ys yi = f zs
u f v = 0 w 0
0 f 0
xs 0 0 ys 0 0 zs 1 0 1 y
center of projection
x
xi = u / w, yi = v / w
Image plane Scene point (xs , ys , zs ) Image point (xi , yi , f) z
C f
Transformation From Lengths to Pixels Transformation uses: • image center (x0, y0)
y
ypix y0
Image center
• scaling factors kx and ky
x0
xs xi = f zs ys yi = f zs u ' α x 0 v' = 0 α y w' 0 0
x0 y0 1
xs 0 ys 0 with zs 0 1
Image point (xi , yi , f)
x xpix
x pix
xs + zs x0 = k x xi + x0 = f k x zs
y pix
y s + z s y0 = k y yi + y0 = f k y zs
α x = f kx α y = f ky
x pix = u ' / w' then
y pix = v' / w'
Internal Camera Parameters u ' α x v' = 0 w' 0 α x s 0 α y 0 0
xs s x0 0 x pix = u ' / w' α x = f kx y s α y y0 0 with zs y pix = v' / w' α = − f k y y 0 1 0 1 x0 0 α x s x0 1 0 0 0 y0 0 = 0 α y y0 0 1 0 0 = K [I 3 | 03 ] 1 0 0 0 1 0 0 1 0
• αx and αy “focal lengths” in pixels • x0 and y0 coordinates of image center in pixels •Added parameter s is skew parameter • K is called calibration matrix. Five degrees of freedom. •K is a 3x3 upper triangular matrix
From Camera Coordinates to World Coordinates y C
x Image plane
Image point (xi , yi , f)
Z
z
Y (xs , ys , zs ) M (Xs , Ys , Z s )
O
X
From Camera Coordinates to World Coordinates 2 x X
i j
k
z
M
C
I
(Xs , Ys , Z s ) in world cordinate system (xs , ys , zs ) in image cordinate system
CO = T translation vector J O
K CM = CO + OM xS i + y S j + zS k = Tx i + Ty j + Tz k + X S I + YS J + Z S K
xS = Tx + X S I.i + YS J.i + Z S K.i xS Tx I.i y = T + I.j S y z S Tz I.k
K.i X S J.j K.j YS J.k K.k Z S J.i
Z
Homogeneous Coordinates xS Tx I.i J.i K.i X S y = T + I.j J.j K.j Y S y S z S Tz I.k J.k K.k Z S xS I.i y I.j S = z S I.k 1 0
Tx X S Ty YS J.k K.k Tz Z S 0 0 1 1 J.i J.j
K.i K.j
xS y S = R z S 0T3 1
X S T YS 1 Z S 1
From Camera Coordinates to World Coordinates 3 x X
i j
k
z
C
I
M (Xs , Ys , Z s ) (xs , ys , zs )
J
CM = OM - OC
O
K
Z
xS i + yS j + z S k = ( X S - X C ) I + (YS - YC ) J + ( Z S - Z C ) K
xS = ( X S - X C ) I.i + (YS - YC ) J.i + ( Z S - Z C ) K.i ~ x cam = R (X - C)
~ (T = -R C)
~ C is vector OC expressed in world coordinate system
Homogeneous Coordinates 2 ~ • Here we use - R C instead of T
xS y S = R z S 0T3 1
XS ~ - R C YS 1 ZS 1
Linear Transformation from World Coordinates to Pixels • Combine camera projection and coordinate transformation matrices into a single matrix P u' v ' = K [I 3 w'
xS y S = R z S 0T3 1
xs y | 0 3 ] s zs 1
XS ~ - R C YS 1 ZS 1
u' R ⇒ v' = K [I 3 | 0 3 ] T 0 3 w' XS u' Y ⇒ v' = P S ZS w' 1
XS ~ - R C YS 1 Z S 1
x = PX
Properties of Matrix P • Further simplification of P: R | 0 3 ] T 0 3
x = K [I 3
[I 3
R | 0 3 ] T 0 3
[
~ - R C X 1
~ - R C ~ ~ = R R C = R I | C 3 1
[
] [
]
]
~ x = K R I3 | -C X
[
~ P = K R I3 | - C
]
• P has 11 degrees of freedom: ~ • 5 from triangular calibration matrix K, 3 from R and 3 from C • P is a fairly general 3 x 4 matrix •left 3x3 submatrix KR is non-singular
Calibration • 1. Estimate matrix P using scene points and their images • 2. Estimate the interior parameters and the exterior parameters
[
~ P = K R I3 | - C
]
§ Left 3x3 submatrix of P is product of uppertriangular matrix and orthogonal matrix
Finding Camera Translation • Find homogeneous coordinates of C in the scene • C is the null vector of matrix P § P C = 0:
[
P = K R I3
~ | -C
]
1 0 0 0 1 0 0 0 1
X c 0 X c Yc 0 Yc = Z c 0 Z c 1 0
• Find null vector C of P using SVD § C is the unit singular vector of P corresponding to the smallest singular value (the last column of V, where P = U D VT is the SVD of P)
Finding Camera Orientation and Internal Parameters • Left 3x3 submatrix M of P is of form M=K R § K is an upper triangular matrix § R is an orthogonal matrix
• Any non-singular square matrix M can be decomposed into the product of an uppertriangular matrix K and an orthogonal matrix R using the RQ factorization § Similar to QR factorization but order of 2 matrices is reversed
RQ Factorization of M 1 0 0 c' 0 s ' c' ' − s' ' 0 R x = 0 c − s , R y = 0 1 0 , R z = s ' ' c' ' 0 0 s c − s ' 0 c' 0 0 1
m33 m32 , s= 2 • Compute c = − 2 2 1/ 2 2 1/ 2 ( m32 + m33 ) (m32 + m33 ) •
Multiply M by Rx. The resulting term at (3,2) is zero because of the values selected for c and s • Multiply the resulting matrix by Ry, after selecting c’ and s’ so that the resulting term at position (3,1) is set to zero • Multiply the resulting matrix by Rz, after selecting c’’ and s’’ so that the resulting term at position (2,1) is set to zero
M R x R y R z = K ⇒ M = K R Tz R Ty R Tx = K R
Computing Matrix P • Use corresponding image and scene points § 3D points Xi in world coordinate system § Images xi of Xi in image
• Write xi = P Xi for all i • Similar problem to finding projectivity matrix H (i.e. homography) in homework
Improved Computation of P • xi = P Xi involves homogeneous coordinates, thus xi and P Xi just have to be proportional: x i × P Xi = 0 • Let p1T, p2T, p3T be the 3 row vectors of P v'i p T3 Xi − w'i p T2 Xi T T xi × P X i = w'i p1 Xi − u 'i p 3 Xi u'i p T2 X i − v'i p1T X i
p 1T X i T P X i = p 2 X i p T3 X i
0T4 ⇒ w'i XTi − v'i XTi
− w'i X Ti 0T4 u 'i X Ti
v 'i XTi p1 T − u'i X i p 2 = 0 0T4 p 3
p1 p2 is a 12 × 1 vector p3
Improved Computation of P, cont’d • Third row can be obtained from sum of u’i times first row - v’i times second row 0T4 T w ' X i i − v'i X Ti
− w'i X Ti 0T4 u 'i XTi
v'i X Ti p1 − u 'i X Ti p 2 = 0 0T4 p 3
• So we get 2 independent equations in 11 unknowns (ignoring scale) • With 6 point correspondences, we get enough equations to compute matrix P Ap=0
Solving A p = 0 • Linear system A p = 0 • When possible, have at least 5 times as many equations as unknowns (28 points) • Minimize || A p || with the constraint || p || = 1 § P is the unit singular vector of A corresponding to the smallest singular value (the last column of V, where A = U D V T is the SVD of A)
• Called Direct Linear Transformation (DLT)
Improving P Solution with Nonlinear Minimization • Find p using DLT • Use as initialization for nonlinear 2 minimization of ∑ d ( x i , PX i ) i
§ Use Levenberg-Marquardt iterative minimization
Radial Distortion • We have assumed that lines are imaged as lines • Not quite true for real lenses § Significant error for cheap optics and for short focal lengths
Scene
Image
Corrected image
Radial Distortion Modeling • In pixel cordinates the correction is written xc − x0 = L(r ) ( x − x0 ) (x0, y0)
(x, y) (xc, yc)
yc − y0 = L ( r ) ( y − y0 ) with r 2 = ( x − x0 ) 2 + ( y − y0 ) 2 L(r ) = 1 + κ 1 r + κ 2 r 2 + ...
Distorted line And ideal line: (x, y) (x’, y’)
• Minimize f (κ 1 ,κ 2 ) = ∑ ( x'i − xci ) 2 + ( y 'i − yci ) 2 i using lines known to be straight (x’,y’) is radial projection of (x,y) on straight line
References • Multiple View Geometry in Computer Vision, R. Hartley and A. Zisserman, Cambridge University Press, 2000, pp. 138-183 • Three-Dimensional Computer Vision: A Geometric Approach, O. Faugeras, MIT Press, 1996, pp. 33-68 • “A Versatile Camera Calibration Technique for 3D Machine Vision”, R. Y. Tsai, IEEE J. Robotics & Automation, RA-3, No. 4, August 1987, pp. 323-344