Czech Technical University, Prague The Center for Machine Perception
Matching of 3-D Curves using Semi-di erential Invariants Tomas Pajdla
Luc Van Gool
Computer Vision Laboratory Czech Technical University Karlovo namest 13 121 35 Prague, Czech Republic
[email protected]
ESAT { MI2 Katholieke Universiteit Leuven Kardinaal Mercierlaan 94 3001 Leuven, Belgium
[email protected]
October 3, 1996
Reference [1] Tomas Pajdla and Luc Van Gool. Matching of 3-D curves using semi-di erential invariants. In 5th International Conference on Computer Vision, pages 390{395, Cambridge, MA, June 1995. IEEE Computer Society Press. This publication can be obtained via anonymous ftp from ftp://cmp.felk.cvut.cz/pub/cvl/ARTICLES/PAJDLA/iccv95.ps.gz
Czech Technical University, Faculty of Electrical Engineering Department of Control Engineering, Center for Machine Perception, Computer Vision Laboratory 121 35 Prague 2, Karlovo namest 13, Czech Republic, FAX +42 2 290159, phone +42 2 24357465 http://cmp.felk.cvut.cz
Matching of 3-D Curves using Semi-di erential Invariants Tomas Pajdla
Luc Van Gool
Computer Vision Laboratory Czech Technical University Karlovo namest 13 121 35 Prague, Czech Republic
[email protected]
ESAT { MI2 Katholieke Universiteit Leuven Kardinaal Mercierlaan 94 3001 Leuven, Belgium
[email protected]
Abstract
A method for matching 3-D curves under Euclidean motions is presented. Our approach uses a semidi erential invariant description requiring only rst derivatives and one reference point, thus avoiding the computation of high order derivatives. A novel curve similarity measure building on the notion of -reciprocal correspondence is proposed. It is shown that by combining -reciprocal correspondence with the robust Least Median of Squares motion estimation, the registration of partially occluded curves can be accomplished. An experiment with real curves extracted from 3-D surfaces demonstrates that curve matching can be successfully performed even on data from a simple and cheap 3-D sensor.
1 Introduction
A method for matching 3-D curves independently of the Euclidean motion they may undergo (i.e. they may rotate and translate arbitrarily in three-dimensional space) is given. It is assumed that 3-D coordinates of curve points are available. In our work, these are obtained through the use of an active range scanner, but they could of course be obtained through shapefrom-motion or stereo as well. Space curves will often lie on a surface, what causes additional problems with occlusions. Hence, it seems crucial to generate descriptions that are not only invariant under 3-D motions, but that also remain useful under conditions of incomplete visibility. An approach based on a global L2 similarity measure was published by Schwartz and Sharir [7]. Their algorithm accomplished matching of curve segments to complete curve models, but except for some occlusion of the curves to be recognized, no other gross errors were allowed. A method based on local invariant descriptions, curvature and torsion, was introduced by Kishon, Hastie, and Wolfson [3]. In their algorithm, matching hypotheses are generated by geometrical hashing and veri ed by Least squares reg-
This work has been done during the visit of T. Pajdla at ESAT, K.U.Leuven. Support by Esprit Basic Research Action `VIVA' and the Belgian project IUAP-50 on Robotics and Industrial automation is gratefully acknowledged.
istration. Gueziec and Ayache [2] have extended the approach by Kishon et al. using an iterative B-spline tting method to obtain stable curvature and torsion. They also compared the performance of di erential descriptions (curvature and torsion) and a mixed descriptions (curvature, length of the vector between a reference and a curve point, and angles between a reference direction and a curve tangent vector) and found the second one to be more stable. As to the rst issue { invariance under motions { the description presented in this paper can be viewed as an alternative to the approach used in [2]. There, an elaborate technique for computing stable 3rd derivatives is developed but nally a combination of curvature with some semi-di erential invariants is used. We explicitly introduce the framework of semi-di erential invariance (invariants combining low order derivatives at more than one point) thus avoiding the combination of dicult problems of both approaches { stable computation of high derivatives and reference point selection (e.g. avoiding 2nd derivatives altogether). The second issue { matching occluded and noisy curves { is handled by the use of a robust Median of Least Squares motion estimation and a segment to segment rather than a curve to curve similarity measure building on the -reciprocal correspondence. This paper is organized as follows. In the next section we introduce semi-di erential descriptions of curves and discuss nding reference points. Section 3 describes how geometrical hashing generates matches between curves. Since hashing alone might generate a number of false matches we evaluate a curve similarity measure (3.2) on robustly registered curves (3.1, 3.3). Finally, section 4 presents an experiment.
2 Semi-di erential descriptions of curves
The invariant description of the curve should be such that it is the same, irrespective of the viewpoint under which it is observed. In order to arrive at an invariant, the number of measurements, e.g. coordinates of points or derivatives, has to surpass the number or group parameters, here 6 parameters of 3-D Euclidean motions. When
derivatives of the curve coordinates with respect to the curve parameter are involved, both the group of Euclidean motions and the e ect of \reparametrization" have to be taken into account when counting the group parameters [9]. If only measurements in a single point are to be considered, then up to the third derivative of that point's coordinates have to be used, in order to generate 3 independent invariants (12 measurements minus 6 Euclidean group parameters and 3 reparameterization parameters yields 3 independent invariants). On the other hand, considering two points and their rst derivatives also yields sucient information (126-2=4 independent invariants). This combination of measurements is considered in this paper. For a discussion on other alternative descriptions, the reader is referred to [4]. We use the following three independent invariants i1 = kx ? x1 k ; ? x1 ) x(1) ; i2 = (x(1) (1) kx k kx ? x1 k (1) ? x 1 ) x1 ; i3 = (x(1) kx1 k kx ? x1 k where x1 plays the role of a xed reference point, whereas x slides along the curve to generate invariants for all curve points. x(1) and x(1) 1 stand for the tangent vector at x and x1 , respectively. The rst invariant is the distance between x and x1 , whereas the latter two give the cosine of the angles between x ? x1 and x(1) and x(1) 1 , respectively. The choice of these invariants was motivated by the experiment. In particular, this set has better noise immunity than the alternative sets of three independent invariants, e.g. three angles between x ? x1 , x(1) , and x(1) 1 . Notice that the invariants require only the rst derivatives in two distinct points, x and x1 , on a curve.
2.1 Finding reference points
Semi-di erential invariants in two corresponding points on di erent curves equal only if the same, i.e. corresponding, reference points are used. Indeed, it is assumed that at least one point on the curves is already matched. In order to assure that the same reference points are used, a reference point selection strategy is called for. In this paper we exploit a highly robust but brute force strategy already proposed in [2]. This strategy uses all point samples on model curves as candidate reference points. In fact, this is equivalent to generating invariant descriptions for all pairs of points on the curve requiring O(n2 ) storage space. On the other hand, the \test curves" extracted from the scene for matching can be described using only one, arbitrarily chosen reference point, since every point has to have its corresponding point on some model curve1.
1 Here we assume that the model curves are not occluded. If this is not true, the same, brute force, strategy using all points as reference points has to be applied to the test curves.
In [5] we develop an alternative approach allowing to reduce the storage space costs from O(n2 ) to O(n). It utilizes the continuity of the curve by choosing reference points as the points having a constant distance along the curve to the point where the curve is being described. Thus the reference point x1 is \attached" to the point x in an invariant manner and \slides" behind and/or in front of x as the latter moves. A second strategy in [5] replaces the distance along the curve by the angle between the tangents in x and x1 . This strategy better adapts to the level of curve detail.
3 Curve matching
Ecient matching techniques based on geometrical hashing have been previously used in [3] and [2]. Here, the scheme described in [2] is followed. For each point pair, point x - reference point x1 on a model curve, we compute the invariants (i1 , i2 , i3) and create the description entry containing (Ic , Ip , Irp ), where Ic is the index of the model curve and Ip resp. Irp denote the index of the point resp. the index of the reference point on the curve. Then, a three dimensional table indexed by the indices (i1 , i2 , i3 ) is built and lled with the pointers to the corresponding description entries. In the matching phase, an arbitrary, but only one reference point is chosen on the \test" curve to be matched and the invariants for all the points on the curve are computed for that particular reference point. The correspondences between test and model curves are then established by addressing the hash table through the invariants and reading out the description entries. Actually, all model curves with entries within some uncertainty ellipsoid form hypothetical correspondences. The above procedure generates hypothetical correspondences with model curves for all choices of reference points. But, since only one reference point on the test curve is used, we know that all correspondences have to have the same reference point on the model curve. For each model curve we nd the reference point on it, say Irp , as the one getting the maximum number of votes from the test curve. Only those hypothetical correspondences having their reference point indices equal to Irp form the nal set of matching candidates between curves.
3.1 Robust motion estimation
The motion estimation from point correspondences can be naturally formulated as minimizing the sum of squared distances between the points. However, straightforward application of the Least Squares estimator (LSQ) would certainly fail due to outliers resulting from mismatched points. To remedy the situation we introduce a two step robust motion estimation algorithm, but rst we discuss LSQ a little further. LSQ Euclidean motion estimation from a set of registered points, say fxi g and fyi g, i = 1; 2; ; n aims at obtaining a matrix of rotation, R, and a vector of translation, t, so that the overall mean squared error is minimized. The theorem given by Umeyama in [8] says that the optimal Euclidean transformation mini-
mizing the mean squared error
e2 (R; t) = n1
X n
X n
kyi ? (R xi + t)k2 ;
=1 is given by R = U S V T ; t = y ? R x , where x and y are the centroids of the fxi g and fyi g data sets respectively, and U D V T is a singular value decompon X sition of the matrix xy = n1 (yi ? y )(xi ? x )T . i=1 The matrix S is given by
S=
=1
ri2 = n1
i
i
I
if det(UV ) = 1 diag(1; 1; ; 1; ?1) if det(UV ) = ?1
The LMSq estimator as proposed by Rousseeuw and Leroy [6] provides robust estimation even if a large number of outliers is present. The trick P is to replace the standard least squares criterion min ni=1 ri 2 by its robust version min median(fri2 g). Such proposed estimator can cope with up to 50% of outliers. The proposed LMSq motion estimation algorithm consists of two steps: 1. Robust LMSq t and outlier detection. From the population of all correspondences m random triples of corresponding points are drawn and the LSQ optimal Euclidean motion is generated for each triple. The median of squared errors with respect to all points is stored together with the transformation. When a certain number of samples is exceeded [6], the transformation minimizing the median criterion among all drawn transformations is chosen. The residuals are used to eliminate outliers. 2. Trimmed LSQ t. After outliers are removed an ordinary LSQ t is performed for all inliers providing the nal optimal transformation.
3.2 Similarity of curves
The L2 curve matching metric for the model curve ized by their arc-lengths, with the reparameterization o set s0 is given by
y(s) and the test curve x(s + s0 ), both parameterl
2 t 0 ky(s) ? R x(s + s0) ? tk ds :
min L2 (x; y) = min s0 R;
Z
The major drawback of such a metric is that it is susceptible to gross errors since even local errors in the arc-length estimation accumulate and nally can completely spoil the similarity measure. To remedy the situation we introduce the similarity measures building on the L2 metric but avoiding the use of a global arc-length parametrization. Let us assume that the optimal R and t were found and all what remains to be done is to evaluate the similarity. The point of departure for curve comparison is a measure we coin the global similarity of curves (GSC).
It takes into account the local error in the matching as well as the di erence in lengths of the curves. Curves x(s) and y(s) are sampled following their arc lengths in equidistant intervals ds providing two point sets X = fxi g and Y = fyj g. Then, the following formula Pm d(xi; Y ) + Pn d(yj ; X ) ?1 i=1 j =1 GSC (X; Y ) = ; 2 min(m; n) d(xi ; Y ) = 8ymin2Y kxi ? yj k ; j
de nes the GSC measure. Here, m and n are the numbers of points in the sets X and Y respectively. The function d(xi ; Y ) assigns to the point xi from the set X the distance to its closest point in the set Y . Notice that the relation \closest point" does not obey, in general, symmetry. See Fig. 1. The similarity meax(s) y(s) sure for partially overlapping curves (PSC) is a robust version of the λ CPY GSC measure. Unlike y GSC, where all points x(sx ) y contribute to the similarity, here only -reciprocal ∆s x λ CPX points are considered. x(s ) In short, \-reciprocal x points" are the points for Figure 1: Arc length which the closest point distance 4sx(sx ; R; t) of relation is almost sym- nonreciprocal points remetrical. lated by the closest point If the curves x(sx ) relation. and y(sy ) are exactly registered the points in corresponding parts of curves coincide. It implies that the distances between coincident points are zero and the closest point relation is symmetrical there. Such points are called the reciprocal points (RP). If a small Euclidean motion is applied to one of the curves, the symmetry is broken and two locally continuous closest point relations emerge [4]: sy = CP X (sx ) and sx = CP Y (sy ). Fig. 1 illustrates these relations. Let the function 4sx(sx ; R; t) = sx ? sxy = sx ? CP Y (CP X (sx )) measure the arc-length distance between the point x(sx ) and the point x(sxy ). Similarly, 4sy (sy ; R; t) is de ned. The local continuity of 4sx(sx ; R; t) and 4sy (sy ; R; t) in parameters of the motion assures that the arc-length di erence is small for small perturbations of R and t, thus yielding the following rule of thumb: locally consistent correspondences are only -reciprocal points i.e. points having 4sx(sx ; R; t) and 4sy (sy ; R; t) . The similarity measure for partially overlapping curves (PSC) is then given by 0 X d(x ; y ) 1?1 i i B CC 8 (xi ;yi )2RP B@ PSC (X; Y ) = m+n A RP
= f(x; y) j x 2 X; 4sx(sx (x)) ; ^ y 2 Y; 4sy (sy (y)) g;
where m and n stand for the number of -reciprocal points in the X and Y set respectively, and RP denotes the set of all -reciprocal point pairs.
Curves No. 1, 1,
Curves No. 2, 2, 2
140
125 1
130 120
120
1
150
150
200
200
250
250
300
300
350
350
400
400
450
450
500
500
2
−>y
50 100
−>y
115
50 100
110
110
100 105
90
200
80
100 −20
250
0
0
20
300
50
40
x
Curves No. 3, 3,
700
240
260 x
100
280
60
Curves No. 4, 4,
700 145
160
140
155
4
3
135
150
180
9
180
160
160
140
140
11 5
−>y
1
−>y
8
120 100
4
200
5
3 9
67
220
60 −100
240
−50 0
110 −20
200
60
100
280
50 x
130
220
13
0
260
135
115
12
1 2
80
4
140
120 4
67
120 100
80
3
125 10
3
2
−>y
200
145
−>y
130 8
10
200
In this section we present an experiment. The goal is to bring two cylindrical soft drink cans into registration through the matching of feature curves detected on their surfaces.
Figure 3: Candidate point matches produced by hashing are joined by lines. A set of 3-D point samples of cans (Fig. 2c) was measured using the one-shot Illumination Coded Range Finder developed previously at ESAT [10] (Fig. 2a). Since the 3-D point set sampled from the surface is in one-to-one correspondence with its projection on the image (Fig. 2a), the topology of the surface can be de ned in terms of a 2-D triangulation of image points. Here, a 2-D Delaunay triangulation was used. Once the triangulation is obtained a C 0 surface can be constructed by a linear interpolation over the triangles. The curves (Fig. 2d) are extracted by backprojecting the edges found in the gray level image (Fig. 2b) onto the surface. Since our description requires rst order derivatives, we construct a C 1 continuous curve passing through the points. Vertices of a polygonal approximation are subsequently smoothed with a B-spline. Spline knots are distributed over a parameter interval according to the length of the polygonal segments providing a parametrization invariant to Euclidean motions. Finally, the curves are uniformly resampled following their arc length with a constant step ds assuring that no two strings of samples are systematically shifted by more than ds=2. The curves detected for the can on the left are hereafter called the model curves, while those found for the can on the right are called the test curves. Each model curve is described using all its points as reference points and the hash table is lled with that description. The test curves are described using only one arbitrarily selected reference point but for both orientations of the curves. This is necessary since di erent orientations of the curves induce sign changes of i2 and i3 (see eq. (1)). The matching via indexing
Curves No. 1, 1,
Curves No. 2, 2,
Curve−Curve Similarity Table model/test
135
1
2
3
4
5
6
7
8
9
Curve−Curve Similarity Table 10
11
12
13
model/test
1
2
3
4
5
6
7
8
9
10
11
12
13
126 1
1
2.22 0.00 0.02 0.02 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1
2.22 0.20 0.22 0.20 0.17 0.15 0.66 0.36 0.32 0.12 0.20 0.24 0.43
122
2
0.00 3.53 0.04 0.06 0.26 0.00 0.08 0.02 0.06 0.32 0.15 0.08 0.05
2
0.32 3.61 0.14 0.30 0.32 0.25 0.44 0.25 0.31 0.43 0.26 0.29 0.51
120
3
0.02 0.03 0.68 0.14 0.04 0.01 0.01 0.02 0.02 0.03 0.01 0.01 0.01
3
0.15 0.24 1.27 0.26 0.26 0.29 0.27 0.33 0.37 0.43 0.31 0.31 0.78
4
0.02 0.05 0.15 0.85 0.08 0.00 0.02 0.03 0.03 0.05 0.02 0.01 0.01
4
0.18 0.49 0.22 0.86 0.25 0.14 0.22 0.13 0.20 0.27 0.22 0.48 0.43
5
0.01 0.10 0.06 0.08 0.40 0.00 0.12 0.06 0.13 0.15 0.04 0.03 0.03
5
0.21 0.28 0.59 0.26 0.32 0.39 0.31 0.31 0.44 0.18 0.42 0.44 0.42
6
0.01 0.00 0.01 0.01 0.01 0.06 0.00 0.04 0.01 0.01 0.00 0.00 0.00
6
0.20 0.23 0.18 0.18 0.72 0.80 1.23 0.20 0.31 0.27 0.27 0.00 0.45
7
0.01 0.00 0.01 0.01 0.01 0.03 0.00 0.04 0.01 0.01 0.00 0.00 0.00
7
0.13 0.26 0.24 0.09 0.33 0.25 0.49 0.16 0.19 0.29 0.24 0.24 0.43
8
0.00 0.05 0.02 0.03 0.13 0.00 0.16 0.12 0.62 0.10 0.00 0.02 0.02
8
0.20 0.23 0.38 0.19 0.52 0.56 1.14 0.44 0.63 0.22 0.00 0.39 0.68
9
0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00
9
0.24 0.23 0.28 0.24 0.09 0.25 0.13 0.16 0.17 0.29 0.24 0.37 0.48
0.00 0.07 0.02 0.03 0.12 0.00 0.46 0.05 0.16 0.09 0.07 0.00 0.06 10
0.21 0.26 0.30 0.29 0.50 0.35 0.83 0.34 0.29 0.28 0.24 0.00 0.45
2
130
124
1
2
125 120
118 −>y
−>y
115 116
110 114 105
112
100
110
95 90 −45
108
−40
−35
−30
−25
252
−26
260 −20
250
106
250
−24
270
−22
254 256
−20
x
Curves No. 3, 3,
(a)
Curves No. 4, 4,
(b)
Robust residua: rs = 71.24, MLSqCutOff = 15.00, 2.5*rs = 178.10 145
155
Robust residua: rs = 2.10, MLSqCutOff = 10.00, 2.5*rs = 5.26
180
10
160
9
3
140
4
150
8
140
135 145
7 120 3
−>y
−>y
130 140
6
4
100
125
5
135 120
80 4 130
115
60 3
110
235 −20
−15
240 −10
−5
245
125
230 −5
5
x
40
2
235
0
20
1
x
0 0
Figure 4: The best PSC matching curves after 10 iterations of the ICP registration algorithm. produces a number of candidate matches, Fig. 3. The candidate matches enter the LMSq motion estimation algorithm providing the rst rough estimate of curve registrations. The registrations are then trimmed by the modi ed ICP algorithm giving the nal transformations of Fig. 4. On the registered curves, the GSC (Fig. 5a), and the PSC (Fig. 5b) similarity measures were evaluated. Notice how PSC similarity increased compared to GSC similarity for the curve \k", which has a substantial part missing (Compare the third diagonal entries in the tables 5a, 5b). Of course, the similarity also increased for other curves since partial matches are more frequent. The global registration of cans (Fig. 7) is computed using the three best PSC matching curves, i.e. "C", "o", and "e". For the comparison we also plot the registration obtained by using the LMSq motion estimation applied to all candidate matches produced by hashing, Fig. 6. The plot of residuals on Figure 5c shows that the MLSq algorithm alone has used only 198 matches while eliminating 1135 outliers. Even though, the contamination is much higher than 50%, the registration is still rather acceptable. On the other hand, the nal registration (Fig. 7) using the -reciprocal points of the three best matching curves is very accurate, with no outliers, Fig. 5d.
5 Conclusion
We have shown that simple semi-di erential invariants can be successfully used to accomplish robust curve matching. Moreover, the combination of -reciprocal correspondence with Least Median of Squares motion estimation allows to accurately regis-
200
400 600 800 1000 # of inliers = 198, # of outliers = 1135
(c)
1200
1400
0 0
50
100
150 200 250 # of inliers = 308, # of outliers = 0
(d)
300
350
Figure 5: a) Global curve similarity measure (GSC). b) Similarity measure for partially overlapping curves (PSC). c) Residuals of the global LMSq registration using all candidates produced by hashing. d) Residuals of the global LMSq registration using only -reciprocal points for the three best PSC matching curves. ter partially occluded curves irrespective of gross errors in their extraction. The use of the one-shot method for the acquisition of range data (i.e. based on a single projection pattern) in combination with robust curve matching should allow the propounded system to be used in dynamic and relatively uncontrolled scenes.
References
[1] P.J. Besl and N.D. McKay. A method for registration of 3-d shapes. IEEE PAMI, 14(2):239{256, February 1992. [2] A. Gueziec and N. Ayache. Smoothing and matching of 3-D spatial curves. Int. J. of Computer Vision, 12(1):79{104, 1994. [3] E. Kishon, T. Hastie, and H. Wolfson. 3-D curve matching using splines. J. of Robotic Systems, 8(6):723{743, 1991. [4] T. Pajdla and L. Van Gool. Euclidean invariant descriptions of curves from range images. ESAT MI2 Tech. Rep. KUL/ESAT/MI2/9417, Katholieke Universiteit Leuven, 1994.
200
10 11
8 11
4
3
140
10
4
9
1 22
5 5 3 6 76 7 12
1
140
4
9
−>y
160 −>y
9
160
9
180
120
8
8
180 8 10
200
5
10
4 5
1 2
120
13
3
1
3 67 6 7 12
2
13
100
100
80 220
60 −60 −40
240
80
y
160
120
8
8
180 8 10
200
12
4
10
1
120
2 2
13
1
3 6 67 7
12
100
100
80 220
60 −60 −40
240
80
