Jun 2, 1993 - We use a recursive dynamic strategy for exploring machine parts. .... robotic observers a.s they provide a means for tracking the continuous, discrete and symbolic aspects of the ..... The images are copied from the video card buffer and then processed. ..... The table is a planar surface, parallel to the floor.
Interm ediate Results in A ctive Inspection and Reverse Engineering Tarek M . Sobh, Jonathan Owen, Chris Jaynes, Mohamed Dekhil, and Thomas C. Henderson 1
UUCS-93-014 Department of Computer Science University of Utah Salt Lake City, U T 84112 U S A June 2, 1993
Abstract In previous work [18], we have proposed a new design for inspection and reverse engineering environ ments. W e have investigated the use of the dynamic recursive context of discrete event dynamic systems (D R F S M D E D S ) to guide and control the active exploration and sensing of mechanical parts for industrial inspection and reverse engineering, and utilized the recursive nature of the parts under consideration. In our recent work, we construct a sensing to C A D interface for the automatic reconstruction of parts from vi sual data. This report includes previous results and describes this interface in greater detail, demonstrating its effectiveness with a reverse-engineered, machined part.
'This work was supported in part by DARPA grant N00014-91-J-4123, NSF grant CDA 9024721, and a University of Utah Research Committee grant,. All opinions, findings, conclusions or recommendations expressed in this document are those of the author and do not. necessarily reflect the views of the sponsoring agencies.
C o n te n ts 1
In tro d u c tio n
4
2
O b je c tiv e s a n d Q u e stio n s
4
3
S en sin g fo r I n s p e c tio n a n d R e v e r s e E n g in e e rin g
5
3.1
D iscrete E v ent D y n a m ic S y s t e m s ............................................................................... ...................................
5
3.2
Sensing S t r a t e g y .....................................................................................................................................................
3.3
T h e D y n a m ic R ecursive C o n te x t for F in ite S ta t e M achines
4
5
8
..............................................................
.9
.................................... ............................................................................................................
9
3.3.1
Definitions
3.3.2
D R F S M R e p r e s e n t a t i o n ........................................ ... ............................................................................
9
3.3.3
A G ra p h ic a l Interface for D eveloping D R F S M , s .........................................................................
10
3.3.4
IIow to use D R F S M ? ...........................................................................................................................
14
3.3.5
A pplying D R F S M in F e a tu re s ex tra c tio n
14
....................................................................................
S en so ry P ro ce ssin g
15
4.1
T w o D im ensional Im age P r o c e s s i n g .................................................................................................................
16
4.1.1
E x tr a c tin g C o n t o u r s ...............................................................................................................................
17
4.1.2
Z c r o - C r o s s i n g s ..........................................................................................................................................
18
4.1.3
C o n to u r P r o p e r t i e s ...................................................................................................................................
18
4.2
Visual O bserv atio n of S t a t e s ...............................................................................................................................
19
4.3
Deciding F e a tu re R e l a t i o n s h i p s ........................................................................................................................
20
4.4
C o n s tr u c tin g th e Recursive R e l a t i o n .............................................................................................................
21
4.5
E x tra c tio n of D e p th Inform ation an d W orld C o o r d i n a t e s .....................................................................
23
4.6
C a m e r a C a lib ratio n
27
4.7
D e p th E s tim a tio n using Illum ination Table
.............................................................................................................................................. ..............................................................................................
30
4.7.1
Table C o n s t r u c t i o n ...................................................................................................................................
30
4.7.2
M o d i f i c a t i o n s ..............................................................................................................................................
31
S en sin g to C A D In te rfa c e
32
5.1
C o n to u rs to S p l i n e s .................................................................................................................................................
33
5.1.1
T h in n in g C o n t o u r s ...................................................................................................................................
34
C o n to u rs to M achined F e a t u r e s ........................................................................................................................
36
5 .2 . 1
T h e A l g o r i t h m ..........................................................................................................................................
36
5.2.2
D a t a P o ints to Arcs and Lines
36
5.2.3
Arcs and Lines to M achined F eatu re s
5.2.4
Results
5.2
......................................................................................................... ...........................................................................................
38
.........................................................................................................................................................
41
6 P u t t i n g it A l l T o g e t h e r 6.1
T h e F irst E x p e rim e n t
41 ..........................................................................................................................................
43
6.2
Results for th e F ir s t E x p e r i m e n t .................................... ...............................................................................
45
6.3
T h e New E x p e r i m e n t ..............................................................................................................................................
45
6.4
R un n ing th e New E x p erim e n t ............................................................................................................................
45
2
7
8
9
6.5
E x p e rim e n ta l results, A u to m a te d Bracket I n s p e c t i o n ..........................................................
49
6.6
E x p erim en tal Results, T est P i e c e ..................................................................................................
53
S u m m a ry o f C u rre n t D ev elo p m e n ts
53
7.1
G oals and M e t h o d o l o g y ....................................................................................................................
53
7.2
C u r re n t D e v e l o p m e n t s ........................................................................................................................
53
7.3
P a s t, C u rre n t, and F u tu re A c t i v i t i e s ...........................................................................................
55
7.3.1
C om p leted a c t i v i t i e s .............................................................................................................
55
7.3.2
C u rre n t a c t i v i t i e s ....................................................................................................................
55
7.3.3
F u tu r e activities
56
....................................................................................................................
In te g r a tio n E ffo rts
'
56
8.1
R obotics and S e n s i n g ............................................................................................................................
56
8.2
C o m p u te r Aided Design and M a n u f a c t u r i n g ............................................................................
56
8.3
VLSI, U n certain ty Modeling, and L anguages
57
.........................................................................
58
C o n clu sio n s
10 A p p e n d i x A : S a m p l e G U o e O u t p u t
60
11 A p p e n d i x B : S a m p l e C a l i b r a t i o n C o d e O u t p u t
62
12 A p p e n d i x C : C o m p a r i s o n b e t w e e n H o u g h t r a n s f o r m a n d c u r v a t u r e t e c h n i q u e
63
3
1
I n tr o d u c tio n
Developing fram ew o rk s for inspection and reverse engineering app lication s is an essential ac tivity in m an y engineering disciplines. Usually, too much tim e is s p e n t in designing h a rd w a re a n d softw are env iron m ents, in order to be able to a t t a c k a specific problem . O ne of t h e p urp oses of th is work is to provide a basis for solving a class of inspection an d reverse engineering problem s. C A D / C A M (C o m p u te r Aided Design, M an u fa c tu rin g ) typically involves th e design an d m a n u fa c tu re of m echanical p a r ts . T h e problem of reverse engineering is to tak e an existing m echanical p a r t as t h e point of d e p a r t u r e and to inspect or p ro d u c e a design, and p e rh a p s a m a n u fa c tu rin g process, for th e p a r t. T h e techniques t h a t we explore can hopefully be used for a variety of applications. We use an observer ag e n t to sense th e c u rre n t world e n v iro n m e n t and m ake som e m easu rem en ts, th en supply relevant in form ation to a control m od ule t h a t will be able to m ake som e design choices t h a t will late r affect m a n u fa c tu rin g a n d / o r inspection activities. T h is involves b o th a u to n o m o u s and s e m i-au to n o m o u s sensing. T h e problem of inspection typically involves using a C A D re p re se n ta tio n of th e item to be inspected, a nd using it to drive an inspection tool such as t h e C o o rd in a te M ea su rin g M achine (C M M ). A n exam ple of this is found in [9]. W hile th e work described th e re is intended to allow th e inspection of com plicated sculpted surfaces, we limit o urs to an i m p o r t a n t class of m achined p a rts . W ith in this class, we hope to reduce th e tim e necessary for inspection by m ore t h a n tenfold, ta k in g a d v a n ta g e of th e p a r t ’s recursive n a tu r e and its fe a tu re description. We use a recursive d y n am ic stra te g y for exploring m achine p a r ts . A discrete event dynamic, system (D E D S) fram ew ork is designed for m odeling a n d s tr u c tu r in g th e sensing an d control problem s.
The
d y n am ic recursive c o n te x t for finite s t a t e m achines (D R F S M ) is intro d u ce d as a new D E D S tool for utilizing th e recursive n a tu r e of th e m echanical p a r ts u n d e r consideration. T his p a p e r describes w h a t this m ean s in m ore detail.
2
O b je c t iv e s a n d Q u e s tio n s
T h e objective of this research p ro je c t is to explore th e basis for a co nsisten t softw are an d h a rd w a re envi ro nm en t, and a flexible system t h a t is cap able of p erfo rm ing a variety of inspection an d reverse engineering activities. In p a rtic u la r, we will c o n c e n tra te on th e a d a p tiv e a u to m a tic e x tra c tio n of som e pro p e rtie s of th e world to be sensed and on th e s u b seq u e n t use of th e sensed d a t a for p ro d u c in g reliable descriptions of th e sensed en v iro n m e n ts for m a n u fa c tu rin g a n d / o r description refinem ent purposes. We use an observer ag en t w ith som e sensing capabilities (vision an d touch) to actively g a th e r d a t a (m ea su rem en ts) of mechanical p a r ts . We co njecture t h a t D iscrete Even t D yn am ical S ystem s (D E D S ) provide a good base for defining co nsisten t and a d a p tiv e control s tru c tu re s for th e sensing m od ule of th e inspection and reverse engineering problem . If this is true, th en we will be able to answ er th e following questions : • W h a t is a su itab le alg o rith m to co o rd in a te sensing, inspection, design an d m an u fa c tu rin g ? • W h a t is a su itable control s tra te g y for sensing th e mechanical p a r t? • W hich p a r t s should be im plem ented in h a rd w a re vs. software?
4
• What, are suitable language tools for constructing a reverse engineering and/or inspection strategy?
D E D S can be sim ply described as : D y n a m ic sy stem s (typically asynchronous) in which s t a t e tra n s itio n s are triggered by dis crete events in th e sy stem . It is possible to co n tro l and o b se rv e hybrid sy s te m s (system s t h a t involve continuous, discrete and symbolic p a ra m e te rs ) u n d er u n c e rta in ty using D E D S fo rm u latio n s [13, 16]. T h e ap p licatio ns of this work are num ero us: a u to m a tic inspection of m echanical or electronic co m p o n ents and re p ro d u ctio n of m echanical p a rts . M oreover, th e experience gained in p erform in g this research will allow us to s tu d y the. subdivision of the solution into reliable, reversible, and an easy-to-m odify softw are and h a rd w a re en viro nm ents.
3
S e n s in g fo r In s p e c t io n a n d R e v e rs e E n g in e e r in g
In this section we describe th e solution m eth odo log y for the sensing m odu le an d discuss th e c o m p o n en ts separately.
T h e control flow is described and th e m eth o d s, specific e q u ip m e n t and p rocedu res are also
discussed in detail. We use a vision sensor ( B / W C C D cam era) and a co o rd in a te m easu ring m achine (C M M ) w ith the necessary softw are interfaces to a Sun S p a rc s ta tio n as the sensing devices. T h e o b je c t is to be inspected by th e co o p e ra tio n of th e observer c a m e ra and th e p rob ing C M M . A D E D S is used as th e high-level fram ew ork for exploring th e m echanical p a r t. A d y n am ic recursive c o n te x t for finite s t a t e m achines (D R F S M ) is used to exploit th e recursive n a tu r e of th e p a r ts u n d e r consideration.
3.1
D iscrete E vent D yn am ic S ystem s
D E D S are usually m odeled by finite s t a t e a u t o m a t a w ith partially observable events to g e th e r w ith a m echanism for enabling an d disabling a su b set of s t a t e tra n s itio n s [3, 12, 13]. We propose t h a t this model is a su itab le fram ew o rk for m a n y reverse engineering task s. In p a rtic u la r, we use th e model as a high-level s tr u c tu r in g technique for o u r system . W e a d v o c a te an ap p ro a c h in which a stabilizable sem i-a u to n o m o u s visual sensing interface would be capable of m ak in g decisions a b o u t th e s ta te of the observed m achine p a r t and th e probe, th u s providing b oth sym bolic and p a r a m e tric descriptio ns to th e reverse engineering a n d / o r inspection control m odule. T h e D E D S -b a sed active sensing interface will be discussed in th e following section. M o d elin g a n d C o n s tru c tin g a n O b se rv e r T h e ta s k s t h a t th e a u to n o m o u s observer sy stem executes can be modeled efficiently within a D E D S fra m e work. We use th e D E D S m odel as a high level s tr u c tu r in g technique to preserve an d m ake use of th e info rm atio n we know a b o u t th e way in which a m echanical p a r t should be explored. T h e s t a t e and event description is associated w ith different visual cues; for exam ple, a p p e a ra n c e of o b jects, specific 3-D m ove m e n ts an d s tru c tu re s , intera ctio n between th e to uch ing p rob e and p a r t, an d occlusions. A D E D S observer
5
serves as an intelligent sensing m odule t h a t utilizes existing info rm ation a b o u t th e task s and th e environ m ent to m ake inform ed tra c k in g an d correction m o vem en ts and a u to n o m o u s decisions reg arding th e s t a t e of th e sy stem . In o rd e r to know th e c u r re n t s t a t e of th e exploration process we need to observe th e sequence of events o ccurring in th e system an d m ake decisions reg arding th e s t a t e of t h e a u t o m a t o n . S ta t e am biguities are allowed to occur, however, th ey are required to be resolvable a fte r a bounded interval of events.
The
goal will be to m ake th e sy stem a stro n g ly o u t p u t stabilizable one a n d / o r c o n s tru c t an observer to satisfy specific task -o rie n ted visual re q u irem en ts. M an y 2-D visual cues for e s tim a tin g 3-D world behavio r can be used. E x am ples include: im age m otio n, shadow s, color and b o u n d a r y inform ation. T h e u n c e rta in ty in th e sensor acquisition proced u re and in th e im age processing m echanism s should be tak en into consideration to c o m p u te th e world u ncertainty .
•
Foveal and p erip heral vision stra te g ie s could be used for th e a u to n o m o u s “focusing” on relevant aspects of th e scene. P y ra m id vision ap proache s and lo g arithm ic sensors could be used to reduce th e dim ensionality and c o m p u ta tio n a l com plexity for th e scene u n d er consideration. E rro r S ta te s a n d S equences We can utilize th e observer fram ew o rk for recognizing e rro r s ta te s and sequences. T h e id ea behind this recognition ta s k is to be able to re p o rt on v isu a lly in c o rre c t sequences.
In p artic u la r, if th ere is a pre
d ete rm in e d observer model of a p a rtic u la r inspection ta s k u n d er observation, th en it would be useful to d e te rm in e if so m e th in g goes w rong with th e ex p lo ration actions. T h e goal of this re p o rtin g proced u re is to alert th e o p e r a t o r or au to n o m o u s ly supply feedback to th e in spectin g r o b o t so t h a t it can correct its actions. An exa m p le of errors in inspection is unex pected occlusions between th e observer c a m e ra and th e inspection en v iro n m e n t, or pro bin g th e p a r t in a m a n n e r t h a t m ig ht break th e probe. T h e correct sequences of a u t o m a t a s t a t e tra n s itio n s can be form ulated as th e set of strin gs t h a t are acceptable by the observer a u t o m a t o n . T h is set of s trin g s re presen ts precisely th e lang uag e describing all possible visual task evolution steps. H ie ra rc h ic a l R e p re s e n ta tio n F igure 1 shows a hierarchy of th re e subm odels.
M otives behind establishing hierarchies in the D ED S
m odeling of different ex ploration ta s k s include reducing th e search space of th e observer and exhibiting m o d u la rity in th e controller design.
T h is is d one th ro u g h th e designer, who subdivides th e ta s k space
of th e exploring ro b o t into s e p a r a t e s ubm od els t h a t are inherently in d ep en d e n t.
Key events cause th e
tra n s fe r of th e o bserver co ntrol to new su bm od els w ithin th e hierarchical description. T ra nsfer of control th ro u g h th e o bserver hierarchy of m odels allows coarse to fine shift of a tte n tio n in recovering events and asserting s t a t e tran sitio n s. M a p p in g M o d u le T h e o b je c t of having a m a p p in g m od ule is to dispense with th e need for t h e m an u al design of D E D S a u t o m a t a for various p latform task s. In p artic u la r, we would like to have an off line m o du le which is to be supplied with som e sym bolic description of th e ta s k u n d er observation an d whose o u t p u t would be th e
6
code for a D E D S a u t o m a t o n t h a t is to be executed as th e observer agent. A graphical re p re se n ta tio n of th e m a p p in g m odule is shown in F ig u re 2. T h e problem reduces to figuring o u t w h a t is an a p p r o p ria te form for th e task description. T h e error s t a t e p arad ig m m otiv a te d regarding this p roblem as th e inverse problem of d e te rm in in g a c cep tab le languages for a specific D E D S observer a u t o m a t o n . In partic u la r, we suggest a skeleton for the m a p p in g m odule t h a t tra n s fo rm s a collection of in p u t strings into an a u to m a to n model.
Task Language
M a p p i n g M o d u le
DEDS Automaton Transition Conditions
F igu re 2: T h e M a p p in g M odule. T h e idea is to su pp ly th e m a p p in g m od ule with a collection of strin gs t h a t re p resents possible s t a t e tran s itio n sequences.
T h e in p u t highly d ep e n d s on the task un der o bservation, w h a t is considered as
relevant s t a t e s and how coarse th e a u t o m a t o n should be. T h e sequences are in p u t by an o p e r a to r. It should be obvious t h a t the “G a rb a g e -in -g a rb a g e -o u t” principle holds for th e c o n stru c tio n process; in p articular, if the set of in p u t s trin g s is no t re p resen ta tiv e of all possible scene evolutions, th en th e a u t o m a t o n would be a fau lty one. T h e experience and knowledge t h a t th e o p e r a to r have would influence th e o u tc o m e of the resulting model. However, it should be noticed t h a t th e level of experience needed for providing these sets of strin g s is much lower th a n th e level of experience needed for a designer to actually c o n s tru c t a D E D S a u t o m a t o n m anually. T h e description of th e events t h a t cause tra n s itio n s between different sym bols in th e 7
set of s trin g s should be supplied to th e m odule in th e form of a list. As an illu strativ e exam ple, su pp o se t h a t the task und er consideration is sim ple g raspin g of one o bject and t h a t all we care to know is th ree configurations: w h e th e r th e hand is alone in the scene, w h e th e r th ere is an o b je c t in add itio n to th e h and and w h e th e r enclosure has occurred. If we represent th e configurations by th ree s ta te s h, h c and h c, then th e o p e r a to r would have to supply th e m a p p in g m odu le w ith a list of s trin g s in a language, whose a lp h a b e t consists of those th ree sym bols, an d those strings should span the entire language, so t h a t th e resulting a u t o m a t o n would accept all possible configuration sequences. T h e m a p p in g from a set of string s in a regular lan gu age into a m inim al equivalent a u t o m a t o n is a solved problem in a u t o m a t a theory. O n e possible language to describe this simple a u t o m a t o n is : L = h+ h t h + c
and a corresp o n d in g D E D S a u t o m a t o n is shown in F ig ure 3.
F igu re 3: An A u to m a to n for Simple G rasping. T h e best-case scenario would have been for th e o p e r a to r to supply exactly th e lang uag e L to the m ap p in g m o du le with th e a p p r o p r ia te event definitions.
However, it could be th e case t h a t th e set of
string s t h a t th e o p e r a t o r supplies do no t represent th e ta s k language correctly, and in t h a t case som e learning techn iqu es would have to be im plem ented which, in effect, a u g m e n t th e in p u t set of strings into a lan guage t h a t satisfies som e p re-d eterm ined criteria. For exam ple, y* is s u b s titu te d for any s trin g of j/’s having a length g re a te r th a n n, and so on. In t h a t case th e resulting a u t o m a t o n would be co rrect up to a certain degree, d ep e n d in g on the o p e r a t o r ’s experience and th e correctness of th e learning strategy . 3 .2
S en sin g Strategy
We use a B / W C C D c a m e ra m o u nted on a ro b o t arm , and a co o rd in a te m easuring m achine (C M M ) to sense th e m echanical p a r t.
A D R F S M im p le m e n tatio n of a discrete event d y n am ic sy s te m (D E D S)
algo rithm is used to fa cilitate th e s t a t e recovery of the inspection process. D E D S are su ita b le for modeling robotic observers a.s they provide a m eans for trackin g th e c o n tin u o u s , d isc re te an d sy m b o lic asp e cts of the scene und er consideration [3, 12, 13]. T h u s th e D E D S controller will be able to m o d e l and report th e s t a t e evolution of th e inspection process. In inspection, th e D E D S guides th e sensing m achines to the p a rts of th e o b je c ts where discrepancies occur betw een th e real o b je c t (or a C A D model of it) and the recovered s t r u c t u r e d a t a p oints a n d / o r p a ra m e te rs . T h e D E D S form u latio n also c o m p e n sa te s for noise in th e sensor readings (b o th am biguities and u n certainties) using a probabilistic ap p ro ach for co m p u tin g the 3-D world p a r a m e te r s [16].
The
recovered d a t a from th e sensing m o dule is then used to drive th e C A D m odule. T h e D E D S sensing agent
is th u s used to collect d a t a of a p a s siv e elem ent for designing s tr u c tu r e s ; an exciting extension is to use a similar D E D S observer for m oving agen ts an d subsequently design behaviors th ro u g h a learning stage.
3.3
The D yn am ic R ecu rsive C on text for F in ite S tate M achines
T h e D y na m ic Recursive C o n te x t for F inite S ta t e M achines (D R F S M ) is a new m eth odo log y to represent and im ple m e nt multi-level recursive processes using s y s te m a tic im p le m e n ta tio n techniques. By multi-level process we m ean an y processing o p e ra tio n s t h a t are done repetitively with different p ara m e te rs . D R F S M has proved to be a very efficient way to solve m an y com plicated problem s in th e inspection p ara d ig m using an easy n o ta tio n an d a s tr a ig h t fo rw ard im p le m e n tatio n , especially for o b jects t h a t have sim ilar multi-level s tru c tu re s w ith different p a ra m e te rs . T h e m ain idea of th e D R F S M is to reuse th e conventional D E D S F inite S ta t e M achine for a new level after changin g som e of th e t r a n s i t i o n 'p a r a m e t e r s . A fter exploring this level, it will re ta ke its old p a r a m e te rs an d co ntinue exploring th e previous levels. T h e im p le m e n tatio n of such m achines can be g e n e ra te d a u to m a tic a lly by som e modification to an existing reactive behavior design tool called G U o e [4] t h a t is capable of produ cin g code from s t a t e m achine descriptions (drawings) by adding a recursive re p re s e n ta tio n to th e conventional re p resen ta tio n of finite s t a t e m achines, and then g e n e ra tin g th e a p p r o p ria te code for it. 3 .3 .1
D e fin itio n s
• V a r i a b l e T r a n s i t i o n V a l u e : Any variable value t h a t dep e n d s on th e level of recursion. • V a r i a b l e T r a n s i t i o n V e c t o r : T h e vector con tain in g all variable tra n s itio n s values, and is d y n a m ically changed from level to level. • R e c u r s i v e S t a t e : A s t a t e calling a n o th e r s t a t e recursively, and this s t a t e is responsible for changing th e variable tran s itio n vector to its new value according to th e new level. • D e a d - E n d S t a t e : A s t a t e t h a t does no t call any o th e r s t a t e (no tra n s itio n arrow s come o u t of it). In D R F S M , when this s t a t e is reached, it m ean s to go back to a previous level, or q uit if it is th e first level. T h is s t a t e is usually called th e E rr o r-tra p p in g s ta te . It is desirable to have several dead-end s ta te s to rep resen t different types of errors t h a t can h ap p e n in th e system . 3 . 3 .2
D R F S M R e p re se n ta tio n
We will use th e sa m e n o ta tio n and te rm s of th e o rd in ary F S M s, b u t som e new n o ta tio n to re present recursive s ta t e s and variable tran sitio n s. F irst, we p e rm it a new ty p e of tra n s itio n , as shown in F ig ure 4; (from s t a t e C to A), this is called th e Recursive T ran sitio n (R T ). A recursive tra n s itio n arro w (RTA) from one s t a t e to a n o t h e r m ean s t h a t th e tra n s itio n from th e first s t a t e to th e second s t a t e is d one by a recursive call to the second one a fte r changing th e Variable T ransition Vector. Second, th e tran sitio n condition from a s t a t e to a n o th e r m ay contain variable p a r a m e te rs according to th e c u rre n t level, these variable p a r a m e te r s are distinguished from th e c o n s ta n t p a r a m e te rs by th e n o ta tio n V ( p a r a m e te r nam e). All variable p a r a m e te rs of all s t a t e tra n sitio n s c o n s titu te th e Variable T ransition Vector. It should be noticed t h a t no ndeterm in ism is no t allowed, in the sense t h a t it is impossible for two co n c u rre n t tra n s itio n s to occur from th e sam e s ta te . F ig u re 5 is th e equivalent F SM re p resen ta tio n (or th e flat re p resen ta tio n ) of th e D R F S M shown in
9
L> 10
trails. Variables VI
V2
V3
V4
V5
Level 1
12
15
0.03
170
25
Level 2
10
12
0.07 100
35
Level 3
6
8
0.15 50
40
F igu re 4: A Simple D R F S M F igure 4, for th ree levels, and it illu stra te s th e c o m p ac tn ess and efficiency of th e new n o ta tio n for this type of process. 3 . 3 .3
A G ra p h ic a l In te rfa c e fo r D e v e lo p in g D R F S M s
In developing th e fram ew o rk for reverse engineering, it has proven desirable to have a quick and easy m eans for m odifying the D R F S M which drives the inspection process. T h is was accom plished by modifying an existing reactive b ehavior design tool, G lJ o e , to a c c o m m o d a te p ro du cing th e code of D R F S M D E D S . G U o e [4] allows the user to graphically draw finite s t a t e m achines, and o u t p u t th e results as C code. G l J o e ’s original m e th o d was to p arse tran sitio n strin g s using lex /y ac c g en e rated code. T h e user interface is qu ite intuitive, allowing th e user to place s t a t e s with th e left m ouse b u tto n , and tra n sitio n s by selecting th e s t a r t and end s t a t e s w ith left and right m ouse b u tto n s . W h e n the s t a t e m achine is com plete, th e user selects a s t a t e to be th e s t a r t s t a t e and clicks th e “C om pile” b u tto n to o u t p u t C code. T h e code o u t p u t by the original G U o e has an itera tiv e s t r u c t u r e t h a t is no t conducive to th e recursive form u latio n of d y n a m ic recursive finite s t a t e machines. Therefore, it was decided to modify G l J o e to suit ou r needs. M odifications to G U o e include: • O u t p u t of recursive ra th e r th a n ite ra tiv e code to allow recursive s t a t e machines. • M odification of s trin g parsing to accept recursive tran sitio n specification. • E n co d in g of an event parser to prioritize incom ing events from m ultiple sources. • Im p le m e n ta tio n of th e variable tran sitio n vector (V T V ) acquisition (when m ak ing recursive tra n s i tions.)
10
Figure 5: Flat Representation of a Simple D R F S M
F ig u re
6:
G U o e W indo w w / D R F S M
E x am p le code from th e m ach ine in F igure
m ay be found in A p p e n d ix A. We used this m achine in
6
ou r new ex p e rim en t which will be m entioned in a later section. T h e event p a rser was encoded to ensure t h a t th e a u t o m a t o n m akes tra n s itio n s on only one source of inpu t. C u rre n tly ac ceptable events are as follows: • P ro b e - p ro b e is in th e scene. • N o P ro b e - no p ro b e is in th e scene. • P ro b e C lo se - p ro b e is w ithin th e “close” tolerance to th e c u rre n t fe a tu re specified by th e V T V . • P r o b e F a r - p ro b e is fa r th e r from th e c u rre n t fe a tu re th a n th e “close” toleran ce specified by th e V T V . • P r o b e O n F e a tu r e - p ro b e is on th e fe a tu re (according to vision.) • P r o b e N o t O n F e a t u r e - p ro b e is close, b u t no t on th e fe a tu re (according to vision.) • Vision P ro b lem - p a r t s trin g has changed, signifying t h a t a fe a tu re is occluded (need to move the ca m e ra.) • Problem Solved - m oving th e c a m e ra has corrected th e occlusion problem . • T o iich edF ea ture - p ro b e has touched th e fe a tu re (according to touch sensor.) • NoTouch - p ro b e ha.s no t tou ched th e fe a tu re (according to touch sensor.) • Closed Region - c u rre n t f e a tu re con tain s closed region(s) to be inspected (recursively.) • O p en Region - c u rre n t f e a tu re c o n tain s open region (s) to be inspected (iteratively.) • T i m e O u t - m achine h as n o t changed s t a t e within a period of tim e specified by th e V T V . • Done - inspection of th e c u rre n t fe a tu re an d its children is com plete, re tu r n to previous level. A d d itio n al events require th e add ition of su ita b le event handlers. New s t a t e s and tr a n s itio n s m ay be add ed com pletely w ithin th e G U o e interface. T h e new code is o u t p u t from GUoe. an d m ay be linked to th e inspection utilities w ith no m odifications. T h e V T V , or Variable T ra n sition Vector, is a v ector con tain ing variables t h a t m ay be d e p e n d e n t on th e c u rre n t d e p th of recursion. It is cu rren tly read from a file. T h e code produced by th e m achine in F igure
6
was first tested using a te x t interface before being linked
with th e rest of th e e x p e rim en tal code. T h e following is a tra n s c rip t showing th e sim u lated ex plo ration of two closed regions A and B, with A con tain ing B:
inspect [5] "/DEDS => bin/test_drfsm enter the string: A ( B ()) A ( B ()) THE VARIABLE TRANSITION VECTOR
12
100.000000 in state A
50.000000 '/, NoProbe is true
has the probe appeared? n has the probe appeared? n
'/, Probe is true
has the probe appeared? y in state B has the probe appeared? y enter the distance from probe to A: has the probe appeared? y
85
enter the distance from probe to A:
45
'/, Probe is Far Probe is Close
in state C enter the string: A ( B ()) enter the distance from probe to A:
10
is the probe on A? y in state D '/, Probe on Feature
is the probe on A? y has touch occurred? y in state E Making recursive call... THE VARIABLE TRANSITION VECTOR 100.000000
50.000000
in state A has the probe appeared? y in state B has the probe appeared? y enter the distance from probe to B :
'/, Probe is true
95
'/, Probe is Far
45
’ /, Probe is Close
has the probe appeared? y enter the distance from probe to B: in state C enter the string:
A ( B ())
enter the distance from probe to B: is the probe on B? y in state D
10
is the probe on B? y
'/, Probe on Feature
has touch occurred? y in state E in state END in state END Inspection Complete.
13
inspect[6] "/DEDS => T h e o b ta in e d results when linked with th e rest of th e ex pe rim en tal code were as expected. F u tu re m o d ifications m ay include th e ad d itio n of “o u t p u t ” on tran sitio n s, such as “T o u c h O c c u r re d /U p d a te M o d e l” , allowing easy specification of com m u n icatio n betw een m odules. It should be clear, however, t h a t th e code g en e rated by G U o e is only a skeleton for th e m achine, an d h as to be filled by th e users according to the task s assigned to each s ta te . In general, G l J o e proved to be a very efficient and h a n d y tool for g e n e ra tin g and m odifying such machines. By a u t o m a t i n g code g eneratio n, one can reconfigure th e whole inspection process w ith o u t being fam iliar with t h e underlying code (given t h a t all required user-defined events a n d m odules are available). 3 .3 .4
H o w to u s e D R F S M ?
To apply D R F S M to any p roblem th e following ste p s are required: • P ro blem Analysis: Divide th e problem into s ta te s , so t h a t each s t a t e accom plishes a sim ple task. • T ransitio n C onditions: Find th e tra n s itio n conditio ns between th e sta te s . • Explore t h e re p etitiv e p a r t in th e problem (recursive p ro p e rty ) and specify th e recursive sta te s . However, som e p rob lem s m ay no t have this property, in th ose cases a FSM is a b e t t e r solution. • V T V fo rm a tio n : If th e re are different tra n s itio n values for each level; these variables have to be defined. • E rr o r tr a p p in g : Using ro b u s t analysis, a set of possible errors can be established, th e n one or m ore D e a d -E n d s ta te (s ) are added. • D R F S M Design : Using G l J o e to draw th e D R F S M an d g e n e ra te th e correspon din g C code. • Im p le m e n ta tio n : T h e code g en e ra te d by G lJ o e has to be filled o u t with th e ex a ct ta s k of each s ta te , th e e rro r h andlin g ro utin es should be w ritten , and th e required o u t p u t has to be im p lem ented as well. 3 . 3 .5
A p p ly in g D R F S M in F e a tu r e s e x t r a c t i o n
An ex p e rim en t was perform ed for inspecting a m echanical p a r t using a c a m e ra and a probe. A predefined D R F S M s t a t e m achin e wa.s used as th e observer age n t skeleton. T h e c a m e ra was m o u n te d on a P U M A 560 r o b o t a rm so t h a t th e p a r t was always in view. T h e p ro b e could then ex ten d into th e field of view and com e into c o n ta c t w ith th e p a r t, as shown in F ig u re 19. Sym bolic R e p re s e n ta tio n of F eatures: F or th is problem we are concerned with O pen regions (O) and Closed regions (C). A ny closed region m ay con tain o th e r featu res (th e recursive p ro p e rty ). Using p a re n thesis n o ta tio n th e s y n ta x for represen ting fe a tu res can be w ritte n as follow: < feature > :: C ( < subfeature >) | C()
14
Figure 7: An Example for a Recursive Object
< s u b fe a tu re > :: < term > , < s u b fe a tu re > | < term > < term > ::
0
| < fe a tu re >
For exam ple, th e sym bolic n o ta tio n of F igu re 7 is C (0 ,C (0 ,C (),C (0 )),C ()) F igure
8
shows th e g raphical rep resen ta tio n of this recursive s t r u c t u r e which is a tree-like s tru c tu re .
F u tu re m odifications to D R F S M ’s includes allowing different fun c tio ns for each level.
4
S e n s o ry P ro c e s s in g
In o rd e r for th e s t a t e m achine to work, it m u st be aw are of s t a t e changes in th e sy stem . As inspection takes place, th e c a m e ra supplies im ages t h a t are in te rp re te d by a vision processor and used to drive the DRFSM . A B / W C C D c a m e ra is m o u n ted to th e end effector of a P u m a 560 ro b o t arm . T h e r o b o t is then able to position th e c a m e ra in th e workplace, tak e stereo images, and move in th e case of occlusion problem s. T h e p a r t to be inspected is placed on th e co o rd in a te m easuring m achine (C M M ) table. T h e p rob e then explores th e p a r t while th e mobile c a m e ra is able to observe th e inspection and provide scene inform ation to th e s t a t e m achine. T h e vision sy s te m provides th e m ach ine w ith specific info rm ation a b o u t th e c u r re n t s t a t e of th e in spection. T h is is done th o u g h several layers of vision processing and th ro u g h th e c o o p e ratio n of 2D, 2 ^ D , and 3D vision processors. T h e a sp e cts of th e im age t h a t need to be given to the s t a t e m achine are:
15
F igu re
8:
G ra p h for the Recursive O b je c t
• N u m b e r of features. • C o n to u r re p re se n ta tio n of each featu re. • R elation ship s of th e featu res. • D e p th of featu res. • A p p ro x im a te d e p th e s tim a te s between features. • L ocation of th e pro be w ith respect to the p art.
4.1
Two D im en sion al Im age P rocessin g
T w o dim ensional fe atu res of th e p a r t are e x tra c te d using several different visual im age filters. T h e ca m e ra c a p tu r e s th e c u r re n t im age and passes it to th e 2D im age processing layer. A fter th e a p p r o p ria te processing has tak en place, i m p o r t a n t in fo rm atio n a b o u t th e scene is supplied to th e o th e r vision layers and th e s t a t e machine. T h e im ages are ca p tu re d using a B / W C C D c a m e ra which supplies 640 x 480 pixels to a VideoPix video card in a Sun W o rk s ta tio n . T h e 2D processing is intended to supply a quick look a t the c u rren t s t a t e of th e inspection and only needs a single image, c a p tu r e d w ith o u t m ovem ent of th e P u m a . T h e im ages are copied from th e video card buffer an d then processed. T h e im age processing was done using th e IM LIB im age processing library rou tines developed a t th e University of U ta h . T h e main goal of in th e im age processing m odules is to discover featu res. searched for am o n g th e fe a tu re responses.
16
O nce fe a tu re s are discovered, c o n to u rs are
Open Contour ClosedContour
F igure 9: Edge finding in th e two dim ensional im age can give us hints a b o u t where to look for three dim ensional featu res.
T h e open c o n to u r here is g en e rated where two faces m eet.
T h e co rresponding
co n to u r is then explored by b o th the stereo layer and th e C M M machine.
4.1.1
Extracting C o n t o u r s
C o n to u rs are considered i m p o r t a n t fe a tu res t h a t supply th e m achine w ith in form ation necessary to build an o b je c t m odel and drive th e a c tu a l inspection.
T h e re are two types of contours, each w ith specific
pro p e rtie s and uses, in th e expe rim ent. 1
. O pen C o n to u r: A fe a tu re of th e p a r t, like an edge or ridge t h a t does not form a ‘closed’ region. Lighting anom alies m ay also cause an open c o n to u r to be discovered.
2. Closed C o n to u r: A p a r t or im age fe a tu re t h a t form s a closed region, t h a t is, it can be followed from a specific po in t on th e fe a tu re back to itself. A typical closed co n to u r is a hole or th e p a r t boundary. We are concerned w ith finding as m an y “real” co nto u rs as possible while ignoring th e “false” contours. A real c o n to u r would be an a c tu a l fe a tu re of th e p a r t while a false c o n to u r is a t t r i b u t e d to o th e r fa cto rs such as lighting problem s (shadows, reflections) or occlusion (th e p rob e d etec ted as a p a r t fe atu re). If we a re u nab le to supply th e m achine w ith relatively few false c o n to u rs and a m a jo rity of th e real con to urs, the a c tu a l inspection will ta k e longer. T h e m achine will w aste tim e inspecting shadow s, reflections, We avoid m an y of these p roblem s by carefully controlling the lighting conditio ns of th e exp e rim en t. T h e s ta tic e n v iro n m e n t of th e m a n u fa c tu rin g w orkspace allows us to provide a diffuse light source a t a chosen intensity.
However, simple control of the lighting is not enough.
We m u st apply several pre-processing
s tep s to the im ages before we search for contours. 1. T h re sh o ld the im age to e x t r a c t th e known probe intensities. 2. C a lcu late th e Laplacian of th e G aussian . 3. C alcu late th e Z ero-C rossings of the second directional derivative. 4. Follow th e “s tr o n g ” zero-crossing edge responses to discover contours.
17
■ ClosedContours
o
CY
'
f \\
n
. OpenContour (Spuriousresponsefromreflection)
EdgeOverlay
F igure 10: A c o n to u r discovery exam ple
4 .1 .2
Z ero -C ro ssin g s
■
T h e M arr-H ild re th o p e r a t o r [1 1 ] is used to find areas where th e gray-level intensities are chan gin g rapidly. T h is is a derivative o p e r a to r which is sim ply th e thresho lded im age convolved w ith th e Laplacian of a G au ssian. T h e o p e r a to r is given by: A 2G (*,s,) = l ^ -
2 e
a is a c o n s ta n t t h a t scales th e G aussian blur over th e image. For large num bers, a a c ts as a low-pass
filter. S m aller values retain m ore localized fe a tu res b u t produ ce results t h a t are m ore susceptible to noise. This scale can be related to an im age window by: (i) C e A — ►( 2 ) D e B — > (3) D e A — ►(4) E e B — ►(5) E G A — > (6 )
21
Figure 15: The graph associated with the example
T h ese relations can be represented by a g ra p h as shown in F ig u re 15. T h e t a rg e t is to convert this gra p h to an equivalent tree s tr u c tu r e , which is the m ost convenient d a t a s t r u c t u r e to re presen t ou r sy n tax . As a first a t t e m p t , we have designed an algorithm to convert from g ra p h re p resen ta tio n to tree rep resen tatio n by scannin g all possible p a th s in th e g ra p h and p u t t in g weights to each node according to n u m b er of visits to this node. In o th e r words, u p d a te th e d e p th variable of each node by traversing the tree in all possible ways and th en assigning the nodes th e m ax im u m d e p th registered from a traversal, and p ro p a g a tin g t h a t d e p th dow n w ards. T h e n from these d ep th weights we can rem ove th e unnecessary arcs from th e g ra p h by keeping only th e arcs t h a t have a relation between a p a re n t of m a x im u m d e p th and a child, a n d elim in atin g all o th e r p a re n t arcs, th u s yielding th e required tree (F igure 16). However, we have developed a b e tte r algo rith m t h a t scans th e relations, co u n ts th e n u m b e r of o ccur rences for each closed region n am e m entioned in th e left side of th e relations giving an a rra y R A N K (x ), where x £ {A ,B ,C ,...}, an d selects th e relations ( X\ £ x-i) t h a t satisfy th e following condition: R A N K ( * 1) - R A N K ( * 2) = 1 T h is g u a r a n te e s t h a t all re d u n d a n t relations w o n ’t be selected. T h e com plexity of this alg orith m is O (n ), where n is th e n u m b e r of relations. A pplying this algo rithm to t h e relations of F ig u re 14 we have, R A N K (A ) = 0 R A N K (B ) = 1 R A N K (C ) =
1
R A N K (D ) = 2 R A N K (E ) = 2 T h e selected relations will be: B £ A
22
F ig u re 16: T h e tree associated with th e exam ple
C € A D
6
B
E € B Now a r ra n g in g these relatio ns to co n s tru c t the s y n ta x gives: A (B ()) — > A (B () ,C () ) — > A ( B (D ()) , C ()) — > A ( B ( D ( ) ,E ( ) ) ,C ( ) ) which is th e required s y n ta x . A tree represen tin g this s y n ta x is easily co n s tru c te d and shown in F ig ure 16. T h e next s te p would be to in sert th e open regions, if any, and this is d one by trav e rsin g th e tree from the m ax im u m d e p th and upw ards. Any open region can be tested by checking any point in it to see w h e th e r it lies within th e m ax im u m d e p th leaves of th e closed regions’ tree hierarchy (the te st is easily done by ex ten d in g a line and checking how m any tim es it intersects a closed region, as in th e te s t for closed regions enclosures). T h e n the u p p e r levels of the hierarchy are tested in ascending o rd e r till th e root is reached or all open regions have been e x h a u s te d . Any open region found to be inside a closed one while traversin g the tree is inserted in th e tree as a son for t h a t closed region. It should be noticed t h a t this alg o rith m is no t a general g ra p h to tree conversion algorith m , it only works on th e specific, kind of g ra p h s t h a t th e image processing m odu le recovers. T h a t is, th e conversion algorithm is ta ilored to th e visual recursion paradigm .
4.5
E xtraction of D ep th Inform ation and W orld C oordinates
A crude initial model is found using stereo vision. T h e c a m e ra model used is a pinhole c a m e ra as shown in F igure 17, corrected for radial d is to rtio n . D e p th s are found w ith such models using th e disp arity between feature, locations in a pair of views according to th e following form ula: Z = f D / ( x i - x r ).
23
F igu re 17: P inhole C a m e r a M odel where xi and x r are co o rd in a tes on th e im age plane, / is th e focal length, and D is th e disparity. A d dition al aspects of th e c a m e ra m odel a re discussed in th e section on c a m e ra calib ration. T h e stereo alg o rith m c u rren tly in use requires no co rrespon den ce betw een individual fe a tu re s in the stereo im age pair [ 1 ],
Instead , corresp ond ing regions are found, and th e disp arity in th eir centers of
m ass is used for d e p th c o m p u ta tio n .
In ou r ex p e rim en t, closed regions are found in two im ages and
th eir relationships are d e term in e d . Each closed region is described by its b o un dary, a c o n to u r in im age co ord inates. It is assum ed for th e initial m odel t h a t these c o n to u rs are p lan ar. Given this ass u m p tio n , th e p a r a m e te rs p, q, a n d c of a plane m u s t be solved for in th e equa tio n Z — p X + q Y + c.
In o rd e r to do this, each region is split into th re e sim ilar sections in b o th left and right images. T h e ce n te r of m ass is c o m p u te d for each section, and th e sy stem of equ a tio n s solved for p, q, and c. T hese values are sto red w ith th e region for la te r o u t p u t of th e C A I) «_ 1 model. It should be n oted t h a t if th e centers of m ass are collinear, this sy stem will n o t be solvable (three non-collinear p o in ts define a plane). Also, if th e cen te rs of m ass a re close to g eth er, th e error in discretizatio n will cause s u b s ta n tia l erro r in c o m p u ta tio n of plan e p a r a m e te rs . In o th e r words, if th e th re e points are close to g eth er, an error of one pixel will cause a s u b s ta n tia l error in th e c o m p u te d o rien ta tio n of th e plane. T h e effect of a one pixel error is reduced by selecting points t h a t are ’’fa r” a p a r t . T h u s, th e technique used to split regions, determ in in g th e locations of these points, is a crucial p a r t of th e algorithm . T h e m o st obvious and p e rh a p s sim plest technique splits c o n to u rs by dividing th e m into th re e p a rts horizontally (see F igu re 18.) Since m an y m achined fe a tu res (such as holes) will p ro d u c e collinear centers of ma,ss when p artitio n ed this way, a different techn iqu e is used. It is a t t e m p t e d to divide each co n to u r into th ree p a rts of equal length (sec F ig ure 18). O n e region m ay p a rtitio n e d purely by length, b u t to p artitio n th e o th e r exactly would require solution of th e correspondence, problem . F o rtunately, th e effects of e rro r in co rrespon den ce are m a d e m inim al when averaged over a section. T h e first pixel in th e left im a g e ’s region is m atch e d w ith one in th e right im age by tra n s la tin g it along a vector between th e centers of m ass of the regions a n d finding th e closest pixel to this position in th e right image.
24
F igure 18: Region S plitting A lgorithm s In p ractice, this was found to work fairly well for th e o u te rm o s t region.
However, using th e sam e
technique for sm aller inner regions, error is m uch g re a te r since th e th ree cen ters of m ass used to determ in e th e plane p a r a m e te rs are closer to g eth er.
A fu r th e r a s su m p tio n m ay be m ad e however, t h a t the inner
regions are in planes parallel to the o u te r region. Using this assu m p tio n , it is riot necessary to split the regions into th ree p a r ts , and th e plane equ ation m ay be solved for c using the ce n te r of m ass of the entire region. If it is assu m ed t h a t all planes are parallel to th e tab le (world x - y plane), th e o u te rm o s t region may be tre a te d in a like m an n er. For o ur initial e x p e rim en t, several a ssu m p tio n s were m ade: • T h e ro b o t z axis is perpen d icu lar to th e table on which th e ro b o t is m o u nted. • T h e table is a p la n a r surface, parallel to the floor. • T h e C C D plane of the c a m e ra is parallel to th e back of th e c a m e ra case. • All o b je c t co n to u rs are in planes parallel to the table. T h e e x p e rim en tal s e tu p is shown in F igure 19 T h e c a m e r a was oriented with its optical axis a p p ro x im a te ly p erp en d icu lar to th e table. T h is was first done by visual inspection. T h e n , for m ore accuracy, a level was used on th e back of the c a m e ra case and th e r o b o t tool was ro ta te d until the c a m e ra ap p e a re d level. T h e ro b o t tool fra m e was th en recorded (as Left). T h is fr a m e was used consistently for c a p tu r in g im ages for the re m a in d er of th e exp e rim ent.
At
t h a t point, th e problem had been co nstra ine d to finding th e angle between robo t x and im age x. T h is is necessary because th e stereo algo rith m is based on d isp arity only in the im age x direction. To accom plish th e co n stra in e d m otion, th e following algo rith m was im plem ented: • Move th e c a m e ra to th e recorded fram e. • T ake an image.
‘25
• C o m p u te the center of m ass of an o b je c t in the scene (there should be only one) in im age coordinates.
A t this point, the ro ta tio n p a r t of th e tran sfo rm from c a m e ra c o o rd in a tes to world co o rd in a tes is known, and th e tran s la tio n a l p a r t m u st be d eterm in e d . X an d Y c o m p o n e n ts of th e tra n sla tio n are taken to be zero, m ak in g th e world c o o rd in a te origin centered in the Left fram e image. T h e Z co m p o n e n t was d eterm in e d by tak in g an im age pair of a p a p e r c u t-o u t (thickness assum ed to be zero). T h e Z -coordinate of this o b je c t should be th e d ista n c e from the im age plane to th e table. T h is was then used to com plete
Several stereo im age pairs were then ca p tu re d using th e Left and R ight fram es, and then used by the stereo code to p ro d u c e «_1 m odels with th e o b jects described in world coo rd ina tes. For a cube m easuring one inch (‘2 5.4 m m ) on a side, th e resulting «_1 m odel was sim ilar to a cube (lighting effects are observable), and dim ensioned to 26.74m m x 25.5m m x 25.7m m (h x 1 x w). T h is co rresp on ds to p ercen t errors of 5.2, 0.4, and
4.6
1 .2 .
Som e exa m p le im ages and corresp ond ing m odels are shown in late r sections.
C am era C alibration
Real-world ca m e ra s differ su b sta n tia lly from th e ideal c a m e ra m odel typically used for discussion of stereo vision.
Lens d is to rtio n , offsets from th e im age center, etc.
are sources of error in co m p u tin g range
inform atio n. T h e c a m e ra calibration tech niq ue chosen for this p ro je c t tak es m an y of these fa cto rs into account. Developed by Roger Tsai, and im plem ented by Reg Willson of CM'U [20, 22], this technique has been described as: • A ccurate. • R easo n a b ly Efficient. • Versatile. • Needing Only Off-the-Shelf C a m e ra s and Lenses. • A u to n o m o u s (requiring no o p e r a to r control, guesses, etc.) T h e technique solves for th e following p a ra m e te rs : • / - Focal L ength. • k - Lens D isto rtio n Coefficient. • (Cx, Cy) Image C enter. • U n c e rta in ty Scale F a c to r (due to c a m e ra tim ing and acquisition error.) • R o ta tio n M atrix . • T ra n sla tio n Vector. of which we use th e focal length, d is to rtio n coefficient, and im age center. T h e eq u a tio n s used are as follows:
Xu = f- * = f ■i X u = X d ■(1 + k ■{ X j + Yd2)) Yn = Yd . { l + k - ( X j + Y * ) ) X d = d x ■( X f - C x )
27
Figure 20: Raw Image
Figure. 21: C orre cted Image, kappa = .0005
Yd = d y ■( Yj - C y )
where dx and d y are the c e n te r-to -c en ter d istances between ad ja c e n t sensor elem ents on th e C C D plane in the x and y directions (obtained from P an aso nic.) X u and K„are u n d isto rted im age plane co ord inates, x, y, and z are in th e c a m e ra c o o rd in a te sy stem . X j an d Yd are d is to rte d im age plane co ord inates.
T h e effects of the d is to rtio n coefficient can be seen in Figures 20 th ro u g h 23. In the classical ap p ro ach to c a m e ra ca libration, co m p u tin g th e large n u m b e r of p a r a m e te rs requires large scale non lin ear search. In T s a i ’s m e th o d however, the p ro b le m ’s dim ensionality is reduced by using the radial align m ent c o n s tra in t to split the search into two stages [20]. In th e first stage, extrinsic p a r a m e te rs such as T ra n sla tio n and R o tatio n p a r a m e te rs are found. T h e second solves for the intrinsic, p a r a m e te rs ( / , k, etc..)
Figure 22: Corrected Image, kappa = .00357 (used in our experiment)
28
Figure 23: Corrected Image, kappa = .05
F igu re 24: T h resh old ed C alib ratio n Grid T h e im p le m e n ta tio n used accep ts a d a t a file co ntaining points t h a t are known in both im age co ord inates and world coo rdin a tes. For T s a i ’s original p aper, d a t a was o b ta in e d from a calibratio n grid m easu red with a m icro m eter and 400x microscope. For our purposes, a p ap e r grid of 1 m m d ia m e te r d o ts spaced 1 cm a p a r t was m ad e using A u to C a d and a p lo tte r (see F ig u re 24). A p lo tte r was used ra th e r th a n a laser p rin ter in hopes of m inim izing such errors as th e s tre tc h in g effect found in th e o u t p u t of worn laser printers. T h e calibration alg o rith m is qu ite sensitive to sy s te m a tic errors in its in p u t. It should be noted t h a t for T s a i ’s algorithm , th e c a m e r a ’s optical axis should be a t an angle g re a te r th a n 30 degrees from th e plane in which the calibration p o in ts occur. T h e calibratio n d a t a was g en e rated in the following m anner: • C a p t u r e calibratio n im age (dot locations are known in world coordinates.) • T h re sh o ld calibratio n image. • Visit each calibration im age “d o t ” in a s y s te m a tic way: - Select a pixel interio r to the do t. - C o m p u te th e center of m ass of the
8 -connected
region.
- O u t p u t th e world x - y - z , im age x - y inform atio n to th e calibration d a t a f i le . For o u r first ex p e rim en t, 25 calibration points were used. T ypical results are shown in A p p e n d ix B. For fu tu re exp erim ents, a m ore refined calibratio n d a t a collection sy stem may be used, possibly using th e C M M as a tool to g e n e ra te d a t a points. T h is will fa cilitate o u t p u t ti n g stereo range in form ation in the C M M ’s c o o rd in a te system .
29
Contour 1
CaseB
F igu re 25: T h e problem when using stereo in d e p th e s tim a te
4.7
D ep th E stim ation using Illum ination Table
Using stereo techniques for e s tim a tin g the d e p th s of an o b j e c t ’s c o n to u rs can be very ac cu ra te , b u t it is limited in t h a t it c a n n o t c o m p u te the d e p th of an occluded c o n to u r (i.e., th e b o tto m of a hole or pocket). As shown in figure 25, th e alg o rith m will give th e d e p th s for b o th c o n to u rs correctly in case A, while in case B th e d e p th of bo th c o n to u rs will be th e sam e. It was a t t e m p t e d to solve this problem using a pre-defined illum ination tab le t h a t relates the intensity of a po int on the o b je c t to the d istan ce between this point and th e ca m era. W h en th e stere o algorithm d etec ts two nested c o n to u rs w ith the sam e d e p th , this tab le would be used to e s tim a te th e d e p th of the inner region.
T h is m eth o d is very simple to im ple m e nt, b u t it proved to have som e draw b acks.
For
exam ple, it is very sensitive to th e lighting conditions, i.e., any variation in th e lighting conditions will result in th e invalidation of th e look-up table. Also, o b je c ts being observed m u s t have co nsisten t surface properties. In th e following section, a t t e m p t s to overcom e these problem s are described 4 . 7 .1
T ab le C o n s tr u c tio n
T h is table is co n s tru c te d off line before run ning th e ex p e rim en t. T h e following ass u m p tio n s were m ade: • T h e o b je c t is form ed of the sam e m aterial, hence th e illum ination a t any point is th e sam e (assum ing well d is trib u te d light and no shadow s). • T h e sa m e c a m e ra with th e sam e calib ration p a r a m e te rs are used d u rin g th e expe rim ent. • T h e lighting conditions will be the sam e d u rin g th e experim ent. We may consider these to be valid a ssu m p tio n s, since th e m a n u fa c tu rin g env iro n m e n t is totally con trolled, so we know the o b je c t m aterial and we set th e lighting conditions as desired. T h is table will be co n s tru c te d only once, th en it will be used for all o ur exp erim ents, as long as they satisfy ou r assu m p tio n s. However, if we wish to ex am ine an o b je c t with different m aterials, or we w ant to change th e lighting conditions, we will have to c o n s tru c t a new table using th e new o b je c t and the new lighting conditions. To c o n s tru c t this table, th e r o b o t a rm t h a t holds th e c a m e ra is moved vertically
30
in increm ental steps, according to th e required accuracy. A t each increm ent, an im age is taken and the intensity a t the ce nte r of th e im age is m easured, see F igure 26 for th e ex pe rim en tal setup.
F igure 26: C o n s tr u c tin g th e Illum ination Table off-line
4.7.2
Modifications
T h e initial im p le m e n tatio n of th is m eth o d did n o t pro du ce th e expected results because of th e noise in the im ages taken a t each d e p th . Several e n h a n c e m e n ts were ad d e d to this m e th o d to reduce th e effect of noise. F irst, instead of m easu ring th e intensity a t one point, we take th e average of th e intensities of a set of p oints t h a t c o n s titu te s a re c ta n g u la r window. By changin g th e window size, we can control th e s m o o th in g degree of th e m easured intensity. T h e second e n h a n c e m e n t is based also on averaging, by ta k in g several im ages a t each height and ta k in g th e average of th e calculated average window intensities. A fte r applying these two m odifications, the effect of noise was gre atly reduced. A n o th e r m odification was to move the light source with th e c a m e ra to increase the difference in the m easured intensity a t each height, which, in tu rn , should have increased th e resolution of o ur table. O ne last e n h a n c e m e n t was in c o rp o ra te d based on th e p ersp ective-projectiv e tra n s fo rm from world points to im age points. T h e window size used to calculate th e average inten sity a t each height should be the sam e, b u t according to the im age form a tio n e q u a tio n s using th e pinhole c a m e ra model, th e co rresponding window in th e im age will chan ge according to th e d ista n c e between th e c a m e ra (im age plane) and the o b ject.
From F ig u re 27, using simple trig o n o m etry , we get th e following relation between th e im age
window size and th e distan ce z betw een the o b je c t and th e cam era: xl
z2
x2
zl
31
Object
F ig u re 27: C h an g in g W ind ow Size A ssum ing P inhole C a m e r a Model
which shows t h a t th e im age window size is inversely p ro p o rtio n al to th e d istan ce between th e o b je c t and the ca m e ra. So, we have to ca lculate the new window size a t each height, which will be th e n um ber of pixels used for averaging. F igu re 28 shows a g ra p h for the c o n s tru c te d illum ination table used in ou r ex p e rim en t. It shows t h a t the intensity decreases when th e d istan ce betw een the o b je c t and the c a m e ra increases, b u t it also shows t h a t any change in th e lighting condition will give different results for th e illum ination table. T h is m e th o d also has som e pitfalls: F irs t, it is very sensitive to any light change, as shown in the figure. Second, th e difference in illum ination values for two close d e p th s is very small. For exam ple, in our ex p e rim en t, th e to ta l ra ng e of differences within 10cm was less th a n 30 gra y levels. Finally, it still has sm all a m o u n ts of noise a t som e points. We are now developing a n o th e r m eth od for d e term in in g d epth from focus. T h is m e th o d involves calculating d istan ce s to points in an observed scene by m odeling the effect t h a t th e c a m e r a ’s focal p a r a m e te rs have on im ages acquired with a small d ep th of field [7].
5
S e n s in g to C A D In te rfa c e
An i m p o r t a n t s te p in th e reverse engineering process is th e a c c u ra te description of th e real-world object. We g e n e ra te an A l p h a . l model from a com b in atio n of th re e ty pes of scene info rm ation . • T w o dim ensional im ages and fe a tu re contours. • S tereo vision d e p th inform atio n. • Touch info rm atio n from th e C M M (still to be im p lem ented.)
32
F igu re ‘28: T w o Different R esults W h e n C h a n g in g the Lighting C ond ition s By using each sensing m e th o d , we are able to g a th e r enough d a t a to c o n s tru c t the a c c u ra te C A D model necessary for reverse engineering (see F igu re 29.) T h e two dim ensional images provide fe a tu re detection t h a t is in tu rn used by th e stereo system to build fe a tu re models. Finally, the C M M elim inates un ce rtain ty th ro u g h the ex p lo ration of the features. T h e s t a t e m achine and the sensors are able to pro duce a set of d a t a p oints and the respective enclosure relationships. Each f e a tu re is co n s tru c te d in A lpha_l indepen d e n tly and th e final model is a com bin atio n of these featu res. T h is co m b in atio n is perform ed using th e recursive s t r u c t u r e of the o b je c t by form ing the co rresp ond in g strin g for t h a t o b ject and g en e ratin g th e code by parsing this strin g recursively. T h e third dim ension is retrieved from the stere o in form atio n and the illum ination tab le as described before. An exam ple for a re c o n stru cted p a r t is shown in F igure 30. T h is interface is one of th e m ost i m p o r t a n t m odules in this work, since it is th e real link between inspection and reverse engineering.
We have chosen A lpha_l as th e C A D language since it has very
powerful fe atu res, in add ition to th e fa ct t h a t it has interfaces w ith som e m a n u fa c tu rin g m achines, which allows us to actu a lly m a n u fa c tu re a hard copy of the p a r t. T h is will be o u r n ext step , so t h a t , the o u t p u t of o ur n ex t ex p e rim e n t will be a n o th e r p art, hopefully identical to th e original p art.
5.1
C ontours to Splines
In th e initial stage, of o u r sensing to C A D interface, we tra n s la te d th e ranged c o n to u rs we found into spline cu rves. B oth closed and open co n to u rs are represented as ordered sets of points. T h e c o n to u r points are used as control points on a spline curve in th e A lpha_l system . It is i m p o r t a n t no t to use all of the co n to u r points while fitting the spline. In m any cases, th ere are m ore th a n a th o u san d points in th e original image.
33
F igu re 29: T h e role of an internal C A D model in th e inspection m achine
T h is gives an over-con strained solution.
5.1.1
Thinning Contours
We m u s t supply a list of control po ints to A lpha_l t h a t will fit a spline ac cura te ly to th e original real-world featu re. Therefore, we m u st decide which points c o n trib u te to the ac tu a l s h a p e of th e fe a tu re and which points are sim ply re d u n d a n t. Obviously, regions of high c u r v a tu re are i m p o r t a n t to th e overall s h a p e of th e featu re, while low c u rv a tu re regions will n ot play as i m p o r t a n t a role. We fit lines to each c o n to u r an d re present them as polyline seg m en ts in A lph a_l. along its length.
Each line only consists of its en d p o in ts r a th e r th a n all th e im age points
All of the line seg m en ts and splines t h a t m ake up a p a r tic u la r co n to u r are com bined
to g e th e r using th e A lpha_l profile curve. An ex am p le is in F ig ure 31. T h e s tra ig h t lines in this closed c o n to u r are found a n d th e corn er points are used as “i m p o r t a n t ” p oints to th e A lp h a_ l model. P o in ts along th e to p arc are all used so t h a t a spline can be fit to th e m accurately. T h e final region is represented as th e co m bination of th e lines an d curves t h a t m akes up its length.
34
5.2
C ontou rs to M ach in ed F eatu res
A ltho ugh th e lines an d splines re p resen ta tio n proved useful, a n o th e r re p re s e n ta tio n was found to be needed to describe o ur m achined p a r ts . A set of m achineable features, as im plem en ted in th e A lpha_l modelling system [2], h as been selected for this re p resen ta tio n .
T h is set includes profileSides (for describing the
outside curve), profileF’ockets (for describing interior fe atu res g en e ra te d by milling), and holes (for describ ing fe a tu res g en e ra te d by drilling). A ltho ugh not within th e scope of c u rre n t research, th e m e th o d being described is extensible to include o th e r featu res, such as slots, bosses, etc. Using this su b set, m o st p a r ts which m eet th e re q u irem en ts of th e visual processing a lg o rith m s m ay be tra n s fo rm e d to a m achin eab le
«_1
represen ta tio n . 5 .2 .1
T h e A lg o rith m
'
T h e tra n s f o rm a tio n algo rithm is as follows: • in p u t: raw images, p a r t strin g, and ranged c o n to u r re presenta tio n • con vert p a r t s trin g to tree re p resen ta tio n (see F igure 32) • g e n e ra te stock using bo u n d in g box (see F igure 33) • g e n e ra te profileSide for o u te rm o s t c o n to u r (see F igure 34) • class each s u b fe a tu re as positive or negative relative to its predecessors (see F ig u re 35) • recursively descend th e p a r t tree, s ta r t i n g with th e o u te rm o s t c o n t o u r ’s children: — if negative, check for su b fe a tu re s positive relative to it * if none, prod u c e a hole or profile pocket d ep e n d in g on cu rv a tu re * otherw ise, m u s t be a profile pocket with an island, produ ce bo th - otherw ise, check to see if this island needs to be trim m e d • o u t p u t : or_l m odel co m posed of m achineable features N o te t h a t this alg o rith m assum es t h a t th e o u te rm o s t co n to u r is th e highest. T h is lim itatio n can be overcom e by a sim ple check a t th e s t a r t , an d s u b seq u e n t t r e a t m e n t of th e o u te r c o n to u r as a fe a tu re within a blockStock feature. 5 . 2 .2
D a t a P o in ts to A rc s a n d L in es
C o n to u rs a re converted to holes or a A profile curves, defined by co m b in atio n s of arcs and lines. T h is is accom plished using th e c u r v a tu re along th e con to u r.
A reas in which th e c u r v a tu re is zero (or below a
sm all thresh o ld ) for a specified distan ce along th e c o n to u r are considered to be line segm ents. A reas in which th e c u r v a tu re is c o n s ta n t for a specified d istance are considered to be arcs. C u rv a tu re , A:, a t a point on a curve is defined to be th e in s ta n ta n e o u s ra te of change of th e c u r v e ’s slope, , w ith respect to curve length, uS- [15]:
F igure 35: Positive (island) and negative (hole and pocket) featu res
k(s) = d{s)/ds where
ds = \Jdx2 + dy2 Slope is taken to be th e o rien ta tio n of the g ra d ie n t of the difference of G a u ssia n s (D O G ) function. T h e D O G functio n is an a p p ro x im a tio n to the Laplacian as m entioned in the Zero-Crossings section. T h e derivatives of slope are c o m p u te d using a forw ard difference technique, and th e results are sm o o th e d a user-controlled n u m b e r of times. A g ra p h of c u rv a tu re vs. distan ce along a curve can be seen in F igure 36. For each arc seg m en t, a circle is fit using a least sq u ares fit [14], and then the e n d p o in ts of the arc s egm ent are grown until th e distan ce from the co n to u r to the fitted circle exceeds a tolerance. T h is process is re p eated until grow ing has no effect or a n o th e r seg m en t is reached. A sim ilar m e th o d is used for the line segm ents. S egm ent d a t a is s to red as a linked list (see F ig ure 37). A Hough tra n s fo rm technique was considered for fitting curves. A lth o u g h very easy to im plem en t (one was im plem ented in a b o u t an hour for co m parison ), it was found to be too expensive in te rm s of memory. See A p p e n d ix C for a com p ariso n between the technique used, and th e Hough tran sfo rm . 5 . 2 .3
A r c s a n d L in es to M a c h in e d F e a tu r e s
If a negative fe a tu re co ntain s a positive featu re, then it m u s t be a profilePocket with an island (th e positive fe atu re). T h e island is trim m e d to the height of the positive f e a tu re with a profileGroove. If the negative fe atu re c o n tain s no positive fe a tu re and is com posed of only one arc segm ent, th en it m ay be represented by a hole. To be a hole, th e a r c ’s radius m u s t m atch one of a list of drill sizes within a tolerance. If a hole con tain s no o th e r featu res, and the interior of the raw im age is below a threshold, it m ay be considered to be a through-hole. Some asp e cts of m achined fe a tu res are difficult to m easure ac cura te ly using our c u rre n t im age processing algorithm s. For exam ple, fillets between line seg m en ts in a profilePocket m ay n o t be within th e accuracy of our vision, b u t are necessary for machineability. In cases such as these, default values are selected so as to have m inimal deviation from th e reverse engineered model, yet allow the m odel to be m achined. It is an ticip ated t h a t som e asp e c ts (such as cham fers and th rea d s) m ay be d etec ted , alth o u g h n o t accurately measured with vision.
38
Position on curve vs. Degrees/360
typedef struct arc_seg { double C e n t e r [2]; double Radius; > Ar c S e g ; typedef struct line_seg { double M, B; int Swapped; } LineSeg; typedef union segment { ArcSeg Arc; LineSeg Line; } Seg m e n t ; typedef struct element { int Type; int start, finish; Segment Data; struct element *Next; } SegNode, *SegPtr;
Three Holes? profile Pocket?
Two Slots + Two Holes? profilePocket?
F igu re 38: Possible co m b inations In its c u r re n t im p le m e n ta tio n , each co n to u r is t re a te d as a t m o st one m achined fe a tu re (some con to u rs m ay be islands and therefo re p a r t of a n o th e r c o n t o u r ’s fe a tu re ). F u tu re work will allow conto u rs to be m ad e from m u ltip le fe a tu res if a p p ro p ria te . For exam ple, com b in atio n s of drilled holes, slots, and pockets m ay be p rod uced (see F igu re 38), based on a m ach in in g /in s p e c tio n t i m e /c o s t analysis which will include such fa cto rs as tim e needed to select and load tools (o p e ra to r), change tools, etc. T his problem has som e ch aracte ristic s t h a t m ay be best solved th ro u g h artificial intelligence or o ptim iza tio n techniques. 5 . 2 .4
R esu lts
A lth o u g h th e model a t this in te rm e d ia te sta g e is still crude, it was considered to be a useful test to ha,ve a p a r t m a n u fa c tu re d from it. T h is in te rm e d ia te sta g e m odel will late r be u p d a te d w ith CMM. d a t a a„s described in th e section on In teg ratio n Efforts. T h e right p o rtio n of F ig u re 40 shows th e result of apply ing this m e th o d to an actu a l p a r t. T h e original p a r t m odel is shown in th e left po rtio n of t h a t figure a n d th e stereo im age pair used in im age processing is shown in F igure 39. N ote t h a t alth o u g h th e original C A D m odel was available in this case, it is used only for d e m o n s tra tio n purposes, n o t as p a r t of th e reverse-engineering process. T h e reverse-engineered m odel was used to create a process plan and m ach ine a p a r t on th e M on arch V M C -45 milling m achine. T h e original p a r t and re p ro d u ctio n can be seen in F igure 41.
6
P u t t in g it A l l T o g e th e r
So far, we have been talkin g a b o u t each m odu le separately, an d now it is tim e to p u t all these m odules to g e th e r to do som e useful work.
F irst, we will describe a previous ex p e rim en t t h a t was run w ith o u t
using th e ro b o t to move th e ca m e ra, and w ith o u t th e interface to th e C A D sy stem . T h is ex p e rim en t was m en tio ned in o u r last technical re p o rt [17].
T h e n we will describe a new ex p e rim en t t h a t uses all the
m odules described in this rep o rt.
41
F igure 39: Stereo im age pair from
Figure 40: Original and Reverse-Engineered part models
42
Figure 41: Original and reproduction
6.1
T h e F irst E x p e r im e n t
T his ex p e rim e n t was perform ed to in te g ra te the visual system with th e s t a t e m achine. An a p p r o p ria te D R F S M was g en e ra te d by observing the p a r t and g en e ratin g the fe a tu re inform ation. A m echanical p art was p u t on a black velvet b ac kg rou nd on to p of th e co o rd in a te m easuring m achine table to simplify the vision algorith m s. T h e c a m e ra was placed on a s t a t i o n a r y trip o d a t th e base of th e table so t h a t th e p a rt was always in view. T h e p ro b e could th en extend into the field of view and com e into c o n ta c t w ith the p a rt, as shown in F ig u re 42. O nce the first level of th e D R F S M was created , the ex p e rim en t proceeded as follows: F irs t, an im age was c a p tu r e d from th e ca m e ra. N ext, th e a p p r o p ria te im age processing takes place to find the position of th e p a r t, th e n u m b e r of fe a tu res observed (and th e recursive s trin g ), and the location of the probe. A p rog ram using this in fo rm ation produces a s t a t e signal t h a t is a p p r o p ria te for th e scene. T h e signal is read by th e s t a t e m achine and the next s t a t e is produced and re p o rted . Each closed fe a tu re is tre a te d as a recursive problem , as th e probe en ters a closed region, a new level of th e D R F S M is generated with a new tra n s itio n vector. T h is new level then drives th e inspection for the cu rre n t closed region. T h e specific d y n a m ic recursive D E D S a u t o m a t o n gen erated for the te s t was a s t a t e m achine G (shown in F igure 43.)
W h e re the set of s t a t e s X
= { In itia l,E O F ,E rro r ,A ,B ,C ,D } an d th e set of tran sitio n al
events £ = {1,2,3,4,5, 6 ,7,8,9,eof}. T h e s t a t e tra n s itio n s were controlled by th e in p u t signals supplied by in te rm e d ia te vision pro g ram s. T h e r e are four s tab le s ta te s A ,B ,C , and D t h a t describe th e s t a t e of the probe and p a r t in th e scene. T h e th ree o th e r s ta te s , Initial, E rro r, and E O F specify th e ac tu a l s t a t e of the system in special cases. T h e s ta te s can be in te rp re te d as: • Initial S ta te : W aiting for first in p u t signal.
Figure 42: Experimental Setup
F igu re 43: S ta t e M achine Used in Test
• A: P a r t Alone in Scene. • B: P r o b e and P a r t in Scene, p ro b e is far from p a r t. • C: P ro b e and P a r t in Scene, p ro b e is close to p a r t. • D: P ro b e touch ing or o verlapping p a r t (recursive s ta te .) • E rro r: An invalid signal was received. • E O F : T h e E n d of File signal was received.
6.2
R esu lts for th e First E xp erim ent
T w o typical sequences from a p ro bin g ta s k were run.
In th e first sequence (F igu re 44), th e p rob e was
introd uced into th e scene a n d m oved in a legal way (accepted by s ta b le s t a t e s in t h e m achine) to w a rd s the p a r t until c o n ta c t was m ade. N ext, th e p rob e backed off a n d again ap p ro a c h e d until th e p ro b e a n d p a r t o verlapped. T h e a u t o m a t o n was forced into an e rro r s t a t e by a p p ro ach in g from th e o th e r side o f th e p a r t m uch to o fast. T h e prob e was n o t seen until it was to o close to th e o b je c t body. B ecause a tra n s itio n from s t a t e A t o C is invalid, a n e rro r s t a t e is reached. T h e p a r t used was a sim ple one w ith only one hole, t h a t is, it is re p resen te d by : C ( C ( ) ) .
■
A n o th e r sequence was tried o u t (F ig ure 45), t h e p a r t was m ore com plex, th e re p re s e n ta tio n was recov ered to be th e following strin g : C ( C ( ) , C ( C ( ) ) , C ( ) ) . T h e p rob e was in tro d u c e d into th e scene an d moved legally to w a rd s th e p a r t. N ext, th e probe backed off a n d again ap p ro a c h e d until th e p ro b e a n d th e p a r t overlap ped . T h e a u t o m a t o n was forced into an e rro r s t a t e by th e su dd en d is a p p e a ra n c e of th e p ro b e after it was very close to t h e p a r t. B ecause a tra n s itio n from s t a t e C to s t a t e A is invalid, an e rro r s t a t e is re p o rte d . E ac h im age was displayed on a term in a l window as it was c a p tu r e d along w ith t h e co rrespon ding s t a t e o f th e a u t o m a t o n . T h e sam e s t a t e re p re s e n ta tio n s are displayed for different layers in th e D R F S M (i.e., for different featu res).
6.3
T h e N ew E xp erim en t
In o u r new ex p e rim e n t we used a r o b o t a r m (a P U M A 560), a vision sensor ( B / W C C D cam e ra) m o u n ted on th e end effector and a p ro b e to sim u la te th e c o o rd in a te m easu rin g m achine (C M M ) prob e, until th e necessary softw are interface for th e C M M is developed. Also th e re a re several softw are interfaces on a Sun S p a rc s ta tio n , for controlling all these devices (see F ig u re 46.) A D R F S M D E D S alg o rith m is used to co o rd in a te th e m o vem en t of t h e r o b o t sensor a n d th e probe. Feed back is provided to th e ro b o t a r m , based on visual ob servations, so t h a t th e o b je c t u n d e r consideration can be explored. T h is D R F S M was g en e ra te d by G U o e as shown in F ig u re
6.
T h e D E D S co ntrol alg orith m
will also guide th e p ro b e to th e relevant p a r ts of th e o b je c ts t h a t need to be explored in m ore detail (curves, holes, com plex s tru c tu re s , etc.) T h u s , th e D E D S controller will be able to m o d e l , report, a n d guide th e ro b o t an d th e p ro b e to reposition in te llig e n tly in o rd e r to recover th e s t r u c t u r e a n d s h a p e p a ra m e te rs . T h e d a t a a n d p a r a m e t e r s derived from th e sensing ag e n t are fed in to th e C A D s y s te m for designing th e g eo m e try of th e p a r t u n d e r inspection. We used th e a A design e n v iro n m e n t for t h a t p urp ose. Using th e a u t o m a t i c p ro g r a m m in g interface we have developed for a A , we g e n e ra te t h e required code to re co n stru ct th e o b je c t using th e d a t a o b ta in e d by th e sensing m odule.
6.4
R u n n in g th e N ew E xp erim en t
T h e first s te p in ru n n in g this ex p e rim en t was s e ttin g th e lighting con dition s as desired (sam e conditions when c o n s tru c tin g th e reflectance m a p tab le), th e n initializing th e r o b o t a n d th e c a m e ra a n d set th e m to initial positions. T h e ex p e rim e n t s t a r t s by ta k in g im ages for th e o b je c t from tw o positions, to g en e rate tw o sets of co n to u rs to be fed in to th e stere o m odule for d e p th es tim a tio n . Using th e ste re o m od ule with th e assistan ce o f th e reflectance m a p tab le and th e c a m e ra calib ratio n m odule, an initial set of world co o rd in a tes for th ese co n to u rs is g e n e rated . N ext, th e D R F S M D E D S m ach ine drives th e p ro b e an d th e
45
47
Sun Sparc
F ig u re 46: An E x p e r im e n t For Inspection and Reverse E ngineering r o b o t a rm holding th e c a m e ra to in spect th e o b je c t using th e in fo rm atio n g e n e ra te d from th e stere o m odule an d th e relation betw een th e o b j e c t ’s co nto u rs. F ig u re 47 shows th e D R F S M for th is ex p e rim en t. T h is m achine h as t h e following s ta tes: • A : T h e initial s t a t e , w aiting for t h e p ro b e to a p p e ar. • B : T h e p ro b e a p p e a rs , an d w aiting for it to be close. Here, close is a relative m e a s u re of th e d istan ce betw een th e p ro b e and th e c u r re n t featu re, since it d ep e n d s on th e level of th e recursive stru c tu re . F or ex am p le, th e d ista n c e a t th e first level, which re p resents th e o u te r c o n to u rs or features, is larger t h a n t h a t of th e lower levels. • C : P ro b e is close, b u t n o t on featu re. • D : T h e p ro b e a p p e a rs to be on fe a tu re in th e im age, and w aiting for physical tou ch in dicated from th e C M M m achine. • E : (T h e recursive s ta te ) P hysical to uch h as hap p e n ed .
If th e c u r re n t fe a tu re rep resen ts a closed
region, th e m ach in e goes one level deeper to get th e inner fe a tu re s by a recursive call to th e initial s t a t e after ch a n g in g t h e variable tra n s itio n p a r a m e te rs . If th e c u r re n t fe a tu re was an o pen region, th e n th e m ach ine finds any o th e r fe atu res in th e sa m e level. • F : T h is s t a t e is to solve any vision problem h a p p e n s d u rin g th e ex p e rim e n t. For exam ple, if th e p ro b e is occluding one of th e featu res, th e n t h e c a m e ra position can be changed to solve this problem . 48
ClosedRegion
F ig u re 47: T h e D R F S M used in th e new ex p e rim e n t • E R R O R l : Usually, th e re is tim e limit for each p a r t of th is ex p e rim e n t to be done. If for any reason, one of th e m odu les d o e s n ’t finish in tim e, th e m achine will go to th is s t a t e , which will re p o rt th e e rro r a n d t e r m i n a t e th e ex p e rim en t. A set of final world co o rd in a te s for th e c o n to u rs is o b ta in e d and fed t o th e a _ l interface, which in tu rn g e n e ra te s th e required code for g e n e ra tin g an a _ l m odel for th e th e o b je c t. F ig u re 48 shows a block d iag ra m for th is ex p e rim e n t w ith th e results after each step.
6.5
E xp erim en tal resu lts, A u to m a ted Bracket Insp ection
A m etal b ra ck et was used in t h e ex p e rim e n t to te s t th e inspection a u t o m a t o n . T h e piece was placed on th e inspection ta b le w ithin view of th e ca m e ra . Lighting in th e room was a d ju s te d so as to elim ina te reflection an d shadow s on th e p a r t to be inspected. C o n tro l signals t h a t were g en e ra te d by th e D R F S M were co n verted to sim ple English c o m m a n d s and displayed t o a h u m a n o p e r a to r so t h a t th e sim ulated p ro b e could be moved. T h e m achine was b r o u g h t on line a n d execution begun in S t a t e A, t h e s t a r t s t a t e . T h e c a m e r a moved to c a p tu r e b o th 2D a n d 3D ste re o vision in fo rm atio n and a rough a _ l m odel was c o n s tru c te d to describe th e surface, as show n in figure 49. T h e re co n stru ctio n tak es place in s t a t e A o f t h e m achine. T h e co n s tru c te d m odel is used by th e m achine in su b s e q u e n t s ta te s . F or exam ple, th e d ista n c e betw een th e p ro b e and th e p a r t is c o m p u te d using th is m odel a n d th e observed p ro b e location. A fter in itiatin g th e inspection process, t h e D R F S M tran s itio n e d th ro u g h s t a t e s until th e p ro b e reached th e b ra ck et bo u n d ary . T h e s t a t e m achine th e n called for th e closed region to be recursively inspected until finally, th e hole is explored and th e m achine exits cleanly.
49
State A: NoProbe
State B: ProbeFar
StateD: ProbeOnFeature
State C: ProbeClose
State D: ProbeOnFeature
StateE: TouchedFeature
Figure 50: Test Sequence (1) for Second Experiment
State A: NoProbe
State B: ProbeFar
StateE: TouchedFeature
StateD: ProbeOnFeature
StateC: ProbeClose
State B: ProbeFar
State C: ProbeClose
StateA: NoProbe
StateE: TouchedFeature
StateD: ProbeOnFeature
State C: ProbeClose
State D: ProbeOnFeature
StateB: ProbeFar
State C: ProbeClose
StateA: NoProbe
State B: ProbeFar
StateE: TouchedFeature
StateD: ProbeOnFeature
StateA: NoProbe
StateE: TouchedFeature
StateE: TouchedFeature
Figure 51: Test Sequence (2) for Second Experiment
6.6
E x p e r im e n ta l R e s u lts , T est P ie c e
A second ex p e rim e n t was run using a m etal piece t h a t was tooled a t th e U niversity of U t a h ’s A dvanced M a n u fa c tu rin g L ab o ra to ry . T h is piece offers in tere stin g fe a tu re s a n d has a com plex recursive s tru c tu re . T h is allowed us t o t e s t th e recursive n a t u r e of th e s t a t e m achine, and t h e c o rresp o n d in g inspection sequence is shown n ext.
7
S u m m a r y o f C u r r e n t D e v e lo p m e n ts
T h is s u m m a ry concludes th e re p o rt by outlin ing som e of t h e goals an d progress w ithin th e p ro je c t. We first describe som e goals and m ethodology, th e n we o utline c u r re n t and p a s t activities.
7.1
G oals and M eth o d o lo g y
We use a n o b server a g e n t w ith som e sensing capabilities (vision an d to uch) to actively g a t h e r d a t a (m ea s u re m e n ts) of m echanical p a r ts . G e o m e tric descriptions of t h e o b je c ts u n d e r analysis are g en e ra te d and expressed in te rm s of a C o m p u te r Aided Design sy stem . T h e g eo m etric design is th e n used to co n stru c t a p r o to ty p e of t h e o b je c t. T h e m a n u fa c tu re d p r o to ty p e s are th e n to be in spected a n d co m p a re d w ith th e original o b je c t using t h e sensing interface a n d refinem ents m a d e as necessary. T h e app lica tio n e n v iro n m e n t we are developing consists of th re e m a j o r working elem ents: th e sensing, design, a n d m a n u fa c tu rin g m odules. T h e u ltim a te goal is to establish a c o m p u ta tio n a l fram ew o rk t h a t is ca p ab le of deriving designs for m achin e p a r ts or o b jects, inspect a n d refine th e m , while cre a tin g a flexible a nd co n sisten t engineering en v iro n m e n t t h a t is extensible. T h e control flow is from th e sensing m odule to t h e design m od ule a n d th e n to th e m a n u fa c tu rin g co m p o n e n t. F eedback ca n be re-supplied to t h e sensing ag e n t to inspect m a n u fa c tu re d p a r ts , c o m p a re th e m to th e originals a n d con tin u e t h e flow in th e loop until a ce rta in tolera n ce is m et (see F ig u re 52). T h e sy stem is inten d ed to be u ltim a te ly as au to n o m o u s as possible. We s tu d y w h a t p a r ts of th e sy stem can be im plem ented in h ard w a re . Som e p a r ts seem to be inh erently suited to h ard w a re , while som e o th e r p a r ts of th e sy stem m ay be possible t o p u t in h ard w are , b u t e x p e rim e n ta tio n will provide th e basis for m ak in g t h a t decision. P ro v id in g lang u ag e interfaces between th e different c o m p o n e n ts in th e inspection and reverse engineering control loop is an integral p a r t of th e p rojec t.
7.2
Current D ev elo p m en ts
W e use a r o b o t a r m (a P U M A 560), a vision sensor ( B / W C C D c a m e ra) m o u n te d on th e end effector and will be using th e c o o rd in a te m easu rin g m achine (C M M ) w ith th e necessary softw are interfaces to a Sun S p a rc S ta tio n as th e sensing devices. A D R F S M D E D S alg o rith m is used to c o o rd in a te th e m o v em en t of t h e r o b o t sensor and th e C M M . Feedback is provided to th e r o b o t a rm , based on visual observations, so t h a t th e o b je c t(s) u n d er co nsid eratio n can be explored. T h e D E D S con tro l alg o rith m will also guide th e C M M to t h e relevant p a r ts of th e o b je c ts t h a t need to be explored in m ore d etail (curves, holes, com plex s tru c tu re s , etc).
T h u s t h e D E D S controller will be able to m o d e l , r e p o r t , an d guide t h e r o b o t and th e
C M M to reposition in te llig e n tly in o rder to recover th e s t r u c t u r e and s h a p e p a r a m e te rs .
53
F ig u re 52: Closed Loop Reverse Engineering T h e d a t a a n d p a r a m e te r s derived from th e sensing ag e n t are th e n fed into th e C A D sy s te m for designing th e g e o m e try of th e p a r t( s ) u n d er inspection. We use th e a _ l design e n v iro n m e n t for t h a t purpose. T h e goal is to provide a u t o m a t i c p ro g ra m m in g interfaces from th e d a t a o b ta in e d in t h e sensing m o dule to th e a _ l p ro g r a m m in g en v iro n m e n t. T h e p a r a m e tric and 3-D p oint descrip tion s a re to be in te g ra te d to provide co nsistent and efficient surface d escrip tion s for th e C A D tool. For pure inspection pu rp o se s th e c o m p u te r aided geo m etric d escrip tion of p a r ts could be used as a d r iv e r for guiding b o th th e robo tic m a n ip u la to r a n d th e c o o r d in a te m easu rin g m achine for exploring th e o b je c t an d recognizing discrepancies between th e real p a r t a n d th e m odel. T h e c o m p u te r aided design p a r a m e te rs will th en to be used for m an u fa c tu rin g th e p ro to ty p e s . T h e so ftw a re a n d h a r d w a re re q u irem en ts of th e e n v iro n m e n t are th e b a c k b o n e for th is p ro jec t. We selected p a r ts of t h e sy stem for possible h a rd w a re im p le m e n ta tio n . T h e D E D S m odel, as an a u t o m a t o n controller, is very su ita b le for P a t h P r o g ra m m a b le Logic (P P L ) im p le m e n ta tio n . A n u m b e r of th e visual sensing alg o rith m s could be successfully im plem en ted in P P L , saving considerable c o m p u tin g tim e. T h e re is a lot of interfacing involved in co n s tru c tin g th e inspection an d reverse engineering en v iro n m e n ts under co nsideration. Using m ulti-la n gu age o b ject-b ased com m u n icatio n an d control m eth o d o lo g y between th e th re e m a jo r c o m p o n e n ts (Sensing, C A D a n d C A M ) is essential.
54
7.3
P a st, C u rren t, and F u tu re A c tiv itie s
7.3.1
C o m p le te d a c tiv itie s
• Designed t h e D R F S M D E D S fram ew ork for recursive inspection. • Im p lem en ted im age processing m odules for recognizing fe a tu res a n d p ro b e position on t h e p arts. • Designed a n d im p le m e n ted visual s t r u c t u r e recovery techn iqu es for m achine p a r t s (using stereo, c o n to u r a n d illum ination m a p d a t a ,) a n d im p le m e n ted calib ratio n routin es.
.
• Designed a n d im p le m e n ted a sensing to C A D interface for g e n e ra tin g a A code for bodies from d e p th , c o n to u r (and d a t a reduction ,) illum ination m ap , a n d t h e recursive fe a tu re relationships. • Im p lem en ted th e D R F S M D E D S a u t o m a t a for recursive inspection (using ro b o t-h e ld ca m e ra , probe a n d ac tu a l p a r ts .) • Designed sen sor a n d s tra te g y - b a s e d u n c e rta in ty m odelling tech niq ues for th e ro b o t-h e ld ca m e ra , for recovering t h e D E D S tr a n s itio n a l “ev e n ts” w ith u ncertainty. • D esigned a n d im p le m e n ted a m odification to an existing reactive beha v io r design tool (G U o e ) to a c c o m m o d a te “d u m p in g ” th e code of D R F S M D E D S from a gra p hic al interface (used t o d ra w th e inspection con tro l a u t o m a t o n .) • Im p lem en ted fe a tu re identification for s u b seq u e n t m a n u fa c tu rin g (from sensed d a t a , i.e, w h a t does set(s) of sensed d a t a p oin ts “m e a n ” in t e rm s of m a n u fa c tu rin g featu res.) • M a n u f a c tu re d p a r t s from c a m e ra re c o n s tru c te d a A surfaces.
7.3.2
C u r re n t a c tiv itie s
• Designing th e D E D S to VLSI design lang uag e interface (a gra phic al interface). • Designing a n d im p le m e n tin g th e softw are “u n c e rta in ty ” m od u le for su b s e q u e n t h a rd w irin g into a chip. • Using focusing, m o tio n, m o m e n ts , shading, a n d m o re a c c u ra te r o b o t a n d c a m e r a calib ratio n tech niques t o e n h a n ce t h e visual proceesing. • F e a tu re interaction identification for m a n u fa c tu rin g (i.e. how can sensed fe a tu re s b est be represen ted for m a n u fa c tu rin g .) • M odifying th e sensing to C A D interface for allowing C M M sensed d a t a , in ad d itio n to visual d a t a . • Im p lem en tin g th e D R F S M D E D S a u t o m a t a for recursive inspection a n d reverse engineering (using m oving c a m e ra , C M M a n d ac tu a l p arts.) • Im p le m e n tin g “safety ” recursive D E D S for checking th e sensing activities, for exam ple, position s of pro be, p a r t, a n d ca m e ra .
55
v
7.3.3
Future activities
• Im p le m e n t t h e VLSI m o dules for th e D R F S M D E D S controller. • Im p lem en t t h e “U n c e rta in ty ” chip. • M a n u f a c tu re p a r ts from c a m e r a a n d C M M re c o n s tru c te d a _ l surfaces (with fe a tu re in te ra c tio n iden tification built in.) • W ritin g a n d using a c o m m o n sh a re d d a t a b a s e for s to rin g d a t a a b o u t t h e g eom etric m odels a n d the rules specifying th e co m m u n ic a tio n betw een th e different phases. • Im p lem en t sensor-based noise m odeling m odules for th e ro b o t-h e ld c a m e r a a n d t h e C M M (h a rd w are a n d softw are.)
8
In t e g r a t io n E ffo r t s
T h e following explains som e of th e in teg ra tio n efforts w ithin t h e different areas of th e p ro jec t.
8.1
R o b o tics and Sensing
We in ten d to develop a softw are interface for th e C M M m achine, a n d a discrete eve n t d y n a m ic system (D E D S) alg o rith m will be used t o c o o rd in a te th e m o v em en t of th e r o b o t sensor a n d th e C M M . T h e D E D S control alg o rith m will also guide th e C M M to th e relevant p a r ts of th e o b je c ts t h a t need to be explored in m ore d etail (curves, holes, com p lex s tru c tu re s , etc.) As a s t a r t i n g p oint to develop this interface, we will w ork w ith a package c u rre n tly u n d er d evelopm ent a t th e U n iversity o f U ta h .
A P I, th e A u t o m a t e d P a r t Inspection package, developed as p a r t of Mike
van T h ie l’s thesis [21] is a se m i-a u to m a tic feature-b ased p a r t insp ecto r t h a t is fully in te g ra te d w ith th e a A sy stem .
T h is package, som e of which can be seen in F ig u re 53, enables a user w ith an a A model
com p osed of m achined fe a tu res to sim u la te a n d / o r drive th e C M M to in spect t h e m achined p a r t. Using ou r in te rm e d ia te fe a tu re-b ased m odel t o guide th e inspection as if it were th e original, we will be able to in c o rp o ra te th e sense o f to u ch in to o u r knowledge base. W i t h a new, m ore a c c u ra te model, we m ay loop back t o t h e beginning o f th e inspection process until we have c a p tu r e d every a s p e c t of t h e p a r ts we inspect to t h e to lera n ces we desire.
8.2
C om p u ter A ided D esign and M anufacturing
We in ten d t o develop th e C A D interface to be m ore a c c u ra te an d t o accep t m o re com p licated m odels. T h e goal is t o e n h a n ce th e a u t o m a t i c p ro g ra m m in g interface betw een th e d a t a o b ta in e d in th e sensing m odule to th e a _ l p ro g r a m m in g en v iro n m e n t. T h e p a r a m e tric a n d 3-D p oint d escriptions are to be in te g ra te d to provide co n sisten t a n d efficient surface descriptions for th e C A D tool. F o r p u re inspection purposes t h e c o m p u te r aided g eom etric description of p a r ts could be used as a d r iv e r for guiding b o th t h e robotic m a n ip u la to r a n d t h e c o o r d in a te m easurin g m achine for exploring t h e o b je c t a n d recognizing discrepancies betw een t h e real p a r t a n d th e model.
56
v
F ig u re 53: T h e A P I User Interface T h e c o m p u te r aided design p a r a m e te r s are th e n to be used for m a n u fa c tu rin g t h e p ro to ty p e s . C o n siderable effort has been m a d e for a u to m a tic a lly m oving from a c o m p u te r aided g eom etric m odel to a process plan for m ak in g th e p a r ts on th e a p p r o p ria te N C m achines and th en to a u to m a tic a lly g e n e ra te th e a p p r o p ria te m achine in s tru c tio n s [6]. We use th e M o n arch V M C -4 5 milling m achine as th e m an u fa c tu rin g host. T h e a _ l sy s te m p rod uc es t h e N C code for m a n u fa c tu rin g th e p a rts .
8.3
V L SI, U n certa in ty M od elin g, and Languages
T h e softw are a n d h a r d w a re re q u irem en ts o f th e e n v iro n m e n t are th e b ac k b o n e for th is p ro je c t. We intend to select p a r ts o f th e sy stem im p le m e n ta tio n an d s t u d y th e possibility of hard w irin g th e m .
T h e r e has
been co nsiderable effort and experience in VLSI chip design [5, 8] an d one o f t h e s u b -p ro b le m s would be to s t u d y th e need an d efficiency o f m ak in g custom ized chips in t h e en v iro n m e n t. T h e D E D S model, as an a u t o m a t o n , is very su ita b le for P a t h P r o g ra m m a b le Logic ( P P L ) im p le m e n ta tio n .
A n u m b e r of th e
visual sensing alg o rith m s could be successfully im plem ented in P P L , saving considerable c o m p u tin g time. In te g r a te d circuits for C A G D surface m a n ip u latio n is an effort t h a t is a lre ad y underw ay. We intend to in v estigate a new area: t h e possibility of im ple m e n tin g th e D E D S p a r t of t h e sy s te m in in te g ra te d circuitry. A n o th e r i m p o r t a n t p a r t to be im p lem en ted in h ard w a re , is th e “U n c e rta in ty ” chip, which will provide fa s t decisions a b o u t th e accu ra cy of o u r m e asu rem en ts. T h is is i m p o r t a n t for deciding w h e th e r th e p a r t needs m ore inspection ste p s or no t.
T h e u n c e rta in ty m odel d ep e n d s on t h e n a t u r e of th e p a r t being
inspected, th e sensor, t h e s t r a t e g y being used to sense th e p a r t, and th e required accuracy.
57
T h e re is a lot of interfacing involved in co n s tru c tin g th e inspection a n d reverse engineering en v iro n m e n ts u n d e r co n sid eration . Using m ulti-la n g u ag e o b ject-b ased co m m u n ic a tio n a n d co ntrol m e th o d o lo g y between t h e th re e m a j o r c o m p o n e n ts (Sensing, C A D a n d C A M ) is essential. We intend to use a c o m m o n shared d a t a b a s e for sto rin g d a t a a b o u t th e g eom etric m odel a n d t h e rules governing th e in te ra c tio n o f th e different phases in th e re p ro d u c tio n a n d inspection p a rad ig m s [10, 19]. We have a lre ad y used a gra ph ic al behavior design tool [4] for th e a u to m a tic p ro d u c tio n of th e sensing D E D S a u t o m a t a code, from a given control language descriptio n. A sensing — > C A D interface has been developed as well.
9
C o n c lu s io n s
We p ropose a new s t r a t e g y for inspection a n d / o r reverse engineering of m achin e p a r ts a n d describe a fram ew o rk for c o n s tru c tin g a full e n v iro n m e n t for generic inspection a n d reverse engineering. T h e problem is divided in to se n sin g , design, a n d m a n u fa c tu rin g co m p o n e n ts w ith th e underlying softw are interfaces a nd h a r d w a re backbone. We use a recursive D E D S D R F S M fram ew o rk to c o n s tru c t an intelligent sensing m odule. T h is p ro je c t aim s a t developing sensing a n d control s tra te g ie s for inspection a n d reverse engi neering, a n d also a t co o rd in a tin g th e different activities betw een th e phases. T h e developed fram ew ork utilizes existing knowledge to fo rm u la te an a d a p tiv e a n d goal-directed s t r a t e g y for exploring, inspecting, a nd m a n u fa c tu rin g m echanical p a r ts .
R e fe re n c e s [1] ALOIMONOS, J . , AND S h u l m a n , D .
In teg ra tio n o f V isual M o d u le s A n E x te n s io n o f the M a r r
P a r a d ig m . A cad em ic P ress, 1989.
[2] a _ l . c t.l u s e r’s m a n u a l, 1993. U niversity of U ta h . [3] B e n v e n i s t e , A . , a n d G u e r n i c , P . L. H y brid dy n am ica l s y stem s th e o ry a n d th e signal language. I E E E T r a n s a c tio n s on A u t o m a t i c C o n tro l 35, 5 (M ay 1990).
[4] B r a d a k i s , M . J .
R eactive b ehavior design tool. M a s t e r ’s thesis, C o m p u te r Science D e p a r tm e n t,
U niversity o f U t a h , J a n u a r y 1992. [5] C a r t e r , T . M ., S m i t h , K. F ., J a c o b s , S. R ., a n d N e f f , R. M . Cell m a t r i x m ethodologies for i n te g ra te d circuit design. In teg ratio n , The V L S I J o u r n a l 9, 1 (1990). [6] D r a k e , S ., a n d S e l a , S. A fo u n d a tio n for featu res. M e c h a n ic a l E n g in e e r in g 111, 1 ( J a n u a r y 1989). [7] E n s , J . , a n d L a w r e n c e , P . A n investigation of m e th o d s for d e te rm in in g d e p t h from focus. I E E E T r a n s a c tio n s on P a t t e r n A n a ly s i s a n d M a c h in e In tellig e n c e 15, 2 (F e b ru a ry 1993).
[8] G u , J . , AND SMITH, K . A s t r u c t u r e d a p p ro a c h for vlsi circuit design. I E E E C o m p u te r 22, 11 (1989). [9] H s i e h , Y . C . R e c o n s tru c tio n of scu lp tu red surfaces using c o o rd in a te m easu rin g m achines. M a s t e r ’s thesis, M echanical E ngineering D e p a r t m e n t , U niversity of U ta h , J u n e 1993.
58
[10] L i n d s t r o m , G . , M a l u s z y n s k i , J . , a n d O g i , T .
Using ty p es to interface fun c tio n a l a n d logic
pro g ra m m in g . In 1991 S I G P L A N S y m p o s iu m o n P r in c ip le s o f P r o g r a m m in g L an g u a g e s (Ju ly 1990), vol. 10. technical s u m m a r y (s u b m itte d ). [11] M a r r , D . , a n d H i l d r e t h , E . T h e o ry of edge detec tio n . In P roceed in gs R o y a l S o c i e ty o f L o n d o n B u lle tin (1979), vol. 204, pp. 3 01-328.
[12] N e r o d e , A . , a n d R e m m e l , J . B. A m odel for hybrid system s. In P roc. H y b r id S y s t e m s W orkshop, M a th e m a tic a l S c ie n c e s I n s titu te (M ay 1991). Cornell University.
,
[13] OzVEREN, C . M . A n a ly s is a n d C o n tro l o f D is c r e te E v e n t D y n a m i c S y s t e m s : A S ta t e Space A pproach. P h D thesis, M a s s a c h u s e tts I n s titu te of Technology, A u g u st 1989. [14] P r a t t , V . D ire ct least-sq u a res fittin g of algebraic surfaces. In S I G G R A P H ’8 7 (1987), pp. 145-152. [15] SCHALKOFF, R . J . D ig ita l Im a g e P ro c e ss in g a n d C o m p u t e r V ision. J o h n W iley a n d Sons, Inc., 1989. [16] SOBH, T . M ., AND BAJCSY, R . A m odel for observing a m oving agent. In P ro ceedin g s o f the Fourth In te r n a t io n a l W o rk sh o p on I n te llig e n t R ob ots a n d S y s t e m s ( I R O S ’91) (N ovem ber 1991).
Osaka,
Japan. [17] S o b h , T . M ., D e k h i l , M ., J a y n e s , C .,
and
H e n d e r s o n , T . C . A d y n a m ic recursive s tr u c tu r e
for intelligent e x p lo ra tio n . Tech. R ep. UUCS-92-036, D e p a r t m e n t of C o m p u te r Science, University of U ta h , O c to b e r 1992. [18] S o b h , T . M ., O w e n , J . C ., J a y n e s , C ., D e k h i l , M .,
and
H e n d e r s o n , T . C . A ctive inspection
a n d reverse engineering. Tech. R ep. UUCS-93-007, D e p a r t m e n t of C o m p u te r Science, University of U ta h , M arch 1993. [19] S w a n s o n , M ., a n d K e s s l e r , R. D o m ain s : efficient m echanism s for specifying m u tu a l exclusion and disciplined d a t a s h a rin g in co n c u rre n t scheme. In F ir s t U .S ./ J a p a n W o rk sh o p o n P a r a lle l (A ug ust 1989). [20] T s a i , R.
A versatile c a m e ra calib ratio n tech niq ue for high-accu racy 3d m achin e vision m etrology
using off-the-shelf t v c a m e ra s and lenses. In R a d i o m e t r y - ( P h y s ic s - B a s e d V isio n ) , G . H. L. Wolff, S. Shafer, E d. J o n e s and B a r t l e t t , 1992. [21] VAN T h i e l , M . T . Feature B a s e d A u t o m a t e d P a r t In sp e ction . P h D thesis, U niversity of U ta h , J u n e 1993. [22] W
il l s o n ,
R . P ers o n a l co m m u n icatio n , 1993.
59
9
10
A p p e n d ix A : S a m p le G lJ o e O u tp u t
int State_B(VTV_ptr) vtypa *VTV_ptr; int DoneFlag; EventType Event; vtype *newVTV_ptr; int EventMask=0; #ifdef VERBOSE printf("in state B\n"); #endif if (VTV.ptr == NULL) { #ifdef VERBOSE fprintf(stderr," * * * ERROR: null vtv in state B\n"); #endif exit (4 ); >;
EventMask |= TimeOutMask; EventMask |= NoProbeMask; EventMask |= ProbeCloseMask; EventMask |= ProbaFarMask; DoneFlag = FALSE; while ( ! DoneFlag) { Event = Get_DRFSM_Event(EventMask, VTV_ptr); if (Event.type == TimeOut) { DoneFlag = TRUE; if (Event.fn != NULL) DoneFlag = ( * (E v en t.fn ))(); State_ERROR(VTV_ptr); >
else if (Event.type == NoProbe) DoneFlag = TRUE; if (Event.fn != NULL) DoneFlag = ( * (Event.fn)) ( ) ; State_A(VTV_ptr); >
60
else if (Event.type == ProbeClose) { DoneFlag = TRUE;
if (Event.fn != NULL) DoneFlag = ( * (Event.fn) ) ( ) ; State_C(VTV_ptr); >
else if (Event.type == ProbeFar) { > >
11
A p p e n d ix B : S a m p le C a lib r a tio n C o d e O u tp u t
Coplanar calibration (full optimization) data file : a.pts f = 8.802424
[mm]
kappal = 0.003570 Tx = -25.792328, Rx = -134.988935,
■ [l/mm“2] Ty = 77.376778, Ry = -0.127692,
Tz = 150.727371 Rz = -0.068045
[mm]
'
[deg]
R 0.999997
0.000737
0.002416
-0.001188
-0.706972
0.707241
0.002229
-0.707242
-0.706968
sx = 1.000000 Cx = 276.849304,
Cy = 252.638885
[pixels]
Tz / f = 17.123394 calibration error: mean = 0.331365,
standard deviation = 0.158494
62
[pixels]
( 2ncomp+3nass J
fwd_diff 1stderivativebyforward diff. using"dir" fromzcedge, alongcontour
mem: 3ninis+ 3nfloats contours
I T
filter_corners markspikesinthe 1stderivativesothey won’tbeusedinaveraging
f 2nadd. +2nass. +nmult. + 3ncomp.
X smooth smooththe 2ndderivative
smooth smooththe 1stderivative
second_diff computethe2nd derivativeusing forwarddifference
fwd_diff asbefore, but usingsmoothed data
smooth average"dir" overa window "w", "m" times
m(14ncomp. +4nadd. +2nmult. +4nass.)
mark_circles detectcircular arcsincontour where2ndderiv isconstant
E
f 4ncomp. +2nadd. +2nass.
shrinkL_fit_circles sortcirclesby radius, lumptogether similarcircles, then recomputefit
fit_circles leastsquares fittocircular arcs ^
(?ncomp. + lOnass+3nmult. +7n
7ncomp. + lOnass+3nmult. +7nadd. mem: 4ndouble shrink_fit_lines
mark^lines
fitjines leastsquaresfit tolinesegments
delectlinesegs. incontourwhere 1stderiv. isconstant ^ 4ncomp. +2nadd. +2nass.
^
sortlinesbyslope, lumptogethersimilar lines, thenrecompute fit
i 12nass+9nadd+5nmult+6ncomp mem: 3ndouble
( 12nass+9nadd+5nmult+ 6ncomp
circles& lines summary: add. n(42+ m(12)) mult. n(19+ m(6)) comp. n(45 +m(42)) ass. n(57 +m(12)) mem: 3nints 3nfloats 7ndoubles ~ )
F ig u re 54: F lo w c h a rt of th e im plem en ted c u r v a tu re technique
12
A p p e n d ix C : C o m p a ris o n b e tw e e n H o u g h t r a n s fo r m a n d c u rv a t u r e te c h n iq u e
T h e c u r v a tu re techn iq u e as im p lem en ted for this app lication is described in flow chart form in F ig u re 54. Using a sim ilar analysis for a basic H ough tra n s fo rm im p le m e n ta tio n (just to d e te c t circles), shows t h a t it would require: • M M M assig n m en ts (to initialize m em ory ). • NMM — 5 add itio n o p e ra tio n s — 3 m ultip licatio n o p e ra tio n s — 1 s q r t o p e ra tio n s — 6 assign m ents — 3 co m parison s
63
• M M M integers for memory. Where M is the precision. Assuming pixel accuracy, M is approximately N / n . N, for this application, can be taken to be contour length, bounded by ^ N n neaN sampies. Thus, the Hough tranform may be considered of order N 3 while the curvature technique used is at most order N 2. Not included in the Hough evaluation is that it would be necessary to do some sort of mode detection to determine the number of circles found. It is anticipated that the fitting algorithm may be extended to include other conic sections than circles, and additionally that it may be extended to use three dimensional coordinates. While the Hough transform is a very useful technique, we anticipate that its memory and order requirements will grow too rapidly to meet our future needs.
64
