Content Based Image Retrieval for Matching Images of Improvised Explosive Devices in which Snake Initialization is Viewed as an Inverse Problem Scott T. Actonb, Andrew D. Gilliamb, Bing Lib, Adam Rossia a
Platinum Solutions, Reston, VA, USA C. L. Brown Dept. of Electrical & Computer Engineering, University of Virginia, Charlottesville, VA, USA [
[email protected],
[email protected],
[email protected],
[email protected]]
b
ABSTRACT Improvised explosive devices (IEDs) are common and lethal instruments of terrorism, and linking a terrorist entity to a specific device remains a difficult task. In the effort to identify persons associated with a given IED, we have implemented a specialized content based image retrieval system to search and classify IED imagery. The system makes two contributions to the art. First, we introduce a shape-based matching technique exploiting shape, color, and texture (wavelet) information, based on novel vector field convolution active contours and a novel active contour initialization method which treats coarse segmentation as an inverse problem. Second, we introduce a unique graph theoretic approach to match annotated printed circuit board images for which no schematic or connectivity information is available. The shape-based image retrieval method, in conjunction with the graph theoretic tool, provides an efficacious system for matching IED images. For circuit imagery, the basic retrieval mechanism has a precision of 82.1% and the graph based method has a precision of 98.1%. As of the fall of 2007, the working system has processed over 400,000 case images. Keywords: image analysis, image retrieval, active contours, image segmentation
1. INTRODUCTION In terms of weapons of terrorism, improvised explosive devices (IEDs) constitute one of the most serious threats to safety in the civilized world. An IED is formed with a crude circuit, battery power, wiring and explosive charges. As these devices can be detonated remotely, linking a terrorist to a specific device is a difficult task. However, according to [1], IEDs have a “special signature of design that can help us identify the bomb maker and his organization.” Our solution, which is currently in use and will soon exceed one half million images processed, has a basic mode of operation that exploits shape, color, and texture information of the IEDs in a content-based image retrieval (CBIR) framework. The key step of shape extraction is accomplished via a novel active contour, also termed a snake, with a large capture range and low computational cost. One of the most significant drawbacks of snake-based image analysis is the sensitivity to snake initialization. In fact, the majority of 2D active contours and 3D active surfaces in use today require manual initialization. This paper emphasizes snake initialization via an automatic method that we term Poisson inverse gradient initialization. This solution approaches snake initialization as an inverse problem, i.e. given the snake external force, the Poisson inverse gradient selects a closed contour (or surface) that represents a low energy (high quality) initial solution. According to the preeminent inverse problem theoretician Alifanov, “Solution of an inverse problem entails determining unknown causes based on observation of their effects." So, in the case of snake initialization, the effects are the external force vectors that are caused by the object boundary. We would like to approximate this object boundary given the external force vectors. This basic system is augmented with a unique graph theoretic approach for matching circuit board images with no
schematic or connectivity information. Many researchers have converted images into attributed graphs consisting of nodes at objects or regions of interest and edges connecting those nodes [2]. Thus, the image classification problem is transformed into a graph matching problem, which has long been a vigorous area of research in the pattern classification field [3, 4]. One particular interest is a graph matching heuristic termed graph edit distance, proposed by Sanfeliu and Fu [5], which finds the distance between two graphs by determining the cost associated with editing one graph into the other. We adapt this concept to measure the similarity between two circuit boards, given annotated electronic component types and locations within the circuit board. This novel method we term circuit edit distance (CED). In this paper, we introduce in detail both the shape-based CBIR method and the CED matching technique. We additionally present quantitative testing results of both techniques.
2. CBIR WITH VECTOR FIELD CONVOLUTION Our CBIR technique exploits shape, color, and texture information to search and classify IED imagery. Integral to this technique is the segmentation of an object of interest within a given image. This section first presents the novel vector field convolution (VFC) active contour for shape segmentation, followed by the Poisson inverse gradient (PIG) method for snake initialization. We then present a general overview of our CBIR technique. As other CBIR steps (morphological pre-processing, color histograms, FSD representation and the Haar-wavelet textural description) are not the focus of this paper, we will detail only the segmentation step. 2.1. Vector Field Convolution Active Contours To capture the dominant object shape in a given image, we make use of the novel VFC active contour. An active contour captures a desired image feature by minimizing a corresponding force balance equation, typically the weighted sum of contour-based internal forces which ensure a smooth solution and image-based external forces which attract the contour to features of interest. VFC defines a novel external force field with a large capture range and the ability to converge to concavities. As compared to gradient vector flow [6], VFC yields improved robustness to noise and initialization, a more flexible force field definition, and reduced computational cost [7]. Let k (with x-component s and y-component t) be a vector field kernel defined as k ( x , y ) = m ( x , y ) ⋅ n ( x, y ) ,
(1)
where m ( x, y ) is the magnitude of the vector at ( x, y ) and n( x, y ) is the unit vector pointing to the origin. The VFC external force field v( x, y ) is given by the convolution of the vector field kernel k ( x, y ) and the image edge map f ( x, y ) (normalized gradient magnitude):
v( x, y) = f ( x, y ) ∗ k ( x, y ) , = [ f ( x, y) ∗ s( x, y), f ( x, y) ∗ t ( x, y)]
(2)
where ∗ denotes convolution. As the function f ( x, y ) is larger near the image edges, these edges contribute more to the VFC force field than homogeneous regions. The VFC field is highly dependent on our choice of the vector field kernel magnitude m( x, y ) . By considering the fact that the influence from the feature of interest should be less as the particles are further away, the magnitude should be a decreasing function of distance from the origin:
m( x, y) = (r + ε ) −γ ,
(3)
where γ and σ are positive parameters to control the rate of decrease and ε is a small positive constant to prevent division by zero at the origin. This formulation is inspired by Newton's law of universal gravitation in physics which can be viewed as a special case of (3) with γ = 2 and ε = 0 .
It is additionally desirable that the snake move rapidly through the background region in order to quickly capture the foreground object of interest. We therefore introduce a shrinking external force based on statistics of the background region to accelerate the segmentation convergence. Let Asat represent the average background saturation value and Avalue represent the average background intensity value. We define the distance d ( x, y) of a given image pixel located at position ( x, y) , with saturation s ( x, y ) and value v( x, y) , to the background region as d ( x, y ) =| Asat − s( x, y ) | + | Avalue − v( x, y ) | .
(4)
We then normalize d ( x, y) such that its maximum is equal to one. The corresponding shrinking external force points to the center of the image and has a magnitude Shrink ( x, y) = 12 e − k ⋅d ( x, y ) ,
(5)
where k is a damping constant (set to .05 here). The force given by the product of (5) with a unit vector pointing toward the image center is added to the VFC force of (2) to yield the total external force. The external force field (u, v) , consisting of both the aforementioned VFC and shrinking forces, is utilized in the following snake evolution update equations (updates for each ( x, y ) coordinate in the contour parameterized by s and time τ): ∂4x ∂x ∂2x = α 2 − β 2 + u ( x, y ), ∂t ∂s ∂s
∂4 y ∂y ∂2 y = α 2 − β 2 + v ( x, y ) . ∂t ∂s ∂s
(6)
Here, α and β are the typical Kass et al. [8] tension and rigidity parameters, set to α = 0.5 and β = 0.3 in this application. 2.2. Initialization of the snake by solving an inverse problem There are three basic choices in active contour initialization: 1) naïve initialization, 2) manual initialization, and 3) automatic initialization. Naïve initialization defines the initial contour without knowledge of image content, e.g. one may initialize the snake at the image boundary or as a simple geometric shape. Such initializations are generally distant from the object of interest and require an inordinate number of iterations to capture the desired boundary. Further, such a naïve initial contour may lead to divergence, in which the incorrect object is captured or clutter “distracts” the snake from the feature of interest. Manual initialization lets the user define a coarse object boundary, which is then refined by a active contour algorithm. Such manual interaction is tedious and time-consuming, and in applications such as the one at hand (CBIR on hundreds of thousands of images) is not a feasible option. Furthermore, in other applications that involve 3D imagery, manual initialization is difficult if not impossible. We must therefore consider the third option of automatic active contour initialization. In addition to the method presented here, there are at least two automated initialization options for active contours and surfaces. The first method, termed centers of divergence (CoD) [9], places small circular initial contours at points of zero vector divergence within a given external force field, such as those provided by gradient vector flow [6] or VFC [7]. Like the approach presented here, the CoD method is well grounded in using the external force to form an initial segmentation, as the vectors within the force field are pointing towards objects of interest. However, the CoD suffers from over-segmentation and often requires significant post-processing in the form of region merging. Another alternative, termed force field segmentation (FFS) [10], quantizes the external force vector field (into say, eight directional unit vectors) and forms connected components in between opposing vectors to define the initialization for a system of snakes. This approach, while also exploiting the external force system, is sensitive to clutter and broken edges and generally leads to spurious contours. Our approach, termed Poisson inverse gradient (PIG) initialization, is essentially the generation of a crude segmentation solution that has an associated energy which is close to the global minimum. Rather than using the traditional segmentation framework which first defines an external force field and subsequently evolves an active contour to delineate the object boundary, the PIG method instead attempts to solve the inverse problem of determining the object boundary that produced a given external force field. As mentioned in the introduction, we are estimating the boundary that caused the presented external force vectors.
Integral to this technique is the calculation of an external energy field E ( x, y ) associated with a given force field f ( x, y ) . This energy field has the same domain as the image and maps the locations of the image plane to a scalar energy. We find lines of constant value, termed isolines, within this energy field E and define the minimum isoline (isoline of minimum energy level) as our active contour initialization. This minimal isoline represents a minimal energy approcimation of the true optimal contour. Formally, we seek the energy E ( x, y ) that has a negative gradient that is close to f ( x, y ) in the L2 sense: E ( x, y ) = arg min ∫∫ − ∇E ( x, y ) − f ( x, y ) dxdy . 2
(7)
E
The solution to (7), well known to those who have studied electromagnetism, is given by Poisson’s equation: ΔE ( x,y) = − div f ( x , y ) .
(8)
In a linear algebraic sense, the discrete version of (8) defines a well-studied symmetric positive-definite banded sparse system. Possible solutions include successive over-relaxation [11], conjugate gradient techniques [12], fast Fourier transform (FFT) techniques [13] and multigrid methods [1, 14]. Outside of the IED application, we have validated the inverse initialization approach in synthetic image tests. For each test, the CoD method, the FFS method and the PIG method were compared. The number of iterations shown in Table 1 reflects the number of numerical steps (using the same parameters) required by the VFC snake to reach the object from the given initialization. The CPU times are broken down by initialization and the time required by the snake to converge. Root mean squared error (RMSE) with respect to the ideal boundary is also provided. The first set of tests involves 11 oval-shaped objects with varying widths of broken edges. The second set involves 11 C-shaped objects with increasing curvature. To test for sensitivity to noise, 50 images of synthetic shapes at 11 SNR levels (making 550 images total) were initialized by each method. The results for all tests are shown in Table 1. Table 1. Synthetic image tests for automatic initialization of snakes. Methods compared are: centers of divergence (CoD) [9], force field segmentation (FFS) [10] and the inverse approach (Poisson Inverse Gradient – PIG) # of Iterations Needed
Initialization + Deformation = Total CPU Time (sec)
RMSE (pixel)
CoD
FFS
PIG
CoD
FFS
PIG
CoD
FFS
PIG
Broken Edges
113
103
30.0
0.09+2.99=3.08
0.08+2.86=2.94
0.43+0.83=1.26
1.08
9.62
0.66
Curvature Tests
23.5
92.5
12.0
0.03+0.52=0.55
0.11+2.89=3.00
0.20+0.24=0.44
3.04
1.14
0.20
Noise
63.5
72.5
33.6
0.04+1.53=1.57
0.11+3.14=3.25
0.04+0.88=0.92
2.89
2.40
1.52
The inverse method (PIG) excels over the other two automated initializations in fidelity for objects with broken edges, where the FFS method has particular difficulties. In terms of objects of increasing curvature, it is the FFS method that encounters the greatest error, and once again, the inverse method presented here is superior. Finally, the PIG method improves upon both the RMSE and the computational expense of computing an initial segmentation in noise. 2.3. CBIR Overview Each image in the evidentiary database contains a ruler for scale information and a label to identify the case. After locating the ruler with a sequence of morphological operations, the next phase of processing involves locating the main object of interest via image segmentation. Image segmentation uses a simplified image created by area morphology as an initial input [15]. Specifically, an area open-close operation with a scale parameter equal to the anticipated minimum object size is utilized. Then, the VFC snake is applied to demarcate the area of interest. This segmentation provides a subimage for subsequent color and
texture analysis and also yields the constituent shape for shape and area analysis. Color histograms in the hue-saturation-value (HSV) color space are extracted for color description. Within the subimage defined by the segmentation, a count is taken of pixels possessing a given hue, saturation and value. An example image and its corresponding HSV histograms are given in Fig. 1 and Fig. 2, respectively. Here, each histogram is broken into 256 bins and normalized such that the sum of each histogram is one. We additionally calculate and store the mean and standard deviation of each histogram. To describe texture efficiently, the dominant components of a 2D Haar wavelet decomposition are saved for each color band (in HSV space). Similar Haar-based retrieval techniques are found in [16]. For each band, a set of Haar wavelet coefficients are extracted (each 128x128). Essentially 1-D Haar wavelet coefficients along each row are computed by computing the sum and the difference of each consecutive pair of pixels, weighted by 1/ 2 . This 1-D procedure is then followed on the image columns to complete the 2-D decomposition. An example 2-D decomposition of the V band is shown in Fig. 3. The 2-D coefficient matrix is one-dimensionalized by concatenating rows. The first element of this 1-D vector is the d.c. value, representing the average value for the particular band in (H, S, or V). The d.c. value is stored separately and removed from the 1-D vector. For each band, the corresponding vector is sorted in terms of decreasing wavelet coefficient magnitude. The C (~60) coefficients with largest magnitude are retained, recording their sign and initial position within the 1-D vector.
Fig. 1. Example image used for color histogram and wavelet analyses. Note that the background is excluded from the both analyses. 0.1
0.05
0
0
50
100
150
200
250
300
0
50
100
150
200
250
300
0
50
100
150
200
250
300
0.4 0.3 0.2 0.1 0 0.06 0.04 0.02 0
Fig. 2. Hue (top), saturation (middle) and value (bottom) histograms for the image shown in Fig. 1.
Fig. 3. (left) Example 128x128 subimage used for V band decomposition and (right)V band wavelet coefficient magnitudes.
Shape information is gleaned from the segmentation and stored in the form of a set of Fourier shape descriptors (FSDs) [17]. From the sampled snake positions (N=128 in this application), a set of positions (xi, yi) is created with a centroid of (0,0) for translation invariance. Given N shape positions (taken from the snake segmentation) in the form of (xi, yi), the mth FSD is obtained using N −1
FSD(m) = ∑ ( xi + jyi )e−2π jmi / N ,
(9)
i =0
where j is the imaginary number. The first FSD, FSD(0), is discarded to make the FSD representation invariant to scaling. Then, the magnitude is computed, so that the FSD representation is invariant to rotation. An example magnitude plot is shown in Fig. 4. 0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
20
40
60
80
100
120
140
Fig. 4. Image (top) and normalized plot of FSD magnitude (bottom).
3. CIRCUIT EDIT DISTANCE In the case of circuit images, we seek a more specific approach. We have developed two novel similarity measures, termed circuit edit distance (CED) and circuit histogram distance (CHD) for the comparison and classification of circuit boards, assuming that schematic and component connectivity information is unavailable. These methods use structural circuit information to quantify the cost of changing one circuit board into another via a set of edit operations, similar to graph edit distance [5]. 3.1. Circuit edit distance As previously mentioned, CED uses structural circuit information to quantify the cost of changing one circuit board into another. For two spatially aligned circuit boards, this modification is governed by a set of edit operations defined as component insertion, component deletion, and component translation. By summing the costs associated with each edit
operation, we are able to define the distance between two circuit boards. Let us first consider the benefit associated with pairing a circuit component in board A to a circuit component in board B. Let us number components in circuit A according to the parameter m ∈ [1...M ] . Each component has an associated type T (m) and associated position [ x( m), y (m)] . We typically define the position in inches to eliminate any scaling effects. We number components in circuit board B similarly according to the parameter n ∈ [1... N ] . We define the benefit of pairing components m and n as ⎧δ − Δ (m, n) if T (m) = T (n) & Δ(m, n) < δ . BEN A, B (m, n) = ⎨ else ⎩0
(10)
where Δ ( m, n) is the Euclidian distance between components m and n, and δ is a user defined maximum Euclidian distance. The benefit is non-zero only if both components are of the same type and are within some Euclidian distance δ of each other. Given the benefit for every (m, n) combination, we pair components in circuit A to components in circuit B via a greedy assignment algorithm. First, the circuit components corresponding to the maximum benefit are paired. For each subsequent non-zero benefit in descending order, we pair circuit components only if they have not yet been assigned a match. This gives the assignment matrix ⎧1 if m pairs with n . ASGN A, B ( m, n) = ⎨ else ⎩0
(11)
A non-zero entry in the assignment matrix corresponds to a component move edit operation, and has the associated cost C move (m, n) = α [T (m)] ⋅ Δ (m, n) ,
(12)
where α [T (m)] is a weighting function based on the component type of circuit element m. Every row within ASGN A, B containing all zeros corresponds to a component deletion edit operation, while every column within ASGN A, B containing all zeros corresponds to a component insertion edit operation. The corresponding edit operation costs are based on the component types as well, and are given by Cinsert (m) = β [T (m)], C delete (n) = χ [T (n)] ,
(13)
where β [T (m)] and χ [T (n)] are weighting functions based on the component type. By summing all required edit operations, we obtain the CED for a given relative orientation between the two images. As circuit boards images seldom have the same spatial origin and orientation, we determine the minimum CED among all possible origins and orientations. For illustrative purposes, consider the example case shown in Fig. 5. Here, we attempt to transform circuit #1 into circuit #2 via the aforementioned CED algorithm. By inspection, we require a single insertion (Z), no deletions, and 3 minimal moves (A=W, B=X, C=Y). The algorithm correctly measures this effect, calculating associated CED of 1.2. The parameters used in this example are δ = 5 , [α , β , χ ]resistor = [0.1, 1.0, 1.0] and [α , β , χ ] IC = [1.0, 2.0, 2.0] .
Circuit 1
Circuit 2 X
B A
W
C
Y Z
(a) W 0 10 10
X 11 1 6
Y 10 4 1
Z 11 6 4
A B C
(c)
W 5 0 0
X 0 4 0
Y 0 1 4
Z 0 0 1
A W 5
B X 4
C Y 4
B Y 1
A B C
Z C 1
(e)
(d) MOVE COST W X A 0.0 B 0.1 C -
Y 0.1
Z -
INSERT COST W X -
Y -
Z 1.0
DELETE COST
A B C
A B C
W 1 0 0
X 0 1 0
Y 0 0 1
Z 0 0 0
(f)
-
(g) Fig. 5. Example CED illustration. (a) Example circuits, (c) distance, (d) benefit, (e) ordered non-zero benefit, (f) assignment, and (g) edit operation costs.
3.2. Circuit Histogram Distance To augment CED performance, we additionally developed a pre-screening similarity tool termed circuit histogram distance (CHD). CHD quantifies the distance between two IED custom circuit boards based solely on the number of each electronic component type present in a given circuit (e.g. the number of resistors, capacitors, diodes, etc.). In more specific terms, the CHD compares the component type histograms of two unique circuit boards according to the following algorithm. Let T represent the set of component types. Let H Z (t ) represent the component type histogram of circuit board Z, i.e. each component type t ∈ T occurs H Z (t ) in circuit Z. We define the per component type distance between two arbitrary circuit boards A and B, D A, B (t ) ∈ [0,1] , as
⎧0 ⎪ D A, B (t ) = ⎨ min[H A (t ), H B (t )] ⎪1 − max[H A (t ), H B (t )] ⎩
if max[H A (t ), H B (t )] = 0 else
.
(13)
Note if the max(.) expression in (13) is zero, the component does not exist in either circuit board, and the corresponding distance is 0. A D A, B (t ) near zero indicates the number of component type t in each circuit is nearly equal, while a D A, B (t ) near one indicates the number of component type t in each circuit is significantly different. We define the CHD between circuit #1 and circuit #2 as a weighted sum of the per component distance measures,
CHD1,2 = ∑ α (t ) ⋅ D1,2 (t ) , t∈T
(14)
where α(t) is the per-component type weighting function. The sum of α (t) over the set C must equal one, thus ensuring CHD ∈ [0,1] . A CHD near zero indicates circuit boards one and two are similar, while a CHD near one indicates a lack of similarity. We typically define a hard CHD threshold which delineates probable matches from
improbable matches. Two circuits with a CHD below this threshold are considered probable matches, and are then analyzed with the more robust CED technique. Table 2 illustrates an example typical CHD analysis. Table 2. Example circuit histogram analysis. Circuit A Histogram
Circuit B Histogram
Distance D A,B (t)
W eighting α(t)
α(t)•D(t)
Resistors
10
12
0.17
0.25
0.04
Capacitors
10
10
0.00
0.25
0.00
Integrated Circuits
2
1
0.50
0.50
0.25
CHD
0.29
4. RESULTS 4.1. CBIR Results
The CBIR process constructs a matching functional that is a weighted linear combination of FSD distance as in [18], Haar wavelet distance as described in [16], color histogram intersection as performed in [18], and the difference in object area. CBIR matching weights were optimized using gradient descent on a 500 image ground truth database, separated into 25 specific groups of matching images and 7 major “super” groups of matching images. The specific and super group labels and precision results are given in Table 3 and Table 4 respectively. We observe low precision (percentage of matching image out of the top 20 retrieved) for the wire and wire cutter (Component #14) groups, due to the variability within these images. Fortunately, these classes represent the least significant objects in determining the match between IEDs. The circuits (at 81.4%) precision are the most significant. Table 3. Specific group labels for CBIR database and precision for the top 20 images retrieved.
#
Specific Group
1 2 3 4 5 6 7 8 9 10 11 12
Component #1 Component #2 Component #3 Component #4 Component #5 Component #6 Component #7 Wire Component #8 Component #9 Circuit #1 Circuit #2
Retrieval Precision (%) 36.0 24.8 38.8 100.0 36.1 88.0 25.9 16.5 55.6 100.0 60.0 49.6
#
Specific Group
13 14 15 16 17 18 19 20 21 22 23 24
Circuit #3 Circuit #4 Circuit #5 Circuit #6 Circuit #7 Circuit #8 Circuit #9 Component #10 Component #11 Component #12 Component #13 Component #14
Retrieval Precision (%) 38.9 73.5 56.0 75.0 100.0 100.0 68.8 49.5 79.7 65.6 97.0 32.8
Table 4. Super group labels for CBIR database and precision for the top 20 images retrieved.
Super Group Component Group #1 Component Group #2 Wire Custom Circuits Component Group #3 Component Group #4
Precision (%) 23.3 95.5 16.5 81.4 49.5 77.4
We then compared four forms of CBIR for circuit board images (see Fig. 6). First, we automatically detected the images containing circuits (using color and shape information) and then performed CBIR on just circuit images (termed circuit-
specific CBIR). These tests were run for CBIR that excludes the printed circuit board (uses just components) and CBIR using features within the bounding box of the board. We then tested the generalized CBIR with and without the VFCbased segmentation. For query size of two images and five images, results are given in Table 5. To give an idea of the computational cost, the portable Java implementation requires less than one minute for the entire 400,000 image database (on a Standard 2 GHz PC with 1.5GB RAM).
(a)
(b)
(c) (d) Fig. 6. Examples of the four CBIR methods tested for circuit images; (a) circuit-specific CBIR (components – board is masked out); (b) circuit-specific using interior of board and components (bounding box shown); (c) generalized CBIR w/o segmentation; (d) generalized CBIR w/ segmentation (VFC contour shown). Table 5. Precision of circuit image retrieval with query sizes of two and five. Query Size
2
5
Circuit-specific CBIR (components)
78.2%
59.6%
Circuit-specific CBIR (bounding box)
82.1%
63.1%
Generalized CBIR w/o segmentation
70.5%
49.8%
Generalized CBIR with segmentation
71.8%
54.1%
4.2. CED Results
We performed a quantitative analysis of CED for the classification of circuit board imagery. We considered a sample data set of 100 images with annotated component types and positions. 78 circuits from this sample set were manually divided into 13 classes, while the remaining 22 circuits were judged to have no corresponding class. Each of the 78 classified circuits was compared to the entire data set, and the best matches were determined according to the calculated distance measure. Table 6 lists the results for a query size of two images and five images. Every CED algorithm considered had classification performance of above 95%. The algorithm using 90° orientation quantization took only 16 seconds to to successfully retrieve two matching circuits 98.1% of the time. So, in terms of retrieving matching circuits, the advantage of CED (98.1%) over the best CBIR approach (82.1%) is clear. The disadvantage of the CED approach is the additional annotation (component labeling) that is required by the data entry operator.
Table 6. Precision of circuit image retrieval using CED for angular increments of 10, 30, 60 and 90° Query Size
2
5
10°
98.7%
98.4%
30°
98.7%
98.4%
60°
98.1%
96.1%
90°
98.1%
97.6%
To decrease execution time, we can utilize the circuit histogram distance (CHD) as a prescreening operation to eliminate improbable matches. We compared classification performance and execution time for CED analysis with and without CHD prescreening at 10° and 90° rotation increments. We consider a maximum allowable component translation of 0.5 inches, assume circuit pair origins at principle components only, and utilize equal component weighting schemes for both CED and CHD. Circuit pairs with a CHD of less than 0.25 were considered probable matches. Table 7 summarizes the results. For a small tradeoff in classification performance, we can significantly cut execution time. Table 7. CED with CHD Analysis
Classification Method CED only CED w/ CHD
Rotation Increment 10°
Query Size=2
Query Size=5
Execution Time
98.7%
98.4%
30 min
90°
98.1%
97.6%
3.7 min
10°
96.8%
95.5%
7.1 min
90°
96.8%
95.5%
1.5 min
Finally, we consider the effects of analyzing the CED algorithm with CHD prescreening for primary components only. We call these primary components (e.g., integrated circuits) the principle components. Not only does this aid in decreasing execution time, it also aides in decreasing acquisition time, i.e. We again compared classification performance and execution time for CED analysis with CHD prescreening at 10° and 90° rotation increments, considering all components versus principle components only. We consider a maximum allowable component translation of 0.5 inches, and a CHD threshold of 0.25, and equal component weighting schemes for all methods. Table 8 summarizes the results. We obtain the best 2nd order classification performance and the fastest execution time across all tests using the principle components only, CED with CHD prescreening, and 90° orientation increments. Table 8. CED with CHD Analysis, Principle Components Only
Classification Method All Components Principle Components
Rotation Increment 10°
Query Size=2
Query Size=2
Execution Time
96.8%
95.5%
7.1 min
90°
96.8%
95.5%
1.5 min
10°
99.9%
96.9%
53 sec
90°
99.9%
97.2%
16 sec
5. CONCLUSION This paper provides a specialized approach to CBIR for IEDs that features VFC-based segmentation. The VFC snakes can be initialized by the Poisson inverse gradient method. This initialization approaches the coarse segmentation as an inverse problem, where the snake external force provides the input data. The paper also develops a novel method to compare two circuits using only the (top side) photograph of the circuit and no schematic information. CED, loosely based on graph edit distance, allows a numeric score to be assigned for the potential match of two circuits which can then be used for classification.
REFERENCES [1]
[2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]
[14] [15] [16] [17]
[18]
M. S. Day, P. Colella, M. J. Lijewski, C. A. Rendleman, and D. L. Marcus, "Embedded Boundary Algorithms for Solving the Poisson Equation on Complex Domains," Lawrence Berkeley National Laboratory, LBNL-41811, 1998. T. Pavlidis, Structural Pattern Recognition. New York: Springer-Verlag, 1977. D. Shasha, J. T. L. Wang, and R. Giugno, "Algorithmics and Applications of Tree and Graph Searching," in Proceedings of the ACM Symposium on Principles of Database Systems (PODS). Madison, Wisconsin, 2002. H. Bunke, "Graph Matching: Theoretical foundations, algorithms, and applications.," in Proceedings of the Intl. Conference on Vision Interface. Montreal, Canada, 2000, pp. 82-88. A. Sanfeliu and K. S. Fu, "A Distance Measure between Attributed Relational Graphs for Pattern Recognition," IEEE Transactions on Systems, Man, and Cybernetics, vol. 13, pp. 353-363, 1983. C. Xu and J. L. Prince, "Snakes, shapes, and gradient vector flow," IEEE Transactions on Image Processing, vol. 7, pp. 359, 1998. B. Li and S. T. Acton, "Vector Field Convolution for Image Segmentation using Snakes," in IEEE International Conference on Image Processing, 2006, pp. 1637. M. Kass, A. Witkin, and D. Terzopoulos, "Snakes: Active Contour Models," International Journal of Computer Vision, vol. 1, pp. 321-331, 1988. X. Ge, X. Ge, and J. Tian, "An automatic active contour model for multiple objects," in Proceedings of the Intl. Conference on Pattern Recognition, 2002. C. Li, J. Liu, and M. D. Fox, "Segmentation of external force field for automatic initialization and splitting of snakes," Pattern Recognition, vol. 38, pp. 1947, 2005. Y. Saad, Iterative Methods for Sparse Linear Systems, 2nd ed: Society for Industrial and Applied Mathematics, 2003. K. A. Atkinson, An introduction to numerical analysis, 2nd ed: John Wiley and Sons, 1988. L. Giraud, R. Guivarch, and J. Stein, "Parallel Distributed FFT-Based Solvers for 3-D Poisson Problems in MesoScale Atmospheric Simulations," International Journal of High Performance Computing Applications, vol. 15, pp. 36-46, 2001. T. Matsumoto and T. Hanawa, "A Fast Algorithm for Solving the Poisson Equation on a Nested Grid," The Astrophysical Journal, vol. 583, pp. 296-307, 2003. S. T. Acton, "Fast Algorithms for Area Morphology," Digital Signal Processing, vol. 11, pp. 187, 2001. C. E. Jacobs, A. H. Finkelstein, and D. H. Salesin, "Fast multiresolution image querying," in Proceedings of the 22nd annual conference on Computer graphics and interactive techniques: ACM Press, 1995. H. Kauppinen, T. Seppanen, and M. Pietikainen, "An experimental comparison of autoregressive and Fourierbased descriptors in 2D shape classification," IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 17, pp. 201, 1995. J. Tang, S. R. Avula, and S. T. Acton, "DIRECT: a decentralized image retrieval system for the national STEM Digital Library," Information Technology and Libraries, vol. 23, pp. 9(7), 2004.
