Image Matching Using Colour Edge Cooccurrence ...

2006 IEEE International Conference on Systems, Man, and Cybernetics October 8-11, 2006, Taipei, Taiwan

Image Matching Using Colour Edge Cooccurrence Histograms Wenjing Jia, Huaifeng Zhang, Xiangjian He, and Qiang Wu, Members, IEEE Abstract- In this paper, a novel colour edge cooccurrence histogram (CECH) method is proposed to match images by measuring similarities between their CECH histograms. Unlike the previous colour edge cooccurrence histogram proposed by Crandall et al. in [21, we only investigate those pixels which are located at the two sides of edge points in their gradient direction lines and at a distance away from the edge points. When measuring similarities between two CECH histograms, a newly proposed Gaussian weighted histogram intersection (GWHI) method is extended for this purpose. Both identical colour pairs and similar colour pairs are taken into account in our algorithm, and the weights are decided by the larger distance between two colour pairs involved in matching. The proposed algorithm is tested for matching vehicle number plate images captured under various illumination conditions. Experimental results demonstrate that the proposed algorithm can be used to compare images in real-time, and is robust to illumination variations and insensitive to the model images selected.

I. INTRODUCTION Colour histograms have been widely used as a stable representation over changes in view for image retrieval and object detection. Traditional histogram-based approaches, such as the one proposed by Swain and Ballard in [12], did not take into account the geometric relationships between pixels. Instead, these approaches counted only the number of pixels with certain colours, and therefore were easy to compute and insensitive to small changes in viewing positions. However, they are likely to produce false positives because they do not include any spatial information and two apparently different images may have same colour histograms. Funt and Finlayson [3] used histograms of the colour ratios of neighbouring pixels, and hence introduced a slight amount of spatial information in their histogram representation. Recently, more efforts have been made to incorporate spatial information into colour information. Pass and Zabih [1 1] defined the colour coherence vectors (CCV) to compare the number of coherent pixels and the number of noncoherent pixels of same colours in a model image and a target image. A more attractive approach is to use the colour Manuscript received March 15, 2006. This work was sponsored by the Australian Research Council under Large ARC-Discovery Grant DP045 1666. Wenjing Jia is with the Faculty of Information Technology, University of Technology, Sydney, PO Box 123, Broadway, NSW, 2007, Australia

([email protected])

Huaifeng Zhang is with the Faculty of Information Technology, University of Technology, Sydney, PO Box 123, Broadway, NSW, 2007, Australia

([email protected]) Xiangjian He is with the Faculty of Information Technology, University

of Technology, Sydney, PO Box 123, Broadway, NSW, 2007, Australia ([email protected])

Qiang Wu is with the Faculty of Information Technology, University of Technology, Sydney, PO Box 123, Broadway, NSW, 2007, Australia

([email protected])

1-4244-0100-3/06/$20.00 C2006 IEEE

cooccurrence histograms (CCHs), also called the colour correlograms in [7]. Chang and Krumm [1] originated an object detection algorithm using the CCHs. The CCH records the number of pairs of coloured pixels that are separated at certain distances in an image space. In [2], Crandall and Luo proposed a more flexible spatial-colour joint probability function, called the colour edge cooccurrence histogram (CECH), for object detection. The CECH is computed for points on the boundaries of two neighbouring regions. Above histograms constrain the spatial relationships of the pairs of coloured pixels in an object, but allow for some degrees of distortion. When applied for image retrieval [7], partially occluded objects recognition [1] and non-rigid colour object detection [2], desirable experimental results have been obtained. In this paper, a novel colour edge cooccurrence histogram algorithm is proposed for image matching purpose. The major contributions of the paper include the following three points. (1) Unlike the CECH algorithm shown in [2], our CECH algorithm only cares about the spatial-colour distribution near edge points. This not only improves the computation efficiency but also prevents solid colour regions from dominating the histogram [2]. (2) Our CECH is only computed in regard to the pairs of pixels which are located on two sides of edge points along their gradient direction lines. This makes the resultant CECH histogram invariant to rotation and deformation of objects in images. (3) Instead of using the conventional histogram intersection to measure the similarities between CECH histograms, we propose an extended Gaussian weighted histogram intersection (GWHI) method to take into account the matchings between both identical colour pairs and similar colour pairs. We limit our goal to develop a general algorithm for matching various compound colour objects, such as car number plates, which have a specific set of multiple colours arranged in specific spatial layouts. The appearance of such colour object images may vary dramatically from object to object and from scene to scene. Since the characters included in images and the orientations of characters are unknown, the features used for object matching must be invariant to characters, rotations, and potential deformations. It must also be robust to colour variations due to illuminant changes and inherent colour differences from one instance of the object to another. Stable matching results for plate images that contain different characters should be able to be given using a fixed model character image. The overview of the proposed algorithm is as follows. First, a general-purpose gradient detection is performed on each input image to obtain its gradient map. Second, the

2413

Authorized licensed use limited to: University of Technology Sydney. Downloaded on February 18, 2009 at 22:39 from IEEE Xplore. Restrictions apply.

colour edge cooccurrence histogram is constructed in gradient directions at edge pixels that have strong gradient value. Next, the colour set of each image is converted to CIE Luv colour space to facilitate the succeeding CECH comparison. Last, the matching rate is obtained by comparing two CECH histograms using the extended GWHI method. Accordingly, the remaining parts of the paper are organised as follows. The proposed algorithm is illustrated in Sections II and III. Section II gives the details about CECH generation and the related colour quantisation issue. Section III explains how to measure the similarities between CECH histograms. In Section IV, the proposed algorithm is tested and the experimental results are analysed. Conclusions and future work are given in Section V. II. COLOUR EDGE COOCCURRENCE HISTOGRAM (CECH) A. From Colour Histogram to CECH Histogram In the approaches of histogram-based image matching, each image is firstly represented as a colour histogram, and then similarities between images are measured by comparing the resultant histograms. In the ideas of using the colour cooccurrence histogram (CCH) for image retrieval [7] and object recognition [1], instead of collecting statistic information of colour only, each histogram also includes some spatial information between pairs of coloured pixels. Each element of a CCH histogram, C denoted by CCH(Q7, 2,D), holds the frequency of occurrences of the pairs of pixels which are located in a predefined spatial structure D and meet the condition that the first pixel takes the colour value of cl, and the second pixel takes the colour value of 63. In [1], the direction of the structure element D was ignored, and only the distance magnitude d, measured by Euclidean metrics, was tracked in order to make the histograms invariant to rotation in the image plane. In this paper, direction information of involved pixel pairs is also employed. Instead of using an absolute direction information as in [4], a novel structure element is designed which direction varies with the change of edge direction in order to preserve the rotation-invariant characteristic. The structure element is defined as two positions which are located on the two sides of an edge point in its gradient direction line and at a distance away from the edge point. The resultant colour edge cooccurrence histogram, which examines colour distribution along the edges of an image, is also called the colour edge cooccurrence histogram (CECH), as in [2]. However, unlike the CECH histogram in [2], where the connected component analysis is first performed in order to obtain regions and the histogram is then computed for two coloured pixels which are located in the boundaries of neighbouring regions, our CECH directly examines colour distribution at edge points in their gradient direction lines. The general definition of our CECH is essentially the same as the grey-level cooccurrence matrix, introduced by Haralick et al. in [6]. In this paper, we follow the naming convention used in [1] and call it as a "histogram" rather than a "matrix", since the term of "histogram" better conveys the

where c1 Then,

=

(0'0)RGB and c2 00

'P

b0C>

(0,0,0)

Bin value adds one. CECH with d=4 Fig. 1. Computing the CECH histogram at the edge point p.

concept that it is used to record the frequency of some things. Though the term "CECH" itself contains a "histogram", in order to emphasize this concept, we hereafter use "the CECH histogram" as the short for "the colour edge cooccurrence histogram". B. Computing CECH Histogram Computing the CECH histogram of an image F contains the following steps. An example illustrating the process of computing the CECH histogram from a colour image is given in Figure 1. Step 1: Detecting edges The first step is to obtain the edge map E of input image F = f(x, y). Since it is still hard to obtain a fine edge map, instead we use the pseudo-edge points which have relatively strong gradient intensities. The gradient of an image f(x, y) at location (x, y) is defined as the vector Of Vf

]-

Of Of

=

Gy

(1)

The magnitude and direction of this vector, denoted by and O(x,y) respectively, are

Vf(x,y)

Vf (x, y)

=

mag(Vf)

=

and

O(x, y)

=

tan-l (

G+G

GY) Gx

(2)

(3)

Note that the edge direction at location (x, y) is perpendicular to the direction of the gradient vector at the point.

2414


Computing the gradient of an image is based on obtaining the partial derivatives Of /0x and Of /&y at every pixel location. In this paper, the classic Sobel operator is used to obtain the first-order partial derivative at each point. Pixels that have a gradient intensity stronger than a predefined threshold, denoted by THG, are viewed as "edge" pixels and will be investigated. Thus, the edge map can be written as E(x, y)= 1 if Vf (x,y) > THG 0 otherwise In experiments, Vf (x, y) is first normalised into the range of [0, 255] using the minimum and maximum gradient strength in the image plane. The threshold THG is empirically set as 30 based on statistical analysis. Step 2: Locating pairs of pixels For each non-zero edge point p at location (x, y), two points, denoted by P1 and P2 respectively, are taken. The two points are located on the two sides of the edge point p on its gradient direction line with a fixed distance, denoted by d, away from the reference edge pixel (see Figure 1). Thus, the coordinates of the pixel pair P1 and P2, denoted by (xi, yi) and (X2, Y2) respectively, can be computed as

and

{

Xj Yi

=

d x cosO y+dx sinO

x

-

x+d x cos0 Y2 y-dxsin0 Step 3: Summing up Denote the colour values at points P1 and P2 as c1 and C2 respectively. The corresponding bin of CECH histogram is counted as CECH(-l c, d ) = Pi IICP2 size (Pl P2) 1Pl P2 C F, and Cl {X

11pi -Pll

JP2

l

(7)

(7)

where C is the colour set of the image, and the function size counts the number of elements in a set. All edge points in the whole image plane are scanned. The frequency of occurrences of the same colour pairs in the image with above spatial constraint is recorded and summed in the corresponding bin of the CECH histogram. Step 4: Normalising the CECH Finally, the CECH histogram is normalised using the total number of edge points in the image. C. Reducing the Number of Colours Reducing the number of colours in images is essential in comparing CECH histograms in real-time. Assume the total number of possible colours in the model image is Nc. So, there are Nc possible colour values for each of the first pixel and second pixel. The total number of possibilities of the pixel pairs with different colour pairs is thus Nc x Nc. This is the dimension of the resulted CECH histogram when the structure element D is fixed. Without loss of generality, assume there are the same number of

colours in the target image. If the colour sets of two images are identical, a single bin-to-bin comparison is enough. Thus, the minimum number of comparison operations between two CECH histograms will be (Nc)2. However, if the colour sets of two images are different, for each bin in the model CECH histogram, each bin of the target CECH histogram needs to be compared and matched. Thus, there will be up to (Nc)4 comparison operations in order to work out the similarity between two CECH histograms. It is known that the total number of colours for a 24224. Without reducing bit true-colour image is NC the number of colours, it is unlikely that the histogram comparison operation could be executed in real-time with such a tremendous dimension. Note that the interested objects in this research take very limited number of colours. It is thus likely to quantise each image into a fixed small number of colours while introducing only slight perceptual distortion. The operation for reducing colour number is also called colour quantisation in references. In [1], colour quantisation was simply performed in RGB colour space using a k-means nearest neighbour algorithm. However, the RGB colour space is known to be sensitive to colour variation. Furthermore, the appearance of colours of an object may differ significantly from image to image because of illumination variations, perceptual colour surround effects [2], and so forth. As its consequence, the quantisation must be designed carefully to ensure that perceptually similar colours are mapped to the same quantised colour values. In [2], a perceptual colour naming method was employed in order to handle colour variation. The standard ISCC-NBS Colour Names Dictionary [10] was employed to quantise and name the colour space. Each pixel is assigned to the standard name of the closest ISCC-NBS centroid colour, according to the Euclidean distance in the CIE Lab colour space [5]. This method uses a single colour set for all images and the histogram comparison can thus be performed on a bin-tobin base. However, since the colours of compound colour objects, such as vehicle number plates, only take a very small part of the standard colour set, such quantising method usually causes large perceptible difference in quantised images (see the right column of Figure 2). In our method, Wu's [14] optimal colour quantisation algorithm, which is based on variance minimisation, is employed to reduce the number of colours by introducing only slight colour distortion into the quantised images. Figure 2 gives some examples of colour quantisation results obtained using two methods. Since the target objects, number plates, in this research take very limited number of colours, each image can be quantised into a fixed small number of colours while introducing only slight distortion based on the variance minimisation rule. Eight colours and sixteen colours have been tested to quantise number plate images. Since using sixteen colours did not make much improvement for image matching results, each image is quantised into eight colours using Wu's variance minimisation algorithm. Note that exactly which

2415


^i-;rf~ ~ ~ ~ ~ ~ ~ ~ .^ r>S fi

U

HI

Lw

MPHM

I.

__ _____ p_ ........

k I **

-

7FT X G,MWIl

_ w-1

-

0%

_~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ X : -

N: U~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .k .

:

Fig. 2. Original number plate images (left column) and their quantised counterparts using the variance minimisation algorithm with eight colours (middle column) and using colour merging method (right column).

colours are referred in each quantised image depend on its original colour distribution. Thus, after colour quantisation, there is a short colour mapping table (also called palette) usually in the head of each image file, which maintains information about colours that are used in the image. The data in the quantised image file only need to record the index of colour in the colour table, rather than its detailed colour values. By this way, in our experiments, the total number of comparisons between two CECH histograms is reduced to 84 = 4096, from originally possible 224 224 281, 474,976, 710,656 comparisons. This greatly improves the computation efficiency of image matching using the CECH histogram while satisfactory results can still be achieved, which are to be seen in experimental results.

III. COMPARING CECH HISTOGRAMS After the CECH histogram is obtained, the next problem is how to compare CECH histograms. Normalised L1-norm distance and the histogram intersection have been proposed in the past for comparing histograms. However, both of the metrics demand an exact bin-to-bin identical colour matching between histograms and thus are very sensitive to the colour distortion caused by various illumination conditions under which the images were captured. In practice, the colours of real world images can be distorted both in the scene itself and in the image capturing process. Hence, images with same visual information but with different colour intensities may degrade the similarity level significantly when the conventional HI is used. In order to overcome this problem, Wong et al. [13] proposed a merged-palette histogram matching (MPHM) method. The essence of the method is to extend the intersection from bins of identical colours to bins of similar colours. In their algorithm, as long as the distance between two colours is less than a fixed threshold, say Th, the intersection between the bins of these two colours will be calculated. This algorithm has produced more robust image retrieval results for images captured under various illumination conditions. However, it assumes an identical weight of the contribution between colours which have different similarities with the given colour. When applied to number plate image matching, the algorithm exhibited to be very sensitive to colour variations. In [9], a Gaussian weighted histogram intersection (GWHI) method has been proposed which overcomes the

: ***

L.

-3.3u

-Th

0

4% ftft

Th

d

-ft

3.3a

Fig. 3. Weight functions in three histogram intersection methods.

above problems and has demonstrated to be much less sensitive to colour variations compared with the conventional HI method and the MPHM method [9]. In this method, instead of applying identical weights to different colours, a Gaussian weight function of colour distance has been employed to differentiate the matching between colours which have different distances to the given colour. The GWHI method has shown to be less sensitive to colour variations. However, this method only addresses intersecting of histogram bins between two colours. For the purpose of comparing two CECH histograms, where there are two pairs of colours involved in the intersection, the GWHI method is further extended to facilitate intersection of two pairs of colours. A. Gaussian Weighted Histogram Intersection (GWHI) Assume all colours of pixels in the model image are taken from a colour set, denoted by Cm, and the colours of pixels in the target image are taken from another colour set, denoted by CT (the subscripts "M" and "T" denote for "model" and "target" respectively). Note that, the colour sets of two images can be either identical or different. In this paper, the colour sets associated with each image are the colour tables of images after colour quantisation, which are generally different from each other. Jia et al. [9] proposed applying a Gaussian weight function of colour distance to the conventional HI method in order to describe the relationship between the colour distance and the weight as [9]:

GWHI

E3 CCT E3 miTn(hm (C), hT('))*CXp(

C, ECM

ej

2j2 (8)

with -BW