image retrieval based on color matching - CiteSeerX

IMAGE RETRIEVAL BASED ON COLOR MATCHING Petteri Kerminen Pori School of Technology and Economics, TUT Information Technology P.O. Box 30 28601 Pori ABSTRACT In this paper we present properties of an image retrieval system based on color matching. The implemented procedure is presented as a simplified pseudocode presentation of the system. Once histogram has been calculated and saved on the disk, it can be used for analysis. The simulation results are also presented. I.

INTRODUCTION

Large image databases are difficult to browse with text search, if the user wants to search images which have a similar color content. Retrieval based on color matching solves this problem. Color based matching evaluation can be made using the following steps: 1.

2. 3. 4.

5. 6.

7.

8. 9.

If a query image has already been analyzed and data stored into a database, jump to 7. Read the query image from the disk. Make a color space conversion if needed. Analyze the query image pixel by pixel and store the values in a quantized histogram. Change bin hit values to relative values (percentage). Save the histogram data with additional information (average color, dominant colors) and image name in the database. Calculate color differences and histogram difference on a higher level (8 - 64 bins) between query image and database images. Select the best candidates to closer evaluation (64 - 512 bins). Select matching images.

These steps are independent from image format and selected color space. The only possibility to build a histogram is exhaustive pixel by pixel analyze. Once it has been done and

Moncef Gabbouj Tampere University of Technology Signal Processing Laboratory P.O. Box 553 33101 Tampere saved on the disk, it can be used for analysis. Comparison between images is not possible, if the histograms don’t have the same number of bins and are not converted to the same color space. II. COLOR SPACES The following paragraphs contains discussion about three color spaces: RGB, L*a*b* and HSV/I. The properties of these color spaces and their advantages and drawbacks for our purposes are presented. A. RGB Color Space The RGB color space (Red, Green and Blue) is well supported on image processing programs. Some image formats (for example RAW and TIFF) use the RGB color system, so it is not difficult to make a program that reads an image from a disk to the computer’s RAM. There are two main disadvantages with the RGB color space. 1. 2.

The RGB color space is not perceptually uniform [1]. All components (R, G, B) have equal importance and so those values have to be quantized with the same precision.

The perceptual similarity problem can be avoided by using a weight matrix (A) during the distance measure stage between two histograms [2]. dhist(x,y)= (x-y)’× A × (x-y),

(1)

In (1) A is a weight matrix, x and y are histograms. If a different number of bins is used on different stages of evaluation, all stages need an A matrix of their own. For example a 64 bin histogram requires a weight matrix which size is 64×64.

Problem 2 means that possible bin amounts are cubes of an integer (i.e. 13=1, 23= 8, 33=27, 43=64, 53=125, 63=216, 73=343…). If these steps are too big there is one way to avoid the problem; a predefined color map that is common to all images is taken to use. Before a histogram can be built an image has to be converted into this quantized color space. Actually this quantized color space corresponds to the histogram’s bins and values can be added to histogram during the conversion. There are some algorithms [3] on how color space quantization can be optimized. B. HSV/I Color Space HSV (Hue, Saturation and Value) or HSI (Hue, Saturation and Intensity) are preferred in [4], because RGB values are correlated and depend on changes in light intensity. HSV values can be obtained from RGB values as shown in (2) [5]. If {r,g,b} is defined so that r,g,b {0 -1}, then h ^0 - 6}, s ^0 - 1} and v ^0 – 1}. v = MAX(r,g,b) mmm = v-MIN(r,g,b) s = mmm / v r1 = (v – r) / mmm g1 = (v – g) / mmm b1 = (v – b) / mmm if r = max and g = min h = 5 + b1 else if r = max and g != min h = 1 – g1 else if g = max and b = min h = 1 + r1 else if g = max and b != min h = 3 – b1 else if b = max and r= min h = 3 + g1 else h = 5 – r1

zed HSV color space is presented. The huescale is divided into 18 sectors (20° for each one), so the primaries (red, green and blue) and yellow, magenta and cyan are divided into 3 sub-divisions. C. L*a*b* Color Space The L*a*b* (Brightness, red-green and yellowblue content) system gives quantitative expression to the Munsell system of color classification [6]. The transform from RGB can be made as follows:

 X  0.490 0.310 0.200  R   Y  = 0.177 0.813 0.011 G  (3)       Z  0.000 0.010 0.990  B   100Y L* = 25  Y0

1/ 3

  

− 16, 1 ≤ 100Y ≤ 100 (4)

 X  1 / 3  Y  1 / 3   −    a = 500  X 0   Y0    *

(5)

 Y 1 / 3  Z 1 / 3    b* = 200   −  (6) Z 0    Y0     In (4) – (6) X0, Y0 and Z0 mean tristimulus values for reference white. L*a*b* color space is best according to perceptual similarity. It is not dependent on any particular device. Colors can be set as them are perceived when operating a repro system. [7] In the analysis L*a*b* is divided into 7 L* levels, 5 a* levels and 5 b* levels. The problems with L*a*b* color space is quantization. From Fig. 2 can be seen, that on each edge the quantization should be coarser, because the (2)

The HSV color space can be quantized easier than the RGB color space. In Fig. 1 a quanti-

Figure 1: Quantized HSV color space 18 hues, 3 values, 3 saturations and 4 grays Å 166 bins.

Figure 2: CIELAB color space

volume should be the same for each subspace. In our tests the volume is smaller for values near the edges. III. ANALYSIS In our system the RGB components are analyzed pixel by pixel and their real combination is the most important thing (i.e. if a point has the values {150,10,150} it is light violet). Adaptive histogram methods allow to compare histograms with different numbers of bins, but this increases computational complexity and it is not a desired property when similarity analysis will be made on-line over a large image database. Grayscale and color images can not be compared together, because it is not possible to build a color histogram from a grayscale image. If different numbers of bins will be used on different stages of the retrieval process, all these histograms should be made, when the image is inserted to the database. Another technique is to calculate a coarse histogram from a more accurate histogram during the analyzing process. This will take time and all improvements to the retrieval process should be used. A system’s computational complexity doesn’t increase very much even if a color space conversion is needed, because this will be made only when histograms are built in an off-line process. However, this conversion must be fast enough, because a query image which is not included in the image database has to be analyzed on-line. A. Computational Complexity For M×N size images (with nb bins in a histogram) computational complexities for the histogram formation part are: The read time is M×N×3 for each histogram type. In addition to the operations which are shown in Tables 1 and 2, normalization has to be done. It requires M×N×3×nb operations. Table 1: Operations required in the average case (M×N). Conv=Color conversion, Tf=To float, Com=Comparison, Ad=Addition, As=

Assign. Color system RGB64 RGB125 HSV166 Lab175

Conv

Tf

Com

Ad

As

34,2 30

3 3

4,5 6 11,5 7

1 1 1 1

1 1 1 1

Table 2: Operations required in the worst case (M×N). Conv=Color conversion, Tf=To float, Com=Comparison, Ad=Addition, As=Assign. Color Conv Tf Com Ad As system RGB64 9 1 1 RGB125 12 1 1 HSV166 46 3 22 1 1 Lab175 30 3 14 1 1 For M×N size images computational complexities for the histogram analysis part are proportional to the number of bins in the histogram. Thus a histogram read takes nb time units and comparison (substracion + square root + absolute value + addition + assign) altogether 5×time units. Thus time consumptions is for 64 bins RGB 64×6 time units, for 125 bins RGB 125×6 time units, for 166 bins HSV 166×6 time units and for 175 bins L*a*b* 175×6 time units. B. Simulation Results MATLAB simulations have been done for all color spaces on 150 MHz Pentium PC with 64 MB RAM and Windows NT 4.0 Workstation operation system. Histograms have been built for 2041 images (average size 240×320). Table 3: Histogram build times Color space Total time RGB64 17h 38min RGB125 21h 19min HSV166 7d 4h 36min Lab175 2d 7h 34min

Time / image 31.1 s/image 37.6 s/image 304 s/image 98 s/image

These times were proportional to the computational complexity analysis. HSV takes much more CPU-time than other methods. The histogram build process will get much more speed when it is implemented with C-langua-

ge. The speed of this part has to be fast enough, because a search image is maybe analyzed on-line (if it is not on the image database). Histogram comparisons have been made between these histograms and analysis times are shown in Table 4. Table 4: Histogram comparison times Color space Total time Time / image RGB64 1min 5s 32 ms/image RGB125 1min 31s 45 ms/image HSV166 1min 37s 48 ms/image Lab175 1min 39s 49 ms/image From these results and histogram build time results can be seen, that histogram analyses are 972 - 6333 times faster than the histogram build part. The HSV 166 algorithm has been implemented with Borland C++ and tested with 2000 pseudo images on the same test computer. Time consumption decreased significantly. Histogram build times for 1000 pixel and 10000 pixel images are 45 seconds and 2 minutes 29 seconds respectively. The analyzing time is 7 seconds for both image sizes. System analyzing software has been used during this last step. It showed that with the 150MHz Pentium processor the CPU speed is not a bottleneck but the disk read speed and disk read delay are. The highest CPU load was only 70% and on average it was on the 45% level. C. Visual Simulation Results The most important factor when analyzing a color based image retrieval system, is the visual color similarity of retrieved images. There are visual results of simulation over the 2041 image database. In Fig. 3 a-d an artificial image is used as a search image. In Fig. 4 a-d a natural image is used as a search image.

Figure 3: Simulation results from the first test case. First RGB 64, second RGB 125, third HSV 166 and fourth L*a*b*. In each case the upper left hand corner has a query image and the best matches follow that image first from left to right in the upper row and then in the lower row.

With large image databases (~100 000 images) the histogram comparisons alone take 350 seconds with the speed, which has been achieved in these tests. A normal usage situation requires much faster response, because the users are not willing to wait so long. Speedup indices and the database structure should be designed so that similar images can be found from the database without exhaustive search methods. Database management systems require that the number of bins has to been reduced. This step is under development. V. ACKNOWLEDGEMENTS This work has been funded by the European Union – ERDF, Technology Development Centre Tekes, Alma Media, Helsinki Telephone Company, Nokia Research Center and Ulla Tuominen's Foundation. REFERENCES [1] Vellaikal, A., Jay Kuo, C.-C. and Dao, S., Content-Based Retrieval of Color and Multispectral Images Using Joint Spatial-Spectral Indexing, SPIE Vol. 2606, pp. 232-243. [2] Sawhney, H. S. and Hafner, J. L., Efficient Color Histogram Indexing, IEEE 1994, pp. 66 – 70 [3] Christy, J., ImageMagick Color Reduction Algorithm, http://www.wizards. dupont.com/christy/www/quantize.html, 1998

Figure 4: Simulation results from the second test case. Order of images as in figure 3. IV. CONCLUSIONS The visual similarity "goodness" of retrieval processes differs for each color system. Depending on query images either color content RGB 64, RGB 125 or HSV 166 produced the best result. When natural images were used as a query image usually the best result was got either from RGB 125 or HSV 166. In case of a constructed query image RGB 64 produced the best results (not in the case of Fig. 3).

[4] Ng, V., Cheung, D. and Fu, A., Adaptive Color Histogram Indexing, SPIE Vol. 2606, pp. 202- 211 [5] Smith, J. R. and Chang, S-F., Automatic Image Retrieval Using Color and Texture, http://www.ctr.columbia.edu/ ~jrsmith/html/pubs/PAMI/pami_final_1.htm, 1995-6 [6] Jain, A. K., Fundamentals of Digital Image Processing, Prentice-Hall, Inc. 1989, pp. 67 - 71. [7] LinotypeCPS Company, North America, CIE Lab color space, http://www.linocolor .com/colorman/sp_ciela_2.htm