1
Real-time Gradient Orientation Pattern Matching Wattanit Hotrakool, Prarinya Siritanawan, and Toshiaki Kondo (
[email protected]),(
[email protected]),(
[email protected]) School of Information, Computer and Communication Technology (ICT) Sirindhorn International Institute of Tehnology, Thammasat University 131 Moo 5, Tiwanont Road, Bangkadi, Muang, Pathum Thani, Thailand, 12000 Tel: +66(0)29869009, +66(0)25643221-9, Fax: +66(0)29869112-3
Abstract—This paper describes a comparison between newly introduced a gradient orientation pattern template matching method (GOPM) and the traditional template matching method in real-time implementation. These techniques are aimed at real-time eye tracking for assisting the ophthalmologist to perform eye treatment and surgery. GOPM is a technique of template matching that use normalized gradient information of image as inputs for matching instead of intensity information as used in traditional block template matching. A strength of GOPM over traditional block matching is that GOPM provides a robustness to changing lighting condition. GOPM can perform template matching very well during the lighting condition changes, whereas the traditional method fail to achieve the best matching position. Moreover, both GOPM and traditional method required small computation time so both methods can be implemented to real-time applications.
I. Introduction Template matching is one of computer vision techniques used for extract the information from the image or video. The key concept of template matching is to use a desired shape, called "template", to scan over the entire image and find the best corresponding point. The template matching technique has many applications, such as traffic camera to track road [1], face recognition, and character recognition. The application which this paper originally developed for is to use in ophthalmologist camera as an eye tracker. This paper presents a comparison between two templates matching methods in real-time implementation, which are a newly proposed gradient orientation pattern matching (GOPM) and the traditional block template matching technique. We employ the sum-of-square differences(SSD) criteria for both methods. The main focus of this paper is 1)to show that GOPM can provide more robustness to both globally and partially changing lighting conditions over the traditional method and 2)to show that GOPM can be implemented in real-time application. Because this paper based on a project to develop an eyetracking system, the test application used for comparison in this paper will be an eye-tracking in real-time under various lighting conditions. II. Block Template Matching Technique There are a numbers of block matching technique used for template matching. The techniques used in this paper
are 3.1 sum of squared difference (SSD) technique, and 3.2 gradient orientation pattern matching [2],[3] technique. The SSD matching technique provide good performance under normal lighting conditions with ease in implementation. Whereas the GOPM is the newer technique in block template matching and can provide the great robustness to both globally and partially changing lighting condition. A. Sum of Square Difference Technique The sum of square difference technique is the another popular intensity-based block matching technique. In this technique, a small reference image (template) is searched for in another image. The performance of the matching technique depends on the features employed in template, and block size. This method is intensity-based block matching according to (1). N/2
~ = SSD(~ p, d)
X N/2 X
(I1 (x + i, y + j)
−N/2 −N/2
−I2 (x + i + u, y + j + u))
2
(1)
Where p~ denotes a point (x, y) in the image coordinate, d~ denotes a intensity displacement (u, v) of the corresponding points between two images, N is the block size, I1 is a block partitioned from the template, and I2 is the block partitioned from image being searched. The point where has lowest intensity displacement d~ is the best matching point in its local search region. B. Gradient Orientation Pattern Matching Technique The gradient orientation pattern matching (GOPM)is also a matching technique based on gradient vectors of image[4]. The difference between GOPM and SSD is that we use unit gradient vectors, obtained through the normalization of image’s gradient vector, for establishing image correspondences. So We do template matching by use only the direction of gradient vectors. The normalized gradient in x and y directions are obtains from normalization step as q nx (x, y) = Ix (x, y)/ Ix2 (x, y) + Iy2 (x, y) + 1 q ny (x, y) = Iy (x, y)/ Ix2 (x, y) + Iy2 (x, y) + 1
(2)
The +1 terms is added to prevent the zero-division during normalization. Then the normalized gradient vectors nx and ny are used separately using sum of absolute difference as PN/2 ~ = PN/2 GOP M _nx (~ p, d) j=−N/2 i=−N/2 |nx1 (x + i, y + j) − nx2 (x + i + u, y + j + v)| PN/2 ~ = PN/2 GOP M _ny (~ p, d) j=−N/2 i=−N/2 |ny1 (x + i, y + j) − ny2 (x + i + u, y + j + v)|
we propose the implementation of sum-of-square difference (SSD) method. The most similar location provides the center of that best match coordinate. The procedure is shown as in Fig.2
(3)
where nx1 , ny1 are the unit gradient vectors of template I1 and nx2 ,ny2 are the unit gradient vectors from the image I2 . Other notations are the same as those in (1). Gradient orientation pattern matching can be obtained by combine these two terms. ~ = GOP M _nx (~ ~ + GOP M _ny (~ ~ (4) GOP M (~ p, d) p, d) p, d) The advantage of GOPM over the others techniques is GOPM provides a good robustness to intensity changes. Because GOPM use only direction information of gradient, so it is not depend on intensities at all.
Figure 2: Hierarchy of block template matching process
III. Procedure Since the method proposed in this paper is originally be a part of eye motion tracking system used for ophthalmic surgery purpose, the objective of this experiment is to test the robustness to changing lighting condition of the conventional block template matching(SSD) and gradient orientation pattern matching(GOPM). In order to response the surgeon work flow, all implementation are in real-time. The program is developed using OpenCV library based on C/C++ programming [5],[6] running on computer with Intel Core2 Duo 1.66 GHz processors and 4 GB of memory. For the simulation in laboratory, we apply our work to the actual ophthalmic surgery and sample video sequence in various situation. The sample input from the surgery camera and our sample video sequence is shown in Fig.1
1) Initialize the Template and Sample Image: In order to track the eye motion perfectly, the template need to be initialized at the beginning of the surgery process [7]. In laboratory, we initialize the inner pupil to be template wherefore the pupil is the darkest and obvious component of the eye. Therefore it is the most appropriate range to track the center of the eye. 2) Sum-of-Square Difference Template Matching Technique (SSD): The sample image and template being initialized from previous step are used for SSD method. This method will compare the intensity informations of the sample image and template to find the best matching position (1). However, since the intensity informations are directly used, so the change in lighting condition, which globally affect the intensity of the entire image, highly affect the result of SSD template matching method. B. The Gradient Orientation Patterns Matching
(a)
(b)
Figure 1: Sample input from the (a) surgery camera (b) our sample video sequence
In this paper, we introduce 2 template matching approaches; Block Template Matching and the Gradient Orientation Patterns Template Matching. A. Template Matching Block template matching is the conventional intensitybased matching techniques by measuring the similarity between each part of image and the template. In this paper,
The Gradient Orientation Patterns Template Matching (GOPM) can be performed by following 3 steps, the first step is to extract the Gradient Orientation Information (GOI) ,the second step is to normalize the GOI, and the third step is to perform the template matching on the GOI. 1) Gradient Orientation Information (GOI) Extraction: The GOI extraction procedure is shown in Fig.3 Gradient vector can be extracted by using edge detection. Therefore, in order to extract the gradient orientation information (GOI), we apply the Sobel operation in x and y directions to both sample image and template. The outputs are shown in Fig.4 2) Normalize and Scaling the Gradient Vector: After obtaining the Gradient vector, we express the GOI using the unit gradient vector by normalize and scaling the GOI using (2). The results are as shown in Fig.5.
3) Gradient Orientation Pattern Matching Technique (GOPM): By separate the x-direction unit gradient vector and y-direction unit gradient vector, the GOPM is the addition of the results of preforming template matching to x-direction gradient y-direction gradient (4). IV. Results This work compares the introduced method which is GOPM to the traditional template matching method which is SSD. Since this eye tracking module is designed for the real ophthalmic surgery, therefore it need the high robustness to any environment. By attempting to test both methods to various lighting conditions in real-time video sequence, the test result is as be shown in Table I. A. Eye Recognition Efficiency Eye recognition efficiency is determined by precision error, which is the percentage of time that matching result fails to locate the best matching position in the video to time of total length of output video. The lower precision error means higher recognition efficient. From Table II, in the normal lighting condition, SSD is able to track most of the eye position in real-time with very low precision error whereas GOPM contains small precision error with currently unknown reason. However in case of changing lighting condition, the precision error of SSD is rapidly increase whereas there is almost no effect to the precision error of GOPM. The reason of this result is the fact that GOPM uses normalized information as inputs so they got no effect from image intensity. B. Obstrcle Robustness This characteristic is highly required in a surgery, since the ophthalmic surgery tools usually disturb the eye shape and may cause the error of the best matching. In this experiment, we compare the result between the SSD and GOPM (as shown in Table II. Both methods are still functional in case of the eye is covered by a half. However
Figure 3: Hierarchy of the Gradient Orientation Information Extraction
in case of closed eye, out-of-frame, or obstacle covers the entire eye, both method is unable to keep the eye position to be tracked. C. Blur Robustness The blurriness condition is also be concerned in this experiment. Both methods provide the property of blur robustness and can find the best matching position even though the camera focus is not proper. D. Lighting-Condition Robustness The GOPM provides the property of lighting-condition robustness and can find the best matching position during the changing lighting condition, whereas SSD fail during changing lighting condition as mentioned in section IV-A (be shown in Fig.6). E. Scaling Robustness Both method are not provide the property of input scaling robustness. F. Computation Time The computation time of both method highly depend on the size of the input video, and also depend on the performance and condition of the computer. In general, both method’s computation time are almost equal to each other in any cases. In case of normal size video, both methods computation time is low enough for using in realtime implementation. V. Conclusion and Future Work The result clearly shows that the GOPM performance is much better than that of traditional template matching technique during a changing lighting condition, with average precision error of GOPM is much less than of template matching. These methods also provide the robustness to obstacles and blurriness. The computation time of both methods are small enough to be use for a real-time application. In conclusion, GOPM can be used in place of the traditional template matching technique for better perfomance under varying lighting condition. As future work, we will improve the performance of GOPM further.
(a)
(b)
(c)
(d)
Figure 4: Gradient vector of (a) sample in x-direction, (b) template in x-direction, (c) sample in y-direction (d) template in y-direction
Table I: Computational time and precision error of Block Template Matching and GOPM Input
Video in constant light condition Video in changing light condition Video in extreme light condition Video of real eye surgery
Block Template Matching Average Precision ErTotal Output Computation ror (s) Length (s) Time (ms) 78.35 1.44 127
Average Computation Time (ms) 62.48
GOPM Precision Error (s)
Total Output Length (s)
4.78
123
78.63
97.09
136
63.09
17.01
132
77.73
34.11
93
64.32
9
86
103.33
0
82
77.7
0
77
Table II: Comparison between Block Template Matching and GOPM Average Precision Error(%) Obstacle Robustness Blur Robustness Lighting-condition Robustness Scaling Robustness Computation Time (ms)
Block Template Matching 30.283 Yes Yes No No 50-350
GOPM 7.366 Yes Yes Yes No 50-350
References [1] Xiangguo Lin, Jixian Zhang, Zhengjun Liu, and Jing Shen, "Semi-automatic road tracking by template matching and distance transform", Urban Remote Sensing Joint Event, 2009. [2] R. C. Gonzalez, R. E. Woods, "Digital Image Processing", Prentice Hall,(2nd Edition) 2002. [3] M. Svedlow, C. D. McGillem, and P. E. Anuta, "Experimental Examination of Similarity Measures and Preprocessing Methods Used for Image Registration", Symposium on Machine Processing of Remotely Sensed Data, 1976. [4] T.Kondo, and W.Kongprawechnon, "A Matching Technique Using Gradient Orientation Patterns", unpublished [5] Gary Bradski, and Adrian Kaehler, "Learning OpenCV", OReilly Media, 2008. [6] http://opencv.willowgarage.com/wiki/FullOpenCVWiki [7] JIN Min, and SHI Lei, "Research of moving targets detection and identification", Second International Conference on Intelligent Computation Technology and Automation, 2009. (a)
(b)
(c)
(d)
Figure 6: Shown the robustness to lighting condition (a) SSD in normal condition, (b) GOPM in normal condition, (c) SSD in darken condition, (d) GOPM in darken condition
. (a)
(b)
(c)
(d)
Figure 5: Gradient orientation pattern after normalize the GOI of the (a) sample in x-direction, (b) template in x-direction, (c) sample in y-direction and (d) template in y-direction
