2Information Technology Dept., Armstrong Atlantic State Univ., Savannah, GA, USA. Abstract: - In this paper, a fuzzy template matching ... object shape to a list of stored shape representations. (templates) [9]. There were various ..... They can be discriminated by people according to their distances to the matched points, i.e., ...
Fuzzy Template Matching for Printing Character Inspection BIN CHEN1, LEI HE2 1 Chengdu Institute of Computer Application, Chinese Academy of Sciences, P.R.China 2
Information Technology Dept., Armstrong Atlantic State Univ., Savannah, GA, USA
Abstract: - In this paper, a fuzzy template matching method is proposed and the application on real time printing character quality inspection is illustrated. The method is robust in handling CCD images with poor quality due to illumination variation, dithering and motion blur, etc. Based on multi-level thresholding, fuzzy templates are constructed offline for all characters in certain font type and size. In inspection, multi-level thresholding is first applied on the real time input CCD images to construct the binary images at different levels to adapt to different quality problems. Then the dynamically updated fuzzy templates are used to match with the binary images at different levels to identify different quality problems. Experimental results and industrial applications on currency printing system are rather satisfactory. Key-Words: - Machine Vision, Fuzzy System, Template Matching, Real Time Quality Inspection, Multi-level Thresholding
1 Introduction Compared with the printed character recognition that has been studied for decades, the printing character quality inspection is a rather new area that attracts more and more interests as so many products currently are labeled with letters and digits, such as currency, invoices, circuit boards, instruments, glass/plastic containers, just name a few. For current character printing systems, the printing quality can’t be perfect due to the system errors generated by friction, unevenly oil spreadness, paper pressure, etc. Generally the system errors result in problems as missing or repeating characters, ink congestion (the rightmost “6” in Fig.1(a)) or empty (the middle “5” in Fig.1(b)), etc. Manual inspection is both time and cost consuming. Therefore, it becomes important to employ an automatic online printing character inspection system for real time printing quality inspection. In such online character inspection system, the camera capturing the images must be running stably, cheap and easy to manipulate. Common CCD camera systems can be utilized, which produce 8-bit gray scale images with the spatial resolution approximately 0.25 mm/pixel (100 dpi). However, the CCD image quality is not good due to low resolution and illumination variation. Moreover, in many cases the characters that are printed on inhomogeneous background comprising various image patterns have very poor local contrast. In order to handle above difficulties and inspect the printing character quality accurately, we propose a fuzzy template matching method for the online printing character quality inspection.
(a) Ink congestion
(b) Ink empty
Fig.1 Examples of printing quality flaws Template matching is the simplest object identification method by comparing the whole input object shape to a list of stored shape representations (templates) [9]. There were various versions that have been widely used in object recognition or detection applications [5][6][10][11]. Here we also apply the template matching for printing character quality inspection due to its simplicity. However, the accuracy and robustness requirements for quality inspection are far stricter than general character recognition. A small variation between the template and the input image generally don’t affect the recognition result, but it is significant enough to change the decision on quality inspection. Thus the common template matching methods can’t be applied directly on printing quality inspection applications. Penz and Bajla [1] use the feature points in sample characters to construct the templates and match them with the extracted input character feature points for character recognition and inspection in real time. The templates are for recognition purpose and they emphasize the discrepancy between different characters rather than the differences between the correct and defected characters. Therefore, the performance in quality inspection is not good. Hu and Wee [2] inspect the characters on plastic wheel by
template matching with the templates being constructed by character boundaries. The statistics of the distances between the points on the input character edge and the template edge points is used to inspect the printing quality. This method limits the quality inspection only on the character edges. In [3], morphology operations are used on the template matching to inspect the printing character quality on chip resistors. The morphology dilation and erosion are used to remove the noise dots and preserve only the mismatch dots with quality problems. However, this method cannot differentiate the thin character with the character having single thin stroke. The former should be considered good because all the strokes in the character have the same thickness, while the later is not due to different stroke thickness. Fuzzy logic is the best mechanism to handle this type of dilemma, as illustrated in many different applications [5][7][8]. In this paper, we construct fuzzy templates based on multi-level thresholding and dynamically change the fuzzy templates in the process of matching with input images for the robust and accurate printing quality inspection. Section 2 gives the algorithm description. Experiments on currency character quality inspection are illustrated in Section 3. Section 4 draws the conclusion.
2 Algorithm Description Input image Compute fitting, ink congestion and empty matrix Fit, Light, Dense matrix Update fuzzy membership value Enhance matrix
Sample images Determine thresholding and skeleton points Fuzzy template construction Multi-level thresholding
Skeleton image
The flowchart of the proposed algorithm is shown in Fig.2. The algorithm consists of two main blocks: fuzzy templates construction and fuzzy template matching, which can be further divided into several sub-steps as shown in the figure.
2.1 Fuzzy Template Construction The first step in fuzzy template construction is the skeleton image generation, i.e., binary images with only skeleton points as the templates. The binary templates for letters and digits can be generated from a set of normal image samples because the printed characters in different sample images have very small variations in orientation, location and size. The skeleton image is required to extract the most important (skeleton) component of the character and keep strokes as even as possible. To determine the skeleton points for a letter or digit, all sample images containing the character are firstly thresholded to binary images, in which the threshold is obtained by maximizing the between-class variance of gray levels [4]. Then for each point, the appearance rate as a character point in all the binary sample images is calculated. If the appearance rate is larger than a predefined number (e.g., 70% in the applications), it’s a skeleton point in the binary template. In the second step, multi-level thresholding is applied on the sample images to construct several other binary images, which can be used to determine the importance of each point in the binary template for the quality inspection. In practice, the lower level thresholding (smaller threshold), the more important points can exist in the thresholded image, i.e., skeleton points are darker than boundary points. In Fig.3, different threshold values (T) are used to illustrate the effect of multi-level thresholding. The higher the T value, the thicker the template is. Moreover, multi-level thresholding is also used to inspect different quality problems in later template matching, such as ink congestion and ink empty.
Multilevel binary images
Fuzzy template Compute ink congestion Fuzzy and empty membership degree Fuzzy value template assignment matching Defect identification Fig.2 Algorithm flowchart
Fig.3 Illustration of multi-level thresholding
in later matching step. Two fuzzy template examples are shown in Fig.4. Six types of points are shown in the templates, and their FMV are assigned from the highest positive value to the lowest negative as: the solid points (1000), “*” (300), “+” (100), “.” (-100), “—“ (-300), and “=” (-1000). In practice, the FMV can be assigned in more levels according to the complexity of the images.
2.2 Fuzzy Template Matching The first step of fuzzy template matching is to compute the fitting matrix (Fit), ink empty matrix (Light) and ink congestion matrix (Dense). The fitting matrix (Fit) indicates the matched points between the input image and the template and will be used to dynamically update the FMV of the fuzzy template. Fit is obtained by multiplying the standard binary image (Stand) of the input image and the fuzzy template (FZT) generated in section 2.1: Fit(x,y) = Stand(x,y)*FZT(x,y) (1) Here the threshold of Stand is computed as in section 2.1 for the skeleton image construction [4].
(a) Original image
Fig.4 Fuzzy logic based template examples After the importance of each point in the template is determined in the multi-level thresholding, a fuzzy membership value (FMV) is assigned to each template point, which is used to construct the fuzzy template. For example, skeleton points of a character are the most important for quality inspection, thus they are assigned with the highest FMV. The FMV decreases from the skeleton points to boundary points. Furthermore, the background points are assigned negative FMV and their magnitudes are also decided according to their importance on the quality inspection, which were determined through a set of samples with quality problems. The higher the absolute value, the more important the point is. The FMVs of the points on both the template and background are assigned as initial values to construct the fuzzy template and will be dynamically updated
(b) Low level thresholding
(c) High level thresholding
Fig.5 Illustration of low and high level thresholding The problems appear in printing process can be divided into the ink congestion and ink empty, thus the goal of the proposed algorithm is to detect these two problems. Besides the Stand, two other binary images of the input image, together with two corresponding matrixes are generated to identify these quality problems. For the two images, one is the low level thresholding binary image (LowVal) and the other is the high level thresholding binary image (HighVal). The low and high level thresholds are determined according to the character point number ratio between the thresholded binary images and the standard binary image (e.g., 90% and 110% in the application). Two examples are shown in Fig.5. The ink empty (Light) and congestion (Dense) matrixes are computed as follows: Light(x,y) = FZT(x,y) – FZT(x,y)*HighVal(x,y) (2)
Dense(x,y) = （-1）*LowVal(x,y)*FZT(x,y) (3) In the second step, the FMV for all the points in the fitting matrix are dynamically updated according to the location of the matched points between the input image and the fuzzy template. The idea here is that the neighbors of the matched points have decreased effect in quality inspection. The updated fitting matrix is known as the enhance fuzzy matrix (Enhance), which is obtained by adjusting the FMV of points surrounding high FMV matched points, as shown in equation (4). Enhance' (x, y), if Enhance' (x, y) > 0 Enhance(x, y) = (4) 0, otherwise
In this section, the proposed fuzzy template matching method is illustrated by the currency digits printing quality inspection. Two input images are shown in Fig.6, from which we can see the digit “9” in Fig.6(a) has ink empty problem and the digit “6” in the rightmost of Fig.6(b) has ink congestion problem.
(a)
(b)
Fig. 6 Two examples for quality inspection
, where
Enhance' (x, y) =
Fit(x, y) + AD , if Fit(x ± 1, y)orFit(x, y ± 1) > TF otherwise Fit(x, y),
, AD is the FMV adjustment value, which is set according to application requirements. If the application focuses on the character edge inspection, AD should be decreased. TF is the predefined threshold of FMV in the fitting matrix. After dynamic fuzzy template updating, the ink congestion and empty matrix are also updated using the equations (5) and (6). This step is to remove the mismatched points with no quality problem from the Light and Dense matrix by reducing the FMV of these points. Light(x,y) = Light(x,y) – Enhance(x,y) (5) Dense(x,y) = Dense(x,y) – Enhance(x,y) (6) Finally, two measurements for the ink empty (EmptyDeg) and congestion (CongDeg) are computed based on the updated Light and Dense matrix, which are used to evaluate the defection level of the quality problem. In fact, these two values indicate the density degree of ink printed on the character. A small value of EmptyDeg means light ink empty problem and a high EmptyDeg value indicates critical ink empty problem. CongDeg has similar meaning on the ink congestion problem. Based on these two indicators, printing quality problems can be divided into different levels, such as light defection and severe defection. The values to the separation can be assigned according to application requirements. The ink empty and congestion degree are computed as following: EmptyDeg = Light(x, y), for all Light(x, y) > 0 (7)
∑∑ x
CongDeg =
y
∑∑ Dense(x, y), for all Dense(x, y) > 0 (8) x
y
3 Experiments
(a) Inspection images for “9” in Fig.6(a)
(b) Inspection images for right “6” in Fig.6(b)
Fig. 7 Illustrations for quality inspection in Fig.6 As mentioned in the section 2.2, the matching between the standard binary image with the binary template constructed in the first step of fuzzy template construction can tell us the mismatched points, from which we can determine the points with quality problems. The inspection images produced by
the algorithm are shown in Fig.7 as small images to illustrate the quality inspection process for the digits “9” and “6” in Fig.6. The left images in the first rows in both Fig.7(a) and (b) are the skeleton images (binary templates). On the derived fuzzy templates, the template points have positive FMV above 50 and other points are in background. The middle images show the points with relatively high (e.g. FMV > 200) absolute FMV of the fuzzy templates and the right images show the points with the highest absolute FMV. The three images in the second row are the LowVal, Stand and HighVal binary images from left to right. As stated in section 2.2, they are the binary images resulted from the multi-level thresholding on the input image. The three images in the third row show the results of step 1 in section 2.2. The left image show the Fit matrix in binary format for a clear view, in which the points having positive FMV are white and the points with zero or negative FMV are in background. The middle image is obtained by overlapping the LowVal image (black) on the binary template (white), which is to detect the ink empty problem. The right images are obtained by overlapping the binary templates (black) on the HighVal images (white), which is to detect the ink congestion problem. The white points in the middle and right images are mismatched points, which include both good points and defected points. They can be discriminated by people according to their distances to the matched points, i.e., the mismatched points are good when close to the matched points; Otherwise they are defected. The goal of next step is to detect the mismatched points with quality problems, and compute the empty and congestion degree to identify the defection level. Images in the fourth rows of Fig.7(a) and (b), show the effect of dynamic updating of fuzzy templates. The left images show the Enhance matrix in binary format and they indicate the effect of updating the FMV as stated in Section 2.2, with AD =50 and TM =80. It can be seen that the FMV of the mismatched points with quality problems are not changed. The Light and Dense matrix can be recomputed as stated in Section 2.2, and the resultant images are shown as the middle and right images, in which the white points have positive FMV and other points with zero or negative FMV are shown as gray background. Again, the binary template is shown as the black mask to make the problematic locations clear. It can be seen that there are white points left only on the problematic locations. According to equations (7) and (8), the ink empty and congestion degree can be computed to determine the defection level of the quality problems. In the currency
inspection application, for both the EmptyDeg and CongDeg, 400 and 800 are assigned as the thresholds to determine the light and severe defection, respectively. In this experiment, the values of EmptyDeg and CongDeg in Fig.7(a) are 665 and 0, in Fig. 7(b) are 0 and 3650. Therefore, the printing quality of the characters in Fig.6(a) and (b) is light defection in ink empty and severe defection in ink congestion, respectively. The images in the fifth and sixth row are only used to demonstrate the defected points with high FMV to the user. The left images in the fifth and sixth row show points with relatively high and highest absolute FMV respectively, same as the middle and right images of the first row. They are shown here to illustrate the important points of fuzzy templates for quality inspection, either on the template or in the background. The mismatched points involved in the left images are shown in the middle and right images, indicating Light and Dense points respectively. These points are shown as white points overlapped by the black template. The white points in the middle (ink empty) or right (ink congestion) images of these rows give user a subjective feeling of the printing quality defection degree. To compare the inspection intermediate and final results between good and defected characters, two examples of inspection images are shown in Fig.8. The second digit “5” in the input image (Fig.8(a)) has the ink empty problem and other digits are good. To compare the inspection results, the inspection images for the defected digit and a normal one (the third “5” from left) are shown in Fig.8(b) and (c), respectively. The values of EmptyDeg and CongDeg in Fig8(b) are 635 and 0, in Fig8(c)’s are all 0, which indicate that the second digit has the light ink empty problem and the third digit has no problem. In the printing process, user is required to check the printing quality when some warning signals produced by light defect appear. User makes decision on how to adjusting the printing parameters to make the defect level decreasing to normal. In the case of severe defect, computer turned off the whole printing system automatically. In our currency printing quality inspection application, for a character sample set (around 100,000 characters), the method in [3] succeeds with 97% characters in the case of severe defect quality problem, and with 85% characters in the case of light defect quality problem. Our method has the success rates as 100% and 99.995% on the severe and light defect quality problems, respectively, which performs far better than the method in [3]. Furthermore, running in a PentiumII 450MHz computer, the proposed method can finish the inspection of one character in 2ms, which satisfy the
real time quality inspection requirement. The fuzzy template matching method can detect the subtle quality defects and accurate printing quality inspection is realized.
(a). Input images with both problematic and normal digits
(b). Inspection images of second “5” in (a)
(c). Inspection images of third “5” in (a)
Fig.8 Comparison for quality inspection between good and defect characters
4 Conclusion In this paper, a fuzzy template matching method based on multi-level thresholding for real time printing character quality inspection is proposed. Fuzzy logic is used here to simulate the human vision system for quality problem identification. The fuzzy templates for different characters are constructed offline based on the multi-level thresholding of the sample images. In the online inspection process, the input image are firstly multi-level thresholded to generate different level binary images. Then the dynamically updated fuzzy template is matched with these binary images for quality inspection. The output is the ink congestion or empty defection level. The fuzzy template matching and dynamic template updating approach performs better than traditional
template matching method in rather difficult quality inspection applications. Furthermore, the proposed method can be applied widely in industrial applications, such as invoice, currency, wrapper, and circuit board printing character quality inspections. Currently, an optical system adopting the proposed method is successfully applied in more than thirty note printing factories. Future work includes broaden this algorism to graphics and color image quality inspection. References: [1] H.Penz, I.Bajla, Fast real-time recognition and quality inspection of printed characters via point-correlation, Proceedings of SPIE, Vol.4303, 2001, pp 127-137. [2] Y.Hu, W.G.Wee, A computer vision system for automated inspection of molded plastic print wheels, Proceedings of SPIE, Vol.1468, 1991, pp 653-661. [3] Y.Numagami, Y.Hattori, An algorithm for quality inspection of characters printed on chip resistors, Proceedings of SPIE, Vol.1606, 1991, pp 970-979. [4] Nobuyuki Ostu, A threshold selection method from gray-level histogram, IEEE Trans on SMC, Vol.9, 1979, pp. 62-66. [5] Y.Li, X.Qi and Y.Wang, Eye detection by using fuzzy template matching and feature-parameterbased judgment, Pattern Recognition Letters, Vol.22, No. 10, 2001, pp. 1111-1124. [6] Anil K. Jain, Yu Zhong, Sridhar Lakshmanan, Object matching using deformable templates, IEEE Trans. on PA MI, 1996, pp. 267-278. [7] B.Ravi, M.H.Ang, Fuzzy logic based character recognizer, Proceedings of the Philippine Computing Science Congress (PCSC), 2000, pp. 69-74. [8] Young Kug Ham, Sang Young Lee, Rae Hong Park, Fuzzy-based recognition of human front faces using the trapezoidal membership function, IEEE International Conference on Fuzzy Systems Vol.4 and 5, Mar. 1995, Yokohama, Japan, pp. 1799-1806. [9] R.C. Gonzalez and R.E. Woods, Digital image processing, Addison-Wesley, 2nd Edition, 2002. [10] J.S. Duncan, R. Owen, P. Anandan, L. Staib, T. McCauley, A. Salazar, and F. Lee, Shape based tracking of left ventricular wall motion, Computers in Cardiology 1990, Chicago, Illinois, pp. 23-26. [11] I. Cohen, N. Ayache and P. Sulger, Tracking points on deformable curves, Proc. Second Euro. Conf. Computer Vision 1992, May 1992.
