Statistical inference in ALPR system - IEEE Xplore

Statistical Inference In ALPR System B. Y. Amirgaliyev

C. A. Kenshimov

Z. Y. Baibatyr

IITU Almaty, Kazakhstan [email protected]

IITU Almaty, Kazakhstan [email protected]

ABY Applied Systems Almaty, Kazakhstan [email protected]

K. K. Kuatov

M. Z. Kairanbay

A. K. Jantassov




Abstract—The paper describes an inference method for Automatic License Plate Recognition (ALPR) system. Proposed approach is aimed at improving the output of ALPR system which specifically operates with multiple video frames. In our work ALPR is considered as a string line output mechanism. These outputs are swept through the analysis and transformed to single result. Enhancement is reflected in reliability of the ALPR outcome.

list of concrete issues that may come with recognized number: partial recognition (missed or unrecognized characters), occasional addition of extra symbols which is noise, incorrectly identified characters. Nevertheless having these problems we count on high frequency of the correctly recognized symbols on the different extracted plate samples.

Keywords—Automatic License Plate Recognition, comparison of the plate numbers, appearance frequency of the character.

In common different proposed ALPR systems, when dealing with only one image, face the problem of insufficient confidence in recognition. The idea of processing of several frames becomes helpful here. In our previous work [2] we proposed realization of ALPR system which works quite well for general cases. But it still required some improvements in detection, segmentation and character recognition. Current work is devoted to these problems. Following research paper [3] describes the approach that improves the accuracy of recognition. Algorithm checks the similarity of newly recognized characters with the ones that are stored. In their work, the recognition process is handled until the vehicle disappears from scene. Another work [4] outlines recognition problems that come from low quality frames which are captured when car is in motion. To get a better result their algorithm utilizes multiple recognition results. They score likeliness between each character and the registered character templates. One more paper [5] makes use of several license plate representations to improve recognition result. In their work authors demonstrate that the single image based recognition provides wrong assignments, but the subsequent majority voting scheme ensures that the final character classification is correct.

I.

INTRODUCTION

One of the problems that the most of ALPR systems share is imperfect accuracy rate. This may be caused by partial detection of license plate, impossibility of proper symbol segmentation or wrong character recognition because of defective image. Challenges like weather condition or illumination level, video or image quality, unreadable license plate view position, damaged or muddy plates, and many other factors, all these somehow or other effect on precision of ALPR system. In the following paper [1], different approaches of ALPR and their limitations are discussed by considering image size, success rate and processing time as parameters. It is hard to come up with the robust algorithm that could provide high confidence in recognition just by dealing with one image. But it is much easier if we can consider several similar images. Particularly it is worth to process video frames rather than working with one static image. Strictly speaking they form a sequence of images that can be analyzed and provide better recognition result. Here we rely on the fact that in most cases picking more serial frames with the license plate will give a higher confidence in detection, segmentation and as well as character recognition. It means that it is possible that one of the ALPR parts may succeed on some images and may not on another. Using statistical analysis we can improve decision making. Let’s assume we have an ALPR subprogram that can extract license plate number from a given frame. The main concept is to assemble extracted plate numbers from sequential frames and process this data to get one reliable result. We are able to achieve that by paying attention to the appearance (occurrence) frequency of characters in their particular positions on a license plate. Motivation of this work can be expressed through the list of possible weaknesses of ALPR. The

II.

III.

RELATED WORKS

TOOL FOR ANALYSIS

The best matching between two license plates - this algorithm or function performs comparison of two strings (license plates) and provides matching data. Matching data consists of pointer to one of two strings, number of offset shifts for pointed string, number of matched characters and number of offset shifts of the first string. The first accepted input string parameter can come with several underscores ( _ ) at the beginning. This character represents offset shift. The output of the function points to one of the given strings which should be modified out of the algorithm by adding underscores. And as

long as we reuse this algorithm several times the second parameter which might be altered afterwards may come as the first parameter next time. That is why the first string can have additional underscores in front of it. To simplify matching operation we dispose of these underscores only inside the function and for the later convenience keep the information about its offset shifts in the output of the function. The idea of the algorithm is to find maximal number of column-wise matched characters of two strings. The algorithm starts by choosing the shortest string which is moved from left to right along the longest one. If string lengths are equal then the first string is simply chosen as the shortest. The first comparison starts with two positions (characters) of the shortest string shifted to the left and ends with two positions shifted to the right. When moving the shortest string the algorithm compares characters column by column. It proceeds a comparison operation defined number of times which can be expressed by the following formula

NC = LS - SS + 1 + 4 ,

(1)

where NC denotes the number of string comparisons, LS is the longest string length, SS is the shortest string length. During each iteration pairs of characters are compared. If the current number of correctly matched pairs are greater than previous then this information along with pointer to one of the strings, its offset shifts and also number of offset shifts of the first string are temporarily stored. Pointer to string which further must be shifted is chosen by the following principle: if at the time of comparison the first character of the shortest string was to the left than the longest one then the longest string is pointed; if at the time of comparison the first character of the longest string was to the left than the shortest one then the shortest string is pointed. Pointed longest or shortest string may represent either first or second input string. This is supposed to be declared at the beginning. After all iterations function returns its best matching data. In Fig. 1 we suggest visualizations of input and output.

Figure 1. Visualizations of input (left) and output (right) in “The best matching between two license plates” algorithm

IV.

ALGORITHM WORKFLOW

Extracted from images, the set of license plate numbers are hypothetically the same. Except that some of them may be incorrectly recognized in terms of certain characters. One of the objectives is to get rid of the irrelevant information, particularly referring to the license plate format checking operation. Each plate number is considered as a string line data. So later on we can call them a list of string lines. It is important to mention that the output which came from ALPR subprogram may also contain specific symbol that denotes unrecognized character. It is obligatory for our algorithm to be aware of the position of

unknown symbol. Despite this sometimes is impracticable to locate missed characters at the beginning or the end of plate. So the algorithm expects that acquired license plate may come clipped from the left or right. Our algorithm is made up such that it will acquire several outputs from ALPR subprogram. When it finished with that it will start to analyze plate samples. The first question that we encounter is how many frames do we need to get a better result. The answer is as many as possible, but without essential reduction of the speed of whole ALPR system. We experientially deduced that 10 frames is enough to reach required result. For recognition of one plate 10 frames give us no greater than 10 extracted plates to analyze. The order of these extracted license plates matters if we know upfront which direction car is moving, towards the camera or in opposite direction. Moving direction can be calculated using coordinates of a license plate samples. Importance of the order comes from the idea that the distance between license plate and camera tells us where we should start to sort plates that were extracted from images. Simply if license plate is close to camera, we assume that we can get a better recognition. Below in Fig. 2 we provide examples of license plate samples.

Figure 2. Examples of license plate samples for analysis

In our work, when we mention statistical analysis we imply some logical derivation of single output from the various inputs. Inputs may consist of correct and incorrect data. Without knowledge which one is right and which one is wrong we derive output by relying on high frequency of the correct data. Being more concrete, the objective is to form the table of aligned plate numbers such that their characters are maximally matched in columns. Well aligned license plates fill each column with characters. These columns were already defined as an assembly of correct and incorrect data. Each character which occurs in one column is examined as possible symbol for a specific position on the final derived plate. Since character may turn up several times in a column means that it has own appearance frequency. Main steps of plate samples analysis: 1. Sorting string lines in a list and alignment of their characters. 2. Counting the appearance (occurrence) number of characters. 3. Getting candidates with proper plate format. 4. Calculation of the appearance frequency values for verified plate numbers.

5.

Choosing candidate with the highest frequency of characters rate.

This algorithm begins with the sorting string lines which is necessary for alignment of plate characters. By alignment here we imply the shifting of whole string line on a several positions to the right. Let’s say car was moving towards the camera and we acquired N samples (where 1 < N < 11), so we traverse across the sequence of plates from the bottom to the top. By taking the last sample (Nth) we compare it with the rest samples above, and look for the best resemblance. As long as we deal with comparison of two strings, to find the best match we apply simple intuitive approach. For two plates using “The best matching between two license plates” function we calculate maximum number of equal characters and the corresponding shift position. As it was already mentioned the algorithm takes the last plate sample and finds the best matched plate with it by examining number of matched characters. Then it replaces the best matched one with the next plate located above the last sample with index N-1. In addition to form the table of aligned characters one of these two strings can be shifted to the right. The same way we reiterate for the next each N-2 samples (without Nth and the one which in the end will turn up at the top of the list). One of the worst possible scenario is when at some point there will be no matching at all. In such situation we can divide the list and start to consider them separately. After sorting to make string lengths equal it is necessary to find the longest string and append the others with trailing underscores. Finally we obtain table of aligned plate symbols. Besides plate numbers this table may contain underscores ( _ ) as well as asterisk ( * ) symbol which denotes missed or unrecognized character. Having this table makes task more appealing now. Examples of tables are shown in Fig. 3.

Figure 3. Examples of aligned plate numbers which were extracted from serial frames

The next step is a calculation of the appearance (occurrence) number of unique character. Number of appearance is a quantity of repetitions of one symbol in particular column. These values are necessary for appearance frequency data. It is expected that one column can provide several occurrence values. Objective is not only to collect appearance numbers but also represent table data more compactly so the further format verification process is managed conveniently. Idea is to keep column structure and instead of having repeated values in them we store unique characters that showed up in their particular column. It can be considered as a matrix where column shows

character position and row addresses unique symbol for this position. Some of the lower rows may address nothing which makes this matrix - sparse. Examples of such matrixes are shown in Fig. 4.

Figure 4. Examples of compact representation of table data in a sparse matrix

This new representation allows us to easily traverse across columns and form combination of characters or plate candidate for format verification. So conceived, that columns in matrix which are arranged according to the table can contain underscores. We do not consider asterisk symbol as a possible character. In case if asterisk is the only symbol in a column algorithm stops working without any inferences. Format verification stage takes advantage of a new matrix of characters. The main task is to wisely combine column characters and check if obtained string line matches license plate format. To form the combination of characters algorithm starts with the first column and selects its element on the top, then it goes to the second column and picks its element on the top, then it proceeds the same way until it reaches the end of the row. When it reaches the end of the row it can perform format checking on this combination. If it passes verification the string will be saved. If the last column of the sparse matrix has any another nonempty character then algorithm substitutes last symbol of examined string with it. And the same way will perform format checking again. Strictly speaking algorithm looks over all elements in each column and makes substitution according to the position of character. When picking the element from the matrix we directly check whether it is a digit or character and hereupon can reject improper formats or prune some wrong symbol combinations upfront. As long as character recognition may misread “O” symbol as “0” (zero) or vice versa we also test both possibilities even if one of them doesn’t appear. Format checking itself is performed simply by examination of each symbol in a string for plate templates. Underscores are omitted during the format checking. Anyhow if candidate is passed, we keep underscores for frequency of characters data evaluation. At the format verification stage plate candidates with inappropriate formats will be dropped. There is still possibility of having several leftover candidates. We use appearance frequency data to choose only one candidate with higher confidence rate. Since we have already calculated the number of appearances of each character in columns it is only takes to compute their frequency values for a specific candidate. Appearance frequency of unique character from one column can be counted by the following formula

AF = NA / NS ,

(2)

where AF is appearance frequency, NA is a number of appearances and NS denotes the number of plate samples. Number of appearances is a quantity of repetitions of one

symbol in particular column. Number of plate samples is initial quantity of extracted license plates from N frames. Summing up appearance frequency values and dividing it by the length of the string will give us the confidence rate that we can use as a statistical measure. The goal is to compute the confidence rates for every license plate which was verified for format. Normally algorithm returns only one candidate, but it is possible to have several derived plates. In such situations we resort to confidence rate values. According to the algorithm, license plate with the highest confidence rate is counted as a final result. Candidates that are shown in Fig. 5 were derived from samples that are demonstrated in Fig. 2.

Figure 5. Examples of a derived candidate with the highest confidence rate

V.

CONCLUSION

To summarize, we have proposed inference method for ALPR system which specifically operates with multiple video frames. Given approach improves the result of ALPR through the use of statistical analysis. In this work, we showed that the algorithm works efficiently even if all recognized license plates have wrongly identified or missed characters. Our solution could be applied with different ALPR realizations that work with sequential frames. It only takes to adapt format verification stage. The goal of the performed work is to provide reliability to ALPR technology. Despite the fact that this method helps in most cases it has own drawbacks that cannot be solved. One of them is it can’t anticipate correct symbol that should be chosen if ALPR mechanism did not provide such data. Another obstacle comes if again ALPR does not provide position of a missed character. We applied statistical analysis to make precision of ALPR system more reliable and accurate. REFERENCES [1] C. Patel, D. Shah and A. Patel, “Automatic Number Plate Recognition System (ANPR): A Survey,” International Journal of Computer Applications (0975 – 8887). Volume 69– No.9, May 2013. [2] B. Amirgaliyev, M. Kairanbay, C. Kenshimov and D. Yedilkhan, “Development of automatic number plate recognition system,” COMPUTER MODELLING & NEW TECHNOLOGIES 2014 18(3) 193-197 (Received 1 March 2014, www.tsi.lv). [3] I. Tsai, J. Wu, J. Hsieh and Y. Chen, “Recognition of Vehicle License Plates from a Video Sequence,” IAENG International Journal of Computer Science, 36:1, IJCS_36_1_04 (Advance online publication: 17 February 2009). [4] K. Nakao, K. Sugimoto, M. Saitoh and T. Okazaki, “Development of a License Plate Number Recognition System Incorporating Low-Resolution Cameras”, Takasago Research & Development Center, Technical Headquarters, Kobe Shipyard & Machinery Works. Technical Review Vol. 45 No. 3 (Sep. 2008). [5] M. Donoser, C. Arth and H. Bischof, “Detecting, Tracking and Recognizing License Plates”. Institute for Computer Graphics and Vision Graz University of Technology.