Morphology Based Handwritten Line Segmentation Using Foreground and Background Information Partha Pratim Roy Computer Vision Centre, Universitat Autònoma de Barcelona, 08193, Bellaterra (Barcelona), Spain.
[email protected]
Umapada Pal Computer Vision and Pattern Recognition Unit, Indian Statistical Institute, Kolkata-108, India
[email protected]
Abstract Currently text line segmentation is an important stage of research in historical document processing. Because of inter-line distance variability and base-line skew variability, line segmentation in unconstrained handwritten document is very difficult. The line segmentation task gets complicated, when overlapping or inter-penetration situation occurs between two consecutive text lines. In this paper we propose a method mostly based on morphological operation and run-length smearing algorithm (RLSA) to segment individual text lines from unconstrained handwritten document images. Here at first RLSA is applied to get individual word as a component. Next, the foreground portion of this smoothed image is eroded to get some seed components from the individual words of the document. Erosion is also done on background portions to find some boundary information of text lines. Finally, using the positional information of the seed components and the boundary information, the lines are segmented. We tested our scheme on images of five different scripts and we obtained encouraging results from the experiments. Keywords: Handwritten Document, Mathematical morphology, RLSA, Handwritten Line Segmentation.
1. Introduction At present text line segmentation is an important topic of research in historical document processing area. It costs to the quality of word and character segmentation greatly. Segmentation of unconstrained handwritten text line is difficult because of inter-line distance variability and baseline skew variability. Components of two consecutive textlines may be touched or overlapped in unconstrained handwritten text. In Indian languages such situation occurs frequently because of several modified characters. These overlapping or touching characters complicate the line segmentation task greatly. There are many methods for text line segmentation [1,3-4,6,10,13-14,16]. Global projection analysis of black pixels is often used for text line segmentation. But this method will not work properly when
Josep Lladós Computer Vision Centre, Universitat Autònoma de Barcelona, 08193, Bellaterra (Barcelona), Spain.
[email protected]
overlapping or skewed text lines occur. Modification of this method is done by some researchers using partial projection method [16]. Input image is divided into vertical stripes and based on the projection profile on these stripes segmentation is done. Pal and Dutta [10] proposed a modified technique of stripe-based method using water reservoir concept. Some studies decompose the text into individual components [14]. By means of hierarchical clustering procedure the components are grouped into individual text lines. This method cannot assign a character in its correct group properly when overlapping or interpenetration situation occurs in text lines. Techniques based on statistical modeling [4], thinning [13], linear programming [15], level set [3], HMM [6] etc. are also used for text line segmentation. In this paper we propose a scheme to segment unconstrained handwritten document pages of Indian scripts into individual text lines. We have used the foreground and background information to segment each line. Here, at first, a horizontal Run Length Smearing Algorithm (RLSA) is applied on the input image. The threshold for RLSA is computed based on the height information of the text lines and it is determined using water reservoir concept. Next, the foreground portion of this smoothed image is eroded to get some seed components from the individual words of the document. These seed components generally represent the central portion of individual words of a text line. This erosion also reduces the touching effect of modified characters and makes the line segmentation task easier. Erosion is also done in background region to find the upper and lower boundary information of a text line. Finally, using the positional information of the seed components and the boundary information, individual lines are segmented. Rest of the paper is organized as follows. In Section 2 properties of different scripts used in our experiment are discussed. Estimation of text line height is described in Section 3. In Section 4, we briefly explain our proposed method, used for line segmentation. The experimental results are discussed in Section 5. Conclusion is given in Section 6.
2. Properties of Different Scripts Used in Our Experiment In our scheme we consider text lines of Devnagari, Bangla, Oriya, Gujarati and English scripts for our experiment. We briefly discuss properties of Devnagari, Bangla, Oriya and Gujarati scripts here. Among Indian scripts, Devnagari is the most popular script in India and the most popular Indian language Hindi is written in Devnagari script. Nepali, Sanskrit and Marathi are also written in Devnagari script. Moreover, Hindi is the national language of India and the third most popular language in the world [9]. In modern Devnagari script there are 14 vowels and 37 consonants. These characters may be called basic characters. Bangla, the second most popular language in India and the fifth most popular language in the world, is an ancient Indo-Aryans language. Bangla script alphabet is used in texts of Bangla, Assamese and Manipuri languages. Bangla is the national language of Bangladesh. Also Bangla is the official language of West Bengal State of India. The alphabet of the modern Bangla script consists of 50 basic characters (11 vowels and 39 consonants).
Figure 1. Basic characters of (a) Bangla and (b) Devnagari alphabet are shown. First eleven characters are vowels and rest is consonants in both the alphabet sets.
In Devnagari and Bangla scripts, most of the characters have a horizontal line at the upper part. See Fig.1 where basic characters of Devnagari and Bangla scripts are shown. When two or more characters sit side by side to form a word, these horizontal lines touch and generate a long line called head-line. In Devnagari/Bangla script a vowel following a consonant takes a modified shape, which, depending on the vowel, is placed at the left, right (or both) or bottom of the consonant [2]. These are called modified characters. A consonant or vowel following a consonant sometimes takes a compound orthographic shape, which we call as compound character. A set of modified characters of Bangla and Devnagari scripts is shown in Fig.2. Gujarati is a popular language spoken by about 46 million people in the Indian States of Gujarat, Maharashtra, Rajasthan, Karnataka and Madhya Pradesh. There are 46 basic characters (12 vowels and 34 consonants) in Gujarati. Oriya is another popular language and script of India. This language is used mainly in the Orissa State of India as it is the official language of Orissa State. The alphabet of
the modern Oriya script consists of 52 basic characters (11 vowels and 41 consonants). Like Devnagari and Bangla scripts modified and compound characters are also present in Oriya and Gujarati scripts. Since modified characters may sit at the top or bottom of the consonant in these scripts, words of two consecutive text lines may touch because of these modified characters. Such touching through modified characters complicates our line segmentation task.
Figure 2. Examples of Bangla and Devnagari modified characters.
3. Estimation of Text Line Height To compute height information of the text lines in a document page, we apply water reservoir concept. The water reservoir principle is as follows. If water is poured from top (bottom) of a component, the cavity regions of the component where water will be stored are considered as top (bottom) reservoirs [8]. For an illustration see Fig.3. For each reservoir we compute its height. By height of a reservoir we mean the perpendicular distance of the base point (the deepest border point of a reservoir) from the water flow level of the reservoir. From each component of a document page, heights of the different reservoirs obtained are computed. A height histogram is computed from these reservoirs heights. In handwritten document there exists a variety of character size and many touching may occur because of handwriting style of different individual. As a result connected component analysis can not give proper text line height information. To get proper height information we take the average height (HL) of those reservoirs whose heights lie in the right half of the height histogram. This is done to ignore the small reservoirs in our height computation. This HL gives an idea about the height of a text line and it is very useful to determine different parameters for smoothing algorithm and to decide the structuring element of morphological operation used in our segmentation approach.
Figure 3. A top water reservoir and its different features are shown. Water reservoir is marked by grey shade.
4. Proposed Line Detection Algorithm We have used binary image for our work and to convert the original grey-level document images into binary image, we have applied the algorithm due to Otsu [7]. Binary image may contain some small components and we have removed such small components for line segmentation. The original image and the resultant binary image are shown in Fig.4(a) and 4(b).
4.1.
Foreground Smoothing
RLSA algorithm links together neighboring black/white areas that are separated by less than a distance (which represents the smoothing threshold). In other word, it replaces a sequence of background pixels between two object pixels by object pixel value in a specified direction. Normally, it is used to fill the background pixel-run of length less than a certain pre-defined threshold in horizontal or vertical direction. In our approach, this method is applied only in horizontal direction, i.e. row-byrow smoothing is done. The threshold for the smoothing is given as 2.5*HL in our experiment. The smoothed image is shown in Fig.4(c). From the figure it can be noted that middle part of the text line are mostly black because of RLSA. Sometimes because of overlapping of text from two consecutive lines, two or more components may touch vertically due to smoothing. Based on the horizontal histogram of each smoothed component we detect such touching. When two smoothed components of two different text lines touch then we generally get a peak and valley shape. We analyze the histogram to find possible valleys between peaks. Generally, the peaks are obtained from parts of different text lines that touch. If a valid valley is found in between two peaks then the valley region is marked as possible touching area. Now, we analyze a touching area to detect whether such touching is formed due to RLSA or not. To do that we consider the corresponding area of the initial image (before RLSA is done) and trace its contour. Starting from the top most point of the considered area of image if we can reach its bottom most point by the tracing, then we conclude that the touching was formed before RLSA. To segment such touching we analyze the contour points of that area and based on the structural shape of the contour the touching point is detected. Using angular information of contour point and run-length information we segment a touching. If the touching is formed due to RLSA then we replace that touching area by corresponding area of initial image.
4.2.
in morphological image processing from which all other morphological operations are based. For details about this see [12]. After RLSA, we will have a smoothed image, where the foreground part belongs to black text regions and background part consists of white regions. By erosion, we determine some important information from foreground and background portion which are very helpful in our line segmentation purpose.
Morphological Operation for Foreground and Background Information Extraction
We have used morphological operations, mainly, erosion to extract the useful foreground and background information. Erosion is one of two fundamental operations
(a)
(b)
(c) Figure 4. (a) Example of Bangla handwritten document image. (b) Binary Image after Pre-processing of (a). (c) Horizontal RLSA result of (b).
4.2.1. Background Information Extraction The background region of run-length smoothed image is eroded to extract some obstacle lines. These obstacle lines will act as “Separator Lines” (SL) between two consecutive text lines. The shape of the structuring element for erosion is chosen as rectangular and its height and width are 0.5*HL and 5*HL, respectively. These threshold values are determined from the experiment. The anchor point is set at the centre of the structuring element. Background eroded image of Fig.4(c) is shown in Fig.5(a). From each of the eroded components of background region, we take the upper and lower profiles in each column and we compute the middle point from these profiles in each column. A line fitting algorithm of these mid-points is used and the resultant line is the ‘SL’. Different SLs obtained from Fig.5(a) are shown in Fig.5(b). The left and right end points of a SL are extended horizontally in both directions till (i) it touches the
foreground parts or (ii) it touches the left or right profiles of smoothed foreground image or (iii) it finds another SL within a vertical distance (HL). These extended SL lines of Fig.5(b) are shown in Fig.5(c). Note that, we have computed the left and right profiles of the smoothed image and the profile information has been used for line extraction.
(a)
RLSA as discussed in Section 4.1. To detect the FSC portion coming from such touching part, we scan FSC region column wise. The columns where height of FSC is bigger than HL are removed, so that touching portions will not affect our line segmentation method. In some cases, a very small FSC may be obtained. We delete these small FSCs also for better line segmentation. Generally, FSC should lie on text line, but because of touching and modified characters, some FSC may appear on the portions between two text lines. To take care of such situations the FSC components which touch a SL are also removed. Remaining FSCs of R are used for line extraction and we call such FSCs of an image as the candidate FSC. The candidate FSC components, obtained from the image given in Fig.4(a) are shown in Fig.6. We use these candidate FSCs for line segmentation purpose.
(b)
Figure 6. Foreground seed components are shown by black regions.
4.3.
(c) Figure 5. (a) The Eroded portions of background are shown by black region (b) Separator Line (SL) obtained by joining the mid-points of upper and lower profiles of eroded components. (c) Extended SLs are shown along with text image.
4.2.2.
Foreground Information Extraction
Morphological erosion has also been applied in the foreground part of the smoothed image to extract foreground seed component (FSC). By a FSC, we mean an isolated eroded component obtained from smoothed foreground part. These FSC components are generally the representative of word components in the document. The structuring element is chosen for foreground erosion as rectangular in shape with height 0.5*HL and width 0.65*HL. The anchor point is set at the centre of the structuring element. Let, R be a set of all FCSs of an image. In handwritten documents, the text lines sometimes touch or overlap each other because of the modified characters of scripts as well as for ascending and descending parts of characters. As a result, the smoothed text lines also touch and sometimes we may get a big FSC although we delete some touching
Line Segmentation
By joining of candidate FSC components we will get the segmented lines. The SLs guide FSC joining to get proper segmentation. For each candidate FSC, we compute two reference points named as “left” (LR) and “right” (RR) reference point. LR of a FSC is found by computing centre of gravity of partial set of pixels of the FSC which lie between leftmost columns to a column upto a width (HL) from leftmost column. RR is calculated similarly from rightmost column of FSC. If the width of a FSC is less than 2* HL, then LR (RR) of a FSC is computed from the left (right) half of the FSC. The left and right reference points of the candidate FSCs of Fig.6 are shown in Fig.7. The two reference points of each FSC are marked by black points on the FSC.
Figure 7. Left and right reference points are shown by black dots on each FSC.
Let A and B be two candidate FSC components. For joining B with A, we compute a searching zone (as shown
in Fig.8) from the right reference point (RR) of A. The length (L) of the searching zone is taken as 10*HL and the width (W) of the searching zone is determined as 1.5*HL. We do not consider full rectangular area at the beginning part of the searching zone. Beginning part is upto a length HL from RR and the searching zone is triangular in shape in this part. Rest of the searching part of length 9*HL is rectangular in shape. This is done so that two vertical candidate FSCs will not be joined. See Fig.8, where searching zone of RR of ‘A’ is shown by hatched lines. We will join B with A, if it satisfies the following conditions: (i) The LR of B lies in the searching zone of RR of A. (ii) The searching zone of RR of A will not cross any SL.
Figure 8. Searching zone of the right reference point ‘A’ is shown here.
In the first condition, we consider the foreground positional information of FSC and this is done in terms of component overlapping. In the second condition we utilize SLs which are based on background information. If the above conditions are satisfied, we join two components A and B. As soon as B is joined to A, we try to join other FSC which is nearer to B and satisfy the above two conditions for B. In this way all possible FSC of a line are joined. After joining of the FSC we extend the LR point of the leftmost FSC of the joined FSC towards left and RR point of the rightmost FSC towards right. During this extension, if we can reach border of the image, then a text line is detected. In this way, FSCs of individual lines will be clustered by joining them through a line segment to get individual text line. To get the characters of a text line, we collect all the RLSA components which pass through this line and character portions of these RLSAs in the segmented line. Sometimes, some small RLSAs may not touch this line. We cluster such RLSA components to their nearest line. If in a single line, two or more line segments are obtained due to longer distance between two FSCs, we group them together in a single line by checking the positional relationship of the leftmost and rightmost points of these line segments along with the information of gap from the line lie above it. If the line-gap is similar for these line segments from a segmented line lies above them, we group these line segments into one line. By line-gap, we mean the distance between two consecutive lines, obtained after FSC joining. Line-gap between first two lines of Fig.9 is shown by arrow. Note that the joining of FSC is done in top-to-bottom fashion starting from topmost line. Line joining result of Fig.7 is shown in Fig.9.
Figure 9. Result of FSC joinings.
5. Experimental Results and Discussions For experiment, 125 handwritten document images were considered from individual of different professions. These documents are collected from different persons. These data are collected from five different languages: Bangla, Devnagari, Oriya, Gujarati and English. We noted that these dataset contain varieties of writing styles. For the experiment we considered single column document pages. To check whether a text line is detected correctly or not we draw a marker on the text line after its detection. By viewing the results on the computer display we calculate the line detection accuracy manually. From the experiment we have found that on an average 92.68% of cases our system can detect the text line properly. Line detection results of our proposed scheme are shown on different scripts in Fig.10. We considered 520, 195, 406, 210 and 115 text lines of Bangla, Devnagari, English, Oriya and Gujarati scripts and we noted that 94.23%, 93.33%, 96.06%, 94.76% and 93.04% of text lines are correctly identified, respectively, from our experiments. For the experiment of English document, we also considered some images from IAM database [5] as well as some images from examination answer sheets. From our experiment we noted that most of the errors occur when the two consecutive text lines are very near to each other. Our system detects all the lines of a document accurately when a gap is present between consecutive lines. The SLs are very useful in preventing joining of a FSC with that of next line which is very near. If there is a missing of SL and the FSC candidates of two consecutive lines are too close then an erroneous result occurs and this is the main drawback of our approach. Our method computes the global text-size and the line detection thresholds are dependent on text-line height. If there exist multi-size text lines in a single document, this method needs to be modified for detecting the size of structuring element. Layout understanding methodologies may help detecting different uniform text zones and different text line size can be computed from those individual zones. However, we noted that hand-written size variation in a document page of a single writer is very rare in our documents. There is not much work on handwritten text line extraction on Indian languages. Recently Basu et al. [1] proposed a method for text line extraction and they
obtained 90.34% accuracy on Bangla script and 91.44% on English script. We obtained overall 92.68% accuracy from our proposed approach when tested on five different scripts.
6. Conclusion In this paper, a script independent line segmentation algorithm is developed from off-line unconstrained handwritten documents. The proposed scheme is developed based on morphological operation on the foreground and background portion of the document. We tested our scheme on different scripts like Bangla, Devnagari, Oriya, Gujarati, English etc. and obtained encouraging results.
7. References [1] S. Basu , C. Chaudhuri , M. Kundu , M. Nasipuri and D. K.
[2] [3] (a)
[4] [5] [6] [7] [8] (b)
[9] [10] [11]
(c)
[12] [13]
[14]
(d) Figure 10. Results of line segmentation are shown on (a) Bangla (b) Gujarati (c) Oriya (d) English text.
[15] [16]
Basu, Text line extraction from multi-skewed handwritten documents, Pattern Recognition, vol. 40(6), 2007, June, pp.1825-1839. B. Chaudhuri and U. Pal, “Skew angle detection of digitized Indian Script documents”, IEEE PAMI, vol. 19, 1997, pp.182-186. Y. Li, Y. Zheng, D. Doermann and S. Jaeger, “A new algorithm for detecting text line in handwritten documents”, In Proc. 10th IWFHR, 2006, pp.35-40. J. Liang, I. Philips and R. M. Haralick,“A statistically based highly accurate text-line segmentation method”, In Proc. 5th ICDAR, 1999, pp.551-554. U. Marti and H. Bunke, “A full English sentence database for off-line handwriting recognition”. In Proc. 5th ICDAR, 1999, pp. 705 - 708. S. Nicolas, Y. Kessentini, T. Paquet and L. Heutte, “Handwritten Document using Hidden Markov Random Fields”, In Proc. 8 th ICDAR, 2005, pp.212-216. N. Otsu, A Threshold selection method from grey level histogram, IEEE Trans on SMC, vol.9, pp.62-66, 1979. U. Pal, A. Belaïd and C. Choisy “Touching numeral segmentation using water reservoir concept” Pattern Recognition Letters, vol.24, 2003, pp. 261-272. U. Pal and B. B. Chaudhuri, “Indian script character recognition: A Survey”, Pattern Recognition, vol. 37, 2004, pp. 1887-1899. U. Pal and S.Datta, “Segmentation of Bangla Unconstrained Handwritten Text”, Proc. 7th ICDAR, 2003, pp.1128-1132. U. Pal and P. P. Roy, Multioriented and curved text lines extraction from Indian documents. IEEE Trans. on SMC. Part B. vol.34, 2004, pp.1676-1684. J. Serra, Image Analysis and Mathematical Morphology. Academic Press, London, 1982. S. Tsuruoka, Y. Adachi and T. Yoshikawa, “The Segmentation of a Text line for a Handwritten Unconstrained Document using Thinning Algorithm”, In Proc. of 7th IWFHR, 2000, pp. 505-510. W. Xiaoying and C. G. Leedham,“Seperating lines and words in unconstrained handwriting”, In Proc. 8 th IGS, 1997, pp. 117-118. B. Yanikoglu and P. A. Sandon, “Segmentation of off-line cursive handwriting using linear programming”, Pattern Recognition, vol.31, 1998, pp. 1825–1833. A. Zahour, B. Taconet, P. Mercy and S. Ramdane, “Arabic hand-written text-line extraction”, In Proc. 6th ICDAR, 2001, pp. 281-285.
