An enhanced handwritten signature verification model ...

3 downloads 0 Views 3MB Size Report
Apr 12, 2017 - The signature verification process is the distinction of genuine signatures of the signer and forged ... the window is enlarged as the classification accuracy of the genuine to forged signature increases ...... The first advantage.
International Journal of Image and Data Fusion

ISSN: 1947-9832 (Print) 1947-9824 (Online) Journal homepage: http://www.tandfonline.com/loi/tidf20

An enhanced handwritten signature verification model applied on off-line benchmark data sets Mostafa A. Salama & Walid B. Hussein To cite this article: Mostafa A. Salama & Walid B. Hussein (2017) An enhanced handwritten signature verification model applied on off-line benchmark data sets, International Journal of Image and Data Fusion, 8:4, 332-353 To link to this article: https://doi.org/10.1080/19479832.2017.1315542

Published online: 12 Apr 2017.

Submit your article to this journal

Article views: 12

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=tidf20 Download by: [197.51.99.27]

Date: 20 November 2017, At: 13:58

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION, 2017 VOL. 8, NO. 4, 332–353 https://doi.org/10.1080/19479832.2017.1315542

RESEARCH ARTICLE

An enhanced handwritten signature verification model applied on off-line benchmark data sets Mostafa A. Salama

and Walid B. Hussein

Downloaded by [197.51.99.27] at 13:58 20 November 2017

Department of Computer Science, British University in Egypt, Cairo, Egypt ABSTRACT

ARTICLE HISTORY

The automatic, highly accurate and instantaneous authentication of a static off-line handwritten signature is highly required in financial, legal and governmental sectors. The genuine signature features are extracted by models like scale-invariant feature transform (SIFT) and speeded-up robust features (SURF), then the common patterns among these features are recognised for verifying new signatures. Although SIFT/SURF-based matching models are commonly used because of their robustness and low computational complexity, their tolerance to the unconstrained signature styles of large variations is not very satisfactory. This work presents three main drawbacks of the traditional application of this model in signature verification and proposes a set of modifications to customise this model to fit the nature of the handwriting domain. These modifications consider initially the limited size of the descriptor window in SIFT/SURF model, then consider the single point-to-point and the crisp matching of the points-of-interest in the tested signature pairs. The experimental work is applied on two benchmark data sets that contain a set of genuine and skilled forged signatures of multiple users. Furthermore, a comparative analysis is applied to show the reflection of each modification in enhancing the accuracy percentage of the signature verification.

Received 6 February 2017 Accepted 31 March 2017 KEYWORDS

Signature authentication; handwritten signature; feature extraction; similarity detection

1. Introduction For many decades, behavioural biometrics such as handwritten signature are the main and simpler tool of personal identification rather than those related to biological biometrics, like fingerprint retina scanning, and DNA analysis (Hafemann et al. 2015). Moreover, this method is unlike traditional techniques that are based on keycards and passwords which may be lost, stolen or even forgotten (Jain and Patil 2014). Also the process of collecting signatures is non-invasive and does not require hardware of high cost. Handwritten signature contains complex geometric patterns/curves representing the identity of a person. In general, the signature is a behavioural biometric that records the physical traits of the ballistic movements of the signer, therefore it is a unique identity for each individual (Taneja and Kaur 2015). The verification of this kind of signatures is very important in business and financial transactions, bank checks and CONTACT Mostafa A. Salama in Egypt, Cairo, Egypt

[email protected]

© 2017 Informa UK Limited, trading as Taylor & Francis Group

Department of Computer Science, British University

Downloaded by [197.51.99.27] at 13:58 20 November 2017

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

333

contracts (Dhaka et al. 2009, Biradar and Panchal 2015), forensic science studies (Chambers et al. 2015), and legal document verification and authentication (Warasart and Kuacharoen 2012). The signature verification process is the distinction of genuine signatures of the signer and forged signatures. The signature forgery is categorised into three types which are random, simple and skilled forgery. The first two types do not need talented or practised forger, while the skilled forgery requires a careful practice by the forger to replicate the genuine signature (Gupta 2014). A common method for fraud prevention is the visual signature verification by the eye of an expert. Although this visual inspection approach is widely used in most of the daily life signatures contained operations, it lacks essential criteria for accuracy and security. This is due to its dependency on many human factors like the experience, the mood and even the working conditions of the verifier. Meanwhile, forensic question document examiners apply manual authentication of a high number of documents, this process is considered as a time- and a manpowerconsuming process (Halder et al. 2014). Also, it is becoming challenging for the verifier to differentiate the complex intra-personal variations, which are the differences between signatures of one person in different sessions, and other fraud variations (Reddy et al. 2006). During the last decade, different feature extraction and classification methods have been developed in the signature verification process. Lately, scale-invariant feature transform (SIFT) (Lowe 1999, Ruiz-del-Solar et al. 2008) and speeded-up robust features (SURF) (Bay et al. 2008, Malik et al. 2014) methods are utilised in signature verification to detect the corners/points-of-interest in the handwritten text (Kaur and Choudhary 2015a). The features extracted from the SURF and SIFT methods are robust to noise and insensitive to the variation in scale and rotation of the signature and also consider the lines and corners existing in the signature (RuizShulcloper and Kropatsch 2008). The utilisation of the SIFT/SURF-based methods in off-line handwritten signature verification can be summarised as follows: these methods initially detect a set of points-of-interest in the signature, then build a window around each point-of-interest. The orientation and magnitude of each point in this constructed window represent a set of features that describe the point-of-interest. Then, the features of a genuine signature are compared to those of a tested signature to detect the degree of matching between both signatures. The used matching method, i.e. Euclidean distance method, searches for each feature in the genuine signature for the best match (of minimum distance) that exists in the tested signature. Two points-of-interest are matched if the distance between them is the minimum and if it satisfies a user-defined distance threshold. Finally, the tested signature is classified as genuine if the number of matches exceeds specific userdefined matching threshold. However, these are three main drawbacks in the current utilisation of the SIFT/SURF methods in off-line handwritten signature verification. These problems appear because the current application of the feature extraction and matching algorithms ignore the nature of the signatures as a set of simple lines and treat them as complex images. ● The first problem in this verification process is that the window describing each

detected point-of-interest is fixed to a specific size. This fixed size window may not

334

M. A. SALAMA AND W. B. HUSSEIN

Downloaded by [197.51.99.27] at 13:58 20 November 2017

be sufficient to include all the required information that uniquely identifies this point-of-interest in the signature. ● The second problem is ignoring the correlation between the neighbour points-ofinterest in the matching between two signatures, where the traditional matching algorithm searches for the highest point-to-point match between the points-ofinterest in the template and tested signatures. For example, three points-of-interest on the same line can distinctly characterise the text representing this line rather than a single point-of-interest. ● The third problem is the crisp matching between signatures, where two signatures are considered matched if the counted number of point-to-point matches between the two signatures exceeds a user-defined number. Also the assigned user-defined threshold is not the same for every signer, as it must be dependent on the degree of the existing intra-personal variations among the genuine signatures of every signer. This work proposes a modified version of the SIFT/SURF feature extraction and matching algorithms for enhancing the signature verification accuracy. These modifications customise the verification algorithms to be more convenient for the domain of signature images. ● Adjusting the descriptor window size of SIFT/SURF method: the first enhancement

is the variation of the size of the window around the point-of-interest to include more details needed to differentiate between this point in a template genuine signature and a corresponding point in a tested signature. The results show that as the window is enlarged as the classification accuracy of the genuine to forged signature increases until a peak, then the accuracy starts to decrease again. ● Replacing the single-point matching by multi-point matching: the second modification is applied in the matching between two signatures, where instead of applying point-to-point matching between these signatures, every three neighbour points-ofinterest are used together in the matching. This enhancement takes into account the nature of the handwritten signature, where three neighbour points would include more complementary and discriminative information rather one point. ● Fuzzifying the match between signatures: finally, the third modification replaces the crisp classification of signature by a more fuzzy approach. Initially, for each group of three neighbour points in the template signature, the shortest Euclidian distance between this group and a corresponding group in the test signature is calculated. Then, the average value of the calculated distances corresponding to every group of point in the template signature is detected; this value represents the distance between the two signatures. This enhancement removes the dependency on user-defined threshold. The results of these modifications show clear enhancement in the classification accuracy results, between 95% and 100% accuracy results, with respect to the traditional process of verification. The work applied here is based on the database of off-line genuine and skilled forged signatures extracted in Galbally et al. (2015). The rest of this paper is organised as follows: Section 2 presents the previous work of applying machine learning here. Section 3 describes the problems existing in the current

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

335

application of SIFT/SURF methods in signature verification and proposed enhancements. The experimental work and discussion appear in Section 4 and finally the conclusion is in Section 5.

Downloaded by [197.51.99.27] at 13:58 20 November 2017

2. Literature review Image-processing and pattern recognition techniques are applied for reaching handsfree, low-cost and higher accuracy verification and authentication of signatures. The first step in signature verification is the extraction of the important features characterising the signature. Image-processing techniques are employed to extract the features of genuine signatures, which are divided into two types: off-line and on-line features. The on-line features extraction requires touch screen instruments for generating dynamic features like pen pressure, speed of signature or time (Liwicki et al. 2011). While, the off-line features extraction requires a scanned two-dimensional image of the signature. Although the on-line verification system is of high accuracy, off-line system is more reliable in many real-time situations. For example, off-line systems are highly applicable in the cases of signed documents and checks. And this kind of system provides a cost reduction in the applied security system rather than the special instruments requirement for on-line signature verification. Also, it is more user interactive and less invasive system (Ramadas and Geethu 2015). These reasons increase the challenge against the proposal of a competing off-line signature features extraction and verification methods. The extracted features in off-line signature verification methods can be partitioned into three categories which are global, local and mask features (Rathi et al. 2012). The global features represent the entire structure of the signature like the height, width and height-to-width ratio features. The local features focus on the signature detail analysis by applying a grid segmentation on the image of the signature. The mask features present information about the curves in the signature to consider their angles and intersections (Viriri 2014). However, the implementation of global features extraction methods provides limited information for verification because they are sensitive to the variations in the signature style. And local features are difficult to be used in off-line signature verification (Abikoye et al. 2011). Moreover, both global and local feature ignore angles, lines and curvatures in the signatures (Neamah et al. 2014, Sayantan and Sushila 2014). Meanwhile, the mask features are highly related to the location and direction of the principal axes of the genuine direction (Soran et al. 2012). Therefore, if the intra-personal variations are quite significant in the signer signatures, then the accuracy of the method degrades. On the other hand, grid and directional features cannot capture the difference between two same objects with slight variations (Zhu et al. 2013). In general, global, local and mask features ignore the variation of the genuine signatures of the same user that includes the size, the position and the rotation of the signature. The study in Shah et al. (2015) presents approaches for the extraction of local features, after partitioning the image into virtual zones. These approaches are modified direction features (MDFs), angular features, Gabor features and pseudo-dynamic features. MDF represents the transition from foreground to background of the signature in each image’s zone. The angular features are extracted by slicing the image into halves according to the geometric centre of the darkest pixels in the images. Gabor features are the Gabor

Downloaded by [197.51.99.27] at 13:58 20 November 2017

336

M. A. SALAMA AND W. B. HUSSEIN

coefficients calculated on each image’s zone. Although the local features bring piecewise information of the signature details rather than those by global features, they are quite sensitive to any variation in the scale and orientation of the signature. The SIFT/ SURF methods are proposed as a solution for this kind of variations in the handwritten domain. These methods have been used in the detection of plagiarism in handwritten assignments delivered by the students (Celar et al. 2015). Also, these methods are applied in the verification of user signatures (Adeyemo and Abiodun 2015). The second step in signature verification is the detection of the forged signatures based on the extracted features. The machine-learning methods proposed for this step can be categorised into simple and complex methods. The example of simple methods is the distance measures, and the example of complex methods is the classification techniques (Vivaracho-Pascual et al. 2015). Distance measures like Euclidian distance and Gaussian empirical rule use only genuine signatures in the training to define a specific threshold for classifying unknown tested signatures. These models match a genuine signature to the tested signatures and accept those higher than the defined threshold as genuine one. More complex machine-learning classification techniques like support vector machines (SVM), neural networks (NN) and Hidden Markov Model (HMM) models are applied to reach a higher level of accuracy (Karouni et al. 2011, Das and Roy 2015, Hatkar et al. 2015, Ooi et al. 2016). These techniques use the genuine and forged signatures in the training of the learning algorithm. Accordingly, the use of a simple classification model like Euclidean distance should be more convenient in the domain of signature verification. The clear reason for this conclusion is that the skilled forged signatures should be unknown in the training phase of the algorithm. Where every day, a new skilled forger can try to replicate the original signature, so distance methods can appear as a general solution for all types of forgeries.

3. Material and method 3.1. SIFT/SURF feature extraction This section shows the current feature extraction approaches in SIFT/SURF techniques. Both techniques are based on the same principles, but with two different algorithms. Although the results from both techniques have a high degree of consistency, SURF reduces greatly the computational cost of the SIFT method. The two techniques depend on scale space representation and contain mainly two steps. The first step is the detection of the signature’s points-of-interest. The second step is utilising these points to extract the dominant features of the signatures.

3.1.1. SIFT feature extraction SIFT technique generates four images (i.e. octaves) of the original image {I1, I2, I3 and I4}, where Ic is the intensity matrix of the c octave. Each octave of these images is a 50% sized-down version of the preceding one. This means that the sizes {I2, I3 and I4} are of 50% of the sizes of {I1, I2 and I3}, respectively, as shown in Figure 1. While, f1 has the same size of the original image.

Downloaded by [197.51.99.27] at 13:58 20 November 2017

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

337

Figure 1. Four octaves of the original signature’s image.

The Gaussian blurring filter G with filter scale σn = kn * σn-1 is convoluted by each octave, for five different filter scales, n = the values {0, 1, 2, 3 and 4}, where the initial scale σ0 is equal to 1.6, and k is a predefined constant value equals √2. This step forms a pyramid space of images with different sizes and scales. Lðp; kn  σn Þ ¼ Gðp; kn  σn Þ:IðpÞ:

(1)

Then the difference of Gaussian (DoG) between each two successive layers is applied in each octave as shown in Equation (2). DoGðpÞ ¼ Lðp; kn  σn Þ  Lðp; kn1  σ n1 Þ:

(2)

The potential points-of-interest are the local extrema centred in a 3x3x3 neighbourhood window. The final points-of-interest are those exceeding a specific threshold and belong to those points extracted from applying the Harries corner detection algorithm on the original image. The orientation and magnitude of each point in the image are calculated based on the Gaussian blurring of the original image at different scales. Then a descriptor window of size 16x16 pixels enclosing each point-of-interest is partitioned into sixteen 4x4 subregions. For each sub-region, the orientations of the points involved in this sub-region form a histogram of 8 dominant orientations (0, 45, 90, 135, 180, 225, 270 or 315). As a result, a feature vector of 128 values (i.e. 16 sub-regions * 8 orientation bins) is extracted in the descriptor 16x16 window.

3.1.2. SURF feature extraction The SURF feature extraction method starts by applying the Hessian matrix filter H of size 9x9 (s0 = 9) and scale σ0 = 1.2 on the original image I. Afterwards, Hessian matrices of different sizes and different scales σn = sn * (σn-1/sn-1), 15x15 (s1 = 15), 21x21 (s1 = 21) and 27x27 (s1 = 27) are applied on I. Dealing with different sized filters reduces the computational complexity exists in applying the same sized filter on different sizes of

338

M. A. SALAMA AND W. B. HUSSEIN

the image, as in the SURF method. Hessian matrix expresses the local changes in the area of each point p in x and y directions. It is based on the second-order derivative of Gaussian filter as shown in Equation (3).   L ðp; σ n Þ Lxy ðp; σn Þ Hðp; σn Þ ¼ xx ; (3) Lyx ðp; σn Þ Lyy ðp; σ n Þ where Lii(p, σn) is the convolution of the image with the second-order derivative of the Gaussian filter in ith as shown in Equation (4).

Downloaded by [197.51.99.27] at 13:58 20 November 2017

Lii ðp; σn Þ ¼ IðpÞ  δðσn Þ=δði2 Þ:

(4)

The next step is to extract the local maxima of the Hessian matrix in the 3x3x3 neighbourhood to explore the points-of-interest. The weighted Haar-wavelet responses (dx and dy) in x and y directions are calculated for each point-of-interest in a circular neighbourhood of radius 6*σ centred around the point-of-interest of corresponding scale σ. Like SIFT method, a descriptor window of size 16x16 around each point-of-interest is partitioned into sixteen 4x4 sub-regions. For each sub-region, the π/3 zone surrounding the dominant orientations is considered, where the responses orientation (dx and dy) of most of the points in this zone contained in each sub-region. This further saves the processing cost of the 8orientations calculations in the SIFT method. Finally, the responses orientation (dx and dy) and responses magnitude (|dx| and |dy|) of the points in the selected π/3 zone are gathered to form four values pattern for each 4x4 sub-region. Therefore, a feature vector of 64 values is extracted to represent the 4 calculated values for each sub-region in the 16x16 descriptor window.

3.2. Drawbacks of the current SURF/SIFT features extraction and matching models 3.2.1. Descriptor window drawback The first problem is the fixed 16x16 descriptor window around each point-of-interest. In any domain other than signature verification, the details around any interest are high such that a window of this size is sufficient to include all the required information. While in the handwritten signatures of random curves, characters and even words, a larger sized window may provide a better description of the point-of-interest. For example, Figure 2 shows a feature in a fingerprint image, ff, and a feature in a signature, fs. The window surrounding the ff holds many details than the one surrounding fs. In the fingerprint image, the window shows different changes appear around the point-ofinterest. While, the window in the signature image holds only the edge of a small portion in the first character. The simplicity of the signature nature leads to a wide distribution of the key points and lowers the features characterising each one. 3.2.2. One-one matching drawback The second problem is that the matching between features is applied as a one to one (1-1) matching. Each point in one signature is compared to each point in the other signatures, and the point that shows minimum distance is considered as the match.

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

339

Downloaded by [197.51.99.27] at 13:58 20 November 2017

Figure 2. A feature in a fingerprint image and a feature in a signature image.

This methodology leads to two problems, the first one is the dependency on a userdefined threshold that varies according to the signer, the scanning conditions and the experience of the signature verifier. The second problem is that the (1-1) matching is not a fully sufficient solution. Such that if two successive points in one signature, preferred near each other, are matched together to two points in the other signature, then the matching will include the correlation between the two points in the matching.

3.2.3. Crisp matching drawback The third problem is the matching process between the features of two signatures. In this process, the extracted SIFT/SURF – features/descriptors of each point-of-interest are matched to those of the point-of-interest in another signature. The accuracy of these methods depends on counting the number of matches between each pair of signatures. Usually, the verification method is successful when the number of matches between genuine–genuine pair is higher than that of the genuine–forged pair. The dependency on the number of matches is not considered as the best solution for handwritten signatures because of two main reasons. First of all, the matching of the SIFT/SURF points-of-interest of the same object in two different images work well even if these two objects have different size, angle and orientation. In the case of signatures signed by the same user, the features of points-of-interest in any two signatures are not identical even for the same portions. For example, the way of writing the character ‘M’ in two genuine signatures for the same user may vary according to different writing conditions. Secondly, the number of matched points-ofinterest between two signatures may be high, but the relative number of incorrect matches is also high. Moreover, the number of points-of-interest in the two images and difference between them is not considered. For example, in the matching between the two signatures in Figure 3, there are only 4 correct matches out of 12 matches. Accordingly, the classification accuracy that is based on only the number of matches may show misleading results in many cases. These problems deteriorate the accuracy of classifying genuine and forged signatures correctly. The work proposed here uses a different approach in extracting and utilising the SIFT/SURF features.

Downloaded by [197.51.99.27] at 13:58 20 November 2017

340

M. A. SALAMA AND W. B. HUSSEIN

Figure 3. Matching of features between two genuine signature images.

3.3. The proposed SIFT/SURF features extraction and matching model The model proposed here introduces a modification in the process of extracting the SIFT/SURF features of the signature image. The other two modifications are applied on the way of matching between two signatures images. This model is applied in order to customise the traditional signature verification model based on SIFT/SURF features to fit the nature of handwritten signatures.

3.3.1. Enhancement of the extracted SIFT/SURF features The first part of the proposed model is varying the fixed size 16x16 grid window describing a point-of-interest in the image in SIFT/SURF methods. The SIFT/SURF features are usually extracted from images of real-life objects of high details, so clearly a 16x16 grid is sufficient. While for a signature, the case is different where the corner in a line/curve describing a character could appear in a larger window than just an 8x8 grid window. Although the SIFT/SURF methods are size-invariant methods, varying the size of the image will not solve this problem. This is because decreasing the image size will diminish or hide many points of interest that exists in the original image. And so decreases the opportunity to detect these points that could represent the identity of the users’ signatures. That is to say that the main cause of the problem is not the size, but the simplicity of the signature, and decreasing the size will cause the loss of the signature details than decreasing the simplicity of the image. A parametric study applied to the SIFT method in this work as a proof of concept based on varying the size of the descriptor window. SIFT method uses Lowe’s Keypoint Descriptor of 16x16 grid window, and each 4x4 sub-grid is represented by 8 orientation bins. Figure 4 shows Lowe’s Descriptor of 8x8 grid window. Variable sized windows are tested against an image of a specific size as shown in Figure 5. The sizes tested are 8x8, 16x16, 32x32, 64x64 and 128x128 grids to check out the classification accuracy for each size. This enhancement applied to the last step in the SIFT/SURF method as shown in Figure 6.

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

Downloaded by [197.51.99.27] at 13:58 20 November 2017

Figure 4. Lowe’s Keypoint Descriptor (with 2x2 descriptors over 8x8).

Figure 5. Variable sized windows surrounding a feature are considered.

Figure 6. The SIFT/SURF method enhancement.

341

Downloaded by [197.51.99.27] at 13:58 20 November 2017

342

M. A. SALAMA AND W. B. HUSSEIN

3.3.2. Multi-point matching The modification in this step is applied in the matching process by changing the singlepoint to multi-point matching. If a single point-of-interest in the image holds a piece of information, then two successive points-of-interest near each other must hold more and complementary information. In other words, two or more neighbour points-of-interest could describe the patterns of a handwritten character better than a single point. For example, the ‘M’ character in Figure 7 in one signature that is characterised by three points-of-points of interest is matched to three points-of-interest in the second signature. Accordingly, another matching method is applied rather than the existing 1-1 matching between features. This matching is referred to as the 2-2 matching and 3-3 matching as shown in Figure 8. For example, in the 2-2 matching method, for each point-ofinterest fg1 in the reference signature G, the next successive neighbour point fg2 is detected. The reference or template signature is selected as the genuine signature that has the lowest difference, the highest similarity, to all other genuine signatures. The detection of the successive points is on based the construction a chain of points, such that each point is the nearest (2D Euclidian distance) to the next point in the chain. The 2-2 matching process is applied on the first and second points in the chain, then on the second and third points in the chain and continues until all points-of-interest are considered. The method searches for two successive points-of-interest (fs1, fs2) in the

Figure 7. 3-3 multi-point matching between two signatures.

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

343

Downloaded by [197.51.99.27] at 13:58 20 November 2017

Figure 8. Replacement of the one-to-one matching enhancement.

tested signature S. This pair of points (fs1, fs2) has the minimum Euclidean distance Em to the successive pair of points (fg1, fg2) in the template signature G as shown in Equation (5). This equation calculates the absolute difference between the descriptors of a pair of points in G and the descriptors of the corresponding pair of points in G. ! X         Em fg1 ; fg2 ¼ min"ðf ;f Þ2S dg1  ds1 jþj dg2  ds2 ; (5) i

s1 s2

i

i

i

i¼1...n

where the parameters of this equation are as follows: ● dg1, dg2, ds1 and ds2 are the descriptors of the points-of-interest fg1, fg2, fs1 and fs2,

respectively. ● n is the number of descriptors (features describing each point). The value of n is

128 in the SIFT method and the value of n is 64 in the SURF method. ● The sign ‘|’ refers to the absolute value of the difference between the descriptor of

one point and the descriptor of the other point. ● For the pair of successive points (fg1, fg2) in G, min"ðfs1 ;fs2 Þ2S method calculates the

distance values for every pair of successive points (fs1, fs2) in S, and returns the distance of the minimum value. For example, finding the minimum match between pair of neighbour points-ofinterest in two signatures should reflect the in-between similarity better than a match between only one point-interest in these signatures.

3.3.3. Fuzzy matching The final issue to handle here is the method of ranking signatures based on the training on a set of genuine signatures. The ranking is based on measuring the match between a genuine signature and a tested signature. One of the matching methods between two different signatures is based on measuring the Euclidean distance between points-of-interest in both signatures. Every point in a signature is tested against all the points in the reference signature, and the point with the minimum distance is selected under a condition that the measured distance is not above a specific threshold. Some points-of-interest do not have a match according to the predefined threshold. On the other hand, some points are

Downloaded by [197.51.99.27] at 13:58 20 November 2017

344

M. A. SALAMA AND W. B. HUSSEIN

not matched correctly or not considered as consistent points. The match between signatures is based on counting the number of all correct and incorrect matches under a specific threshold. In this case, the correct match is counted as one and the incorrect match is counted as zero. In this work, the average value of the Euclidean distance points in all matches is considered as the matching value. In this case, the matching is converted from a crisp method based on a specific number to a fuzzy method based on a distance value as shown in Figure 9. This distance value is the summation of the value of the Euclidean distances between points-of-interest in the two matched signatures. This replacement will exclude the dependency on a threshold that may vary from user to another based on his/ her way of writing the signature. Instead, a marginal region is detected in the training phase between the matching values of genuine–genuine and genuine–forged signatures. Accordingly, the distance between two signatures MGS is calculated as shown in Equation (6). MGS is the sum of the distances Em, as calculated in Equation (5), of all pair of successive points (fg1, fg2) in G and their corresponding pair of points in S. X   Em fg1 ; fg2 : (6) MGS ¼ "ðfg1 ;fg2 Þ2G Then MGS value is measured for every pair of signatures in the training set. The values corresponding to genuine–genuine and genuine–forged signatures’ matching are plotted. It is expected that the average value of the MGS values of all the genuine–genuine signature pairs will be lower than that of the genuine–forged signature pairs. Finally, a margin is placed between the each type of pairs to extract a threshold ε. When testing any new signature, a match MGS is calculated between a genuine signature and the unknown tested signature, and, if MGS is above the threshold ε, this signature will be considered as a forged one.

4. Experimental results and discussion The proposed approach is applied on two benchmark data sets extracted in Galbally et al. (2015) and in Kaur and Choudhary (2015b). The forged signatures in these

Figure 9. Enhancement of the ranking method of signatures.

Downloaded by [197.51.99.27] at 13:58 20 November 2017

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

345

data sets are skilled forged signature, which increases the challenge against the proposed system. The experimental work applied for every data set includes the training and the testing phases. In the training phase, the SIFT/SURF features are extracted from a set of five genuine signatures that is considered as the training set. The match between each pair of signatures in the training set is calculated according to Equation (6). The genuine signature having the lowest Euclidean distance, highest match, to the other four genuine signatures is selected as a reference signature. This signature holds most of the common patterns in the genuine signatures. The matching value MRG between the reference signature R and the other genuine signature G is considered as a threshold V above which any signature will be considered as a forged signature. The testing phase will examine a set of signatures against the reference genuine signature in the training set. The signatures to be tested are classified according to the matching value between each one of these signatures and the reference signature. If the matching value MRF between the reference signature R and any of the forged signatures F is higher than the threshold V, then this signature is classified correctly (true–negative tn). The percentage of the verification accuracy is calculated as the percentage ratio between two numbers as shown in Equation (7): Accuracy ¼

tn þ tp tn þ tp ¼ ; nþp tn þ fn þ tp þ fp

(7)

● p is the number of positive instances that represents the genuine signatures. ● n is the number of negative instances or forged signatures, where n = tn+fn. ● tn is the number of forged signatures that are classified correctly. In this proposed

model, tn is the number of forged signatures whose distance to the reference signature (tn instances) is higher than V. ● fn is the number of forged signatures that are not classified correctly. Since that there are no genuine signatures in the testing two data sets, only in the training data set, the number of positive instances are zero, tp = fp = p = 0. Accordingly, the verification accuracy has the formula shown in Equation (8). Accuracy ¼

tn : tn þ fn

(8)

The accuracy percentage of the verification is 100% if all the signatures in the testing set have a matching value to the reference signature higher than the threshold V detected in the training phase, where tn = n and fn = 0. The accuracy of the proposed model is compared to the traditional signature verification model based on the SIFT/SURF features. The results of these comparisons show the high efficiency and the importance of the three proposed modifications in this model.

4.1. Adjusting the descriptor window size of SIFT/SURF method The first step applied here is to enhance the quality of the extracted features from the SIFT/SURF method. This step is based on varying the sizes of the window around the

Downloaded by [197.51.99.27] at 13:58 20 November 2017

346

M. A. SALAMA AND W. B. HUSSEIN

point-of-interest. The standard size of this window in the SIFT/SURF method is 16x16 pixel around the point-of-interest. The hypothesis to be proved here is that this fixed size window is not fitting the nature of all images. Handwritten words, names and signatures are of a nature different from images of objects of high details like faces, trees and any other types of objects. The test is applied to a set of signatures (genuine and forged) of two signers from the benchmark data set extracted in Galbally et al. (2015). The window size around the point-of-interest in the SIFT features applied in this test is 64x64 pixels. The number of genuine signatures in this set is five, and the one with the highest number of common patterns of the other signatures is selected as reference signature R. The charts in Figures 10 and 11 show the average Euclidean distance between the SIFT features of a reference genuine signature R and four genuine signatures G followed by 30 forged signatures F. The first four points in these charts show the matching distance between the remaining four genuine signatures G and the reference signature R, while the rest of points represent the matching between the forged signatures F and the reference signature R. The vertical axis in these charts represents the average Euclidean distance between each two signatures. The horizontal axis represents the signature indexes, starting from four genuine signatures, then the rest of forged signatures. The verification accuracy for the signer one in Figure 10 is 100%, and for signer two in Figure 11 is 92%. It is clear that the four genuine signatures of signer one have values of small difference. And the first three genuine signatures of signer two have similar values where the fourth genuine signature holds a higher variance than the rest of the signer signatures. The average value of the matching distance is considered as the threshold above which the signature is considered as forged. Another comparative study is applied on the signatures of five different signers from a benchmark data set extracted in Kaur (2015). Each signing user signed five times and a skilled forger signed a set of 50 signatures highly similar to the original genuine signatures. The first 5 signatures are used for training, and the 50 forged signatures are classified by SIFT/SURF features extraction and matching. The size of the window used is varied from 8x8 pixels around the points-of-interest to 256x256 pixels. The chart

Figure 10. Verification of genuine/forged signatures for signer one.

Downloaded by [197.51.99.27] at 13:58 20 November 2017

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

347

Figure 11. Verification of genuine/forged signatures for signer two.

in Figure 12 shows the verification accuracy variation of the five signers forged signatures based on the six sizes of the window around each point-of-interest in the SIFT method. Each curve in this chart represents a single signer, the horizontal axis represents the different sizes from 8 to 256 and the vertical axis represents the corresponding verification accuracy. This chart shows that the maximum verification accuracy of the forgery of the signatures for each signer lies between 64x64 and 128x128 window size. These results confirm that the information exists in a window of size 8x8 or 16x16 pixels is not enough to describe the information of the point-of-interest. Furthermore, another chart in Figure 13 is blotted to show how the margin between reference genuine MRG and reference forged MRF matching value increases as the window size increases. The vertical axis in this chart represents the Euclidean-based measure MRS between the reference signature and the rest of the other signatures. Each curve represents variations of this measure for among the various signatures of the same signer for a specific

Figure 12. The high verification accuracy of using larger Low’s window in the SIFT model than the standard window size.

Downloaded by [197.51.99.27] at 13:58 20 November 2017

348

M. A. SALAMA AND W. B. HUSSEIN

Figure 13. The variation of Euclidean-based matching of reference signature to genuine and forged signatures.

window size. The first grey part in horizontal axis represents the genuine signatures, while the rest of the axis represents the forged signatures. To evaluate how the window size influences the discrimination of genuine and forged signature, a test is applied on the MRG and MRF values in Figure 13. For each window size, the difference between the average values of the MRG values and the average of the MRF values is calculated. The results show that this difference increases as the window size increases from 8x8 pixels to 256x256 pixels as shown in Figure 14. This concludes that the discrimination of genuine signatures out of forged signatures increases as the window size increases.

Figure 14. The variation of the difference between the average of MRG and MRF values according to the change in the windows.

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

349

Downloaded by [197.51.99.27] at 13:58 20 November 2017

4.2. Replacing the single-point matching by multi-point matching The second proposed modification is applied in the matching process between the SIFT features of two signatures. The test here is applied on the same set signatures from five different signers as in the previous section. This modification is tested by comparing different number of matching points in verifying the genuine/forged signatures. The test is applied on the five signers starting from single-point matching and ending by fivepoint matching. Such that for each single point-of-interest in the reference signature R, a single point-of-interest in the tested signature S is detected whose Euclidean distance match MRS is the minimum. Then for each two successive/neighbour points-of-interest in R, two successive points-of-interest in S are detected whose average MRS is the minimum. And so on until a match is applied between pair of five successive pointsof-interest in the two signatures. This test is operated for two different window sizes around the points-of-interest, the standard 16x16 pixels window and larger window size of 64x64 pixels as shown in Figures 15 and 16, respectively. Starting by Figure 15 in the case of using 16x16 window, the verification accuracy increases in using multi-point matching in the signatures of the first, third and fourth signers. While in the case of using 64x64 window and five-points matching as shown in Figure 16, the verification accuracy increases to 85% for the fourth signer and to 100% for the fifth signer. The accuracy in verification in the case of single- or multi-point matching is dependent on the behavioural handwriting of the signer. Some signers could preserve the identical format of each curve and character in his/her signature, while others may only preserve a partial similarity of their characters. In other words, a signer could preserve to have up to five successive points-of-interest identical in character or a curve every time he/she signs. While, the other kind of signers could preserve only a separated features in his/her written characters. For example, Figure 17 shows that the first point-of-interest in the left corner of ‘M’ character appears similar in all images, while the second point at the right corner varies every time the signer writes ‘M’. While for the other type of signers, these two neighbour points do not hold that variance for all sessions of signing.

Figure 15. 16x16 Low’s window: comparing the verification accuracy percentage results of five signers based on single-point and multi-point matching.

350

M. A. SALAMA AND W. B. HUSSEIN

Downloaded by [197.51.99.27] at 13:58 20 November 2017

Figure 16. 64x64 Low’s window: comparing the verification accuracy percentage results of five signers based on single-point and multi-point matching.

Figure 17. The ‘M’ handwritten character per signatures of the same signer in different sessions.

4.3. Fuzzifying the match between signatures The test of the last modification is applied here based on utilising the fuzzy matching between the pairs of points-of-interest in two compared signatures. The first advantage in this modification is that the matching process is independent on any user-defined threshold. As the threshold value should be different from a signer to another, keeping the fitness or suitability of this threshold to every signer is not feasible. The test applied here is similar to the crisp matching applied in the charts of Figures 10 and 11, and the testing data set is the same for signer one and signer two. Except that the matches

Figure 18. Verification of genuine/forged signatures for signer one.

Downloaded by [197.51.99.27] at 13:58 20 November 2017

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

351

Figure 19. Verification of genuine/forged signatures for signer two.

between the features of the reference signature R and the rest of signatures are based on the proposed fuzzy matching modification in this proposed model. The resulted chart for this test appears in Figures 18 and 19 for the same two signers. These two figures show the average Euclidean distance between the SIFT features of a reference genuine signature R and four genuine signatures G followed by 30 forged signatures F. The matching applied here is a user-defined threshold-independent method. Although the verification accuracy of signer one is decreased from 100% to 94%, but the accuracy of verifying the forgery of the signatures is raised from 84% to 100%. Accordingly, in addition to avoiding the user-defined threshold, this modification provides a further enhancement in the verification accuracy of the signatures.

5. Conclusion The signatures of the same person usually vary according to the writing conditions like position, mood, used pen/pencil and even the used paper. Accordingly, the extraction and matching process between points-of-interests in signatures should not be the same as in identical objects in different images. This work presents a set of pitfalls that appear in the utilisation of SIFT model in the domain of handwritten signature verification. These problems can be summarised as the fixed size window surrounding the extracted features, the way of matching between signatures and the dependency on 1-1 matching between descriptors. The proposed model is applying a set of modification on the SIFT extraction and matching model. These modifications include varying the window size, increasing the number of points-of-interest in the matching process and computing the average Euclidean distance of all matches rather than comparing the number of matches. The resulted verification accuracy in this study is compared to the results of the traditional SIFT-based signature verification model to prove the high efficiency of the proposed model. The experimental results of applying the proposed model on benchmark data sets show superior verification accuracy in the order of 95%. The future work is to create a tool that automatically extracts and visualise a signature that holds all the common features among all genuine signatures.

352

M. A. SALAMA AND W. B. HUSSEIN

ORCID Mostafa A. Salama

http://orcid.org/0000-0003-2559-8056

Downloaded by [197.51.99.27] at 13:58 20 November 2017

References Abikoye, O., Mabayoje, M., and Ajibade, R., 2011. Offline signature recognition & verification using neural network. International Journal of Computer Applications, 35 (2), 44–51. ISSN:0975-8887 Adeyemo, A. and Abiodun, A., 2015. Adaptive SIFT/SURF algorithm for off-line signature recognition. Journal of Egyptian Computer Science, 39 (1), 50–56. Bay, H., et al., 2008. Speeded-up robust features (SURF). Computer Vision and Image Understanding Archive, 110 (3), 346–359. doi:10.1016/j.cviu.2007.09.014 Biradar, S. and Panchal, S., 2015. Bank cheque identification and classification using ANN. International Journal Of Engineering And Computer Science, 4 (7), 13237–13242. ISSN:2319-7242 Celar, S., et al., 2015. Classification of test documents based on handwritten student ID’s characteristics. In Proceeding: 25th DAAAM International Symposium on Intelligent Manufacturing and Automation, Procedia Engineering, 100 (1), 782–790. Chambers, J., et al., 2015. Currency security and forensics: a survey. Multimedia Tools and Applications, 74 (11), 4013–4043. Das, S. and Roy, A., 2015. Signature verification using rough set theory based feature selection. Advances in Intelligent Systems and Computing, 411, 153–161. Dhaka, V., Rao, M., and Manu Singh, P., 2009. Signature verification on bank checks using Hopfield neural network. KARPAGAM Journal of Computer Science, 3 (4), 9. ISSN:0973-2926 Galbally, J., et al., 2015. On-line signature recognition through the combination of real dynamic data and synthetically generated static data. Pattern Recognition, 48 (9), 2921–2934. doi:10.1016/ j.patcog.2015.03.019 ISSN:2349-9745 Gupta, S., 2014. Handwritten signature verification using artificial neural network. International Journal of Modern Trends in Engineering and Research, 1 (2), 308–322. ISSN:2349-9745. Hafemann, L., Sabourin, R., and Oliveira, L., 2015. Offline handwritten signature verification – literature review. Computer Vision and Pattern Recognition, Submitted on 28 Jul 2015 (v1), last revised 19 Aug 2015. Halder, B., et al., 2014. Analysis of fluorescent paper pulps for detecting counterfeit Indian paper money. Information Systems Security, Lecture Notes in Computer Science, 8880, 411–424. Hatkar, P., Salokhe, B., and Malgave, A., 2015. Offline handwritten signature verification using neural network. Journal of Information, Knowledge and Research in Electrical Engineering, 3 (2), 449–453. Jain, U. and Patil, N., 2014. A comparative study of various methods for offline signature verification. In: Proceeding: International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT 2014), IEEE, Ghaziabad, 7–8 February 2014, 760–764. Karouni, A., Daya, B., and Bahlak, S., 2011. Offline signature recognition using neural networks approach. In Proceeding: World Conference on Information Technology, Procedia Computer Science, 3, 155–161. Kaur, R. and Choudhary, P., 2015a. Offline signature verification in Punjabi based on SURF features and critical point matching using HMM. International Journal of Computer Applications, 111 (16), 4–11. ISSN:0975–8887. doi:10.5120/19620-1288 Kaur, R. and Choudhary, P., 2015b. Handwritten signature verification based on SURF features using HMM. International Journal of Computer Science Trends and Technology (IJCST), 3 (1), 187–195. Liwicki, M., et al., 2011. SigComp11: signature verification competition for on- and offline skilled forgeries. In: Proceeding: 11th International Conference on Document Analysis and Recognition (ICDAR), IEEE, Beijing, 18–21 September, 1480–1484. Lowe, D.G., 1999. Object recognition from local scale-invariant features. In: Proceeding of 17th IEEE International Conference on Computer Vision, IEEE, Kerkyra, 20–27 September 1999, 1150–1157.

Downloaded by [197.51.99.27] at 13:58 20 November 2017

INTERNATIONAL JOURNAL OF IMAGE AND DATA FUSION

353

Malik, M., et al., 2014. Automatic signature stability analysis and verification using local features. In: Proceedings: 14th International Conference in Frontiers in Handwriting Recognition (ICFHR), IEEE, Crete, 20–27 September, 621–626. Neamah, K., et al., 2014. Discriminative features mining for offline handwritten signature verification. 3D Researcher, 5 (1), 2. doi:10.1007/s13319-013-0002-3 Ooi, S., et al., 2016. Image-based handwritten signature verification using hybrid methods of discrete Radon transform, principal component analysis and probabilistic neural network. Applied Soft Computing, 40 (1), 274–282. doi:10.1016/j.asoc.2015.11.039 Ramadas, S. and Geethu, P., 2015. Comparative study on offline handwritten signature verification schemes. International Journal of Advanced Research Trends in Engineering and Technology (IJARTET), 2 (10), 1317–1322. Rathi, A., Rathi, D., and Astya, P., 2012. Offline handwritten signature verification by using pixel based method. International Journal of Engineering Research \& Technology, 1 (7). ISSN:2278-0181 Reddy, S., Maghi, B., and Babu, P., 2006. Novel features for offline signature verification. Journal of Computer, Communication and Control, 1 (1), 17–24. doi:10.15837/ijccc.2006.1.2268 Ruiz-del-Solar, J., et al., 2008. Offline signature verification using local interest points and descriptors. Progress in Pattern Recognition, Image Analysis and Applications, Lecture Notes in Computer Science, 5197, 22–29. Ruiz-Shulcloper, J. and Kropatsch, W., 2008. Signature verification using local interest points and descriptors. In: Proceeding: 13th Iberoamerican Congress on Pattern Recognition (CIARP 2008), Progress in Pattern Recognition, Image Analysis and Applications, LNCS 5197, Havana, 9–12 September, 22–29. Sayantan, R. and Sushila, M., 2014. Offline signature verification using grid based and centroid based approach. International Journal of Computer Applications, 86 (8), 35–39. ISSN:0975–8887. doi:10.5120/15009-3292 Shah, A., Khan, M., and Shah, A., 2015. An appraisal of off-line signature verification techniques. International Journal of Modern Education and Computer Science, 4 (1), 67–75. doi:10.5815/ ijmecs.2015.04.08 Soran, B., et al., 2012. Tremor detection using motion filtering and SVM. In: Proceeding: 21st International Conference on Pattern Recognition (ICPR), IEEE, Tsukuba Science City, 11-15 November 2012, 178–181. Taneja, B. and Kaur, N., 2015. Biometric system based on off-line signatures. International Journal of Advanced Research in Computer and Communication Engineering, 4 (5), 435–438. doi: 10.17148/ IJARCCE.2015.4594. Viriri, S., 2014. Handwritten signature verification based on enhanced direction and grid features. Advances in Visual Computing, Lecture Notes in Computer Science, 8888, 820–829. Vivaracho-Pascual, C., Simon-Hurtado, A., and Manso-Martinez, E., 2015. On the use of score ratio with distance-based classifiers in biometric signature recognition. In: Proceeding: International Conference on Neural Information Processing (ICONIP 2015), Springer International Publishing, Istanbul, Turkey, 9-12 November 2015, 318–327. Warasart, M. and Kuacharoen, P., 2012. Paper-based document authentication using digital signature and QR code. In: Proceeding: 2012 4TH International Conference on Computer Engineering and Technology (ICCET 2012), Bangkok, 12–13 May 2012, 94–98. Zhu, S., Lei, H., and Zanibbi, R., 2013. Rotation-robust math symbol recognition and retrieval using outer contours and image subsampling. In: Proceedings: Document Recognition and Retrieval XX, SPIE Electronic Imaging, vol. 8658, id. 865805, Burlingame, CA, 4 February 2013, 1–12. doi:10.1117/12.2008383.