Dynamic User Modeling for Sketch-Based User Interface Zhengxing Sun, Bin Li, Qiang Wang, and Guihuan Feng State Key Lab for Novel Software Technology, Nanjing University, 210093, P.R. China
[email protected]
Abstract. This paper presents a strategy of dynamic user modeling for sketchbased user interface. A user model is defined as an incremental decision tree for a specific user. A drawing style modeling process is designed to model user’s temporal drawing performances among inputting strokes, and to predict the possible shape in term of fuzzy matching. A user mediation process is proposed based on relevance feedback to capture users’ drawing intentions and refine the recognition results. Experiments prove efficiency of the method.
1 Introduction Due to the fluent and lightweight nature of freehand drawing, sketch-based user interface have being widely used for exploratory, creative activities in computer aided design, E-learning, networking games and so on [1][2]. However, it is quite difficult and impossible to ask computers to completely understand various sketches, because sketching is usually informal, inconsistent and ambiguous. Hence, it is not surprising that the poor accuracy of recognition engines is always frustrating, especially for the newly added users and composite shapes, though a wide variety of online sketch recognition methods have been produced [3][4 ]. Considering that human cognition is performed incrementally and iteratively, it is most wise choice that computer should share their contexts and understanding to match incrementally their physical and cognitive habits in order to facilitate the processing of computers in a recognition engine for sketch-based user interface. Accordingly, we identify two main questions for online sketchy shape recognition based on our previous researches. The first has to do with the task of grouping users’ strokes into his/her intended shapes. We propose a process of drawing style modeling to model the user’s inputting habits according to the emergent frequency for each of sketchy shapes in his/her drawing styles. The second concerns the user mediation. We adopt a process of users’ interactive feedback by means of the relevance feedback, which is a technique widely used in multimedia retrieval, to capture users’ inputting intends and refine the recognition results incrementally. The rest of paper is organized as following. The strategy of our solution for sketchbased user interface is outlined in section 2. The principle of our user modeling method is introduced in section 3. In section 4, our scheme of relevance feedback method is discussed. Finally, Some experiments and conclusions are given. X
Z. Pan et al. (Eds.): Edutainment 2006, LNCS 3942, pp. 1267 – 1272, 2006. © Springer-Verlag Berlin Heidelberg 2006
1268
Z. Sun et al.
2 Strategy Overview A framework of our online sketch recognition for sketch-based user interface is shown in Fig. 1. As a precondition, stroke preprocessing is firstly introduced to eliminate the noise points and find the segment points based on physical drawing contexts, such as pen speed and curvatures. Stroke segmentation and primitive recognition are then used to divide the stroke into different geometric primitives [5]. To resolve user adaptation problem, a SVM incremental learning algorithm have also proposed [6] to classify users’ input strokes. To recognize the composite sketchy shapes, we have also proposed a method based on Spatial Relationship Graph [6], which employs a constrained partial permutation strategy to reduce the computational cost of graph matching. However, SRG-based matching cost is still higher for realtime interaction in some cases, because it is originally an NP problem. X
X
X
X
Fig. 1. Framework of our strategy for sketch-based user interface
To improve the efficiency and reliability of existing algorithms, dynamic user modeling method is accordingly used to mention human performances. In our work, ‘user model’ is used as a profile to record personal drawing performances. A user model can be denoted as a tree structure T=(V,E), where, V={vi}, vi=(si,ri,GListi) and E={ei}, e=(vm,vm+1) denoting the sequence of the two strokes vm and vm+1, is one edge of the decision tree. A stroke s is defined by its class and direction: the class of a stroke is obtained by the SVM classifier [6] or by the feedback of users and the direction of a stroke is set by one of 8 directions with the equiangular division of a circle respectively. The relationship ri between two continuous strokes can be expressed as the direction from the boundary center of the previous stroke vm to that of the current vm+1. In addition, in a user model, each node also includes an attribute ‘possible shapes’, which can be found along this branch and their corresponding possibilities. The list of the candidate objects is expressed as Glist={wi}, where wi=(GClass,weight), ‘Gclass’ is one of objects that can be searched from the current node of user models, and ‘weight’ is the visiting frequency of the candidate object. These two values are obtained by learning from users’ mediation by relevance feedback or users’ historic dataset. Each time he/she is drawing, user model is used as an assistance of sketch recognition to predict the “possible shapes” and updated incrementally based on statistical calculation of his/her historical drawing properties and his/her mediation. B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
Dynamic User Modeling for Sketch-Based User Interface
1269
Based on definition of user model, the process of dynamic user modeling is twofold, as shown in Fig. 1. On the one hand, we define a process of drawing style modeling to model user’s drawing habit for every type of shapes, as shown in the left side of Fig. 1. On the other hand, a process of user’s mediation for spatial constitute of sketch is proposed, as shown in the right side of Fig. 1. These will be discussed in following sections. X
X
X
X
X
X
3 Drawing Style Modeling Drawing style modeling is defined to build a specific tree for each specific user according to the sequence of inputting strokes and three attributes of each stroke: the type of the stroke class, the direction and the relationship between two consecutive strokes and model user’s temporal drawing habit for every type of shapes. Along with the training process, the decision tree will incrementally grow and adjust to the user’s style in stroke sequence and construction of the composite shapes. When a composite shape is being drawn in a sequence of strokes, each stroke may be tried to match along the branch. If the matching is successful, the possible shape can be predicted or recognized, and at the same time, the weight of related nodes in the user models are adjusted. Otherwise, a new branch of the tree is created. Once a composite shape does not exist in the template, the strokes are collected and added to the system after shape regularization. During modeling, there is a risk that a decision tree would become more and more complex and. Accordingly; we design a Lottery Rule to ignore/remove some branches that are hardly used based on their visited frequency. All of the temporal styles in a model are ordered by their lotteries and every style of a model may keep a list of lotteries. When a new drawing style is added, the lotteries of some styles that has same model ID with the new one will be consumed. When user model need to be depressed, the styles with smaller lotteries could be ignored or removed according to some specific requirements. When being sketched in a sequence of strokes, a composite shape is tried to match every stroke along all branches of the decision tree. However, there exists a contradiction between the accuracy of calculating the attributes of strokes when building the decision tree and the illegibility of these strokes when drawing and predicting it. Therefore, a fuzzy matching method is introduced to search the ‘possible shapes’ that the user intents to draw as described as follow. Firstly, two fuzzy attributes, the direction and the type of stroke, are preprocessed in terms of a ‘subjective array’ R of fuzzy stroke classes with 14 columns and 14 rows [6] and a ‘subjective function’ of fuzzy direction of stroke [6] respectively. Then, let g’ be the weight (visit frequency) of the current node and GList be the list of the candidate objects (possible shapes) of the current node, the weight of each candidate object in this node is g’⋅Glist. All possible shapes from all visited nodes are ranked by their maximum weights (some objects may have different weights along different branches). When the next stroke is input, the fuzzy directions of this stroke can be calculated as s1 and s2, the fuzzy relations to the previous stroke are r1 and r2, and the possibility of being a particular stroke type is Rij(i,j=1,2,…,14). Consequently, the possibility of the next searching node is: g = g ' sm rn Rij [6]. X
B
X
B
B
B
B
B
B
B
B
B
1270
Z. Sun et al.
4 User Mediation by Feedback To capture users’ input intentions, we bring relevance feedback method into online sketchy recognition to improve and boost up dynamic user modeling. The main idea is described as shown in the right side of Fig. 1. During drawing, users can check up the candidates according to his/her interests and make his/her judges by appearances of instances as the relevance of the candidates iteratively. The system will then employ the feedback algorithm to refine the candidate results. The process is over until the user gets the desired sketch object. The information about users’ feedbacks can finally be added to user model in terms of “historical information analysis”. The contents what users stressed would be the topological relationship between primitives during his/her inputting. Accordingly, we represent a drawing shape as a set of topological graphs and convert it to a vector-based representation, as shown in Fig. 2. The adjacency matrix of the topological graph is firstly derived from graph and its eigenvalues can be computed. The absolute values of eigenvalues are sorted to obtain the spectrum descriptor [7], which is the set of graph eigenvalues and can be calculated from the eigenvalues of its adjacency matrix, and the topological features of a sketch are finally converted into multi-dimensional vectors. In our research, eight types of spatial relationships are considered and eight different spectrum descriptors will accordingly be got and can be used as feature vectors for a sketchy shape. X
X
X
X
Fig. 2. Diagram to convert the topological features to a vector representation
However, the dimension of the spectrum descriptors will be same with the number of primitives and the vertices in its topological graph. Therefore, we do the map of different multi-dimensions of the spectrum descriptors into one dimension by means of the principle of dimensionality reduction of vector. Regarding an n-dimensional vector as a point in the n-dimensional Euclidean space, the multidimensional spectrum descriptors can be reduced to the Euclidean norm. Given a drawing with feature vector Fd=(fd1,fd2,…,fd8) and a template with feature vectors Ft=(ft1,ft2,…,ft8), the similarity between them can then be calculated as follows:
⎧0, if Dis ≥ ω; Similarity= ⎨ where,Dis = ⎩1 − Dis ω, if Dis < ω.
8
∑ ( fd i =1
i
− fti )
2
(1)
where, “Dis” is Euclidean distance between two vectors, ω is a threshold introduced from our experiments to normalize the similarity and make “Similarity” fall into the range of [0,1]. We regard the shapes with the highest similarities as the candidate result set, and return the shapes to users with similarities descending. The relevance feedback algorithm used in our research is the strategy of vector adjustment [7]. The objective of this approach is to construct a new feature vector X
X
Dynamic User Modeling for Sketch-Based User Interface
1271
point that is “closer” to relevant shapes. The best-known approach to achieve feature vector adjustment is based on a formula as follow [8]: Let Srel={Oj|Oj is a relevant object} be the set of relevant shapes of the user feedback, Let Fj, Fnew and Fold be the feature vector of Oj, the new feature vector and the old feature vector respectively. The new feature vector point changes incrementally over the original feature vector point as follows, which is moved towards the relevant points: X
B
B
B
B
B
B
B
B
B
B
B
B
B
B
Fnew = α × Fold +
β Srel
×
∑F
O j ∈S rel
(2)
j
The speed at which the old vector point moves towards relevant feature points is determined by the parameters α , and β , where α + β = 1 . The purpose of retaining part of the original vector point is to avoid “overshooting” and to preserve some part of the user supplied shape with the hope that it contributes important information to guide the retrieval. In our system, we set α = 0.2 and β = 0.8 . After feature vector adjustment, we can do similarity calculation again with new feature vector and the new candidates can be obtained after recalculation according to similarity descending, which get closer to the user intention.
5 Experiments and Evaluation Our experiments have selected 150 general standard composite objects [6], where each standard object can be drawn in several drawing styles. We have collected 537 styles for all objects, and 10 drawing shapes for each style of one object. For our experiments of the dynamic user modeling, we collect one, three, five or seven types variations of styles for every object as different training data, Denoted as T1 (537 samples), T3 (1611 samples), T5 (2685 samples) or T7 (3759 samples) respectively, and three types as the test data (1611 samples in total). The precision of recognition using the fuzzy matching and the accurate matching is comparably shown in Table 1, measured if the object that user drawing appears in top k of the candidate objects. X
B
B
B
B
B
B
B
B
X
X
Table 1. Comparison between fuzzy matching (F) and Accurate matching (A) Training Dataset TB1B TB3B TB5B TB7B
Top 1 F 77.72 81.81 83.43 85.23
A 47.61 76.61 85.35 89.14
Top n accuracies in the candidate objects list (%) Top 3 Top5 Top7 F A F A F A 88.08 55.49 91.37 58.2 92.68 60.34 92.99 81.87 95.47 95.1 96.96 86.10 93.67 89.26 96.46 91.1 97.7 91.50 94.60 92.43 96.96 93.9 98.6 94.29
The results of experiments show that our user modeling method would be effectual from two aspects: the precision will increase along with the increment of training number, and the fuzzy matching is more powerful than the accurate matching, especially when the training data set is very little. On the other hand, our feedback experiments have also proved that as the number of iteration of feedback increases, the recognition results are refined step by step.
1272
Z. Sun et al.
6 Conclusion In this paper, we have exploited a principle of the dynamic user modeling for sketchbased user interface. Based on the definition of a user model, the process of dynamic user modeling can then be divided into two sub-processes: drawing style modeling and user mediation by feedback. The former generates incrementally the model for a specific user according to his/her temporal drawing habits of a specific shape and recognizes dynamically the ‘possible shapes’ by means of fuzzy matching based on the visiting frequency of every recorded drawing style. The latter adjusts iteratively the weights of similarity calculation of spatial/topological features of sketch by means of relevance feedback. When a user wants to input a similar shape with his/her historic shape, our methods can predict the resulting shape more efficiently and make dynamic pruning to cut off the noisy or exceptional nodes.
Acknowledgement This paper is supported by the grants from the National Natural Science Foundation of China [Project No. 69903006 and 60373065] and the Program for New Century Excellent Talents in University of China [Project No. NCET-04-04605].
References 1. Landay J A and Myers B A, Sketching Interfaces: Toward More Human Interface Design, IEEE Computer, vol. 34, no. 3, 2001, pp. 56-64. 2. Zhengxing Sun and Jing Liu, Informal user interfaces for graphical computing, Lecture Notes in Computer Science, Vol. 3784, 2005, pp. 675-682. 3. Levent Burak Kara, Thomas F, Stahovich, Sim-U-Sketch: A Sketch-Based Interface for Simulink, Proceedings of AVI-2004, pp. 354-357. 4. Newman M W, James L, Hong J I, et al, DENIM: An informal web site design tool inspired by observations of practice, HCI, vol. 18, 2003, pp. 259-324. 5. Wenyin Liu, Xiangyu Jin and Zhengxing Sun, Sketch-Based User Interface for Inputting Graphic Objects on Small Screen Device, LNCS, vol.2390, 2002, pp. 67-85. 6. Zhengxing Sun, Wenyin Liu, Binbin Peng, et al, User adaptation for online sketchy shape recognition, LNCS, Vol. 3088, 2004, pp. 303-314. 7. Xiaogang Xu, Zhengxing Sun, Binbin Peng, et al, An online composite graphics recognition approach based on matching of spatial relation graphs. IJDAR, vol. 7, no.1, 2004, pp. 44-55. 8. Shokoufandeh A, Dickson S, Siddiqi K, and Zucker S, Indexing using a spectral encoding of spatial structure, Proc. of the CVPR, 1999, pp. 491-497. 9. Rocchio J. J., Relevance feedback in information retrieval, The SMART Retrieval System, Experiments in Automatic Document Processing, Prentice Hall, 1971, pp. 313-323.
