Researching Affective Computing Techniques for

Researching Affective Computing Techniques for Intelligent Tutoring Systems Malinka Ivanova Technical University of Sofia Sofia, Bulgaria [email protected] Abstract— Intelligent Tutoring is applied as a supportive tool in decision making, problem solving, knowledge receiving. Adding computer vision for affect recognition leads to the adaptation of tutor behavior not only to the cognitive level of a student but also to his emotional state that could improve quality of learning. In this paper the current research in the area of facial expression and gesture recognition in the context of intelligent tutoring is examined with the aim to facilitate educational society in building of affective intelligent tutoring systems. Keywords—intelligent tutoring system; affective computing; facial expression recognition; gesture recognition

I.

INTRODUCTION

The aim of an Intelligent Tutoring System (ITS) is to support each individual student in his/her cognitive process through emulation of adequate teachers’ behavior. A typical ITS possesses simple or more complex functions related to adaptation of the course content and knowledge, adaptation of the pedagogical scenarios, adaptation of the content presentation to the students’ learning level and emerging learning needs. Such ITS does not care about student’s emotional state and does not render the possible relation between emotion and learning. A number of research articles in areas like neuroscience, education, decision making support and psychology discuss cognition, emotions, attention, motivation and learning. Sylwester writes that “Emotion drives attention, which drives learning, memory and problem solving and almost everything else we do [1]”. Other authors like Li and Mao prove that emotions are an important factor for motivation and also that motivation drives learning [2]. Researchers divide emotions into two main groups: positive and negative – both of them have their impact on learning. Kort et al. declare that positive emotion could enhance learning while negative emotions can disable the learning process [3]. Rishi discusses the influence of emotional state on attention in task doing – negative emotions (anger, anxiety, or distress) do not allow focusing on the learning item or moving the attention to the new one and in this way the learning performance is decreased [4]. Positive emotions like joy and pride could facilitate thinking and learning. The author proposes a rulebased dynamic method for ensuring the best emotional conditions for learning, including detection of emotions and provoking suitable affective state for performance improvement. As it can be seen emotions influence on the learning process and on so important engineering activities like

978-1-4799-0152-4/13/$31.00 ©2013 IEEE

decision making [5] and problem solving [6]. Chaffar and Frasson present a system ESTEL (Emotional State towards Efficient Learning system) that has features to predict the optimal emotional state for learning according to the learner’s personality [7]. It can induce the appropriate emotions to improve the processes memorization and comprehension through applying different techniques like guided imagery, music and images. As it can be seen the common affective states that could play a catalyze role for learning are positive emotions like: joy, confident, pride, anxious, self-gratification. On the other side the emotional feedback given by students influence on tutor affective status and behavior that could change the future teaching/learning path. Therefore, an ITS would be more effective if the computer tutor like the human one has the capability to understand and self-adapt to the students’ affective state. This fact is taken into consideration in the development of enhanced ITS’ version called Affective Tutoring System (ATS) [8]. In this way the computer tutor adapts content, pedagogical methodology and presentation not only according to students’ achievements and learning needs, but also in correspondence to their emotional adjustment. Nowadays, several methods for emotions’ detection during students’ learning are used in ITSs. This includes: (1) applying a quiz and self-reporting about the current student’ mood, (2) facial expression recognition and analyzing the affective student’ state, (3) voice recognition and analyzing the sound characteristics, (4) text typing on the keyboard and understanding the dynamics of text writing, (5) using different sensors on the student’ body to measure blood pressure, pressure on the chair, temperature, etc., (6) analyzing the brain waves. One part of contemporary ITSs is based on one method and other part combines different methods for receiving a more realistic picture about the current emotional state of a student. In this paper the accent is given on facial expression and gesture recognition techniques and their utilization in contemporary ITSs. Recent scientific achievements including technical solutions and good practices are examined. Several open source approaches of libraries and software are explored. All research is made in the context of the emotions role for ITSs improvement and enhancing the learning quality.

25-27 September 2013, Kazan National Research Technological University, Kazan, Russia

2013 International Conference on Interactive Collaborative Learning (ICL) Page 611

II.

FACIAL EXPRESSION RECOGNITION TECHNIQUES

The face characteristics and its muscle motions are described with a set of parameters which are utilized for recognition of facial emotions. Several sets with such parameters are created (Facial Action Scoring Technique, Emotional Facial Action Coding System, Maximally Discriminative Facial Movement Coding System, Facial Electromyography, Affect Expressions by holistic Judgment, FACS Affect Interpretation Database [9]), but the most used are the following two: the Facial Action Coding System (FACS) presented by Ekman and Friesen [10] and the set with Facial Animation parameters (FAPs) which is included in the MPEG4 Synthetic/Natural Hybrid Coding (SNHC) standard [11], [12]. Anyway, the MPEG4 standard does not give information about some facial behavioral characteristics that differentiate the posed from spontaneous emotions. MPEG4 is applied for preparing animations of facial avatars, but it does not count the changes in surface texture like shape changes, bulges and wrinkles that are important for FACS action units description. The differences between posed and spontaneous expressions are based on emotions appearance and their temporal characteristics (onset-apex-offset). Posed and spontaneous expressions can be recognized by the movement of given facial components and by their movement dynamics. Ekman talks also about micro facial expressions and squelched expressions [13]. Micro facial expressions are observed in the cases when people are trying to mask their real emotions. Their duration is very short about 1/25-1/15 of a second, but they are complete expressions (they have onset, apex and offset). The squelched expressions begin their showing but they are immediately stopped and changed to other expression. The squelched expressions are uncompleted and their duration is longer than micro expressions. At this moment several automatic recognition systems for micro expressions are developed [14], [15], but still the meaning of micro facial expressions for educational society are not researched. Instead of that there are a wide range of good practices at the implementation of facial expression recognition systems working with posed and spontaneous expressions, including in the area of ITSs. The main modules of an automatic facial expression recognition system are related to the processes of preprocessing (noise reduction, contrast normalization), face and facial components detection, including tracking and facial expression recognition, feature extraction, expression classification and post-processing (Figure 1).

Preprocessing

-Detection (Face & facial components) -Tracking (Face & facial components)

Feature extraction

Classification for facial expression recognition

Postprocessing

Figure 1. Architecture of an automatic facial expression recognition system

The aim of the pre-processing is to improve the image quality in the case when there is noise or the contrast between

978-1-4799-0152-4/13/$31.00 ©2013 IEEE

background and objects is low. The developed methods are classified into two main classes: local and global methods [16]. Local methods improve contrast by increasing the contrast ratio or by applying local histogram modification (local histogram equalization, local histogram stretching, nonlinear mapping methods). Global methods are related to the approaches of the histogram modification. Typical example of global methods are histogram equalization and its variants that are very often applied methods in practice to increase the image contrast by giving the most probable intensity values to pixels [17]. Face recognition is a process for identification of people faces on images. Face detection is a process for finding the position of a given face whose existence in the image is already specified. Face tracking is a process of following a detected face in a sequence of images or tracking a detected face across frames of a video sequence. Facial expression recognition is a process for identification the affective states of the detected people faces. Until now over than 150 methods for face detection are known, but recently two main groups of methods are often applied in practical solutions: image-based methods and geometrical-based methods. Image-based methods use a sliding window through a set of examples to detect the face. Among the often used image-based methods are: Eigenfaces, Principal Component Analysis (PCA), Fisher’s Linear Discriminate. Also, several other methods for reduction of dimensions in an image space exist like distribution-based model [18], Bayesian based model [19], and others. Through geometrical-based (feature-based) methods could be detected not only frontal faces, because the face features are independent from the head pose and illumination. The researchers distinguish two groups of geometrical-based methods realized through top-down approach and bottom-up approach. Top-down methods find the faces in an image through skin color and this approach is very fast in a face detection process. Typical segmentation algorithms for faces extraction are region growing, Gibbs Random Field Filtering, others. Bottom-up methods find facial features like eyes, mouth, nose, hair line that are invariant according to the pose, illumination, scaling. Recognition of facial expressions could be achieved through different algorithms, including: Active Appearance Model [20], Principal Component Analysis and Non-negative Matrix Factorization [21], face recognition using 2D model or 3D model [22]. Through the process of feature extraction a representation of the face shape, color, texture, motion or different face components is formed. The extracted features are utilized for categorization of expressions. A wide variety of methods are developed and successfully evaluated: feature extraction by color boosting [23], features extraction by automatically detecting local texture information, global texture information and shape information [24], lip feature extraction using fieldprogrammable gate arrays [25], texture feature extraction 26], Active Appearance Models and Active Shape Models [27]. A classifier is used for a categorization of expressions to be performed. There are many classifiers with models of pattern distribution and related parametric and non-parametric techniques for automatic expression recognition. The most



used classes are action units (AU) [28] and prototypic facial expression [29]. Anyway, these two classes could be applied together for the realization of a hierarchical recognition system [30]. For classifiers training different machine learning approaches are applied and often used are Neural Network [31], Support Vector Machine (SVM) [32], Hidden Markov Model (HMM) [33], Sparse Network of Winnows [34]. Post-processing techniques could be performed to enhance the accuracy of the facial expression recognition, for example the multi-channel deconvolution method for clearing the ambiguities at emotions transition [35]. Researchers have performed enormous work, developing algorithms, methods and techniques for recognition of facial expressions and they continue to improve their quality. One part of researchers create algorithms for recognition of emotions and they clearly present the domains for implementation of their innovation. Another part of researchers develop integrated solutions suitable for implementation in educational situations. In the next section existing approaches for facial expression recognition with general purposes including educational scenarios and specially created ITSs with integration of a method for recognition of student’s emotional state are summarized. III.

FACIAL EXPRESSION RECOGNITION IN TUTORING SYSTEM

Fan et al. have developed a facial expression analysis system that could be applied in ITSs and also in robotic tutors [36] with aim the performance of an intelligent tutor to be improved after an analysis of the student’s affective states. This system integrates the following components: for face detectionNeural Network algorithm is applied and improved by preliminary image segmentation, checking for geometric shapes and color pixels, including algorithms for facial feature location - for eye detection, mouth detection and the Active Contour Modeling algorithm; for face recognition – Principal Component Analysis is used; for facial feature extraction algorithms of active contour modeling; for facial analysis fuzzy logic to classify the facial expressions is used. The authors note that their system possesses disadvantages related to the needed time for face recognition (around 1 minute) that could delay its performance in real time as in the case of ITSs. The recognized emotions are: normal, surprised, sad, happy, distinguished, angry, puzzled. Bartlett et al. have developed an automated system for recognition of spontaneous facial expressions in real time [37]. The authors suggest that it will be useful for appliyng in several areas including tutotring systems. This facial expression recognition system is based on FACS and detects frontal faces from video sequences. The solution includes methods like Gabor filters and Gentleboost for machine learning, and cascade training procedure. The authors have experimented with datasets containing different face behavior: talking, moving, rotating and all faces are successfully detected. Grimm et al. are other researchers working on spontaneous facial expressions and their recognition when persons are active or inactive speakers in a dialogue [38]. The authors are

978-1-4799-0152-4/13/$31.00 ©2013 IEEE

focused on a dialogue strategy and emerging emotions exploring humans interactions with aim to improve computer tutors. A novel technique combining multi-scale, multiorientation Gabor filtering and Principal Component Analysis is applyed for extraction of features. Artificial Neural Networks are used for features classification and a neuro-fuzzy method for the estimation of emotions. The approach is based on FACS and meta-features like open and closed eyes, smilling and non-smilling mouth that are easy for observation. The experiments show that 72.9% is the average recognition rate for emotion classification in categories and 80.1% for emotion space classification. Ratliff and Patterson propose a framework for recognition of face emotions based on FACS and Active Appearance Model (AAM) [39]. Six emotions - fear, joy, surprise, anger, disgust, sadness plus neutral are correctly recognized from pictures with percentages between 60 and 100. AAM has a posibility to model the face shape as well as the face texture and to support the preparation of a feature sets for classification of facial expressions. Other researchers elaborated the AAM tecnique for real time localization of facial features in the case when data of intensity and depth exist [40]. Whitehill et al. propose a solution for facial expression recognition of students automatically detected in real time during lectures [41]. Authors suppose that traditional feedback to a teacher in the form of questions asking or asking for a sentence repeating is not so effective because of time wasting for clearing of understanding points or due to the delayed response of students who miss other important points in the meantime. They prove that the method for automatic expression recognition is suitable for usage in ITSs to measure the students’ difficulties observed at lecture time and to adapt the speed of presented learning materials according to the students’ affective state. Also, it is shown that the spontaneous expressions on students’ faces during lecture presentation could be predicted and in this way the lesson speed could be determined. It is considered that the first fully functioning Affective Tutoring System (ATS) is Easy with Eve and it is applied in primary school students in their mathematics learning [8]. It proposes a solution for students’ emotion detection (neutral, smiling, laughing, surprised, angry, fearful, sad, disgusted) and according to that an animated agent shows corresponding emotions using appropriate tutoring strategies. A case-based method is applied to adapt the tutor actions to the students’ affective states. The used cases are taken from the teaching practice of a human tutor. The module for facial expression analysis is fast enough to work in real time. It is started in the background during the students’ interactions with the ATS and the information is updated after every one change in the facial expression. Banda and Robinson present a multimodal solution for affect analysis (FACS cognitive and affective mental states in 6 classes - agreeing, concentrating, disagreeing, interested, thinking and unsure) in the scope of an ITS, fusing automatic facial expression recognition, head gestures and audio analysis [42]. The aim of the system is to generate such affective response of the intelligent tutor that could keep the students’



attention and motivate them for learning during the started session. Several strategies for tutor interventions according to time, frequency and nature are discussed, including the meaning of the affective content in the intervention. The used system for the affective state recognition works on three levels: action unit detection that consists of face tracker from a video, Gabor image transformation and neural networks algorithms; encoding of detected action units and their conversion in gestures through usage of hidden Markov models; emotions modeling via mental dynamic Bayesian networks and emotions prediction through the history of six previously observed gestures. FaceReader software is used to analyze the emotional face expressions during an assessment session [43]. Additionally two different researchers are involved in the role of observers and analyzers of emerging emotions over the students faces. The experiment is performed with students from a Greek University in their studying in introductory Informatics course. The assessment quiz with 45 multiple choice questions has to be taken in 45 minutes. The conclusion of the authors is that the software FaceReader possesses 87% efficacy and it is suitable for integration in a learning environment. The emotions that are often expressed by students during quiz time are: neutral, angry, and sad. Facial expression recognition in combination of gaze direction tracking in a ITS are applied for detection of student's distraction [44]. In real practice the technique like gaze tracking is enough to be used as a signal for attention loosing during a learning session. The authors suggest that facial expression recognition techniques could lead to the realization of a more effective process for distraction identification - for example when a student is talking with somebody else, is looking at other object than a computer display or is looking around for a long time. IV.

GESTURE RECOGNITION

In this part the importance of head and hand movements in learning situations and their relations with given emotional states are studied. For recognition of head gestures and hand movements around the face the above mentioned techniques for face recognition and facial expression recognition could be applied as well as several other methods typical for hand movement tracking and recognition like non-rigid hand movement analysis [45], Eigen dynamics analysis [46], crowdsource method [47]. The importance of hand gestures and posture for modeling the affective state of a student in the context of an ITS is presented in [48]. The spontaneous gestures during the performance of academic tasks are studied and connected to 4 different emotions: boredom, flow, confusion, frustration. The results show: (1) that the dominant emotions are flow and boredom, (2) that a big part of the observed gestures are combination of movements, (3) that one and the same gestures are related to different emotions. The accuracy of the created model is tested through leave-one out cross method and it is calculated to 43.10%. A gesture detection system identifying the unintentional hand gestures and in this way mapping the affective states of

978-1-4799-0152-4/13/$31.00 ©2013 IEEE

students is presented in [49]. Unintentional movement of hand gestures around the face during lecture time are video recorded and mapped with the affective states of students. The gestures are categorized in 7 groups: head scratch, nose itch, lip, touch, eye rub, chin rest, lip zip and ear scratch. After post-experiment interview with students, these 7 gesture groups are connected to the following 6 affective states: recalling, satisfied, thinking, tired, bored, concentrating and their relationship is modeled via a Bayesian network. V.

DISCUSSION

The performed overview in this paper shows that researchers and educators are interested in not only the student's cognition but also in the psychological component of student's characteristics during his learning. Their aim is to humanize the computer tutor giving techniques for its vision and adaptation to the current cognitive level and to the affective state of a student. In the examined practices the developed intelligent tutors include only facial expression recognition techniques for emotions identification or possess several channels for better understanding of students’ emotions combining recognition of facial expressions, eye-tracking, gesture recognition, voice analysis, others. Anyway, at this moment the intelligent tutors with computer vision are not so often used in educational practice despite of the enormous research and created applications in the areas of facial expression recognition and analysis and tutoring systems. The open source community has its contribution in this context proposing libraries and software in support of the building of an affective tutoring system: x OpenCV (http://sourceforge.net/projects/opencvlibrary/) – a library for computer vision with than 2500 algorithms for image processing, face detection and recognition, tracking, feature extractions, others. It possesses interfaces in C, C++, Python, Java and can be delivered on different operational platforms. x Discrete Area Filters (DAF) Face Detector (http://www.semanticvisiontech.com/) – a library using facial features tracking scheme based on Discrete Gabor Jets and modified LDA. It is free for non-commercial and educational purposes. x Face Tracking SDK (http://chenlab.ece.cornell.edu/projects/FaceTracking/) – the system uses Gaussian Mixture Model (GMM) for color distribution modeling and a logrithmic search for the template deformation. It is written in C++ for Windows and Linux platforms. x PyVision (http://sourceforge.net/apps/mediawiki/pyvision/index.php ?title=Main_Page) - is a Computer Vision Toolkit for rapid prototyping, analyzing and evaluation of different computer vision algorithms. It is written in Python and includes several other tools for face detection and labeling, eye picker, etc. x FaceAPI (http://www.seeingmachines.com/product/faceapi/) – API toolkit for real time face tracking, facial features extraction



x

x

x

x

and facial expression recognition. It can be integrated in other software for non-commercial use. Face detector (http://kolmogorov.sourceforge.net) – software for vision perception and machine learning consists of face detector and video labeling. It is written in C++ and can be used on Windows platform. CTAT (Cognitive Tutor Authoring Tools) (http://ctat.pact.cs.cmu.edu/) is an open source platform for creation of cognitive (model-tracing) and examplebased tutors. It is developed in Human-Computer Interaction Institute at Carnegie Mellon University to propose the authors without programming skills flexible way for building behavior graphs (learning paths), deployment and delivery of intelligent tutoring in a wide range of domains. The tutor follows example-tracing rules to interpret the students’ behavior, taking knowledge from examples at solving the specific problem. Anyway, the authors have to understand Java or Flash techniques to design the learner graphical interface. The advanced authors have to use special rules for building the cognitive tutors. GnuTutor (http://gnututor.com/) is an open source conversational ITS based on AutoTutor and created in Institute for Intelligent Systems at University of Memphis. The tutor is in form of an animated agent and it speaks and recognizes natural language using latent semantic analysis. It is written in C# and requires cross-platform for running like Mono and .NET. xPST (Extensible Problem-Specific Tutor) (http://code.google.com/p/xpst/) is an open source ITS developed by a group from Iowa State University. ITS observes the students' behavior proposing suitable learning path, guidance and feedback (hints, error messages). There is a graphical authoring tool for intelligent tutors building by authors without the programming skills. The system is written in Java and it needs a Firefox plugin the problem to be seen in the browser. There is an extension for tutors authoring in 3D games.

Nowadays, researchers are still looking for good answers of the question "How emotions understanding could support teaching and learning?" On Figure 2 the directions of facial expression and gesture recognition empowering for tutoring and learning improvement are summarized. As it can be seen the recognition of student's affective state is used to: x measure the student difficulties and abilities for apprehending the learning content, x

identify when a student is distracted, bored or ready to back on a learning item,

x

adapt one learning session to the student's personal characteristics like learning speed and duration, the way for perception of content complexity/simplicity, the preferred learning path,

x

assess and compare the current emotional charge of a students and his cognitive capabilities,

978-1-4799-0152-4/13/$31.00 ©2013 IEEE

x

drive attention allocation on a learning item and student' s motivation to continue his learning session or go on another,

x

predict the future student learning behavior arranging learning items with suitable speed, time for learning, content complexity, content digital format, emotional charge, attention allocation and motivational elements. Student' affective state recognition measure - difficulties - abilities

identify - distraction - boredom - readiness for learning

adapt - speed - duration - content - learning path

assess - emotional charge and cognitive capabilities

drive - attention allocation - motivation

predict - learning behavior - characteristics of a learning process

Figure 2. Affect recognition for educational purposes

The intelligent tutoring systems with computer vision take advantages of the existing facial expression recognition and gesture techniques, but other emerging questions are related to "What kind of emotions are experienced by students?" and "What kind of emotions are important for education?" In Table 1 a summary is performed to demonstrate some of the used techniques for affect recognition through face and gestures near to face in the context of intelligent tutoring and also what kind of emotions are in the scope of research. TABLE 1. Authors

Techniques

Recognized emotions normal, surprised, sad, happy, distinguished, angry, puzzled

Fan et al. [36]

Neural Network algorithm, Active Contour Modeling algorithm; Principal Component Analysis, facial analysis - fuzzy logic

Bartlett et al. [37]

Gabor filters and Gentleboost, cascade training procedure, datasets containing different face behavior: talking, moving, rotating

spontaneous facial expressions in real time

Ratliff and Patterson [39]

Active Appearance Model

fear, joy, surprise, anger, disgust, sadness, neutral

Banda and Robinson [42]

face tracker from a video, Gabor image transformation and neural networks algorithms; hidden Markov models; dynamic Bayesian networks and emotions prediction through the history of six previously observed gestures

FACS cognitive and affective mental states in 6 classes agreeing, concentrating, disagreeing, interested, thinking and unsure



Some of authors use the popular Neural Network algorithm, Active Contour Modeling algorithm; Principal Component Analysis, Active Appearance Model to recognize posed emotions like: fear, joy, surprise, anger, disgust, sad, neutral. Other researchers use existing and improved techniques to recognize spontaneous facial expressions in real time. The third group is interested in recognition of cognitive and affective mental states and emotions prediction applying standard and improved methods - face tracker from a video technique, Gabor image transformation and neural networks algorithms; hidden Markov models; dynamic Bayesian networks. VI.

CONCLUSION

[10]

[11]

[12]

[13]

In this work, special focus is given on solutions for emotions gathering suitable for applying in ITSs through facial expression recognition and recognition of gestures close to the face. There are many techniques for recognition of posed emotions from pictures, video sequences and in real time situations. Techniques for recognition of spontaneous emotions are still under development. Just a few cases are known related to the micro expressions recognition and their importance/or not for educational purposes is not studied. The existing solutions are created using one or several classification databases that means that facial expression recognition approaches are not universal among different cultures. The performance rates could be improved. Despite of the big array of existing research still affective tutoring systems are not so popular among the educational society. It needs easy for use automated authoring tools for creation of such intelligent tutors as well as a flexible framework for applying in different educational scenarios.

[14]

[15]

[16] [17]

[18]

[19]

[20]

REFERENCES [1] [2] [3]

[4]

[5]

[6]

[7]

[8]

[9]

R. Sylwester, "how emotions affect learning," Educational Leadership, 52(2), 1994, pp. 60-66. X. Mao and Z. Li, "Agent based affective tutoring systems: A pilot study," Computers & Education, 55(1), 2010, pp. 202–208. B. Kort, R. Reilly, and R.W. Picard, "An affective model of interplay between emotions and learning: Reengineering educational pedagogybuilding a learning companion," In Proceedings of International Conference on Advanced Learning Technologies (ICALT 2001), August 2001, Madison, WI, http://vismod.media.mit.edu/pub/tech-reports/TR547.pdf O. P. Rishi, "Intellectual intelligent tutoring system: The ITS with emotions," International Journal of Engineering and Technology, 1(1), April 2009, pp 1793-8236. H. I. Ahn and R.W. Picard, "Affective cognitive learning and decision making: The role of emotions," Proceedings of the First international conference on Affective Computing and Intelligent Interaction, 2005, pp. 866-873. M. Spering, D. Wagener, and J. Funke, "The role of emotions in complex problem-solving," Cognition and Emotion, 19(8), 2005, pp. 1252-1261. S. Chaffar and C. Frasson, "Inducing optimal emotional state for learning in intelligent tutoring systems," Intelligent Tutoring Systems Lecture Notes in Computer Science, Vol. 3220, 2004, pp 45-54. S.T.V. Alexander, A. Sarrafzadeh, and S. Hill, "Easy with Eve: A functional affective tutoring system," http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.100.3683&rep =rep1&type=pdf M. A. Sayette, J. F. Cohn, J. M. Wertz, M. A. Perrott, and D. J. Parrott, "A psychometric evaluation of the Facial Action Coding System for

978-1-4799-0152-4/13/$31.00 ©2013 IEEE

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28] [29]

assessing spontaneous expression," http://www.uned.es/psico-doctoradoenvejecimiento/articulos/Ellgring/Sayette2001.pdf P. Ekman and W. Friesen, "Facial Action Coding System: A technique for the measurement of facial movement," Consulting Psychologists Press, Palo Alto, 1978. Y. Zhang, Q. Ji, Z. Zhu, and B. Yi, "Dynamic facial expression analysis and synthesis with MPEG-4 facial animation parameters," IEEE Transaction on Circuit and Systems for Video Technology, 18(10) October 2008. http://www.ecse.rpiscrews.us/~qji/Papers/face_animation_proof.pdf M. Pardàs and A. Bonafonte, "Facial animation parameters extraction and expression recognition using Hidden Markov Models," Imaging of the Signal Processing: Image Communication Journal, 2002, pp. 675688. P. Ekman, "Darwin, deception, and facial expression". http://www.evenhappier.com/darwin.pdf T. Pfister, X. Li, G. Zhao, and M. Pietikainen, "Recognising spontaneous facial micro-expressions," International Conference on Computer Vision (ICCV), 2011. http://tomas.pfister.fi/files/pfister11microexpressions.pdf S. S. Polikovsky, Y. Kameda, and Y. Ohta, "Facial micro-expressions recognition using high speed camera and 3D-gradients descriptor," 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP-09), London, UK, 2009. R. Jafri and H. R. Arabnia, "A survey of face recognition techniques," Journal of Information Processing Systems, 5(2), June 2009, pp. 41-68. C. G. Ravichandran and V. Magudeeswaran, "An efficient method for contrast enhancement in still images using histogram modification framework," Journal of Computer Science 8(5), 2012, pp. 775-779. K. K. Sung and T. Poggio, "Example-based learning for view-based human face detection," IEEE Transaction on pattern analysis and machine intelligence, 20(1), January 1998, pp. 39-51. B. Moghaddam, T. Jebara, and A. Pentland, "Bayesian face recognition," Pattern Recognition, 33(11), November 2000, pp. 17711782. B. Abboud, F. Davoine, and M. Dang, "Facial expression recognition and synthesis based on an appearance model," Signal Processing: Image Communication, 19(8), September 2004, pp. 723–740. L. Zhao, G. Zhuang, and X. Xu, "Facial Expression Recognition Based on PCA and NMF," Proceedings of the 7th World Congress on Intelligent Control and Automation, June 2008, pp. 6826-6829. A. F. Abate, M. Nappi , D. Riccio, and G. Sabatino, "2D and 3D face recognition: A survey," Pattern Recognition Letters, 28 January 2007, pp. 1885-1906. S. S. Sugania and K. J. Peter, "Feature extraction for face recognition by using a novel and effective color boosting," International Journal of Engineering and Advanced Technology, 1(4), April 2012, pp. 145-148. X. Feng, B. Lv, Z. Li, and J. Zhang, "A novel feature extraction method for facial expression recognition." http://www.atlantispress.com/php/download_paper.php?id=259 D. Nguyen, D. Halupka, and P. Aarabi, "Real-time face detection and lip feature extraction using field-programmable gate arrays," IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics, 36(4), August 2006, pp. 902-912. M. C. Mohan, V. V. Kumar, and K.V. Subbaiah, "A new method of face recognition based on texture feature extraction on individual components of face," International Journal of Signal and Image Processing, 1(2), March 2010, pp. 69-74. T. F. Cootes, G. Edwards, and C.J. Taylor, "Comparing active shape models with active appearance models," in Proceedings of British Machine Vision Conference, 1999, pp. 173-182. G. Donato, M. S. Bartlett, J. C. Hager, P. Ekman, and T.J. Sejnowski, "Classifying facial actions," IEEE Transactions Pattern Analysis and Machine Intelligence, 21(10), 1999, pp. 974-989. P. Ekman, "Emotion in the human face," Cambridge University Press, 1982. M. Pantic and L. J. M. Rothkrantz, "An expert system for multiple emotional classification of facial expressions," Proceedings of 11th



[30]

[31]

[32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

IEEE International Conference on Tools with Artificial Intelligence, 1999, pp. 113-120. C.C. Chibelushi and F. Bourel, "Hierarchical multistream recognition of facial expressions," ProceedingsVisual Image Signal Process., 151(4), August 2004, pp. 307-313. H. Rowley, S. Baluja, and T. Kanade, "Neural network-based face detection," IEEE Transactions on Pattern Analysis and Machine intelligence., vol. 20, 1998., pp. 22-38. E. Osuna, E. Freund, and F. Girosi, "Training support vector machines: an application to face detection," In Proceedings of Computer Vision and pattern recognition, 1998, pp. 45-51. F. Samaria and S. Young, "HMM based architecture for face identification," Image and Vision Computing, vol. 12, 1994, pp. 537583. N. Littlestone, "Learning quickly when irrelevant attributes abound: a new linear-threshold algorithm," Machine learning, vol. 2, 1998, pp. 285-318. G. Krell, R. Niese, and B. Michaelis, "Facial expression recognition with multi-channel deconvolution," in Proceedings of the 7th International Conference on Advances in Pattern Recognition, 2009, pp. 413-416. C. Fan, M. Johnson, C. Messom and A. Sarrafzadeh, "Machine vision for an intelligent tutor," Proceedings of IEEE International Conference on Computational Robotics and Autonomous Systems, 2003. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.5.2645&rep=r ep1&type=pdf M. S. Bartlett, G. C. Littlewort, M. G. Frank, C. Lainscsek, I. R. Fasel, and J. R. Movellan, "Automatic Recognition of Facial Actions in Spontaneous Expressions." http://people.ict.usc.edu/~gratch/CSCI534/Bartlett_JMM06.pdf M. Grimm, D. G. Dastidar, and K. Kroschel, "Recognizing emotions in spontaneous facial expressions." http://grimmm.de/research/Publications/Grimm06%20Recognizing%20 Emotions%20in%20Spontaneous%20Facial%20Expressions%20%5BIS YC%5D.pdf M. Ratliff and E. Patterson, "Emotion recognition using facial expressions with active appearance models," Proceedings of the 3rd IASTED International Conference on Human Computer Interaction 2008, pp. 138-143.

978-1-4799-0152-4/13/$31.00 ©2013 IEEE

[40] G. Fanelli, M. Dantone, and L. Gool, "Real time 3D face alignment with random forests-based active appearance models," 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 2013, pp.1-8. [41] J. Whitehill, M.Bartlett and J. Movellan, "Automatic facial expression recognition for intelligent tutoring systems," Proceedings of the CVPR Workshop on Human Communicative Behavior Analysis, 2008, http://mplab.ucsd.edu/~jake/its08.pdf [42] N. Banda and P. Robinson, "Multimodal, Affect Recognition in Intelligent Tutoring Systems," ACII'11 Proceedings of the 4th international conference on Affective computing and intelligent interaction, Volume Part II , 2011, pp. 200-207. [43] V. Terzis, C. N. Moridis, A. A. Economides, "Measuring instant emotions based on facial expressions during computer-based assessment," Personal and Ubiquitous Computing, 17(1), January 2013, pp 43-52. [44] Mark ter Maat, "How to detect a loss of attention in a tutoring system using facial expressions and gaze direction." http://hmi.ewi.utwente.nl/verslagen/capita-selecta/CS-Maat-Markter.pdf [45] H. Fei and I. Reid, "Probabilistic Tracking and Recognition of NonRigid Hand Motion." http://www.robots.ox.ac.uk/~fei/AMFG.pdf [46] H. Zhou and T. S. Huang, "Tracking articulated hand motion with eigen dynamics analysis," Proceedings of the 9th IEEE International Conference on Computer Vision, Vol. 2, Octomber 2003, pp. 11021109. [47] I. Spiro, G. Taylor, G. Williams, and C. Bregler, "Hands by hand: crowd-sourced motion tracking for gesture annotation." http://cims.nyu.edu/~bregler/acvhl10_hands.pdf [48] D. M. Bustos, G. L. Chua, R. T. Cruz, J. M. Santos, and M. T. Suarez, "Gesture-Based Affect Modeling for Intelligent Tutoring Systems," Artificial Intelligence in Education Lecture Notes in Computer Science Vol, 6738, 2011, pp 426-428. [49] A. R. Abbasi, N. V. Afzulpurkar, and T. Uno, "Exploring Un-Intentional Body Gestures for Affective System Design." http://cdn.intechopen.com/pdfs/5193/InTechExploring_un_intentional_body_gestures_for_affective_system_design. pdf

