EEG-Based Personalized Digital Experience

EEG-Based Personalized Digital Experience Olga Sourina, Yisi Liu, Qiang Wang, and Minh Khoa Nguyen Nanyang Technological University, Nanyang Ave, Singapore {eosourina,liuy0053,wang0586,raymondkhoa}@ntu.edu.sg

Abstract. To make human computer interfaces more immersive and intuitive, a new dimension could be added. Real-time brain state recognition from EEG including emotion recognition and level of concentration recognition would make an access to information more adaptive and personalized. Modern EEG techniques give us an easy and portable way to monitor brain activities by using suitable signal processing and classification methods and algorithms. We proposed a new subject-dependent fractal-based approach to brain state recognition and innovative applications based on EEG-enable user’s interaction. The algorithms of the “inner” brain state quantification including emotion recognition would advance research on human computer interaction bringing the proposed novel objective quantification methods and algorithms as new tools in medical, entertainment, and even digital art methodology applications, and allowing us an integration of the brain state quantification algorithms in the human computer interfaces. In this paper, we describe our fractal-based approach to the brain state recognition and its EEG-enable applications such as serious games, emotional avatar, music therapy, music player, and storytelling. Keywords: HCI; BCI; fractal dimension; emotion recognition; serious game; music therapy; storytelling.

1 Introduction Modern Human Computer Interfaces (HCI) could use both verbal and non-verbal types of human communication. Information could be entered to the computer by the user through the input devices such as keyboard, mouse, haptic device, microphone, touch response display, etc. The modern input devices help to create more intuitive and seamlessness user interfaces. To make human computer interaction even more seamlessness, the information could be entered by cameras, sensors-based tracking systems, and by biofeedback sensors. Then, the corresponding algorithms could process the gathered information depending on the application. For example, human emotions could be recognized from face, gesture, or from the combination of biological signals and could be used in different applications such as music therapy, emotional avatar, etc. In this paper, we describe our approach to add one more dimension to human computer interaction. It is so called communication by “brain power”. EEGbased communication was used for many years in Brain Computer Interfaces (BCI) and neurofeedback games for medical applications. Electroencephalogram (EEG) is a non-invasive technique recording the electrical potential over the scalp which is C. Stephanidis (Ed.): Universal Access in HCI, Part II, HCII 2011, LNCS 6766, pp. 591–599, 2011. © Springer-Verlag Berlin Heidelberg 2011

592

O. Sourina et al.

produced by the activities of brain cortex, and reflects the state of the brain [1]. Recently, new portable wireless EEG reading devices have entered to the market, and it made possible to develop applications personalized and adapted to the user. It could be adaptive games, music therapies, emotion-enable interaction in 3D collaborative environment, etc. In this paper, we propose a new fractal based approach to brain state recognition including emotion recognition and concentration level recognition, and innovative applications based on the EEG-enable user’s immersion and interaction. The proposed fractal based method is a general method, and it could be used as a basis for the development of other brain states recognition algorithms as well. The proposed and implemented applications consist from two parts: fractal based brain state recognition algorithms and interactive application systems. In the proposed algorithms, complexity of the signal is estimated by calculating fractal dimensions values changing over time. The algorithms of concentration level recognition and emotion recognition use just one fractal feature per channel that allows us to implement real-time EEG-enable applications. We implemented real-time applications such as concentration games, emotionenable web-based music player, music therapy, and storytelling. In Section 2.1, we review Brain Computer Interfaces (BCI) and neurofeedback techniques. In Section 2.2, emotion recognition algorithms are described. Then, the fractal-based approach to the brain state recognition is given in Section 3. Real-time EEG-enable applications including serious games and emotion-based personalized interaction are described in Section 4.

2 Related Work 2.1 CI and Neurofeedback Brain Computer Interfaces (BCIs) are the systems allowing the user to control computers/devices directly with the user’s brain signals. As an acquisition of EEG signals becomes more and more convenient due to new wireless portable devices appeared recently in the market, the applications of BCI spread even to entertainment industry. Thus, now applications of BCIs could be used not only by disabled people but by any person and even at home. Neurofeedback is a special technique that is used for medical treatments of different psychological disorders. A therapy is often embedded in the neurofeedback system, and the user makes an effort to improve his/her brain state following the system feedback. The feedback could be audio, visual, or even haptic. Thus, the user by doing some exercises recommended by the doctor or just by playing the serious games with neurofeedback learns how to improve his/her brain plasticity. Some researches demonstrated that the EEG distortion can reflect psychological disorders such as Attention Deficit Hyperactivity Disorder (ADHD) [2-3], Autistic Spectrum Disorders (ASD) [4-5], Substance Use Disorders (SUD) including alcoholics and drug abuse [6-7], etc. Neurofeedback could be used for treating these disorders as an alternative approach to medical treatments. These disorders could be treated with QEEG protocol based on power spectrum analysis, as different frequency bands reflect different brain functions [8]. They also could be treated by event related potential (ERP) analysis. Particularly, Slow Cortical Potential (SCP) analysis had shown its usability in ADHD treatment [9], and P300 component training could be used for drug abuse rehabilitation [7].

EEG-Based Personalized Digital Experience

593

Recently, the use of nonlinear features of EEG signals became popular due to the nonlinearity of the EEG signals and the possibility of nonlinear EEG feature extraction in real time. In [10-11], based on the two well-known algorithms, i.e. Box-counting [12] and Higuchi [13], concentration level recognition algorithms were proposed and applied in neurofeedback games, and the efficiency of the algrithms was shown. There is a growing demand for real-time brain-state recognition that could be used in medical applications. 2.2 Emotion Recognition Algorithms EEG-based emotion recognition is a relatively new research topic, and it attracted more attention from the research community since recognition of the “inner” feelings could be very important in medical applications, entertainment, marketing, etc. There are two types of the EEG-based emotion recognition algorithms: subject-dependent and subjectindependent ones. The algorithms of both groups consist of feature extraction and classification parts. To our best knowledge, the developed algorithms are off-line processing ones. They differ by the feature extraction methods used such as Fast Fourier Transform [14], Higher Order Crossings [15], etc, by the classifiers employed such as Support Vector Machine (SVM) [16], Neural Networks [17], etc, by the number of electrodes needed to identify the emotional states, and by types of emotions recognized. Currently, there is a demand for real-time emotion recognition algorithms that could be used in medical applications, serious games, and entertainment. In [18], we proposed and implemented a real-time fractal-based algorithm recognizing 7 emotions and using only 3 channels. Fractal dimension algorithms were applied to compute fractal-based features. The proposed algorithm is subject-dependent. The predefined thresholds are calculated during the training session. The satisfied, pleasant, happy, frustrated, sad, fear and neutral emotions could be differentiated. Since the discrete emotions can be mapped to the 2D emotion model, and fractal dimension values can be mapped to 2D emotion model as well, more emotions that are defined in 2D model could be distinguished.

3 Real-Time Brain State Recognition The overall structure of the real-time brain state recognition application systems is shown in Fig. 1. The mental state of the user is recognized from his/her EEG in real time. An overall recognition algorithm used in the real-time applications consists from the following steps: data acquisition and pre-processing such as data filtering, then, feature extraction, and subject-dependent classification. Then, the command is sent to the feedback system based on the recognition results.

Fig. 1. Diagram for non-invasive BCI system

594

O. Sourina et al.

3.1 Data Acquisition and Pre-processing EEG data are collected by Emotiv device with 14 electrodes located at AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4 standardized by the American Electroencephalographic Society [19]. The device sampling rate is 128Hz. The collected data are filtered by a 2-42 Hz bandpass filter to get the major waves of EEG [20]. 3.2 Fractal Dimension Features Extraction and Classification The next step after the data pre-processing is feature extraction. Fractal dimension (FD) is used to measure the complexity and irregularity of the object based on an entropy analysis. Entropy is a measure of the disorder in physical systems, or an amount of information that may be gained by observations of the disordered systems. A common practice to distinguish among possible classes of time series is to determine their so-called correlation dimension. The correlation dimension, however, belongs to an infinite family of fractal dimensions [21]. Hence, there is a hope that the use of the whole family of fractal dimensions may be advantageous in comparison to using only some of these dimensions. The concept of generalized entropy of a probability distribution was introduced by Alfred Renyi [22]. There are various methods to calculate fractal dimension (FD). In [23-24], the generalized Renyi approach based on Renyi entropy and calculation of the whole spectra of fractal dimensions to quantify brain states were studied. In our real-time applications, we use only Hausdorff dimension. We implemented two well-known Box-counting [12] and Higuchi [13] algorithms calculating fractal dimension. Both of them were evaluated using mono-fractal Brownian and Weierstrass functions where theoretical FD values are known [10]. Higuchi algorithm gave better accuracy as FD values were closer to the theoretical FD ones, and it was finally chosen to compute the FD features. We apply a sliding window and calculate one FD value per sample per channel. Number of channels used in the recognition algorithm defines a size of the feature vector as follows:

F = {FD1 , FD2 ,… , FDn } .

(1)

where n is number of channels. In the concentration level recognition algorithm, we have one feature as only one channel is used [10]. In the emotion recognition algorithm, there are 3 features in the vector as 3 channels are used [18]. Thus, in our approach we use one FD value per channel. Currently, we implemented a simple real-time subject-dependent classification algorithm based on threshold FD values that are calculated during a short training session. Note that off-line processing with SVM classifier of the EEG labeled with emotions and concentration levels gave us similar accuracy as the real time implementation algorithm used thresholds.

4 Real-Time Applications 4.1 Emotion-Based Digital Experience

We proposed and implemented a real-time fractal-based emotion recognition algorithm where the calculated fractal dimension values were mapped to 2D Valence-Arousal


595

emotion model [18]. It could be possible to recognize in real time any discrete emotions that could be defined with the 2-dimensional emotion model. Satisfied, pleasant, happy, frustrated, sad, fear, and neutral emotions were recognized in [18]. In our algorithm, only 3 channels are used that makes set up of our device faster and more convenient for the user. We implemented the emotion-enable real-time applications such as Emotional Avatar, Music Therapy, Music Player, and proposed Emotion-enable Storytelling. Emotional Avatar. We implemented an application with the EEG-enable avatar [18]. The music stimuli were used for emotion induction as it was proposed in [25]. We used an avatar available with version of Haptek development package for our application [26]. Haptek Activex control provides functions and commands to change facial expressions of 3D avatars. We defined six emotions by changing the parameters controlling the facial muscles of the Haptek emotion avatar. Those emotions are: fear, frustrated, sad, happy, pleasant and satisfied. In the application, emotions of the user are recognized from EEG and visualized in real-time on the user’s avatar with Haptek system. In Fig. 2 (a), the user was listening to music pieces for emotion induction, and the algorithm recognized “sad” emotion that was visualized in real time on the user’s avatar. The avatar emotions are changing according to the emotions that the user is feeling during the music listening. In the EEG-enable emotional avatar application in collaborative 3D virtual environment, the current user emotion could be visualized on his/her avatar or could be used in interaction with other avatars or even objects in virtual environments. Music Therapy. Second, we implemented an EEG-enable music therapy web-site. Music therapy can help the patients deal with the stress, anxiety and depression problems, for example, the patients’ anxiety could be released by listening to music during the surgery [27]. Our real-time EEG-based human emotion recognition algorithm is used in the music therapy system implemented on the Web. We proposed and developed the system that works as a therapist. The music choice and duration of the music is adjusted based on the patient’s current emotion recognized automatically from EEG. For music therapy helping in pain management, happy (positive/high aroused) songs are played to the user to distract his/her attention from the pain they feel. The user’s emotion state is recognized in real time from his/her EEG data. If the happy emotion is not induced by the current song, the player would automatically switch to another one. For music therapy dealing with depression, pleasant (positive/low aroused) songs are played to the user to make him/her feel relaxed. The song would be changed according to the EEG-based feedback. Fig. 2 (b) shows the implemented EEG-enable music therapy website. The user has EEG device Emotiv and an earphone for listening music during the therapy session. EEG-enable Music Player. Another personalized EEG-based system proposed and implemented is a web-enable music player. In this application, the user’s current emotion state is recognized, and then, the corresponding music is played according to the identified emotion. Songs are categorized into six emotion types: fear, sad, frustrated, happy, satisfied, and pleasant. Information about the current emotional state of the user and the music being played is given on the display of the player. For example, if the emotion state is recognized as pleasant, then the music which is categorized into the genre of pleasant music is played to the user.

596

O. Sourina et al.

(a)

(b)

Fig. 2. (a) “Sad” emotion is recognized and visualized on the user 3D avatar implemented with Haptex [26]. (b) The user accesses the music therapy website.

Interactive Storytelling. When people are exposed to the traditional story telling mediums such as reading a novel or watching a movie, they are just passive participants. Even in the games, the players need to follow the predetermined development of plot of the roles they played. However, since development of human computer interfaces including BCI, the EEG-enable interaction could be used in storytelling. The interactive storytelling gives the users a chance to enjoy more active experiences. For example, a story could be changed in real-time according to the user’s current emotion recognized from EEG. In interactive story telling entertainment, it is the user playing the leading role who decides the following storyline [28]. The interactive story telling can be implemented by different ways, for example, processing the text typed by user and based on the understanding of the text, the corresponding development of plot will be given [29]. Cavazza et al [30] took emotion into consideration to drive the interactive story telling system, and speech-based emotion recognition was used to decide the behaviors of each character and the interactions between different characters. Our real-time EEG-based emotion recognition provides another way to realize the interactive story telling system in entertainment. With the emotion recognized from the users’ brain activity, the story development of a movie or a game could be automatically directed by the user. In our preliminary study, we change a story line based on the user emotion detection. If the user is feeling unhappy, the story line could be changed. It would be possible to keep the emotion state of the user according to the predefined graph, for example, to keep the user excited for some time or to scare the user to get optimized emotional personalized experience through the story. 4.2 EEG-Enable Serious Games

The EEG-based serious game design includes two parts: signal processing algorithms and 2D or 3D game part. Raw EEG signals collected by the device from the user brain are filtered and analyzed by signal processing algorithms, and the resulting values are interpreted in the game as an additional game control using just the “brain power”. A therapeutic effect of such games could consist from combination of a distraction effect of the game and an effect from the learning by the user/patient how to control the game by voluntarily changing his/her brain state, for example, by learning how to improve the user’s concentration level. We developed concentration games such as


(a)

597

(b)

Fig. 3. (a) Screenshot of “Brain Chi” game. (b) Screenshot of the “Escape” game

“Brain Chi” and “Dancing Robot”, a game for stress management named “Pipe”, and behavior learning game “Escape”. They are simple single-player games implemented with the game engines SDL, Panda3D, and Adobe Flash CS4 correspondingly. The recognized relaxation/ concentration/ stress level values from EEG could be interpreted in the games as any visual/audio effects or even as a behavior change of the game characters. We also did a preliminary study on how to use the EEG-enable serious games for pain management and have got some promising results. Brain Chi. In the “Brain Chi” game, the relaxation/concentration level of the user is associated with radius of a “growing/shrinking” ball. It allows the “little boy” character to fight enemies by “growing” the ball as shown in Fig. 3 (a). In our implementation, the concentration and relaxation levels could be easily associated either with concentration training or relaxation training depending on the therapeutic purpose of the game. Dancing Robot. In the “Dancing Robot” game, the relaxation/concentration level is associated with the “robot” character behavior. When the concentration level of the user increases, the 3D robot character starts to move faster. If the user is fully relaxed, the robot stops dancing. The change of the quantified level of the user concentration level is interpreted as a “faster/slower” movement of the “robot”. Pipe. The “Pipe” game is adapted after the popular ‘Pipe’ game and is implemented as a traditional neurofeedback game. The two bars are used to indicate the level of the user stress and the game time left. In the “Pipe” game, water flows faster when the player’s stress level increases, and hence it makes the game playing more difficult. Escape. “Escape” game has an educational purpose. The story in the game requires the player to solve the educational puzzles to get the passwords for unlocking the doors. In this game, EEG could be used as an alternative way to get the passwords when the players can not figure out how to solve those puzzles. The user has to stay concentrated for the specified time, and the password will be given to him. If the player uses “brain power” help, the overall game time allocated for the player to escape is reduced. The screenshot of the game is shown in Fig. 3 (b). The user could go through the portal shown if he/she solves the puzzle.

598

O. Sourina et al.

5 Conclusion In this paper, we study one more possible dimension in human computer interfaces that is based on brain state recognition from EEG. It could allow the user to have more personalized experience during the computer interaction. We proposed fractal based approach to recognition of brain states including emotion recognition and concentration level recognition. Using just one fractal dimension feature per channel and the simple machine learning algorithm allows us to implement real-time brain state recognition algorithms with acceptable accuracy. Based on the algorithms, we proposed and implemented novel applications making the user experience more intuitive and personalized. We also work on the improvement of the real-time filtering of artifacts of different origin. The work described in the paper is a part of the project EmoDEx presented in [31]. Acknowledgments. This project is supported by the grant NRF2008 IDM-IDM004020 “Emotion-based personalized digital media experience in Co-Spaces” of National Research Fund of Singapore.

References 1. Nunez, P.L., Srinivasan, R.: Electric Fields of the Brain, 2nd edn. Oxford University Press, Oxford (2006) 2. Lubar, J.F., Swartwood, M.O., Swartwood, J.N., O’Donnell, P.H.: Evaluation of the effectiveness of EEG neurofeedback training for ADHD in a clinical setting as measured by changes in T.O.V.A. scores, behavioral ratings, and WISC-R performance. Biofeedback and Self-Regulation 20(1), 83–99 (1995) 3. Fuchs, T., Birbaumer, N., Lutzenberger, W., Gruzelier, J.H., Kaiser, J.: Neurofeedback treatment for attention-deficit/hyperactivity disorder in children: A comparison with methylphenidate. Applied Psychophysiology Biofeedback 28(1), 1–12 (2003) 4. Coben, R., Linden, M., Myers, T.E.: Neurofeedback for autistic spectrum disorder: A review of the literature. Applied Psychophysiology Biofeedback 35(1), 83–105 (2010) 5. Kouijzer, M.E.J., van Schie, H.T., de Moor, J.M.H., Gerrits, B.J.L., Buitelaar, J.K.: Neurofeedback treatment in autism. Preliminary Findings in Behavioral, Cognitive, and Neurophysiological Functioning. Research in Autism Spectrum Disorders 4(3), 386–399 (2010) 6. Saxby, E., Peniston, E.G.: Alpha-theta brainwave neurofeedback training: An effective treatment for male and female alcoholics with depressive symptoms. Journal of Clinical Psychology 51(5), 685–693 (1995) 7. Sokhadze, T.M., Cannon, R.L., Trudeau, D.L.: EEG biofeedback as a treatment for substance use disorders: Review, rating of efficacy, and recommendations for further research. Applied Psychophysiology Biofeedback 33(1), 1–28 (2008) 8. Demos, J.N.: Getting Started with Neurofeedback. WW Norton & Company, New York (2005) 9. Gevensleben, H., Holl, B., Albrecht, B., Schlamp, D., Kratz, O., Studer, P., Wangler, S., Rothenberger, A., Moll, G.H., Heinrich, H.: Distinct EEG effects related to neurofeedback training in children with ADHD: A randomized controlled trial. International Journal of Psychophysiology 74(2), 149–157 (2009) 10. Wang, Q., Sourina, O., Nguyen, M.K.: EEG-based “Serious” Games Design for Medical Applications. In: Proc. 2010 Int. Conf. on Cyberworlds, Singapore, pp. 270–276 (2010)


599

11. Sourina, O., Wang, Q., Liu, Y., Nguyen, M.K.: A Real-time Fracal-based Brain State Recognition from EEG and Its Application. In: Proc of Biosignals 2011, Rome, Italy, pp. 82– 91 (2011) 12. Block, A., Von Bloh, W., Schellnhuber, H.J.: Efficient box-counting determination of generalized fractal dimensions. Physical Review A 42(4), 1869–1874 (1990) 13. Higuchi, T.: Approach to an irregular time series on the basis of the fractal theory. Physica D: Nonlinear Phenomena 31(2), 277–283 (1988) 14. Ishino, K., Hagiwara, M.: A feeling estimation system using a simple electroencephalograph. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 4205, pp. 4204–4209 (2003) 15. Petrantonakis, P.C., Hadjileontiadis, L.J.: Emotion recognition from EEG using higher order crossings. IEEE Transactions on Information Technology in Biomedicine 14(2), 186– 197 (2010) 16. Zhang, Q., Lee, M.: Analysis of positive and negative emotions in natural scene using brain activity and GIST. Neurocomputing 72(4-6), 1302–1306 (2009) 17. Takahashi, K.: Remarks on emotion recognition from multi-modal bio-potential signals. In: 2004 IEEE International Conference on Industrial Technology, IEEE ICIT 2004, 8-10 2004, vol. 1133, pp. 1138–1143 (2004) 18. Liu, Y., Sourina, O., Nguyen, M.K.: Real-time EEG-based Human Emotion Recognition and Visualization. In: Proc. 2010 Int. Conf. on Cyberworlds, Singapore, pp. 262–269 (2010) 19. : American electroencephalographic society guidelines for standard electrode position nomenclature. Journal of Clinical Neurophysiology 8(2), 200–202 (1991) 20. Sanei, S., Chambers, J.: EEG signal processing. John Wiley & Sons, Chichester (2007) 21. Hentschel, H.G.E., Procaccia, I.: The infinite number of generalized dimensions of fractals and strange attractors. Physica D: Nonlinear Phenomena 8(3), 435–444 (1983) 22. Renyi, A.: On a new axiomatic theory of probability. Acta Mathematica Academiae Scientiarum Hungaricae 6(3-4), 285–335 (1955) 23. Kulish, V., Sourin, A., Sourina, O.: Human electroencephalograms seen as fractal time series: Mathematical analysis and visualization. Computers in Biology and Medicine 36(3), 291–302 (2006) 24. Kulish, V., Sourin, A., Sourina, O.: Analysis and visualization of human electroencephalograms seen as fractal time series. Journal of Mechanics in Medicine and Biology, World Scientific 26(2), 175–188 (2006) 25. Sourina, O., Kulish, V.V., Sourin, A.: Novel Tools for Quantification of Brain Responses to Music Stimuli. In: Proc of 13th International Conference on Biomedical Engineering, ICBME 2008, pp. 411–414 (2008) 26. Haptek, http://www.haptek.com 27. Stevens, K.: Patients’ perceptions of music during surgery. Journal of Advanced Nursing 15(9), 1045–1051 (1990) 28. Crawford, C.: Chris Crawford on interactive storytelling. New Riders, Berkeley (2005) 29. Mateas, M., Stern, A.: Natural Language Understanding in Façade: Surface-Text Processing. In: Göbel, S., Spierling, U., Hoffmann, A., Iurgel, I., Schneider, O., Dechau, J., Feix, A. (eds.) TIDSE 2004. LNCS, vol. 3105, pp. 3–13. Springer, Heidelberg (2004) 30. Cavazza, M., Pizzi, D., Charles, F., Vogt, T., André, E.: Emotional input for characterbased interactive storytelling. In: Proc of the 8th International Conference on Autonomous Agents and Multiagent Systems, Budapest, Hungary, pp. 313–320 (2009) 31. IDM-Project: Emotion-based personalized digital media experience in Co-Spaces (2008), http://www3.ntu.edu.sg/home/eosourina/CHCILab/projects.html