Music Inspired Framework for Remediating Emotional ... | ScienceDirect

Procedia Computer Science Volume 88, 2016, Pages 469–474 7th Annual International Conference on Biologically Inspired Cognitive Architectures, BICA 2016

Music Inspired Framework for Remediating Emotional Deficits in Autism Margaret Tan1* & Neha Khetrapal1† 1

Institute of High Performance Computing, A*STAR [email protected], [email protected]

Abstract Autism Spectrum Disorders (ASD) is a lifelong communication disorder that limits the abilities of diagnosed individuals to relate socially and interpret emotional cues. Thus, it is important to have early interventions in the domains of social and affective functioning. Recent research efforts have focused on the innovative applications of Assistive Technologies (AT) for rehabilitation efforts. However, despite excellent preliminary findings, the efficacy of AT remains limited. This paper aims to fill the identified efficacy gap by proposing a framework incorporating music as a therapy which will be developed into a technological application to help children with autism to deal with their emotional dysfunctions. The proposal is also based on findings which show that this special population prefers and has successfully used technological devices such as the iPad for learning new skills. Keywords: Autism, Adaptive Living, Brain, Music Therapy, Social and Communicative Deficits

1 Introduction Autism Spectrum Disorders (ASD) is a lifelong developmental disorder characterized by impairments in social interaction, communication and presence of restricted repetitive behaviors (Rogers et al., 2013). Particularly, they have problems with identification of emotions from facial expressions and tone of voice (Stewart et al., 2013). Erratic identification and inappropriate expressions of emotion puts the affected individuals at a comparative social disadvantage as these impairments hinder social adjustments and affects cognitive learning negatively. Individuals with autism exhibit peculiar cognitive strengths (which we could harness for promoting their wellbeing) such as keen attention to details, islets of musical capability and other well developed abilities in the domain of numbers, auditory pitch processing and keen use of computer applications (Itzchak et al., 2013). Therefore different types of Assistive Technologies (AT) have been proposed to remediate these emotional deficits (Ploog et al., 2013). The technologies are aimed towards teaching recognition of emotional cues in facial expressions. A challenge inherit in these technologies, is the * †

Masterminded EasyChair and created the first stable version of this document Created the first draft of this document

Selection and peer-review under responsibility of the Scientiﬁc Programme Committee of BICA 2016 c The Authors. Published by Elsevier B.V.

doi:10.1016/j.procs.2016.07.466

469

Music Inspired Framework for Remediating Emotional Deﬁcits in Autism

M. Tan & N. Khetrapal

generalization of learning. It is still unclear whether individuals with autism are able to successfully transfer their learned skills across domains (de Marchena et al., 2015). We aim to fill this gap by developing a framework for rehabilitation of their emotional deficits based on one of their cognitive strengths, that is, affinity for music. This paper describes the theoretical framework and charts the course of typical and atypical emotional development. We then propose an experimental design to validate our new framework.

2 Review on Emotions: Implications for Autism Infants from early on are confronted with the task of making sense of auditory stimuli that enters their cognitive systems such as classifying an episode of auditory stimulation as a speech or nonspeech stimulus. As they get older and have amassed more experience, they learn to provide linguistic labels to these bits of ‘speech’ stimulation, for instance, labeling consonants versus vowels. These linguistic labels get increasingly complex with age (e.g. words). Parallel developments also occur for the domain of emotions. Infants are able to ‘interpret’ emotions from music and the tone of voice as they navigate the social world. The undifferentiated understanding of emotions from music and tone of voice implies that these two systems are not differentiated in the early stage of life (McMullen & Saffran, 2004). What leads to the differentiation? The infant is gradually able to understand that speech is often coated with paralinguistic cues where differences in the way a word is produced signals different emotions, e.g., the use of high pitched voice signal anger. In a nutshell, the inference of emotions at the early stages occur in a pre-linguistic manner where these ‘emotions’ are devoid of language labels. The constant coupling of babies’ facial emotional expressions, reflecting their inner state, and their mother’s imitation of these expressions forms the basis for later emerging emotion labeling ability (Allen & Heaton, 2010). However, socially communicative cues like the tone of voice and the use of facial expressions are less informative for children with ASD. This reduced inability puts these children at a disadvantage for kick-starting the process of emotion labeling. At advanced developmental stages, much later than the typical developmental milestone, some children with autism who are high on verbal IQ (Quintin et al., 2011) may be able to hack this process or develop compensatory strategies. Most of the children with autism are unable to partition their inner emotional milieu and categorize it with the help of labels making them susceptible to problems for overt communication and inner emotional regulation. This is because linguistic labels attached to particular all-encompassing emotion experiences helps a typical individual bring the ‘labeled’ experience to the forefront of his consciousness in order to be able to talk about it for the purpose of communication or ponder over it internally for emotional regulation.

3 Assistive Technologies for Autism Research has shown that these special individuals prefer technological applications over people as applications are simple and easy to manipulate after discovering the underlying patterns. It is thus understandable why communicating with people is difficult as in most instances human communication is not patterned by a rule book. In this context, ATs have been developed to closely align with their keenness for using technological applications. It has been found that individuals with autism are good at systematizing (Baron-Cohen, 2010). For this reason, computer based methods for teaching emotion recognition (e.g. methods that focus on distinguishable emotional signatures like a downturned mouth for sad) has shown some degree of success (Golan et al., 2006). But emotional recognition is much more than just dissection of fragments of emotional signatures as many modalities enrich an ‘emotional episode’ in the real world. These episodes encompass both the expression as well as detection of emotions. The modalities support emotional recognition and expression, both of which

470



are intricately dependent upon tone of voice, facial expressions, eye gaze, heart rate and other brain activity. The latter two contribute towards the internal emotional milieu. Advances in computing technology have taken advantage of these emotional multi modalities in order to develop systems that can detect affective states and provide intelligent responses (Calvo & De’Mello, 2010). Individuals with autism are good at systematizing, thus this mode of functioning spills over to the social world as a means of analyzing social stimuli. Unfortunately, the social world is not scripted by set rules, for example, people may have different ways of expressing happiness in different cultures and an apparent rule book detailing display rules for different emotions is bound to fail here. Even within one culture people are likely to express their emotions differently. This is where the problem arises for individuals with ASD. To overcome this to some extent, portable digital assistants (PDA) has been devised (Bishop, 2003). Other recent ecological developments include detecting emotions in individuals with autism based on their physiological indices and using that for modulating the behavior of the robot, targeted towards teaching social skills (Conn et al., 2008). Wearable portable technologies geared towards self-affect sensing have also been developed with the goal of increasing self-awareness. For example, the Galvactivator is a sensory glove that converts varying levels of skin conductance to the brightness of a glowing LED (Picard & Scheirer, 2001). As skin conductance varies with different emotional states, the glove helps the wearers to reflect on the inner ‘affective’ states perhaps suitable for people with autism who lack personal emotional terminology for introspection. Another attempt at intervening for the lack of appropriate ‘mental state’ words, was described by El Kaliouby et al. (2006) through the development of a ‘selfcam’ that consists of a small video-camera worn over one’s chest. The camera analyses the mental state of the wearer in real time and communicates the analyzed information back to the wearer through the visual, auditory or the vibratory mode. Virtual Environments (VE) have proven to be another area of research holding important implications for interventions (Bellani, 2011). VE has the advantages of mimicking specific social situations in which the users can participate with the aim of learning social role-play or taking turns in conversations. The predictable nature of these immersive environments serves as a learning ground for individuals with ASD who are unable to cope with the dynamic nature of the social environment. Efficacy studies show that the affected individuals are able to participate and learn simple social skills following immersion (Parsons et al., 2006).

4 Music Inspired Framework Biologically, music has special ‘healing’ features as it helps the neurons of the brain to communicate in a more effective and synchronized manner (Thaut, 2005) much like a set up where several metronomes set to the same tempo but with different phases will synchronize and end up sharing the same phase, after being placed on a table. Once energy travels from one metronome to the other though the table, all metronomes synchronize (Schon & Tillmann, 2015). This metaphor aptly describes what may happen to the brain after exposure to music. Music with its periodic and temporally structured sounds can synchronize the out of sync neurons, as may happen in the autistic model (Hove & Risen, 2009), with the help of acoustic and neuroelectric waves, making an individual more receptive to learning (Schon & Tillmann, 2015). Thus, we propose a new rehabilitative framework incorporating music. Individuals with autism crave for emotional experience and music helps to satisfy this unmet inner need. The unmet inner need is likely to be the result of the traditional rehabilitative approach that just focuses on pairings of emotional pictures (e.g. faces) with linguistic labels. Such an approach only aims at teaching emotion recognition (Figure 1a). In comparison, Figure 1b shows our approach that connects music with the inner feelings. Our approach is likely to be successful as research has shown that participants with autism have typical-like physiological responses as measured by changes in the Galvanic Skin

471



Response (GSR) (Allen et al., 2003) in response to music. In other words, with the natural induction of emotion that music can arouse coupled with simultaneous associations of emotional pictures (e.g. music that portrays fear is paired with an ‘fearful’ picture of thunderstorm), we hope to strengthen the inner emotional experiences giving rise to better emotional regulation and communication skills.

Figure 1 Comparison of the Traditional and Music Inspired Approaches

5 Research Methodology Our methodology is motivated by the established system of picture learning that forms the cornerstone of the famous picture exchange communication system (PECS) (Charlop-Christy et al., 2002) and has proven to be successful as a teaching paradigm in autism. We believe that an improved ability to emotionally categorize music with the help of pictures in contrast to linguistic labels will help these diagnosed children develop an inner sense of emotional awareness. We therefore hypothesize that towards the end of the therapy, children will have higher ability to categorize the four basic emotions and think about these emotions in terms of pictures ultimately giving way to better overt communication. To validate the new framework, we will assess their learning ability with the help of a questionnaire. The items on the questionnaire will ask for specific information about the affective functioning of the child from parents or teachers. There are basically three phases on designing the music as AT. In phase 1, we build a repertoire of music library with pieces representing four important emotions such as happy, sad, anger and fear. Each piece of music with duration of 30 seconds will be composed targeting each emotion. Typically developing adults and children will be requested to rate the musical pieces for emotional intensity. 10 musical pieces representing each emotion (n = 40) that receive the highest rating will be chosen for working with individuals with autism. We will also develop appropriate emotional pictures that could be paired with each of the emotional category of music, and a computer platform will be used to present the pairings of emotional pictures and music. In phase 2, we will then pilot with two groups of children diagnosed with autism (aged 7 to 12 years). These individuals will be administered with a baseline questionnaire to assess their emotional awareness and perception. The experimental group will be encouraged to pair music from each of the four emotional categories with consistent emotional pictures. The control group will undergo similar procedure except for one change where instead of music they will be encouraged to pair emotional pictures with emotional words (text) much like the traditional protocol. This group will importantly establish the role of music as the new AT. The experimental music-emotional picture pairing can be equated to an immersive VR experience whereby children with autism experience emotional episodes in a ‘stripped down’ and non-overwhelming manner. The intervention sessions will be carried out for 3 days in a week with one child with at least 30 minutes each day. The entire intervention comprising

472



the music-picture pairing or text-picture pairing will be carried out for 4 weeks where 1 week will be devoted for one emotion (emotions: happy, sad, anger and fear). There will be two additional weeks where the children will be exposed to a new set of emotional music (n = 5/per emotion) and requested to pair consistent pictures in order to assess for generalizations of learning. Finally at the conclusion of the intervention session, the parents and the teachers will complete the same questionnaire for the purpose of comparing pre and post intervention scores. A better performance during the last two weeks and a higher adaptive score on the questionnaire will establish the efficacy of music for ameliorating emotional deficits. In phase 3, the aim is to develop the music application in a smart device (e.g. iPad or iPhone). This smart application will allow children to engage in the rehabilitation exercise at all times either from their homes or intervention centers as they socialize and communicate with their parents and teachers. However, to validate our framework, we need to further understand how the new smart application can be effectively combined with other forms of therapeutic interventions.

6 Conclusion People with ASD show certain atypicalites of social and communicative functioning, yet they also show cognitive strengths in other domains, such as their affinity for music and using computer aided devices. Technologies aligned with the strengths of the affected individuals have been developed as intervention tools to ameliorate emotional recognition deficits. Our proposed framework goes a step further by introducing music as the new AT geared towards strengthening the connections between their inner emotional experiences and the outside world as represented by pictures akin to emotional learning in infancy. The aim of the framework is to provide a long term solution that assists children with autism differentiate basic emotions (happy, sad, anger and fear) so that they are able to introspect upon them and map these to consistent emotional events in the outside world. We acknowledge that our framework for learning emotions is based on music while social interactions in the real world do not take place in a musical context. Thus, we encourage future research to further investigate how findings from our proposed studies could be generalized to the real world.

References Allen, R. & Heaton, P. (2010). Autism, Music and the Therapeutic Potential of Music in Alexithymia. Music Perception: An Interdisciplinary Journal, 27(4), 251-261. Baron-Cohen, S., (2010). Empathizing, Systemizing, and the Extreme Male brain theory of Autism. Progress in Brain Research, 186, 167-175. Bellani, M., Fornasari L., Chittaro L. & Brambilla P. (2011). Virtual Reality in Autism: State of the Art. Epidemiology and Psychiatric Sciences, 20(3), 235-238. Bishop, J. (2003). The Internet for educating individuals with social impairments. Journal of Computer Assisted Learning, 19(4), 546-556. Calvo, R.A. & D'Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18-37. CharlopǦChristy, M.H., Carpenter M., Le L., LeBlanc, L.A. & Kellet, K. (2002). Using the picture exchange communication system (PECS) with children with autism: Assessment of PECS acquisition, speech, socialǦcommunicative behavior & problem behavior. Journal of Applied Behavior Analysis, 35(3), 213-231.

473



Conn, K., Liu, C., Sarkar, N., Stone, W. & Warren, Z. (2008). Towards affective-sensitive intervention technologies for children with autism. Affective Computing: Focus on Emotion Expression, Synthesis and Recognition. Vienna, Austria, pp. 365-390. de Marchena, A.B., Eigsti, I.M. & Yerys, B.E. (2015). Brief Report: Generalization Weaknesses in Verbally Fluent Children and Adolescents with Autism Spectrum Disorder. Journal of Autism and Developmental Disorders, 45(10), 3370-3376. El Kaliouby, R., Teeters, A., & Picard, R.W. (2006). An exploratory social-emotional prosthetic for autism spectrum disorders. Wearable and Implantable Body Sensor Networks. International Workshop on IEEE. Golan, O., Baron-Cohen, S. & Golan, Y. (2006). Reading the mind in films – testing recognition of complex emotions and mental states in adults with and without autism spectrum conditions. Social Neuroscience, 1(2), 111-123. Hove, M.J. & Risen, J.L. (2009). It's all in the timing: Interpersonal synchrony increases affiliation. Social Cognition, 27, (6), 949-961. Itzchak, E.B., Aviva, B. & Zachor, D.A. (2013). Are special abilities in autism spectrum disorder associated with a distinct clinical presentation? Research in Autism Spectrum Disorders, 7(9), 1122-1128. McMullen, E. & Saffran, J.R. (2004). Music and Language: A Developmental Comparison. Music Perception, 21(3), 289-311. Parsons, S., Leonard, A. & Mitchell, P. (2006). Virtual environments for social skills training: comments from two adolescents with autistic spectrum disorder. Computers & Education, 47(2), 186-206. Picard, R.W. & Scheirer, J. (2001). The galvactivator: A glove that senses and communicates sensitivity. Proceedings of the 9th International Conference on Human Computer Interaction. New Orleans, pp. 1538-1542. Ploog, B.O., Scharf, A., Nelson, D. & Brooks, P.J. (2013). Use of computer-assisted technologies (CAT) to enhance social, communicative, and language development in children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 43(2), 301-322. Quintin, E., Bhatara, A., Poissant, H., Fombonne, E. & Levitin, D.J. (2011). Emotion perception in music in high-functioning adolescents with autism spectrum disorders. Journal of Autism and Developmental Disorders, 41(9), 1240-1255. Rogers, T.D., McKimm, E., Dickson, P.E., Goldowitz, D., Blaha, C.D. & Mittleman, G. (2013). Is autism a disease of the cerebellum? An integration of clinical and pre-clinical research. Frontiers in Systems Neuroscience, 7(15), 1-16. Schön, D. & Tillmann, B. (2015). ShortǦand longǦterm rhythmic interventions: perspectives for language rehabilitation. Annals of the New York Academy of Sciences, 1337(1), 32-39. Stewart, M.E., McAdam, C., Ota, M., Peppé, S. & Cleland, J. (2013). Emotional recognition in autism spectrum conditions from voices and faces. Autism, 17(1), 6-14. Thaut, M.H. (2005). Rhythm, music, and the brain: Scientific foundations and clinical applications (Vol. 7). Routledge.

474