Building Autonomous Social Partners for Autistic

0 downloads 0 Views 110KB Size Report
(Eds.): IVA 2012, LNAI 7502, pp. 46–52, 2012. ... animated talking heads for language and speech training. Autonomous .... to videos of children's interactions with Andy. ... coded by a coder trained in the modified SAP coding scheme. Video ...
Building Autonomous Social Partners for Autistic Children Sara Bernardini1 , Kaska Porayska-Pomsta1, Tim J. Smith2 , and Katerina Avramides1 1

London Knowledge Lab., Institute of Education, University of London, 23-29 Emerald Street, London WC1N 3QS, United Kingdom {S.Bernardini,K.Porayska-Pomsta,K.Avramides}@ioe.ac.uk 2 Psychological Sciences, Birkbeck, University of London, Malet Street, London WC1E 7HX, United Kingdom [email protected]

Abstract. We present the design and implementation of an autonomous virtual agent that acts as a credible social partner for children with Autism Spectrum Conditions and supports them in acquiring social communication skills. The agent’s design is based on principles of best autism practice and input from users. Initial experimental results on the efficacy of the agent show encouraging tendencies for a number of children. Keywords: Virtual Social Partners, Pedagogical Agents, Autonomy.

1

Introduction

This paper presents an autism practice-based approach to designing and implementing an autonomous virtual agent that can act credibly both as a peer and as a tutor to support young children with Autism Spectrum Conditions (ASCs) in developing social communication skills. The design of the agent is based on participatory design workshops with practitioners and children as well as the SCERTS framework [7] - an established educational intervention approach aimed to support social communication (SC) and emotional regulation (ER) of children with ASCs through appropriately designed transactional support (TS). Our pedagogical agent is implemented in a virtual environment called ECHOES intended for real-world use in schools as part of their everyday activities. Autism is a spectrum of neuro-developmental conditions that affects three main areas (“triad of impairments” [1]): (i) communication: problems with verbal and non-verbal language; (ii) social interaction: problems with recognising and understanding other people’s emotions and with expressing their own emotions; and (iii) patterns of restricted or repetitive behaviours: problems with adapting to novel environments. We focus on enhancing the social communication competence of children with ASCs because this is the domain with which they typically have the most difficulty [6] and because recent studies indicate that individuals with ASCs and their caregivers consider support for social communication as the most desirable feature of technology-enhanced intervention [8]. Y. Nakano et al. (Eds.): IVA 2012, LNAI 7502, pp. 46–52, 2012. c Springer-Verlag Berlin Heidelberg 2012 !

Building Autonomous Social Partners for Autistic Children

47

The paper is organised as follows: Section 2 discusses why children with ASCs may benefit from virtual agents; Section 3 presents the SCERTS model, which provides the theoretical foundation of our agent’s design; Section 4 describes the design and the implementation of the ECHOES agent. Sections 5 and 6, respectively, report the evaluation of the agent and offer our conclusions.

2

Agent Technology for Autism

Children with ASCs have an affinity with technology and are motivated by computer-based training [8]. Software programs are predictable and structured environments that can accommodate the children’s need for organisational support and their preference for routine behaviours. The anxiety linked with social interaction can be mitigated by the use of artificial peers which are tireless, consistent and positive towards the child regardless of the child’s behaviours. An appropriately designed artificial peer can meet individual children’s needs and allow them to exercise the same skill in different scenarios, from structured situations to gradually more unpredictable contexts, thereby increasing the chances of transferring the learned skills from the virtual to the real world [11,2]. Despite a recent growing interest in the potential of artificial agents, both virtual and physically embodied, for human-computer interaction the efforts have focused primarily on agents with little or no autonomy, with the exception of the Thinking Head project [4], which focuses on developing a talking head that teaches social skills through its ability of realistically portraying facial expressions, and the virtual peers, Baldi and Timo [2], which are 3-D computeranimated talking heads for language and speech training. Autonomous agents carry a significant potential for autism intervention for children, because they can compliment the intensive one-on-one support that the children need, by allowing human practitioners to focus on the most complex aspects of face-to-face interventions, while managing any repetitive tasks and on-demand access. The approach presented in this paper focuses on the development of a fully autonomous agent, i.e. an agent that is able to decide independently how to act best in order to achieve a set of high-level goals that have been delegated to it. In keeping with the classic agent theory of Wooldridge and Jennings [13], in addition to autonomy, the ECHOES agent is equipped with: (i) pro-activeness; (ii) reactivity; and (iii) social ability. The agent’s pro-activeness is important to maintaining the child’s attentional focus and to foster motivation. Reactivity is fundamental to adapting the support to the children’s changing needs as well as cognitive and affective states, while social ability – to maximising the chances of the child to experience a sense of self-efficacy in communicating with the agent.

3

Pedagogical Underpinnings for the ECHOES Agent

In order to identify the social communication skills that a virtual agent needs to possess to act as a credible social partner to children with ASCs and to support

48

S. Bernardini et al.

their social competencies, we draw from SCERTS [6], a comprehensive approach to social communication assessment and intervention in autism. SCERTS identifies the particular skills that are essential for successful social communication and which we argue are also necessary for an ideal virtual agent to possess in order to act as a credible social partner to children with ASCs. These skills are encapsulated in three overarching domains: (i) Social Communication: spontaneous and functional communication, emotional expression, and secure and trusting relationships with others; (ii) Emotional Regulation: the ability to maintain a well-regulated emotional state to cope with stress and to be available for learning and interacting; (iii) Transactional Support : support to help caregivers respond to the childs needs and interests, adapt the environment, and provide tools to enhance learning. SCERTS breaks down each domain into a number of essential constitutive components. Then, for each component, it provides a detailed description of the education objectives to be achieved, the strategies for intervention and the assessment criteria. We build on this operationalisation of social communication for designing the agent’s behaviour. The interaction between the child and the agent is structured around twelve different learning activities and is facilitated by a large multitouch LCD display. The learning activities focus on social communication and, in particular, on the two sub-components of social communication that have been identified by SCERTS as the most challenging for ASCs children: (i) Joint Attention: ability to coordinate and share attention and emotions, express intentions, and engage in reciprocal social interactions by initiating/responding to bids for interaction; and (ii) Symbol Use: understanding of meaning expressed through conventional gestures and words and ability to use nonverbal means to share intentions.

4

The Design and Implementation of the ECHOES Agent

We designed an artificial social partner, called Andy, that can act credibly both as a peer and as a tutor to ASCs children based on (a) the SCERTS model and (b) input from thirty ASCs practitioners, who participated in two workshops organised over the lifespan of the ECHOES project and involving storyboarding tools, group discussions and individual interviews. We now describe the main design choices that we have made to create Andy. Agent’s Intelligence: Among the various domain-independent agent architectures that have been proposed for building agents, FAtiMA [3] is ideally suited to fulfil the design requirements of our agent, because it combines the kind of reactive and cognitive capabilities needed to implement an autonomous, proactive and reactive agent with the socio-emotional competence that we envisaged for our agent in this context. The cognitive layer of FAtiMA is based on artificial intelligence planning techniques [9], while the emotional model is derived from the OCC theory of emotions [5] and the appraisal theory [10]. A FAtiMA agent is characterised by: (1) a set of internal goals; (2) a set of action strategies to achieve these goals; and (3) an affective system composed of emotional reaction

Building Autonomous Social Partners for Autistic Children

49

rules, action tendencies, emotional thresholds and emotion decay rates. The two main mechanisms controlling a FAtiMA agent are appraisal and coping. The agent experiences one or more of the 22 emotions of the OCC model, based on its appraisal of the current external events and its subjective tendencies to experience certain emotions instead of others. The agent deals with these emotions by applying problem-focused or emotion-focused coping strategies. Both the appraisal and the coping work at two different levels: the reactive level, which affects the short-term horizon of the agent’s behaviour, and the deliberative level, which relates to the agent’s goal-oriented behaviour. In ECHOES , each learning activity has an associated FAtiMA agent model. All these models share the same specification of the agent’s affective system, because we want the agent to maintain the same personality from session to session in order to establish a trusting relationship with the child. Andy is a positive, motivating and supportive character that tends to be happy and does not get frustrated easily. We obtained such behaviour by manipulating Andy’s emotional reaction rules as well as the emotional thresholds and decay rates of the OCC emotions available in FAtiMA. We control Andy’s facial expressions and gestures through the specification of Andy’s action tendencies. For example, the agent smiles when it is happy and opens its mouth when it is surprised. While Andy’s personality does not change between activities, the goals that the ECHOES agent actively tries to pursue and its action strategies are specified for each learning activity based on: (i) the high-level pedagogical goals on which the activity focuses and (ii) the specific narrative content of the activity itself. Repertoire of the Agent’s Behaviours: Since the focus of our environment is on supporting children’s social communication, the agent’s actions are either concrete demonstrations of the related skills or actions performed to trigger the child to practice those skills. Specifically, we define the joint attention and symbolic use in terms of three component skills: (i) Responding to bids for interaction; (ii) Initiating bids for interaction; and (iii) Engaging in turn taking. Our agent is able to perform these skills in three different ways: (a) Verbally by using simple language or key words; (b) Non-verbally through gaze and gestures such as pointing at an object or touching an object; and (c) By combining verbal and non-verbal behaviours. Initiating a non-verbal bid for interaction by the agent involves Andy looking at the child, then looking at and indicating an object and then looking back at the child. Our agent is able to make requests, to greet the child by name, to comment on events happening in the garden and to use exploratory actions on objects. This variety of behaviours makes the interaction dynamic enough to keep the child engaged and to foster generalisation, while retaining a degree of predictability that is essential to supporting the childs attentional focus. The practitioners emphasised the importance of providing the children with positive feedback in order to reduce the children’s anxiety of social interactions and to help them experience a sense of self-efficacy. Andy always provides the child with positive feedback, especially if the child follows the agent’s bids for interaction correctly. If the child does not perform the required action, the agent first waits for the child to do things at their own pace

50

S. Bernardini et al.

and then intervenes by demonstrating the action. Andy is a responsive agent. Its responsiveness ranges from physical reactions to actions performed on its body (e.g., Andy laughs if the child tickles it), to the ability to respond to the child’s changing needs and cognitive and emotional states. Ideally, the agent should attune its emotional tone to that of the child in order to keep emotional engagement with the child. Such a sophisticated level of reactiveness requires that the agent is able to assess in real-time the current cognitive and emotional state of the user. Our current system includes a simple user model, which evaluates the child’s current state based on the real-time information from the touch system. Agent’s Physical Appearance and Communication: Following SCERTS’ emphasis on creating opportunities for children with ASCs to play with other children and develop positive relationships with them [6], we chose a child-like physical appearance for the agent. Based on input from the practitioners, we gave our agent a cartoonish look and animated it based on observations of children’s cartoons in order to motivate the children. Our agent uses (a fragment of) the Makaton language [12] to facilitate communication with the child. Makaton is a language programme of signs and symbols to support spoken language and is used with speech. Since auditory information can be challenging for children with ASCs, we kept the spoken language as simple as possible. The agent can perform a good range of positive facial expressions: it can smile, laugh, look surprised, happy, and excited. These expressions are implemented by careful use of movements in the agent’s lips, eyes and eyebrows. They are usually accompanied by body gestures consistent with emotion showed through the face in order to reinforce the message.

5

Experimental Results

To assess the impact of Andy and ECHOES on social communication in children with ASCs, a large scale multi-site intervention study was conducted. The system was deployed in five schools in the UK. 19 children with ASCs participated in the study, during which they played with ECHOES for 10 to 20 minutes, several times a week over an eight week period. The SCERTS Assessment Protocol (SAP) [7] was modified into a finer-grained coding scheme that could be applied to videos of children’s interactions with Andy. The modified SAP coding scheme contains 16 main behavioural categories. Due to space limitations, we will focus only on two main social behaviours, which are severely impaired in children with ASCs: responding to and initiating bids for interaction. Fifteen minute periods during which the children interacted with Andy were identified for analysis from the beginning, middle and end of the intervention period. Each video was blindcoded by a coder trained in the modified SAP coding scheme. Video annotations were applied in “Elan” and moderated by a second coder. Child’s Response to Agent: The mean probability that a child responded to the practitioner’s bids for interaction during the table-top pre-test was 0.62 (SD=0.19) and 0.71 (SD=0.14) after the intervention (slight increase was not

Building Autonomous Social Partners for Autistic Children

51

significant t(14)=-1.644, p=.123). Across the three ECHOES sessions the probability of responding to Andy’s initiations decreased slightly: beginning = 0.57 (SD=0.22), middle = 0.53 (SD 0.25), end = 0.49 (SD 0.24); although this decrease was not significant. However, the probability of responding to the human practitioner’s bids for interaction increased marginally across the three sessions: beginning = 0.74 (SD=0.21), middle = 0.75 (SD 0.21), end = 0.81 (SD 0.21), and neared significance: beginning vs. end, t(18)=-2.017, p=.059. This increase may suggest a comfort with the ECHOES environment that elicited a level of responsiveness in the child not observed during the table-top activity. As Andy is a critical part of the ECHOES environment we can assume his presence contributed to this improvement. Child’s Initiations to Agent: A more advanced social behaviour rarely observed in ASCs children is the initiation of social interactions. The frequency with which the child initiated an interaction during the table-top pre-test was low, 8.63 times (SD 7.94) and did not change by the post table-top session, 7.84 times (SD 10.0): t(18)=.456, p=.654, n.s. Although there was no improvement in the real-world scenario, initiations to Andy numerically increased across the three ECHOES sessions from 4.98 (SD 8.05) to 6.68 (SD 7.68) and 9.58 (SD 13.67). Unfortunately, this group difference did not reach significance (t(18)=1.699, p=.106, n.s.) even though eight children increased their number of initiations to Andy, seven produced the same number and only four decreased. This suggests that the heterogeneity in our ASCs population may obscure a group increase. For a number of children Andy appears to be eliciting a large increase in spontaneous initiations. This is strikingly obvious when examining videos of the ECHOES sessions. For example, one child who showed no initial interest in Andy spontaneously waved and said “Hi Andy!” when the agent walked on the screen in a later session. Such behaviours were extremely surprising to teachers within the school who believed the child in question to be non-communicative.

6

Conclusions and Future Work

In this paper, we presented our approach to designing and implementing an autonomous pedagogical agent for supporting social communication skills of children with ASCs based on principled intervention guidelines, recommendations from autism practitioners and children themselves. We undertook a major evaluation of the ECHOES system involving a significant number of children. A preliminary analysis of child behaviours in relation to Andy shows that, whilst no statistically significant differences in social communication have been observed across all children, some children benefited from their exposure to Andy and the ECHOES environment. Andy’s reciprocal interactions with the children appear to elicit spontaneous social behaviours. With improved versions of the system, especially focusing on a more comprehensive user modelling capabilities, we hope that the trend of improvements in the social behaviours manifested by the children will prove significant.

52

S. Bernardini et al.

References 1. American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders. 4th edn., Text Revision (DSM-IV-TR) (2000) 2. Bosseler, A., Massaro, D.: Development and evaluation of a computer-animated tutor for vocabulary and language learning in children with autism. Journal of Autism and Developmental Disorders 33(6), 653–672 (2003) 3. Dias, J., Paiva, A.: Feeling and Reasoning: A Computational Model for Emotional Characters. In: Bento, C., Cardoso, A., Dias, G. (eds.) EPIA 2005. LNCS (LNAI), vol. 3808, pp. 127–140. Springer, Heidelberg (2005) 4. Milne, M., Luerssen, M., Lewis, T., Leibbrandt, R., Powers, D.: Development of a virtual agent based social tutor for children with autism spectrum disorders. In: Proc. of the International Joint Conference on Neural Networks, pp. 1–9 (2010) 5. Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press (1988) 6. Prizant, B., Wetherby, A., Rubin, E., Laurent, A.: The scerts model: A transactional, family-centered approach to enhancing communication and socioemotional ability in children with autism spectrum disorder. Infants and Young Children 16(4), 296–316 (2003) R Model: 7. Prizant, B., Wetherby, A., Rubin, E., Laurent, A., Rydell, P.: The SCERTS! A Comprehensive Educational Approach for Children with Autism Spectrum Disorders. Brookes (2006) 8. Putnam, C., Chong, L.: Software and technologies designed for people with autism: what do users want? In: 10th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 3–8 (2008) 9. Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach, 2nd edn. Prentice Hall (2003) 10. Smith, C.A., Lazarus, R.S.: Emotion and adaptation. In: Vlahavas, I., Vrakas, D. (eds.) Handbook of Personality: Theory and Research, ch. 23, pp. 609–637. Guilford, New York (1990) L.A. Pervin (Ed.) 11. Tartaro, A., Cassell, J.: Playing with virtual peers: bootstrapping contingent discourse in children with autism. In: Proc. ICLS 2008, pp. 382–389 (2008) 12. Walker, M., Armfield, A.: What is the makaton vocabulary? Special Education: Forward Trends 8(3), 19–20 (1981) 13. Wooldridge, M., Jennings, N.R.: Intelligent agents: Theory and practice. Knowledge Engineering Review 10(2), 115–152 (1995)