Mattheiss, E., Kickmeier-Rust, M. D., Steiner, C. M., & Albert, D. (2010). Approaches to detect discouraged learners: Assessment of motivation in educational computer games. In Proceedings of eLearning Baltics (eLBa) 2010, July 1 – 2, 2010, Rostock, Germany.
Approaches to Detect Discouraged Learners: Assessment of Motivation in Educational Computer Games Elke E. Mattheiss, Michael D. Kickmeier-Rust, Christina M. Steiner, Dietrich Albert Department of Psychology University of Graz Universitätsplatz 2 / III 8010 Graz
[email protected] [email protected] [email protected] [email protected]
Abstract: Motivation is an important aspect in educational games. To create games which are able to motivate players again who get demotivated while playing, a reasonable assessment of motivational states has to be developed. The present paper describes different approaches found in previous e-learning research concerned with this issue. Based on that research a simple and convenient method was created and implemented in the 80Days project demonstrator game, using reaction times, number of errors and number of requesting help to diagnose a lack of attention and respectively a lack of confidence.
1 Introduction Considering motivational aspects in the design of educational systems - like instructional games – is of increasing interest for researchers and developers. Besides the assumption of games being motivating per se, it becomes more and more important to consciously create games which enhance the players’ engagement. Beyond conventional design guidelines for instructional games, a key option is enhancing motivation in gamebased-learning by providing players with motivating interventions if they happen to be demotivated. A major step towards such vision is equipping the system with the capabilities to recognize in real-time whether players are motivated to play the game or not. This kind of assessment is necessary to provide players with suitable tailored motivational interventions just in
the right moment and, equally important, to allow highly engaged players to continue without any interruptions. Therefore, several researchers are working on assessment methods for motivational states. The present outline aims at introducing different approaches of motivational assessment and demonstrating the method realized in a learning adventure game, developed in the context of the European research project 80Days. In this project a demonstrator game was developed, teaching geographical content for 10 to 14-year-old children. The game tells the story of a boy, who gets hijacked by a friendly alien named Feon, who pretends to write a travel guide about Earth (in fact the aliens want to conquer Earth). They fly around Europe with the UFO and collect geographical information e.g. about countries and capitals. The game also includes a simulation part with the players’ task to increase or decrease the danger of a flood. To include motivational aspects in this kind of gameplay prior research was scanned for motivational assessment methods in e-learning environments, which are presented in the next chapter.
2 Approaches to motivational assessment within e-learning environments The easiest way to assess a latent construct such as motivation is relying on subjective measures. Simply asking people how motivated they are in a given situation, delivers fair information about a motivational state. For example, Song and Keller [SK01] used this approach to measure different aspects of motivation within an e-learning environment. Of course, this direct assessment is not free of flaws and seems inappropriate in many game-based learning situations. Imagine a person immersed in a game is suddenly confronted with a pop-up asking “How motivated are you right now?” – This would gravely destroy the players’ immersion, unless queries are not carefully embedded in the games’ storyline and its dialogues. Consequently, methods are required which assess the motivational state in a more subtle way. Some researchers rely on objective physiological measures to identify different motivational states. McQuiggan and Lester [QL06], for example, found a relation between the self-efficacy – an important factor of motivation – and the heart rate of participants while completing problem solving tasks. Although physiological measures have a great potential for the assessment of motivational states – especially in a typical game-based learning situation – they are not realizable in real life settings. From a near-term perspective, a non-intrusive assessment based on the player’s behaviour, which can easily be measured and recorded by the computer system during playing, appears more promising. Therefore,
reliable behavioural indicators and patterns for different motivational states must be found. For this purpose, some researchers gathered behaviour-based expert ratings of the motivation of users of e-learning environments to infer motivation diagnosis rules and patterns. The collected data yielded rules such that (1) non-random mouse movements may indicate a high attention or interest, (2) quick performance may indicate either confidence or a lack of interest (dependent on other characteristics of the behaviour), (3) performing well may justify a high satisfaction [VP02, VP03], (4) hurrying through the virtual environment without spending much time on available information may indicate severe disengagement [CW07a, CW07b], or (5) making extremely quick actions after an error may indicate a loose trial and error behaviour [BCK04]. This previous research paves an expedient way towards an easy assessment of motivation and provides a set of useful indicators. Of course, we have to admit that this kind of expert judgment based rules have limitations in terms of reliability and validity. Another approach that goes without the need for maybe mislead human ratings of user interactions is the technique of engagement tracing introduced by Beck [Be05], which is based on Item Response Theory, and uses response times to model disengagement. The technique models performance advancements with increasing response time, until performance hit a plateau and gradually declines again. The technique allows calculating the engagement according to the response time and other parameters. Another field providing potentially useful information for a motivational assessment is the so-called affect perception in educational software. The approach is based on the analysis of log data in order to find some evidence for different emotional states of a learner. Examples are (1) counting the amount of deleting and re-typing while writing as a measure of certainty, (2) analysing the goal directedness of mouse movements as indicator of concentration or frustration, (3) counting the number of drop-outs suggesting disappointment or boredom [VK03, VK04], or (4) analysing general behaviour (mouse movements, response times, use of agents, use of maps, or use of specific inventories) and cognitive characteristics (error frequency, error persistent occurrence, correct answer frequency, or correct occurrence) [KV05]. In his dissertation, McQuiggan [Qu08] has been describing an inductive framework for affect recognition and expression he named CARE. The framework is able to learn in a first phase models of affect by monitoring self-reports of people engaged in an interactive environment. Based on these reports, in a second phase the framework
enables the generation of new models applying machine learning techniques and in a third step it enables predictions of affective states according to observed data.
3 Motivational assessment in an educational game At the beginning of the paper we asserted the claim that a non-invasive realtime assessment of motivational states and appropriate interventions are key factors of successful and effective educational computer games. Based on the available body of research, as outlined above, we accumulated suitable heuristics into a method of motivational assessment. For the application in an educational game especially assessment methods related to the use of log data are promising, since they unlikely impair the players’ immersion. For this kind of non-invasive assessment the following indicators for different motivational states appropriate for educational gaming situations have been identified, based on the prior research: •
Time measurements: time spent on a specific task tells much about the motivation of players. In general, we can state that the more time learners spend on a task, the more engagement they show. Furthermore, if learners proceed through a task (e.g. reading a text, answering a question, or solving a problem) very quickly they likely lack appropriate attention and feel bored. Quick actions also relate to a systematic trial-and-error behaviour. If the playing time exceeds a certain threshold value, the learners can again have low attention because of being distracted but, at the same time, they also may lack confidence to proceed through the game.
•
Number of backward movements: if a learner tends to go back very often, this likely indicates a lack of confidence and a worry about failure.
•
Requesting hints/help: in general hints are requested when a person is unconfident. Obviously, this is not the only reason to ask for a hint – people also ask for help if they lack required skills or they are “gaming the system” through extensively using help functions, which can be related to a lack of attention and boredom. Crucial for an interpretation is the time spent after a hint has been given. If the learners immediately ask again for help, they are probably “abusing” the help function in order to proceed through the task without spending much effort.
•
Randomness of mouse movements: if people move the mouse towards a specific goal with an obvious intent, they are likely attentive and confident about what they are doing. If the movements are random the players are probably bored or distracted and lack attention. But if their mouse movements are unassertive and slight, they rather lack specific knowledge or, again, confidence.
•
Number of mistakes/correct answers: the performance of learners is - especially combined with other observations - a good indicator for motivation. In general, a person, who fails several times in a row, would rather need a motivational intervention in terms of a confidence enhancement in comparison to a person who succeeds continuously.
•
Certainty of answer correctness: the repeated indication of a low certainty of the given answer to be correct can indicate a low confidence (especially if the player shows a good performance).
The presented interpretation of these indicators is of course not free of flaws, and do not match every person in every situation - especially if the interpretation depends on one single observation - but they describe behavioural pattern frequently found to be related to specific motivational states. Only with the aid of a continuous assessment during the game, the interpretation of the indicators should lead more and more to an appropriate picture of the players’ motivational states. Further it is essential to define threshold values for these indicators highly dependent on the learning situation. Therefore no general scale can be stated, but the single action possibilities in the specific education game have to be assessed carefully to define reasonable thresholds for motivated behaviour (e.g. how long does it usually take to accomplish a specific action, and when should the system trigger an intervention).
Further not all indicators are applicable in every educational game or in every situation within a game. According to the principle ‘keep it simple’, a method was chosen appropriate for the 80Days demonstrator game, which essentially incorporates log data of time measurements, number of requesting help, and number of mistakes to identify players’ attention and confidence as two crucial aspects of motivation [cf. Ke87]. The motivational assessment method (see Figure 1) includes a first step relying on log data information and a second step related to the answer certainty and correctness. The first step is based on a rather conventional procedure in a game-based learning situation: the player is facing a specific task, shows certain behaviour, and accordingly the systems gives some kind of hint or feedback [cf. GAD02; MKSA09]. Since previous research showed that the time after providing feedback has a great potential to diagnose motivational states, this is the essential factor of our assessment of attention and confidence. Players with low attention tend to act very quickly or very slowly, whereas players with low confidence act in an appropriate or too long time period. For the confidence assessment the information about the number of errors and help requests is crucial. A person who acts in an appropriate or even too long time after a system reaction (like a hint) and who requests help very often, likely is – besides simply lacking of knowledge - not confident enough to accomplish the task, which in fact is important for a sustained motivation to play the educational game. Furthermore, a person who seems to consider a system reaction by taking some time and still makes many mistakes would rather need some confidence enhancement interventions in order not to lose the motivation to play. Imagine the described simulation situation, in which the players’ have to increase and respectively decrease the danger of a flood. The players can make different kinds of adjustments (e.g. planting a tree or building a dike) which have an influence on the danger of a flood for the people. The influence is displayed in a increasing or decreasing bar named risk level (see Figure 2). The players also get frequently hints from the friendly alien Feon when they accomplish some wrong actions. Both kinds of feedback about the performance should help the players to solve the task. Now what step 1 of our motivational assessment does, is to differentiate after each feedback trial between 1. players who likely do not pay attention to this feedback (assumed by the time after the feedback to the next action being too short to actually consider it) 2. players who act in an appropriate time after the feedback and 3. those who take more time than we would assume to be required to process the feedback and act again. Players falling in category 1 or 3 are diagnosed to have a lack of attention, and may get an intervention to focus their attention on the relevant information (e.g. Feon saying: “Hey, my friend! Are you sleeping?!“). For players of category 2 or
3 the number of mistakes and requesting for hints is additionally assessed, and in case of a high number for one or both of this indicators a intervention is triggered to work against a lack of confidence (e.g. Feon saying: “Well done, keep your effort.“).
Figure 1: Motivational assessment procedure in the 80Days demonstrator game.
Figure 2: Screenshot of the simulation situation in the 80Days demonstrator game.
The motivational assessment could stop after the first step or (if appropriate in a certain game-based learning situation) continue with the second step as an additional filter for motivational interventions in case of a lack of motivation was identified in step 1. According to the players’ answer certainty (e.g. simply assessed by asking the players how certain they are that their answer is correct) and actual correctness of the behaviour, four different cases can be distinguished. First, if the player exhibits a correct action (e.g., solving a problem) with a high certainty, no motivational intervention is required (because we can assume that correct answers are motivational in their self). Second, if the player exhibits a correct action with a low certainty, the player maybe lacks confidence and therefore a confidence intervention emphasizing the success should be triggered in case a lack of confidence was diagnosed by step 1. Third, if the player acts very quickly (which leads to the conclusion of a lack of attention in step 1) and indicates a high certainty related with a wrong answer, we assume inattentiveness and therefore trigger an attention intervention. The fourth possibility is that the player exhibits a correct action with a low certainty. Dependent on the other indicators, in this case the player may lack
confidence as well as attention and may benefit from a corresponding intervention.
Conclusion In conclusion, the described assessment and intervention procedures are supposed to provide the learners with suitable tailored support and feedback without overflowing the learner with inappropriate and annoying interventions. The presented approach still has to be evaluated in terms of its effectiveness to indentify critical motivational states, by comparing this non-invasive assessment with subjective measures of motivation.
Acknowledgements The research and development introduced in this work is funded by the European Commission under the seventh framework programme in the ICT research priority, contract number 215918 (80Days, www.eightydays.eu) and the Austrian Federal Ministry of Science and Research.
References [BCK04] Baker, R., Corbett, A., & Koedinger, K. (2004). Detecting student misuse of intelligent tutoring systems. In Proceedings of the Seventh International Conference on Intelligent Tutoring Systems, 531-540. [Be05] Beck, J.E. (2005). Engagement tracing: Using response times to model student disengagement. In C.-K. Looi, G. McCalla, B. Bredeweg, and J. Breuker (Eds.) Proceedings of 12th International Conference on Artificial Intelligence in Education, AIED’2005, IOS Press, 88-95. [CW07a] Cocea, M., Weibelzahl, S. (2007a). Cross-system validation of engagement prediction from log files. In E. Duval, R. Klamma, and M. Wolpers (Eds.) Creating New Learning Experiences on a Global Scale, Second European Conference on Technology Enhanced Learning, ECTEL 2007, pp. 14-25, Springer Berlin/ Heidelberg. [CW07b] Cocea, M., & Weibelzahl, S. (2007b). Eliciting motivation knowledge from log files towards motivation diagnosis for Adaptive Systems. In C. Conati, K. McCoy & G. Paliouras (Eds.), User Modeling 2007. Proceedings of 11th International Conference, UM2007, Lecture Notes in Artificial Intelligence, pp. 197-206, Berlin: Springer. [GAD02]Garris, Ahlers, & Driskell (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441-467.
[KV04] Katsionis, G., & Virvou, M. (2004). A cognitive theory for affective user modelling in a virtual reality educational game. In Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics (SMC '04), vol. 2, pp. 1209_1213, IEEE, The Hague, The Netherlands, 2004. [KV05] Katsionis, G., & Virvou, M. (2005). Adapting OCC theory for affect perception in educational software. In Proceedings of the 11th International Conference on Human-Computer Interaction, Mahwah, New Jersey: Lawrence Erlbaum. [Ke87] Keller, J. M. (1987). Development and use of the ARCS model of motivational design. Journal of Instructional Development, 10(3), 2-10. [MKSA09] Mattheiss, E., Kickmeier-Rust, M., Steiner, C., & Albert, D. (2009). Motivation in game-based learning: It’s more than ‘flow’. In Proceedings of 3. Workshop Game-based Learning , DeLFI 2009. [QL06] McQuiggan, S.W. & Lester, J.C. (2006). Diagnosing self-efficacy in intelligent tutoring systems: An empirical study. In M. Ikeda, K. Ashley, & T.-W. Chan (Eds.), ITS 2006, LNCS 4053, pp. 565-574, Berlin: Springer. [Qu08] McQuiggan, S.W. (2008). An inductive framework for affect recognition and expression in interactive learning environments. PhD Thesis, North Carolina State University. [SK01] Song, S. H., & Keller, J. M. (2001). Effectiveness of motivationally adaptive computer-assisted instruction on the dynamic aspects of motivation. Educational Technology Research and Development, 49(2), 522. [VK03] Virvou, M., & Katsionis, G. (2003). Relating error diagnosis and performance characteristics for affect perception and empathy in an educational software application. In Proceedings of the 10th International Conference on Human-Computer-Interaction, HCI 2003, 480-484. [VP03] De Vicente, A. & Pain, H. (2002). Informing the detection o f the students’ motivational state: An empirical study. Proceedings of the 6th International Conference on Intelligent Tutoring Systems. Lecture Notes in Computer Science. Berlin: Springer [VP03] De Vicente, A. & Pain, H. (2003). Validating the detection of a student’s motivational state. In A. Mendez Vilas, J.A. Mesa Gozalez, & J. Mesa Gonzalez (Eds.), Proceedings of the Second International Conference on Multimedia Information & Communication Technologies in Education, pp. 2004-2008.
