Dynamic Cognition as Revealed by Eye Tracking François Vachon School of Psychology, Université Laval Québec, Québec, Canada
[email protected]
Guillaume Hervet Saint-Etienne School of Management Saint-Etienne, France
[email protected]
Benoît R. Vallières School of Psychology, Université Laval Québec, Québec, Canada
[email protected]
Sébastien Tremblay School of Psychology, Université Laval Québec, Québec, Canada
[email protected]
ABSTRACT
In the context of a computer-controlled simulation of radar-based risk assessment, we monitored eye movements to examine what they can reveal about the underlying cognitive processes involved in complex dynamic situations. We adopted an approach of eye-tracking analysis that is in synchrony with specific events of the simulation that engage cognitive functions that are key to the operator placed in complex dynamic situations. We extracted various metrics 1) to examine the ability of monitoring the evolving situation (e.g., gaze position relative to changing objects), 2) to assess the impact of task interruption on cognitive functioning (e.g., pattern of fixations around the interruption), and 3) to pinpoint the sources of error in dynamic decision making (e.g., a combination of measures relative to scanpath, fixations, and pupillary response). This approach illustrates how dynamic, event-based measures of eye movement constitute a gateway to the ongoing cognitive processing in complex dynamic situations, which in turn can serve as a basis for assessing the effectiveness of support technologies designed to augment cognitive work in such settings. Author Keywords
Eye tracking; Cognition; Complex dynamic situation; Dynamic decision making; Change detection; Task interruption. INTRODUCTION
Individuals operating in dangerous or volatile dynamic environments such as air traffic control, emergency response, or risk assessment have to make optimal decisions under severe constraints such as high risk, time pressure, complexity, and ambiguity whereby a high load of information from multiple sources must be continually processed and filtered. In such situations, the role for decision support systems to augment cognition is becoming crucial. Providing cognitive support proves one of most challenging prospects in complex situations evolving in real-time. In fact, without a proper comprehension of the underlying cognitive processes involved, the design and development of such technology may serve only to exacerbate rather than enhance the desired effect. In the context of the mind-eye hypothesis—the correspondence between eye gaze and information processing—the analysis of eye movements has been invaluable in investigating online cognitive processes such as those involved in reading, decision making, and memory (e.g., Pearson & Sahraie, 2003; Zelinsky, 2008). One promising avenue in understanding the key cognitive functions at play in dynamic situations is to exploit eye tracking in a manner that is closely related to the situation events. In the present study, we highlight a range of cognitive processes that can be studied by assessing the disposition, scanning pattern, and physical response of the eyes during the monitoring of a complex dynamic scene. METHODOLOGICAL APPROACH
The present research adopts an experimental methodology that attempts to bridge the gap between basic and applied research by maintaining both empirical control—hence the ability to identify causal relationships—and external realism through the use of a synthetic environment or microworld. We used the Simulated Combat Control System (S-CCS) microworld, a low-level computer-controlled simulation of single ship naval above-water warfare (see Vachon et al., 2011). In a typical S-CCS scenario, a single participant playing the role of a tactical coordinator has to monitor a radar screen representing the airspace around the ship, be sensitive to changes to air traffic in the operational space, evaluate the threat level of every aircraft moving in the vicinity of the ship based on a list of parameters, and take appropriate defensive measures against hostile aircraft. Three cognitive processes were assessed: 1) situation monitoring, tested through the ability to detect critical threat-level changes to aircraft; 2) planning/coordination, investigated by momentarily interrupting the primary task; and 3) dynamic decision making, evaluated through the classification of aircraft according to the level of threat they posed to the ship. Eye movements were recorded with a Tobii T1750 eye tracker. Eye-movement data were analyzed using Tobii’s ClearView software and scripts developed with the programming language Python (Van Rossum & Drake, 2005). Beyond the classical static analysis of pre-defined regions of interest, we adopted an approach of eye-tracking analysis that is in synchrony with specific events. So, for each cognitive process, we derived a set of eye tracking-based metrics aligned around a significant change to an aircraft, the occurrence of task interruptions, or the sequence of information intake that precedes a classification decision. 1
COGNITIVE METRICS Situation Monitoring
The ability to detect critical changes to aircraft is affected by the pattern of eye movements over the dynamic display. Indeed, critical changes were more likely to remain unseen if the changed aircraft was never fixated in the moments preceding or following the change. If one assumes that eye movements can index the allocation of attention over a display (e.g., Rayner, 2009), such findings are consistent with the notion that attention is required for conscious change perception (e.g., Simons & Ambinder, 2005). Pupillometry results suggested that an attended change can also be missed due to a failure of attentional processes. As pupil dilation has been associated to increased attentional processing (e.g., Hoeks & Levelt, 1993), the finding of dilation for changes undetected but fixated suggests that despite the failure to consciously detect fixated changes, some resources were nonetheless devoted to the processing of the changed aircraft. Planning and Coordination
Fixation durations can be considered as an index of the difficulty of processing, where longer durations are usually associated to the difficulty in interpreting the components of an interface (e.g., Goldberg & Kotval, 1999). To assess the extent to which task interruptions altered processing, fixation duration was analyzed over time with the interruption as an anchor. The comparison of the average dwell time (see Saint-Aubin et al., 2007) made in the seconds that preceded and followed each interruption provided us with an index of the extent to which information processing is altered and slowed down by the need to recover from task interruptions (e.g., St. John & Smallman, 2008). Dynamic Decision Making
In order to pinpoint the sources of error in dynamic decision making, we extracted metrics relative to 1) scanpath (measures of search; e.g., Poole & Ball, 2006), 2) eye fixations (measures of processing; e.g., Goldberg & Kotval, 1999), and 3) pupillary response (measures of cognitive load; e.g., Hoeks & Levelt, 1993) during the categorization of aircraft according to the level of threat they posed to the ship. Incorrect classifications were associated to 1) longer-lasting and longer scanpaths that are assumed to indicate less efficient information search, 2) longer and more frequent fixations on key attributes, revealing difficulties in making sense of critical information, and 3) smaller pupil size, often indicative of a low level of cognitive load. CONCLUSION
Eye tracking is particularly suitable to study cognition in dynamic situations and the assessment of support systems as it provides a direct, online, sensitive, and non-invasive index of processing. We developed a set of eye-tracking metrics that are grounded in the sciences of human cognition, psychometrically robust, and ecologically valid. ACKNOWLEDGMENTS
This work was supported by a partnership grant from the National Sciences and Engineering Research Council of Canada with Defence R&D Canada and Thales Canada. We are thankful to Julie Champagne for assistance in data analysis. REFERENCES
Goldberg, J.H., and Kotval, X.P. Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics 24 (1999), 631-645 Hoeks, B., and Levelt, W.J.M. Pupillary dilation as a measure of attention: A quantitative system analysis. Behavior Research Methods, Instruments, & Computers 25 (1993), 16-26. Pearson, D.G. and Sahraie, A. Oculomotor control and the maintenance of spatially and temporally distributed events in visuo-spatial working memory. Quarterly Journal of Experimental Psychology 56 (2003), 1089-1111. Poole, A. and Ball, L.J. Eye Tracking in Human-Computer Interaction and Usability Research: Current Status and Future Prospects. In Ghaoui, C. (ed.) Encyclopedia of Human Computer Interaction. (2006), 211-219. Rayner, K. The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology 62 (2009), 1457-1506. Van Rossum, G. & Drake, F. L. (2005). Python Reference Manual. PythonLabs, Virginia, USA. Simons, D.J., and Ambinder, M.S. Change blindness: Theory and consequences. Current Directions in Psychological Science 14 (2005), 44-48. St. John, M., & Smallman, H.S. Staying up to speed: Four design principles for maintaining and recovering situation awareness. Journal of Cognitive Engineering and Decision Making 2 (2008), 118-139. Vachon, F., Lafond, D., Vallières, B.R., Rousseau, R., and Tremblay, S. Supporting situation awareness: A tradeoff between benefits and overhead. In Proc. CogSima 2011, IEEE (2011), 282-289. Zelinsky, G.J. A theory of eye movements during target acquisition. Psychological Review 115 (2008), 787-835.
2
