Conceptual, Theoretical, Methodological, and

This article was downloaded by: [Dr Gale Sinatra] On: 28 February 2015, At: 10:46 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Educational Psychologist Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hedp20

Defining and Measuring Engagement and Learning in Science: Conceptual, Theoretical, Methodological, and Analytical Issues Roger Azevedo

a

a

Department of Psychology North Carolina State University Published online: 27 Feb 2015.

Click for updates To cite this article: Roger Azevedo (2015) Defining and Measuring Engagement and Learning in Science: Conceptual, Theoretical, Methodological, and Analytical Issues, Educational Psychologist, 50:1, 84-94, DOI: 10.1080/00461520.2015.1004069 To link to this article: http://dx.doi.org/10.1080/00461520.2015.1004069

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions

EDUCATIONAL PSYCHOLOGIST, 50(1), 84–94, 2015 Copyright Ó Division 15, American Psychological Association ISSN: 0046-1520 print / 1532-6985 online DOI: 10.1080/00461520.2015.1004069

Defining and Measuring Engagement and Learning in Science: Conceptual, Theoretical, Methodological, and Analytical Issues Roger Azevedo

Downloaded by [Dr Gale Sinatra] at 10:46 28 February 2015

Department of Psychology North Carolina State University

Engagement is one of the most widely misused and overgeneralized constructs found in the educational, learning, instructional, and psychological sciences. The articles in this special issue represent a wide range of traditions and highlight several key conceptual, theoretical, methodological, and analytical issues related to defining and measuring engagement. All the approaches exemplified by the contributors show different ways of conceptualizing and measuring engagement and demonstrate the strengths and weaknesses of each method to significantly augment our current understanding of engagement. Despite the numerous issues raised by the authors of this special issue and in my commentary, I argue that focusing on process data will lead to advances in models, theory, methods, analytical techniques, and ultimately instructional recommendations for learning contexts that effectively engage students.

Engagement is one of the most widely misused and overgeneralized constructs found in the educational, learning, instructional, and psychological sciences. A recent search of the literature on PsycINFO yielded more than 32,000 articles about engagement in the last 14 years. Engagement has been used to describe everything including student academic performance and achievement; classroom behaviors; approaches to interacting with instructional materials; students’ self-perceptions of beliefs in handling individual and contextual aspects of learning situations; students’ enactment of cognitive, motivational, affective, metacognitive, and social processes, particularly in academic contexts (e.g., classrooms, intelligent tutoring systems); teacher practices in learner-centered classrooms; and features of instructional and learning contexts designed to initiate, sustain, and foster learning. In some cases, the construct is used either implicitly or synonymously and interchangeably with other widely used terms, such as motivation and flow (see Christenson, Reschly, & Wylie, 2012). The construct is widely used by researchers, students, teachers, parents, school administrators, Correspondence should be addressed to Roger Azevedo, Department of Psychology, North Carolina State University, 640 Poe Hall, 2310 Stinson Drive, Raleigh, NC 27695-7650. E-mail: [email protected] Color versions of one or more of the figures in the article can be found online at www.tandonline.com/hedp.

and government officials without proper and accurate definitions. In addition, interdisciplinary researchers who work closely with educational, learning, and psychological researchers are also adopting and using engagement to refer to motivational beliefs, behavioral enactments of cognitive strategies, affective states, persistence, and self-regulation. The term disengagement is also pervasive and is used widely to refer to students’ lack of persistence, task value, curiosity and interest, off-task behaviors, infrequent use of cognitive strategies, lack of meta-awareness, and negative emotions. Despite researchers’ best intentions to coin a construct capable of capturing individual, interpersonal, and contextual factors related to complex human learning, the term stands to lose its meaning, precision, and scientific utility because there is little agreement on a concrete definition and effective measurement of engagement (Sinatra, Heddy, & Lombardi, this issue). A major issue related to the definitional and conceptual aspects of engagement is the manner in which researchers operationalize the construct, describe the conceptual and theoretical foundations of the construct, pose and specify their research questions and hypotheses, and select methodological and analytical techniques and methods used to measure and analyze engagement, as well as the accuracy of the inferences drawn about engagement based on the methodologies and analytical techniques (Betts, 2012;


ENGAGEMENT AND LEARNING IN SCIENCE

Fredricks & McColskey, 2012). In addition to the definitional issues, there is a general lack of consistency in the specification and articulation of the theoretical underpinnings of the construct that propagates the wide misuse of the term. In addition, the research questions and hypotheses posed by researchers in general tend to lack specificity and therefore are not always empirically testable. This is related to the choice of methodological tools and analytical methods, which range widely from traditional self-report measures (B. A. Greene, this issue; Renninger & Bachrach, this issue) that are easy to administer but do not actually capture engagement-related processes in real time, to very precise instruments such as eye-trackers, which provide real-time data of processes that need to be inferred with engagementrelated processes but lack contextual information necessary to infer engagement processes (e.g., Miller, this issue). In general, more attention needs to be devoted to the conceptual, theoretical, methodological, measurement, and analytical issues related to engagement; otherwise, the term will lose its value and be replaced with some other construct. Given the importance of engagement, Gale Sinatra and Doug Lombardi have coedited this special issue of Educational Psychologist by assembling a group of well-established as well as young scholars to represent the various theoretical, methodological, and analytical challenges in defining and measuring student engagement in science. The six articles presented in this special issue of Educational Psychologist offer an array of definitions, as well as novel and sophisticated approaches to measuring engagement. The publication of this special issue is timely and provides the interdisciplinary community of educational, learning, cognitive, instructional, computational, and engineering scientists the opportunity to examine the ways in which traditional techniques of measuring engagement (e.g., self-reports, observations) can be augmented with more recent approaches (e.g., log-files, eye-tracking) to provide a concrete definition and effective measurement of engagement. The research community will benefit by employing the techniques described in these articles because they have the potential to transform contemporary conceptions and the measurement of engagement (e.g., Fredricks, Blumenfeld, & Paris, 2004). As such, the two main goals of my commentary on the six articles included in this special issue are as follows: (a) summarizing each article, emphasizing key findings, and highlighting critical issues; and (b) raising issues, challenges, and questions related to conceptual, theoretical, methodological, and analytical issues.

SUMMARY, KEY FINDINGS, AND CRITICAL ISSUES PRESENTED IN THE ARTICLES Sinatra et al.’s (this issue) introduction article raises critical issues related to the challenge of defining and measuring student engagement in science. They critically and

85

eloquently raise several major issues that will impede the scientific advancement of engagement if researchers do not attend to them. For example, they describe the multidimensional perspective on engagement (i.e., behavioral, cognitive, emotional, and agentic) and suggest a complementary approach that places engagement instrumentation on a continuum. Their approach to clarifying some of the measurement issues is for researchers to consider instrumentation on a grain-sized continuum that ranges from person centered to context centered. They propose that identifying the grain size for measuring engagement is critical because a grain-sized continuum for conceptualizing, observing, and measuring engagement can range from the microlevel (i.e., individual in the moment, task, and learning activity) to the macrolevel (e.g., group of learners in a class, course, school, or community). As such, measuring engagement at the microlevel can include physiological and psychological indices from brain imaging, eye-tracking, response time, and attention allocation. At the macrolevel, measuring engagement can include discourse analysis, observations, ratings, and other in situ contextual analysis of learning, schooling, and practices among and within groups. The authors critique and clarify definitional and measurement issues related to each of the behavioral, emotional, cognitive, and agentic perspectives of engagement. In addition, another important issue raised by Sinatra et al. (this issue) is their call for more focused theoretical work on domain-specific (e.g., psychophysiological arousal that generates a cognitive, behavioral, and emotional response from using a serious game for science, technology, engineering, and mathematics [STEM] learning) and domaingeneral (e.g., attention allocation, metacognitive monitoring, positive and negative emotions) aspects of engagement, as well as for more precise and differentiated measures of science engagement, which in turn will continue to refine the construct’s definition. In terms of domain-specific aspects of science engagement, Sinatra and colleagues review several factors that may impact engagement differently in science compared to other domains, including epistemic cognition; attitude; gender, minority, and identity issues; misconceptions; topic emotions; and involvement in scientific and engineering practices. They conclude their article by highlighting some critical and challenging issues, such as the operationalization and measurement of engagement when students are participating in scientific and engineering practices, dealing with dichotomized constructs used by many researchers (e.g., low engagement, high metacognitive engagement), individual and developmental differences, use of a single measurement method, in situ problems, and pinpointing the source of engagement. The article by B. A. Greene (this issue) provides an overview of research spanning 20 years, as it relates to the measurement of cognitive engagement using self-report measures. She reports and documents her program of


86

AZEVEDO

research including the theoretical underpinnings of selfreport instruments, early measurement findings and struggles with the nature of these instruments, and contributions to our understanding of cognitive and motivational engagement. For example, early on, Greene and colleagues defined their self-report measures of cognitive engagement in terms of type and degree of cognitive strategy use, the use of selfregulatory processes, and the degree of effort exerted. Greene reviews the contributions of other widely used selfreport measures, such the MSLQ (Pintrich & De Groot, 1990), LASSI (Weinstein, 1987), and MAI (Schraw & Dennison, 1994), as well as their weaknesses. In general, Greene highlights contributions of her research to the field, including the role of self-efficacy in predicting cognitive engagement, the importance of perceived instrumentality (i.e., future consequences or future goals) as a motivational variable linked to cognitive engagement, and so on. She also reflects on key issues, such as the fact that survey methods are not sufficiently sensitive to capture the complexities of cognitive engagement involved in all learning tasks. She clearly articulates the drawbacks of using selfreport measures without including other process-related data as she describes her recent work in STEM, which utilizes interviews, observations, and trace data. Her recent work has allowed her to use an observation protocol to detect macrolevel differences in patterns of engagement in science classes and to use experience sampling techniques to measure engagement during studying for college classes. Miller (this issue) cogently articulates and provides extensive empirical psychological evidence regarding the use of reading times and eye movements to measure student engagement. More specifically, his article focuses on selfpaced reading and eye-tracking, which can be used to measure microlevel student engagement during science instruction. These methods suggest a definition of engagement as the quantity and quality of mental resources directed at an object and the emotions and behaviors entailed. This definition is based theoretically on decades of extensive cognitive research and supported by models of reading comprehension and visualization comprehension. He clearly articulates that the use of eye-movement data is based on a number of assumptions, including the assumption that people look longer at words and images presented in a text because they are thinking about those objects more. In addition, whereas self-paced reading and eye-tracking have several strengths, such as precision (i.e., time scale in milliseconds) and detail (e.g., tracing the allocation of attention to specific parts of a text and diagram), they also have limitations, including difficulty of interpretation without additional contextual information. Miller proposes that researchers address the limitation of eye-tracking by using multiple indices. For example, regressions are of particular interest to engagement researchers because students tend to regress more often when they have difficulty understanding the text, and increased regressions are associated with a

difficult conceptual change within a science topic. In general, regressions, by definition, constitute extra effort in that they interrupt the flow of reading and require the reader to exert extra effort to repair errors or reconsider information (Miller, this issue). Last, he proposes that researchers should consider carefully controlled research designs and triangulation of multiple methods as possible ways to take advantage of the strengths of eye-movement data, as well as to address the proposed limitations. The article by Gobert, Baker, and Wixon (this issue) focuses on concretizing engagement by operationalizing and detecting disengaged behaviors while students are using an intelligent tutoring system for science inquiry. They present their ongoing research on how the detection of disengaged behavior is used to intervene with students in real time in order to better understand their science inquiry with the Inquiry-Intelligent Tutoring System. In comparison to the other articles in this special issue, Gobert et al. take a different perspective to defining and measuring engagement—that is, they focus on defining, operationalizing, and detecting disengagement using computational methods, based on their work on automatic detecting of disengaged behavior using log-files as the main source of evidence. Gobert and colleagues describe their extensive work on field observations that can also be used to create automated measures of engagement through data mining on log-files. For example, coders record the affective state and current engaged/disengaged behavior of each student based on time sampling, and includes students’ work contexts, actions, utterances, facial expressions, body language, and interactions with peers and teachers. Despite its advantages, their method is not ideal in studying the development of engagement and cannot provide a continual estimation of student engagement; however, it can provide ground truth for measures of engagement needed to build automated detectors of affect. This article provides evidence that a learner can disengage in several ways, including gaming the system, engaging in off-task behavior, and illustrating a lack of effort by becoming careless and giving wrong answers. Gobert et al. present sample data (i.e., log-files) from middle-school students that were used to develop the automated detector, including a presentation of the algorithm generated from the final model. It should be noted that the use of a data-mining approach presents a purely bottom-up approach to defining disengagement with the Inquiry-Intelligent Tutoring System. As such, their results and conclusions are interesting and can be integrated with other measures of engagement. Last, their article highlights the potential contribution of data mining and machine learning techniques (Baker & Siemens, 2014) in measuring and predicting engagement-related behaviors from trace data (e.g., log-files). Renninger and Bachrach’s (this issue) article focuses on studying triggers for interest and engagement using observational methods. Their article provides extensive research



related to motivation, interest, and engagement in science. The main thrust of their article is that triggering interest establishes engagement; therefore, studying the triggers for interest and engagement are critical. In their article, they consider the use of observational methods to study the triggering process. According to Renninger and Bachrach, interest is a cognitive and affective motivational variable, and engagement includes the study of interest, along with environmental conditions. They make several critical observations that are important to highlight, including the fact that it is possible for a person with low interest to be behaviorally engaged; therefore, triggering interest and supporting its development is likely to be essential to whether an engagement intervention will have the power to change behavior. Following their extensive clarification of interest and engagement, they clearly specify that it is possible to be behaviorally engaged but not interested, whereas it is not possible to have an interest in something without being engaged in some way (e.g., behaviorally or cognitively). Also, they provide a description of the phases of interest development, how they need to be mapped onto the construct of engagement, and the need for educational interventions designed to support engagement. In their article, Renninger and Bachrach (this issue) argue that observational methods can extend traditional experimental work on the generation or triggering of interest that tends to focus on one feature as a trigger for interest (e.g., challenge, novelty) but do not address questions related to describing triggers as co-occurring with one or more other triggers, whether triggers work the same way for all learners, at all times, in a variety of settings with different content, and so on. By using observational methods, they contend that researchers can gather detailed data about what learners are actually doing as they engage in an activity, and these data can augment self-reported data. They provide an excellent example of an observation method with all its stages of data acquisition, reduction, and analysis. However, it is important to note that this approach also has its weaknesses, including lack of generalizability; being context specific, time dependent, and individualistic (idiosyncratic to individual); and reliance on frequency data. Last, a major strength of their research is the potential to significantly contribute to the design of a learning environment that will promote all learners to develop interest and be productively engaged. The last article by Ryu and Lombardi (this issue) characterizes engagement in science learning from a sociocultural perspective and offers a mixed-method approach that combines critical discourse analysis and social network analysis. This approach is deeply rooted in Vygostksy’s work, as well as other work in the areas of self-regulation and motivation (McCaslin, 2009). Conceptualizing engagement from a sociocultural perspective, Ryu and Lombardi discuss the advantages of a mixed methodological approach and how mixed methods can expand and augment our

87

understanding of engagement in certain science learning situations. Through this sociocultural viewpoint, engagement is defined as meaningful change in disciplinary discourse practice, which captures the dialectical relationship between the individual and the collective group. The combined use of critical discourse analysis and social network analysis integrates individuals’ relative position within a group with their situated language use. The authors argue that by combining methods, researchers may overcome some of the limitations of taking only one approach (i.e., either quantitative or qualitative analysis). In addition, better understanding this dialectical relationship through a mixed-methods approach may provide unique insights into designing learning environments that promote engagement in science practices. In sum, these six articles have provided very thoughtprovoking ideas, based on their conceptual, theoretical, methodological, and analytical approaches to defining and measuring engagement in science. For each article, I summarized the major findings, highlighted the major strengths, and raised some issues that need further elaboration. In the following section I raise some theoretical, conceptual, methodological, and instructional issues that are based on these six articles.

ISSUES, CHALLENGES, QUESTIONS, AND IMPLICATIONS FOR FUTURE RESEARCH This special issue suggests two major implications for defining and measuring engagement in science. Each issue, challenge, question, and implication is summarized next, along with some key questions that should provide useful direction and guidance for future research in the area. Definitional, Conceptual, and Theoretical Issues The articles presented in this special issue raise several definitional, conceptual, and theoretical issues regarding the nature and role of engagement. There is a need for researchers to clearly define the construct and to articulate a theoretical framework from which to describe, measure, and analyze engagement. There are three possible paths for research on engagement: (a) develop an overarching and unifying theoretical framework to account for the majority of critical elements of the construct; (b) continue to conduct disparate programs of research, as exemplified in the set of articles in this special issue, due to various issues and constraints (e.g., researchers’ training, idiosyncratic definition of engagement that is based on a particular context, adherence to a particular theoretical framework, interest in examining engagement at different grain sizes and therefore using particular methods); or (c) abandon the construct altogether and continue to follow the second path, but without referring to engagement (in favor of some other construct,


88

AZEVEDO

such as self-regulated learning). Each of these paths has its own strengths and weaknesses, and it is up to researchers to determine what is best for them given the various constraints (e.g., phenomenon of interest and how it may be related to engagement); research context (e.g., laboratory research or applied research in classrooms); theoretical stance; methodological and analytical expertise; human and technological resources necessary to conduct, code, and analyze data; accessibility to and training in the use of innovative techniques; and so on. Given the complexity of the construct, I would favor the challenge associated with the first path, which would entail dealing with several key issues, including the following: (a) understanding the constraints imposed by the nature of the research context (e.g., lab or applied research in the classroom) and one’s personal goals (e.g., conducting basic research, conducting applied research in the classroom, or informing policy); (b) clearly articulating and defining engagement, based on (a); (c) adopting or integrating key elements from frameworks, theories, and models of leading motivational, cognitive, affective, metacognitive, social, and sociocultural research from various disciplines that correspond to the engagement-related phenomena; (d) posting testable research questions and hypotheses; (e) defining the level of granularity and grain size for measuring engagement vis-a-vis contextual factors; and (f) selecting any number of methods and analytical tools that correspond to (d) and that will provide evidence about engagement. It is important to explicitly highlight that the first path is challenging and that many researchers may not be willing to pursue it for a variety of reasons (including their own biases toward some theoretical frameworks, methodologies, and analytical approaches). Such a challenge will require interdisciplinary research efforts currently witnessed in several fields (e.g., Azevedo & Aleven, 2013; Calvo, D’Mello, Gratch, & Kappas, 2014). Despite the path taken by researchers to study engagement, it is evident that each article in this special issue exemplifies different levels of adherence to some overarching aspects, dimensions, or features of engagement (e.g., behavioral, cognitive, motivational, and agentic). However, none of them adhere to a particular theory of engagement because there is none. For example, B. A. Greene (this issue) adopts models based on motivation and cognitive engagement research, Renninger and Bachrach (this issue) primarily use models of interest development and motivation, and Miller’s (this issue) research on response time is based exclusively on extensive cognitive research. In contrast, Gobert and colleagues (this issue) take a bottom-up approach of defining disengagement by using innovative methods from data mining and machine learning. Given the conceptual and theoretical plurality, it is imperative that we define, construct, and test an overarching model of engagement that we can use to generate hypotheses and make assumptions regarding the role, timing, triggers, duration,

and quality of specific processes, mechanisms, and constructs of engagement. Therefore, several issues need to be addressed, including the following: (a) How is engagement the same as or different from other leading theoretical frameworks, models, and theories of self-regulation (e.g., Pintrich, 2000; Winne & Hadwin, 2008; Zimmerman & Schunk, 2011) and contemporary frameworks of socially shared regulation (Hadwin, J€arvel€a, & Miller, 2011)? (b) What engagement-related strategies are students knowledgeable about? How much practice have they had in using them? Are they successful in using them? Do they know if they use them successfully? Can we expect young students to be able to dynamically and accurately monitor and regulate their cognitive, affective, metacognitive, and motivational processes during individual or social processes, or during group interactions with peers and teachers? (c) How familiar are students with the tasks they are being asked to complete? Are they familiar with the various aspects of the learning context or system they are being asked to use? (d) What are students’ levels of prior knowledge? How do developmental and individual differences impact their knowledge and use of key cognitive, metacognitive, motivational, affective, and social processes? What impact will prior knowledge have on a learner’s ability to self-regulate? (e) How do domain-specific aspects of science engagement, such as epistemic cognition, scientific and engineering practices, misconceptions, emotions, attitudes, and gender, influence students’ interest and participation in science-related activities and choice of STEM careers? (f) Do students have the necessary declarative, procedural, and conditional knowledge of metacognitive knowledge and regulatory skills essential to regulate their engagement during learning and problem solving? Do young students have the ability and sophistication to verbally express utterances that represent (and are coded as) individual processes that can be both coded by researchers and externalized to others during collaborative tasks that involve negotiation, shared task understanding, and so on? These are just some of the important issues that future research should address, based on an analysis of the articles published in this special issue. These issues become even more complex when dealing with the social and in situ nature of engagement, exemplified by Ryu and Lombardi (this issue); Renninger and Bachrach (this issue); and, to some extent, B. A. Greene (this issue). The similarity among their programs of research and emerging conceptions of self-regulation, coregulation, and socially shared regulation of learning (e.g., J€arvel€a & Hadwin, 2013) needs to be highlighted. As exemplified in the articles included in this special issue, there is relatively little research about how groups and individuals in groups trigger, engage, sustain, support, and productively monitor and regulate social processes. Accordingly, several articles in this special issue (Renninger & Bachrach, this issue; Ryu & Lombardi, this issue) represent an initial



launching point for conducting engagement research that specifically examines the role of self-regulatory and other regulatory processes in various learning contexts, as well as methods that facilitate or impede individual and shared regulation of learning. An exciting prospect is using computerbased learning environments (Gobert et al., this issue) to successfully support regulation in individual learning and leverage such environments for collaborative task contexts to examine the role of socially shared regulated learning. When we oscillate between individual and social aspects of engagement (depending on the context, theoretical framework, research question(s), hypotheses, methodological and analytical tools, etc.), we face several imminent challenges. Some of the major conceptual and theoretical questions include the following: (a) What are the defining criteria of engagement? (b) How is engagement differentiated from other conceptually similar constructs, such as self-regulated learning, coregulated learning, externally regulated learning, and socially shared regulated learning? (c) How does the contextually bound nature of engagement impact researchers’ ability to clearly and consistently define and operationalize all the constructs, mechanisms, and processes associated with it? (d) Are there clear boundaries between engagement and all other types of learning (e.g., cognitive apprenticeship, problem-based learning, projectbased learning, self-regulated learning; see Sawyer, 2014)? If so, what are they? Or do we agree on a few defining criteria (e.g., engagement is not always intentional or goal directed; learners may exert different levels of cognitive and behavioral effort during learning), whereas other criteria are contextually bound (e.g., engagement is social; contextual triggers are important in initiating and sustaining interest that leads to engagement; there are meaningful changes in disciplinary discourse practices)? (e) How can we extend our current frameworks (as exemplified in the articles in this special issue) beyond their descriptive nature so that we can improve our ability to make predictions about engagement? Methodological and Analytical Issues The second major issue stemming from the articles has to do with the differences among measures of engagement. The research presented by the authors in this special issue include a variety of methodological and analytical tools— some traditional (e.g., B. A. Greene, this issue; Miller, this issue), whereas others (e.g., Gobert et al., this issue) have the potential to forge new directions in the field. However, there are some major issues that need to be addressed as we embark on measuring engagement during various learning activities across a variety of learning contexts (Azevedo, Moos, Johnson, & Chauncey, 2010). There are several issues that should be raised regarding the grain size of measuring engagement (Sinatra et al., this issue). Unfortunately, many researchers use dichotomies to

89

describe engagement-related processes, such as low versus deep engagement, high metacognitive judgment, shallow versus deep processing, high metacognitive engagement, engagement versus disengagement, and so on. These are a few of the terms used by the contributors to this special issue. It should be noted that such dichotomies do not contribute to our definition of engagement and minimize the complex nature of engagement. No wonder the term is misused! As such, one major question is determining the “right” level of granularity for one to understand the role, triggers, emergence, development, and so on, of a particular cognitive, behavioral, or motivational process during task performance, the learning session, and so forth, while interacting alone or with a computer-based learning environment, or in a group with a computer-based learning environment. A related and important issue that was raised by a few of the contributors to this special issue is the coding schemes used to code engagement-related processes. Questions related to coding schemes for engagement that arise immediately include the following: (a) What is the level of abstraction? (b) How many categories and subcategories should be used (and what do they represent?)? (c) How do the categories conceptually differentiate between core aspects of engagement, which may be based on distinct theoretical models (e.g., a macrolevel category for cognitive strategies, another one for emotions, and another one for motivation because they each have a distinct theoretical basis; Azevedo et al., 2010; Pekrun & Linnenbrink Garcia, 2014; Wigfield & Eccles, 2000)? (d) In some cases where categories (e.g., motivational and affective processes) are collapsed, what is the conceptual foundation for collapsing certain processes into the same category? If the purpose of one’s research is to capture and measure the real-time enactment processes, then what should the sampling frequency and timing be between observations? Is the use of inferential statistics justified when analyzing ipsative data? How does the integration of individual and social aspects of engagement complicate these analytical and statistical issues? Traditional approaches to measuring engagement are based on self-report measures that can hinder our understanding of the deployment of engagement processes. As amply acknowledged and stated by contributors to this special issue, as well as others (Azevedo et al., 2010; Winne & Azevedo, 2014), the problem is that self-reports are based on students’ perceptions of how one would or did enact certain processes, and these perceptions often do not align with what actually occurs during learning (Zimmerman, 2008). I propose that researchers continue to integrate interdisciplinary methodologies to capture engagement using trace methodologies, such as log-file data, eye-movement data, physiological measures, and self-report measures of engagement processes. Doing so will extend our methodological paradigm in the area of engagement with new, cutting-edge, interdisciplinary work using innovative tools and


90

AZEVEDO

techniques from educational data mining and machine learning (Baker & Siemens, 2014), educational statistics (J. A. Greene & Azevedo, 2009; Shute, in press), and affective and motivational science (Calvo et al., 2014; Gross, 2013; Pekrun, Goetz, Frenzel, Petra, & Perry, 2011; Scherer, 2013; Schunk, Meece, & Pintrich, 2013). Multiple methodologies are critical to systematically and accurately capture the temporally unfolding cognitive, metacognitive, motivational, and affective processes deployed during learning (e.g., Azevedo et al., 2013). This will involve integrating and extending existing methods, aligning various sensing devices, and using software to build a comprehensive model of the key processes and the conditions under which they are triggered, maintained, sustained, and transferred. The research and analytical methods will allow researchers to determine the validity of these unique means of measuring engagement and their comprehensiveness in accounting for the core engagement processes underlying learning in a variety of contexts. In sum, the proposed research approach will extend current frameworks and research on engagement by addressing the key questions regarding the how, when, why, what, and by whom related to understanding engagement and learning in a multitude of learning and instructional contexts. Researchers have a wealth of measures from which to choose to capture, analyze, and infer different aspects of engagement. In Table 1, I present a sample of existing methods and tools available to researchers, classified according to process and product data, self-reports, and knowledge-construction activities. For each category, I have included several methods and tools, and indicated whether they are suited to measuring cognitive, metacognitive, affective, or motivational processes. For example, eye-tracking data can provide a precise measurement of attention allocation and yield several indices of cognitive processing (see Miller, this issue), which can be used to infer learning strategies (e.g., coordinating informational sources between text and diagrams based on regressions); however, they do not provide direct evidence about metacognitive, affective, and motivational processes. Other methods, such as concurrent think-alouds, can provide evidence of both cognitive and metacognitive processes while sometimes also providing utterances that can be coded as motivational (e.g., I find reading this text challenging) and affective (e.g., I am confused as to what this diagram is trying to illustrate) in nature (Azevedo et al., 2010). In general, product data, including pretests, posttests, and transfer tests, relate to the cognitive domain of learning and engagement and are typically used at specific times. In contrast, self-report measures (e.g., MSLQ, PALS, LASSI, AEQ, ERQ, MAI, and OMQ) provide evidence regarding students’ self-perceptions of a number of cognitive, metacognitive, affective, and motivational beliefs across contexts. Some focus exclusively on one set of processes (e.g., AEQ, on achievement emotions), whereas other gather

perceptions on a number of processes (e.g., MSLQ, for motivation, cognitive strategies), thus explaining the shading in the table across the four cells. It important to highlight a few critical issues related to the methods and tools presented in the table. First, all measures provide varying levels of direct versus indirect measures—for example, direct evidence of the engagement-related processes (e.g., classroom discourse provides evidence of teacher intervention aimed at triggering interest in a new science topic), screen recordings of student-machine interactions with a computerized tutor, and so on. In contrast, some measures provide direct evidence but have to be coded and scored in order to isolate the underlying processes (e.g., utterances from think-alouds), whereas others have to be triangulated with additional data (e.g., eye-tracking with facial expressions of emotions and screen-recordings) in order to provide adequate context to understand the nature of and accurately code engagement-related processes. In other words, the triangulation of multiple data sources is necessary to maximize the inferences made about the complex nature of engagement-related processes during learning. A major challenge for research is determining which measures to use and how to temporally align them in order to make accurate inferences, conditions, triggers, and contributions of individual and combined engagement-related processes. Following the selection of a research method or tool, researchers need to adopt a research strategy in order to measure engagement-related processes by triangulating process, product, and self-report measures (see Table 1). As such, I advance a research strategy, illustrated in Figure 1, aimed to capture, track, model, and examine the unique signatures provided by each type of datum using various trace methodologies (see also Azevedo et al., 2013). I argue that by using complex quantitative statistical methods, machine learning, and data-mining algorithms, we can make inferences about individual cognitive, affective, metacognitive, and motivational behavioral signatures and about the quantitative and qualitative changes in learning and engagement processes during extended learning, in any learning context. Similar to work on self-regulated learning by Azevedo and colleagues (Azevedo et al., 2013; Azevedo et al., 2010), Figure 1 depicts theoretical, conceptual, and methodological assumptions regarding engagement and learning in a variety of learning contexts. First, cognitive, affective, metacognitive, and motivational processes can be detected, tracked, and modeled using online trace methodologies, such as concurrent think-aloud protocols, log-files, eye-tracking data, and physiological sensors (e.g., electrodermal activity), presented in Table 1. Second, students deploy these processes during extended interactions, such as in a 120-min learning session with an intelligent tutoring system, a 90-min science classroom period, or a semester-long course, represented at the bottom of the figure by t0 to tn. Third, the cognitive, metacognitive, affective, and motivational signatures will


91

TABLE 1 Illustration of Different Methods Used to Measure Process, Product, Self-Report, and Knowledge Construction Activities During Engagement Data Type

Method/Tool

Process

Cognition

Metacognition

Affect

Motivation

Screen recordings (video and audio) Concurrent thinkalouds

Retrospective thinkalouds


Eye tracking

Log-files

Facial expressions of emotions Physiological sensors (EDA, EMG, EKG, EEG, fMRI, fNIR)

Product

Pretest-posttesttransfer tests Quizzes

Summaries

Self-Reports

Knowledge Construction

Self-report questionnaires (MSLQ, PALS, LASSI, AEQ, ERQ, MAI, OMQ) Note-taking and drawing Classroom discourse

Note: This is a sample of existing methods available to researchers that can be used to measure and analyze engagement. The physiological measures have been clustered in one row; however, please note that they should each have their own row since they each differ along several dimensions (e.g., temporal resolution). This signifies that the method is ideally suited to capture and measure engagement-related processes.

This signifies that the method is not ideally suited to capture and measure engagement-related processes.

This signifies that the method may be ideally suited to capture and measure engagement-related processes; however, it depends on the context.


92

AZEVEDO

FIGURE 1 Converging multiple sources of data to analyze and infer cognitive, motivational, affective, and metacognitive engagement-related processes prior to, during, and following learning.

have different profiles depending on real-time fluctuations in response to individual differences (e.g., prior knowledge, epistemic stance, motivational beliefs), developmental differences (e.g., middle school student vs. college student), internal conditions (e.g., internal standards for monitoring progress), external conditions (e.g., time to complete task, accessibility to instructional resources, negotiating assessment rubric, or a feedback system), and phases of learning, or some other phasic or stagelike assumptions about learning and performance (represented as vertical arrows). Fourth, several types of trace, self-report, and product data have been identified as critical for examining the complex nature of engagement when learning in a variety of contexts (see Table 1). Each datum type is represented by a different dot and is presented prior to the administration of the pretest, during learning (e.g., associated with each subgoal), and/or after the posttest. Each type of datum can include several methods that provide individual streams of data. Similar to the contributors to this special issue, trace data can include eye-tracking, log-files, observations, and other tools such as several physiological devices (e.g., galvanic skin response). In Figure 1, each dot represents one of these methods, and the spacing between them represents the sampling rate of each process (from milliseconds to seconds, depending on the device). Also, each dot appears after the first horizontal arrow (at pretest) because these data are collected during learner interactions in the particular learning environment. The product data are to the left of the first horizontal arrow because they represent three individual pretest measures

that assess different types of knowledge including declarative, procedural, and mental models. The same three dots are also represented after the last horizontal arrow because they are administered as posttest measures. Note that the administration of self-report measures of motivation and emotions can be presented at pretest, during learning, and at posttest. The ovals and rectangles represent the potential contribution of each datum type (both within and across data channels) and its relative contribution to understanding and predicting the role of engagement processes relative to qualitative changes (e.g., self-efficacy for making inferences) and quantitative changes (e.g., increased frequency in accurately regulating one’s frustration following feedback from a peer or teacher) throughout learning. Last, a major issues lies in determining how many data channels (e.g., concurrent think-alouds alone vs. concurrent thinkalouds + eye-tracking + log-files + physiological data + self-report data) are necessary for one to make valid and reliable inferences regarding the nature of temporally unfolding engagement processes, and which analytical tools to use. The studies in this special issue use a variety of analytical tools that have strengths and weaknesses, and also vary along several key dimensions (e.g., statistical assumptions, number of parameters, accuracy, fit and alignment with theoretical assumptions, ease of use and interpretation, etc.) to analyze individual and social engagement process learning. We can ask how far the articles in this special issue have advanced the current literature and methods used to detect,



track, model, and infer engagement processes. As such, we have some major issues and challenges as we forge a new direction in the study of engagement. For example, do we have adequate analytical and statistical methods to handle the complex nature of these processes during learning individually and with others? Another issue related to level of granularity is the issue of temporal sequencing regarding the deployment of individual and social processes during learning. This is an extremely important issue that has to do with extending the timeframe for learning tasks so that we can capture and investigate the complexity of the underlying individual and social processes. How do these processes relate to learning outcomes in complex and ill-structured science tasks?

CONCLUSION The articles in this special issue represent a wide range of traditions and highlight several key conceptual, theoretical, methodological, analytical, and instructional issues related to defining and measuring engagement. This special issue is timely given the widely acknowledged misuse of the construct and its potential of being replaced by other constructs. This special issue further contributes to the emerging evidence that self-report measures are inadequate in measuring engagement and introduces other methods and analytical approaches that have the potential to advance the field. All the approaches exemplified by the contributors show different ways of conceptualizing and measuring engagement and demonstrate the strengths and weaknesses of each method to significantly augment our current understanding of engagement. Despite the numerous conceptual, theoretical, methodological, and analytical issues raised by the authors of this special issue and in my commentary, I truly believe that focusing on process data will lead to advances in models, theory, methods, analytical techniques, and ultimately instructional recommendations for learning contexts that effectively engage students.

ACKNOWLEDGMENTS I would like to thank Gale Sinatra and Doug Lombardi for their invitation to write the commentary and their comments and feedback on this article. Lastly, I would also like to thank Clark Chinn for his feedback on this commentary.

FUNDING This work was supported in part by the National Science Foundation, the Institute of Education Sciences, the Social Sciences and Humanities Research Council of Canada, and

93

the Natural Sciences and Engineering Research Council of Canada. REFERENCES Azevedo, R., & Aleven, V. (Eds.). (2013). International handbook of metacognition and learning technologies. Amsterdam, The Netherlands: Springer. Azevedo, R., Harley, J., Trevors, G., Duffy, M., Feyzi-Behnagh, R., Bouchet, F., & Landis, R. S. (2013). Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 427–449). Amsterdam, The Netherlands: Springer. Azevedo, R., Moos, D., Johnson, A., & Chauncey, A. (2010). Measuring cognitive and metacognitive regulatory processes used during hypermedia learning: Issues and challenges. Educational Psychologist, 45, 210– 223. http://dx.doi.org/10.1080/00461520.2010.515934 Baker, R., & Siemens, G. (2014). Educational data mining and learning analytics. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed., pp. 253–274). Cambridge, England: Cambridge University Press. Betts, J. (2012). Issues and methods in the measurement of student engagement: Advancing the construct through statistical modeling. In S. J. Christenson, A. L. Reschly, & C. Wylie (Eds.). Handbook of research on student engagement (pp. 783–803). New York, NY: Springer. Calvo, R., D’Mello, S., Gratch, J., & Kappas, A. (Eds.). (2014). Handbook of affective computing. Oxford, England: Oxford University Press. Christenson, S. J., Reschly, A. L., & Wylie, C. (Eds.). (2012). Handbook of research on student engagement. New York, NY: Springer. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59–109. http://dx.doi.org/10.3102/00346543074001059 Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763–782). New York, NY: Springer. Gobert, J. D., Baker, R. S., & Wixon, M. B. (this issue). Operationalizing and detecting disengagement within online science microworlds. Educational Psychologist, 50. Greene, B. A. (this issue). Measuring cognitive engagement with selfreport scales: Reflections from over 20 years of research. Educational Psychologist, 50. Greene, J. A., & Azevedo, R. (2009). A macro-level analysis of SRL processes and their relations to the acquisition of sophisticated mental models. Contemporary Educational Psychology, 34, 18–29. http://dx.doi. org/10.1016/j.cedpsych.2008.05.006 Gross, J. J. (2013). Emotion regulation: Taking stock and moving forward. Emotion, 13, 359–365. http://dx.doi.org/10.1037/a0032135 Hadwin, A. F., J€arvel€a, S., & Miller, M. (2011). Self-regulated, co-regulated, and socially-shared regulation. In B. Zimmerman & D. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 65–84). New York, NY: Routledge. J€arvel€a, S., & Hadwin, A. (2013). New frontiers: Regulating learning in CSCL. Educational Psychologist, 48, 25–39. http://dx.doi.org/10.1080/ 00461520.2012.748006 McCaslin, M. (2009). Co-regulation of student motivation and emergent identity. Educational Psychologist, 44, 137–146. http://dx.doi.org/ 10.1080/00461520902832384 Miller, B. W. (this issue). Using reading times and eye-movements to measure cognitive engagement. Educational Psychologist, 50. Pekrun, R., Goetz, T., Frenzel, A. C., Petra, B., & Perry, R. P. (2011). Measuring emotions in students’ learning and performance: The


94

AZEVEDO

achievement emotions questionnaire (AEQ). Contemporary Educational Psychology, 36, 34–48. http://dx.doi.org/10.1016/j.cedpsych.2010.10.002 Pekrun, R., & Linnenbrink-Garcia, L. (Eds.). (2014). International handbook of emotions in education. New York, NY: Routledge. Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). San Diego, CA: Academic. Pintrich, P. R., & De Groot, E. (1990). Motivatinonal and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82, 33–40. http://dx.doi.org/10.1037/00220663.82.1.33 Renninger, K. A., & Bachrach, J. E. (this issue). Studying triggers for interest and engagement using observational methods. Educational Psychologist, 50. Ryu, S., & Lombardi, D. (this issue). Coding classroom interactions for collective and individual engagement. Educational Psychologist, 50. Sawyer, K. (Ed.). (2014). The Cambridge handbook of the learning sciences (2nd ed.). Cambridge, England: Cambridge University Press. Scherer, K. R. (2013). The nature of dynamics of relevance and valence appraisals: Theoretical advances and recent evidence. Emotion Review, 5, 150–162. http://dx.doi.org/10.1177/1754073912468166 Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460–475. http://dx.doi.org/ 10.1006/ceps.1994.1033 Schunk, D. H., Meece, J. L., & Pintrich, P. R. (2013). Motivation in education: Theory, research, and applications (4th ed.). Upper Saddle River, NJ: Pearson.

Shute, V. J. (in press). Stealth assessment. In J. M. Spector (Ed.), The SAGE encyclopedia of educational technology. Thousand Oaks, CA: Sage. Sinatra, G. M., Heddy, B. C., & Lombardi, D. (this issue). The challenges of defining and measuring student engagement in science. Educational Psychologist, 50. Weinstein, C. E. (1987). LASSI user’s manual for those administering the Learning and Study Strategies Inventory. Clearwater, FL: H & H. Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary Educational Psychology, 25, 68–81. http://dx.doi.org/10.1006/ceps.1999.1015 Winne, P. H., & Azevedo, R. (2014). Metacognition. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed., pp. 63–87). Cambridge, England: Cambridge University Press. Winne, P. H., & Hadwin, A. F. (2008). The weave of motivation and selfregulated learning. In D. H. Schunk & B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). Mahwah, NJ: Erlbaum. Zimmerman, B. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45, 166–183. http://dx.doi. org/10.3102/0002831207312909 Zimmerman, B. J., & Schunk, D. H. (Eds.). (2011). Handbook of self-regulation of learning and performance. New York, NY: Routledge.