Designing Adaptive Interventions for Online ... - Springer Link

Education and Information Technologies 9:4, 355–375, 2004.  2004 Kluwer Academic Publishers. Manufactured in The Netherlands.

Designing Adaptive Interventions for Online Collaborative Modeling RACHEL OR-BACH Department of Computer Science and Information Systems, Emek Yezreel College, Emek Yezreel 19300, Israel E-mail: [email protected] WOUTER R. VAN JOOLINGEN ∗ Faculty of Behavioral Sciences, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands E-mail: [email protected]

Abstract One of the characteristics of collaborative learning is that it offers opportunities for learners to reflect and justify their work, to compare, understand and criticize their peers’ work, and to iterate through these processes as needed. This paper presents a design of a system that supports learners in taking advantage of these collaboration affordances in the context of collaborative modeling. The main focus is the automatic generation of adaptive interventions for the process of qualitative modeling of physical phenomena. Students interact with the learning environment by running a simulation, using visual tools for qualitative modeling, and communicating with each other through special tools and free text. The system tracks and analyses learners’ activities that relate to the subject matter tasks as well as to the communication between the learners and generates interventions accordingly. The layered interventions are designed also to integrate communication and content issues. Keywords: intelligent tutoring systems, computer supported collaborative learning, collaborative modeling, adaptive help, layered interventions

1. Introduction Many educational researchers are attracted presently to collaborative learning as a leading concept in education. Collaborative learning fits in very well with the changing view on learning and the nature of knowledge. The knowledge-construction process is not merely looked upon as an individual affair but rather as a process of interaction and negotiation with other agents in the learning environment such as the teacher, fellow pupils and the learning material (Van der Linden et al., 2001). Collaboration is expected to promote activities like elaboration, justification and argumentation that trigger learning mechanisms. Despite the expectations, there is no guarantee that these activities will occur without additional educational design constraints (Dillenbourg, 1999). The challenge of an intelligent collaborative learning environment is to support the interactions and activities contribut∗ This work has been carried out while Wouter R. van Joolingen was working at the Graduate school of Teaching

and Learning, University of Amsterdam.

356

OR-BACH AND VAN JOOLINGEN

ing to effective learning, as described above, at the same time leaving enough freedom for learners to perform the processes of collaboration and discovery themselves. One means for supporting productive communication is that of external representations. External representations can be used to reify entities of the learning material as well as communication acts. Variation in features of the representational tools, especially issues like salience and constraints of the representation, can have a significant effect on the learners’ discourse and on learning outcomes (Ainsworth, 1999; Löhner and Van Joolingen, 2001; Suthers, 1999). Along with the support that is provided by a special purpose representation both for knowledge construction and for discourse, support interventions might be needed to make the collaborative learning more productive. In a review of state of the art technology for supporting collaborative learning (Jermann et al., 2001), a classification framework is suggested to distinguish between mirroring systems, which display basic actions to collaborators, metacognitive tools, which represent the state of interaction via a set of key indicators; and coaching systems that offer advice on an interpretation of those indicators. The main advantages of collaborative learning are the opportunities for learners to reflect and justify their work, compare, try to understand and critique their peer work, and sometimes repeat and iterate through these processes. In order to enable learners to take advantage of the collaboration affordances, interventions should be designed to allow opportunities for the above, both by allowing time for it and by directing the interventions towards the support of such processes. We believe that support interventions within a collaborative learning environment should be designed to trigger the following processes to mutually enhance each other: (a) Individual and group activities, (b) Content-oriented and communication-oriented interactions. In this respect, Hoadley and Enyedy (1999) identify two categories of media that are common in computer-supported collaborative learning: communication media and information media. These media types map easily on two types of social activities in which learning is grounded: dialogue and monologue. Hoadley and Enyedy suggest the need for interfaces that help students’ transition from dialogue to monologue and back again. Another type of support needed, as was shown by empirical studies of collaborative learning is that learners need support for maintaining focus (Veerman, 2000). Focus maintenance support can involve both content and communication issues. In our case, this is achieved by content related educational goals that serve as good anchors for communication. Coherence and contradictions are a major concern in modeling activities, and conflicts and contradictions are good triggers for collaborative learning. We looked for anchors that are suitable for both naturally invoking discussion and also suitable for anchoring cognitive support. The current paper addresses the question of supporting collaboration in modeling environments. Modeling environments stimulate learners to explicitly build models of their understanding of a (scientific) problem (e.g., Jackson et al., 1995). These models can be communicated with others, and/or used to simulate behavior to express the consequences of learner’s models. In this paper we address the question of how to support the collaborative modeling process by providing interventions based on analysis of the models that learners make and comparing them to each other and to reference models. We propose

ADAPTIVE INTERVENTIONS

357

a method for generating supportive feedback based on an analysis of the progress of the modeling activity by the learners. The paper starts with theoretical background on modeling in science education, collaborative learning, and the use of conflicts as anchors for learning. The next chapter describes the learning environment, including a short description of two earlier studies that influenced the design of the shared learning environment. The remaining chapters describe our method for designing adaptive interventions and a respective discussion.

2. Theoretical Background and Its Implications for the System Design The theoretical background relates to three major topics that together influenced the design decisions. The first, “Science education and modeling”, explains the choice of the skills to be supported – qualitative modeling; the second, “Collaborative learning”, adds the collaboration dimension – collaborative modeling and how it can be supported. The third topic is about conflicts as a trigger for communication as well as an anchor for focusing on the content – debugging in a modeling process.

2.1. Science education and modeling Building models by learners is advocated as a fundamental activity in modern science teaching programs, and an appropriate emphasis for effective use of computers in high school science education (Thomas, 2001). Scientists build models to express and test their theories and to improve their understanding about complex systems. Modeling as practiced by scientists requires much prior knowledge and mathematical ability, but with an appropriate learning environment modeling activities can also be experienced by school students (Jackson et al., 1995). The idea is to provide the learners with visual tools for constructing qualitative models, where minimal prior knowledge from other domains is needed, and where modeling practice can be experienced. Collaboration within scientific domains is advocated for Mathematics, Engineering, and Technology learning (Springer et al., 1999) and for discovery in scientific domains (Okada and Simon, 1997). Learning from the construction and debugging processes can be enhanced by collaboration; by having learners compare, reflect and discuss their constructed models and the relations between the models and the real (or simulated) phenomenon. The visual modeling tools should be designed or chosen carefully to support the construction of the model as well as productive collaboration (Suthers, 2001). Chapter 3 of this paper describes the rationale and the role of such tools for modeling, collaboration and diagnostic assessment in our system design.

2.2. Collaborative learning There are many studies about the various aspects of collaborative learning, both from descriptive and prescriptive views. Studies deal mainly with what is collaborative learning,

358


how individual and social learning relate (Salomon and Perkins, 1998) and how collaboration can be supported. Dillenbourg (1999) stresses the need for supporting collaborative learning and suggests four support dimensions: a) Set up initial conditions, such as group size and tasks nature; b) Use of clear specifications of roles or matching of learners, for example by complementarity’s of their skills or knowledge (Hoppe and Ploetzner, 1999); c) Scaffold productive interactions by semi-structured interfaces, using special buttons, menus or sentence-openers; d) Monitor and regulate the interactions, by some diagnosis and intervention, or provision of some mirroring tools to enable self-regulation. The design decisions we present in this paper relate mainly to the scaffolding of productive interaction and the monitoring of the interactions. In a study to investigate the cognitive skill of coaching collaboration, Katz and O’Donnell (1999) checked why more capable students do not or cannot help their peers. The most common cause was a breakdown of communication between students. It happened because of unclear or ambiguous language, failure to intervene in a timely manner, misdiagnosis of peer errors, and insufficiently directive coaching. Our design deals with each of these problems by structuring the various interactions and by automatic generation of adaptive interventions. The following paragraphs review several systems that address part of these problems along with a comparison to our design. C-CHENE (Baker and Lund, 1996) employed a special communication interface with communication acts buttons that were grouped according to their function: task-oriented or dialog-control ones. In their investigation (Baker et al., 2001) on epistemic interactions for co-constructing scientific notions by school students, one of the conclusions was that the right degree of freedom and constraint on communicative interaction can lead students to concentrate on the most fundamental aspects of the task. In our design we employed a free text input facility along with structured interaction and communication tools that not just separate task-oriented and dialog–control buttons; but direct the task, reify the subject matter structure (relevant objects and their attributes) and exhibit the goal of developing modeling skills through productive collaboration. In Algebra Jam (Singley et al., 1999) two types of facilities are used for supporting communication in team problem solving. The goal is to ease the typing load on the sender and to ease the comprehension load on the receiver. One is “Object-Oriented Chat” for linking the chat topic to a specific object, and the other is “Collabicons” for framing the intention of the message. The “Collabicons” are based on a message typology defined by the authors, and are used also by the system for updating beliefs about a student’s collaborative skill. The design of our communication tools involves a more structured “Object-Oriented Chat” that links the chat topic to both a specific object and a specific chosen attribute of this object. Students can also frame their intention by a predefined special menu which is both content and communication oriented. In our design all these elements are further analyzed along with the free text input for the adaptive interventions. Söller (2001) designed and developed tools for supporting social interaction in an intelligent collaborative learning system. A basic guideline for the design was that structuring the environment by sentence openers or special buttons for communication is not enough, there is also need for support and guidance for effective collaboration. The respective tools


359

enable the analysis of collaborative learning conversation and activity to support the social interaction. Söller (2001) system identifies and targets group interaction problem areas, while our system design addresses subject matter problems in the context of collaboration. 2.3. Conflicts as anchors for learning Conflicts are a natural part of any collaborative activity and can play an important role for promoting reflection and articulation. The inevitability of conflicts and their potential to increase group productivity has generated a large body of research in diverse areas (Sociology, Psychology, Education, Distributed Artificial Intelligence, etc.). Tedesco and Self (2000) define conflicts as the last stage of a three-phase process. The first is difference of views, when agents have different views but have not yet communicated them to each other; the second is disagreement, where agents inform each other about their inconsistent view but a discussion over them does not follow; and finally there is a conflict if the agents involved also try to convince the other about their own points of view. It should be noted that the difference of views might be based on contradictions (or misinterpretations) to the underlying evidence, assumptions, etc. The educational goal then is to develop it into a conflict with potential for reflection and articulation. The design of COLER (Constantino-Gonzalez and Suthers, 2000) is based also on the potential advantages of conflicts for learning during collaborative problem solving. The system employs a coaching agent that encourages students to share and discuss solution components that conflict with components of the group solution. The system utilizes domain specific knowledge to identify semantically important differences between students’ Entity-Relation diagrams (for a given problem regarding conceptual database design). Our design uses conflicts between students’ designs as learning opportunities but also conflicts between the designs and the modeled phenomena (or its simulation), conflicts with the problem statement and conflicts with the modeling formalism. All these conflicts serve as anchors for promoting iterative processes of reflection and collaboration by adapted interventions related to the conflicts. Modeling activities involve search for coherence with aspects of the real phenomenon, the underlying assumptions, and the modeling formalism. So anchoring support interventions in contradictions that can bring up conflicts is beneficial both for the content to be learned and for effective collaborative learning (Or-Bach and Van Joolingen, 2001). 3. The Learning Environment 3.1. An overview The learning environment deals with modeling tasks based on simulations that can be conducted for discovery and validation of a model. The learning environment is based on SimQuest (Van Joolingen and de Jong, 2003) which includes a dedicated visual tool for modeling (Van Joolingen and Löhner, 2001; Löhner et al., 2003). The learning environment is intended to be used by junior-high students during their science class. Students

360


are expected to work on-line in pairs, alternating between individual and collaborative sessions. The simulations represent phenomena in the real world, such as the evolution of temperature over time in a house (as depicted in Figure 1). The main goal is to have learners experience and practice basic qualitative modeling activities, and to build understanding of the modeled domain through these activities. Modeling activities include identifying input variables and state variables, using a symbolic notation for describing a model, discovering chains of influence and debugging an individually constructed model. This chapter starts with the description of the individual learning environment and the various ways that the learners have to interact with the environment. It continues with a short description of two earlier studies, which investigated students’ modeling and communication processes while working with this learning environment. This is followed by a description of the shared environment that employs respective results of those empirical studies. 3.2. Modeling within a simulation based learning environment The individual learning environment as described above consists of the following components (Figure 1): • The problem statement and explanations about the various elements. • A simulation for exploring the phenomenon that should be modeled. • A modeling space for using a formalism consisting of a set of visual objects and rules for constructing qualitative models. 3.2.1. The problem statement The problem statement defines the objective of the modeling activity, that is the research question as well as the context in which this can be answered. 3.2.2. The simulation The simulation contains input slots for defining initial values for the various variables, graphs, and fields displaying the value of output and state variables. Learners can run the simulation and examine changes in values along with graphs that are produced of the major functions that are time dependent. Learners can change the view on the simulation in a graph window. These views are needed for checking relations that cannot be observed directly in the default view of the simulation. Dedicated views exist for different kinds of variables, especially for time dependent variables (displayed as a continuous curve) and time independent variables (displayed as a series of points). 3.2.3. Visual modeling tool The visual modeling tool allows learners to construct a model by connecting nodes that represent variables with arrows indicating relations between the variables. The main goal of this visual representation is scaffolding the basic stages of qualitative modeling. These stages involve (1) identifying the variables that compose the description of the model (2) identifying influences among the variables involved in the phenomenon (3) defining the qualitative nature of the influences (4) defining the

Figure 1. Overview of the learning environment.


361

362


precise specification of relations between the variables. The building blocks to construct the model are variables and links. Variables can be state variables, output variables (time dependent or not) or input variables. State variables change over time as a consequence of their natural development. The rate of change of state variables is determined by other variables. Input variables represent external influences on the model. Output variables depend on input and/or state variables and represent observable phenomena in the domain. The type of the variable (state variable, input or output) is represented by its shape. The color of a variable indicates whether it depends on time. Links are used for specifying influences between variables. Learners are expected to draw arrows showing the direction of influence and to apply properties to the links by context sensitive menu(s). The properties to be applied are: (1) Sign (+ or − to indicate positive or negative influence), (2) The type of relation (dynamic or rate). The type indicates whether it is a relation that holds in any point of time (y = f (z)) or a change over time/rate relation (dy/dt = f (z)). The type property is manifested by a different line type. The use of this modeling tool can be seen in Figure 1. 3.3. Earlier empirical studies – modeling and communication processes Earlier studies investigated the behavior of learners involved in this modeling environment. Löhner, van Joolingen and Savelsbergh (2003) used a setting where dyads of learners (11th grade) worked with the modeling environment face-to-face, sitting together behind a screen. This study showed that learners were capable of creating reasonable good models. Learners using the graphical model representation that was described above on average reached a score of 80% on a similarity measure comparing learners’ models with a model made by an expert. However, looking more closely at the model building process, learners seem to adopt a trial and error strategy, jumping from one relation to another without building a deeper understanding of the model. In another study (Saab and Van Joolingen, 2003), learners collaboratively explored a simulation model, answering questions about the simulation, without building a model themselves. Dyads of learners (10th grade) communicated with each other through a shared environment and chat. It was found that argumentative communication (asking questions, giving arguments) contributed to the inquiry processes of hypothesis generation and interpretation of data. Both stand for a more thoughtful generation of knowledge from the inquiry process. On the other hand, more directive communication, where one learner takes the lead, contributes to the performance of experiments. Hence, for triggering deeper cognitive processes in constructing a model, one needs to trigger argumentative behavior. More directive communicative processes may be a trigger to stimulate the performance of relevant experiments. 3.4. Design of the shared learning environment Students alternate between their individual workspace and a shared one. It is possible to share elements between environments. In the shared workspace learners can post their


363

Figure 2. The communication interface, using a facility common to object oriented chat. The learner can point to an object and add the comments in direct relation with the object pointed to.

models and comment on their own model and that of their collaborator(s). There is also a shared simulation space that learners can use to watch experiments together. Our design is motivated by the following communication goals: • Let the students communicate in a natural and easy way. • Make the communication useful for learning (maintaining focus, conveying the main issues and purposes of the communication, etc.). • Stimulate communicative processes that seem beneficial for the inquiry process. • Enable the system to “understand” the communication, in order to detect anchors for support and instruction. The design of the communication is tailored to make the communication between learners as explicit as possible without being overly obtrusive. What need to be made explicit is what the communication is about and what the intention of the communication is. By making the various aspects that learners can communicate about visually salient in the modeling environment, it will be easy for learners to indicate visually what they are talking about. A learner who wishes to communicate with a fellow learner starts by pointing to the object that the communication is about. This will open an annotation pane that allows the learner to type in free text. On the annotation pane the learner indicates which property of the object is addressed and what the intention of the communication is (see Figure 2). This structure for the communication provides learners with tools to both structure their thought as well as provide constructive help to peers. Tagging the information with the purpose of the message can be done before or after writing the text in order not to interfere with the natural flow of thought. The system uses the tags for leveraging the degree of confidence in the system conclusions regarding the communication. The tags menu is actually

364


a checklist with relevant communication intentions. A learner can use several tags for a specific input, as an input can serve several related intentions, such as “critique” and “suggestion for an experiment”. The communication intentions are derived from the findings in one of the earlier studies mentioned in the previous section (Saab and Van Joolingen, 2003), and consist of argumentative intentions: Justification, Critique, Question, and Directive intentions, Suggestion for Experiment, Reference to problem statement.

4. Help Generation, Diagnosis and Analysis Help generation and diagnosis are highly related, as it makes sense to diagnose only when intervention might be considered. This section deals with both issues; it starts with the use of conflicts as anchors for support and the main guidelines for intervention and help generation; continues with the use of layered interventions, the general process (including the required diagnosis procedures) and the intervention templates; and ends with a scenario. The scenario demonstrates diagnosis and decisions regarding the support generation.

4.1. Using conflicts as anchors for support In providing support for collaborative modeling we focus on the process of debugging models. Debugging or revising a model is a major activity of modeling in which learners try to resolve shortcomings and contradictions in their models. We distinguish between differences and contradictions. A difference occurs when the models of the learners differ, e.g., variables or relations are present in one but not in the other. A contradiction occurs when a learner’s model contradicts the underlying system model. Differences serve as a starting point for discussion, as an indication that underlying contradictions might be involved. Lack of differences does not mean that the model is correct (that there are no contradictions), and this might be a point where an intervention is needed for any learning to occur. Some differences do not necessarily imply that a model is incorrect, but still it is a good starting point for useful discussion. We distinguish in the current paper between the following types of contradictions: a. Contradictions that relate to the formalism of the (visual) modeling environment: building blocks and construction rules. b. Contradictions to the problem statement. For instance learners can model relations that are not present or indicated as non-relevant in the problem statement. c. Contradiction between the learners’ suggested model (as described by the visual representation) to the real model (that the learner can explore via the simulation). For example, the learner(s) model suggests that A increases as B increases, while in the real model A decreases as B increases. The actual context can vary here with respect to whether the learner(s) performed the relevant simulations before. This will affect the help intervention.


365

4.2. Guidelines for help generation The visual tools of the environment provide the opportunities for learners to compare and “see” differences and contradictions. The collaborative setting provides another mean of potential feedback, feedback coming from a peer when the contradictions were not “seen” or were not understood. We believe that only when these opportunities do not lead to progress, some other (minimal) intervention should follow, pointing to a way to “see” the contradiction within the tools that are available in the learning environment, or a way to help each other to interpret it. An opportunity for a support intervention is when a contradiction exists and the system has evidence that the learners are unaware of it. So the main reasoning procedures of the system deal with the detection of differences and contradictions, and with some analysis of the communication. The analysis of the communication involves several sources of information: the object pointed to and the item choice from the respective menu; and the tags that the learner used along with the free text input. The free text input is analyzed by respective keywords. 4.3. Layered interventions The layers relate to the specificity of the intervention, each layer is more specific than the previous one, and hence more intrusive in the learning process. The four layers are: (a) suggestion to discuss a difference, (b) communication requests, (c) suggestion for an anchored discussion and (d) direct help intervention. The system presents also (e) followup interventions in some specific occasions. We describe for each layer the rationale, the audience, the required data for generating the intervention of this layer, and the respective types of templates to be used for presenting the intervention. Table 1 summarizes the various types of templates that are employed. In the layers and templates one sees reflected the types of communicative acts that were mentioned by Saab and Van Joolingen (2003): argumentative and directive. At the first two layers, the system stimulates these acts between learners, in deeper layers, the system itself takes up an argumentative (c) or directive (d) role. 4.3.1. Suggestion to discuss a difference This is the first intervention layer, when the system detects a difference between the visual models constructed by the learners. The rationale is that learners should first try to help each other rather than relying on the system. Therefore the audience is both learners, and the main content is a request to discuss a certain element in the environment. The system chooses a difference that relates to a concept with a higher priority, as will be explained later about the diagnosis procedures. For deciding on the intervention (as opposed to stop or go to deeper intervention layer) and for generating an intervention, the system employs data from the diagnosis process (described later). The analysis of the communication is done by both tracking the use of the pointing device along with the choice of the respective menu item, and also analysis of the free text by the keywords related to the current concept and possible synonyms. The template that is used for presenting the intervention presents a reflection/discussion question such as “Do you see your different opinions about concept?”

366


4.3.2. Communication requests This layer refers to two types of interventions regarding communication requests: A suggestion for critiquing and a request for justification. The rationale is that when learners do not succeed in helping each other without external help, one learner can be encouraged to help the other. The audience is only one learner. The data that is required for the second type of intervention is also the free text input and the respective tagging, to check whether the learner had already given a respective explanation. Two templates are used here “Justify . . .” and “Critique . . .” The resulting critique is expected to be a constructive one, as the learner has to use the communication aids: pointing, choosing the respective property and writing text that might be informative. 4.3.3. Suggestion for an anchored discussion This layer already provides content related help, a suggestion for anchoring a relevant discussion. For instance, learners can be given a suggestion to do a certain experiment that confirms or rejects a certain relation found. Again the audience is one learner. This layer requires a deeper level of diagnosis regarding the type of contradiction involved in the difference that was detected between the learners’ models. The suggested anchor for discussion can be the type of contradiction involved, or a relevant activity in the environment that was performed previously. The use of the diagnosis table, which is described later in this chapter (and shown in Table 1), enables the identification of contradictions that are manifested in the erroneous model (for the respective concept/issue). The manifested contradictions can relate to each of several sources: the formalism, the problem statement and the underlying model as observed in the simulation. Each source requires a different intervention, so the system needs a conflict resolution mechanism whenever there exist more than one source. The default order for suggesting a discussion is according to the columns order of the table. This order implies starting with contradictions to the formalism, where it might be only a learner slip and getting to the higher order types of mistakes/contradictions. The order might change according to previous activities of the learners that the system can also use as an anchor for discussion. This adaptation requires the recording of relevant activities of the learners. The relevant activities are the use (access or pointing for communication) of specific items of the formalism and use of specific views of the simulation (not the default ones). Whenever such a respective anchor exists, the system uses it for anchoring the discussion, instead of using the default order of elimination according to the diagnosis table order. This level employs templates such as “Check and discuss . . .” and “Experiment and discuss . . .”, depending on the type of contradiction. Another type of template is for suggesting a previous activity and bringing up the relevant records. 4.3.4. Direct help intervention This layer is the most informative one for the learner and requires additional analysis, mainly of the underlying model for generating a suggestion of the most appropriate view of the simulation. The rationale is that in the end the system should be able to help the learners out of a problem situation. The following section about the diagnosis procedures elaborates on the required additional analysis. This layer employs more specific templates of the types “Check . . .” and “Experiment . . .”, especially the “Experiment . . .” type for the various views of the simulation.

367


Table 1. The diagnosis table, indicating the various diagnosis options Concepts/issues

Contradictions to Formalism

Problem statement

Simulation

State variable

A variable exists with correct rate links and still not painted as state variable.

A variable that has in the problem statement an explicit wording of “state variable” or wording “has initial value of . . . and changes in time” and is not painted as such.

A state variable exists with no painted input link.

Input variable

1. A variable exists with output links and no input links and still not painted as input variable. 2. A variable has input links and is painted as an input variable.

A variable that was defined in the problem as “input variable” is not painted as such.

A variable is painted as an input variable but its value changes through out the simulation.

Time dependence of a variable

1. A variable that is timedependent is not painted as such. 2. A variable that is not time-dependent is painted as time-dependent.

Rate link

A link was painted into the state variable that was not colored as rate link.

1. A variable that is timedependent is not painted as such. 2. A variable that is not time-dependent is painted as time-dependent. A link was painted into the state variable that was not colored as rate link, when the problem statement includes a respective wording of “rate link . . .”

Influence/link existence

Within the underlying model there is a negative or positive influence between the respective variables, and no link was painted.

Positive influence (+ sign of the link)

X

X

Within the underlying model there is a negative influence between the respective variables, and a link with positive sign was painted.

Negative influence (− sign of the link)

X

X

Within the underlying model there is a positive influence between the respective variables, and a link with negative sign was painted.

368


4.3.5. Follow-up intervention A follow-up intervention is required when the system did not get to the deeper layers (third or fourth) because the model(s) was already corrected by the learner(s), but still there is no evidence that the learners performed the critical experiment in a case where there were contradictions to the simulation in the model. As the respective diagnosis is already done by the system (for a direct intervention) it requires almost no additional computations, but a slightly different template. 4.4. Diagnosis procedures The diagnosis procedures are centered on the diagnosis table (Table 1) that contains the required computational methods for diagnosing the learner’s visual model. Some additional analysis, pertaining to both learners’ communication and some elements of the underlying model, is done for generating the intervention. The purpose of the diagnosis table is to identify a need for intervention that is meaningful according to the system’s educational goals. The rows of the table present the various issues/concepts to be learned, while the columns present the sources of contradictions that may be presented in a learner-constructed model. The cells of the table contain methods for detecting respective possible need for intervention. Some of the cells in Table 1 were filled for demonstration. A cell may contain more than one method. The sources of contradictions imply a suitable intervention. The order of the rows implies instructionally preferred order to deal with the concepts involved, while the order of the columns implies logical order of elimination of possible existing contradictions. Execution of the table’s methods at each stage will produce for each method a list of the respective variables/links where these contradictions were detected for a given learner. The general diagnosis procedure runs as follows: • The diagnosis table is filled-up and updated regularly for both learners. After each significant contribution by one of the learners, the cells in the table are re-evaluated. • Both tables are compared; differences in the table will invoke the first intervention layer. Interventions will be invoked in the priority order defined by the table. • If there are no differences, but there exist contradictions between with the formalism, problem statement or simulation, the second layer of intervention is involved. • If the intervention does not yield a change in the learner’s models, the next-higher layer of support interventions is invoked. Comparison of both tables can show differences between the visual models constructed by the two learners. The detection of differences is required for the first intervention layer. As the table relates to the main issues/concepts to be learned, the differences that the system will intervene about will be only those related to these concepts, in the order implied by the table. The use of this table also ensures that the system intervenes only about differences that have some contradictions beneath. The table also enables the identification of cases where there is no difference, but still there are contradictions (that is a case where the second intervention layer comes in). For a specific concept, the diagnosis table enables the detection of the underlying type of contradiction, which determines the type of intervention. This is required for the third level of intervention, for the system suggestion for an anchored discussion.


369

Table 2 summarizes the types of templates that the system employs for intervention. These templates are invoked by the reasoner in order to generate the appropriate feedback and are linked to both the intervention layer as described above and the kind of contradictions that are detected using the diagnosis table. For the direct help of the fourth layer additional analysis is needed. This is done by use of respective tables for each type of contradiction, such as Table 3 for generating support for the link/relation sign. 4.5. A scenario In the current section we present a hypothetical scenario of interventions that can be generated by our system. The scenario is based on an interaction actually occurring with a prototype of the modeling system without our intervention mechanism. Let’s assume that the model of learner A is correct (Figure 3) and that of learner B is the one shown in Figure 4. The differences that can be seen between the models are: • The model of learner B does not include the variable for the house heat capacity (C_house). • The model of learner B lacks the link between T_inside, the temperature in the house and the amount of heat loss (P_loss). A scenario that uses these two models as a starting point can yield the following interaction between learners and system: 1. Learners show their work to each other, the system detects differences but does not intervene immediately. 2. Learner B adds the variable for the house heat capacity, links it correctly, but assigns an incorrect sign to the link. Learner B uses the communication channel, points to the variable, chose the “type” item from the menu, writes something in the chat window and tags it “explanation”. Then learner B submits the model again to the shared workspace (using the submit/end button). 3. The system notes that a change was made regarding both the variable and the link, but an explanation/justification was given only for the variable (as there was no pointing to the link and no text about link or sign). 4. The system asks for justification regarding the link. 5. Student B gives some explanation but does not change the sign. 6. The system checks for differences and found the sign difference and the one regarding the link between T_inside and P_loss (the loss of heat due to imperfect isolation). 7. The system chooses to start with the missing link as this regards the presence of a link instead of a sign of the link, the latter being lower in the hierarchy of the diagnosis table. 8. The system checks whether there was already some discussion regarding the link, found that there was not and invokes the “suggestion to discuss a difference” layer, using the corresponding template. 9. The learners discuss the difference, resulting in B adding the link.

370


Table 2. The list of intervention templates Template

Layer

Message

Elements to be presented

Reflection/discussion question

A

Do you see your different opinions about object?

Justify communication request

B

Can you explain to other learner why you think your model is right?

Critique communication request

B

Can you explain other learner why his/her model is incorrect/incomplete/. . .

Check and discuss respective formalism

C

Do your think this object is used correctly here?

Pointer to an object.

Check and discuss problem statement

C

Do you think that this object is in line with the problem we are discussing?


Experiment and discuss (with history)

C

Rerun this experiment. Are the results consistent with this link?

Pointer to a link. Experiments history with a hilighted experiment.

Experiment and discuss (without history)

C

Can you find experiments in favor or against this link?

Pointer to a link.

Concept oriented check of formalism

D

Is this object correctly defined according to this description?

Pointer to an object. Formalism description with the relevant concept hi-lighted.

Concept oriented check of problem statement

D

Is this object correctly defined according to this description?

Pointer to an object. Problem statement with the relevant concept hi-lighted.

Experiment with a viewpoint of an input and time-related function

D

Look at this experiment, is it consistent with this link?

Pointer to a link. Execution of a respective experiment in the simulation panel.

Experiment with a viewpoint of two time-dependent functions

D



Experiment with a viewpoint of two time-independent variables/functions

D



Watch the experiment

E




371

ADAPTIVE INTERVENTIONS Table 3. Additional analysis for direct help interventions Link between

How to diagnose

How to intervene

Input and time-related function

Underlying model

Graph of the function (change in time) for two values of the input, while other inputs are kept unchanged

Two time-dependent functions

Underlying model

Have each function on an axis and plot the values for two points of time*

Two time-independent variables/functions

Underlying model

Have them on each axis and present the resulted graph

* Assuming all the functions are monotonic.

Figure 3. Learner A’s model.

Figure 4. Learner B’s model.

10. The system moves to pointing to the sign of the link between C_house and T_inside. 11. The system checks whether discussion has taken place already, and so skipped the “suggest the learners to discuss the difference” intervention (first layer) and went for the second layer. 12. The system checks the correctness of the two models regarding that link, and then intervenes by suggesting learner A to critique the link properties.

372


13. Learner A points to the link, chose the sign property from the respective pop-up menu, writes something in the chat window and tags with two tags of “critique” and “experiment”. 14. Learner B uses the default simulation, having not succeeded to learn from it regarding this link, he resubmitted it with no change to the shared space. 15. The system enters the third layer of anchored discussion. The relevant experiment to conduct was not conducted before, so there was no relevant anchor. The respective type of contradiction, which requires an “experiment” intervention, was already suggested by the other learner, so there was no point to repeat it. So the system entered the fourth intervention layer. 16. The system realizes the need for a direct help regarding a contradiction to the simulation, using the viewpoint of the simulation that demonstrates best the issue of that link sign. For this decision the respective table for additional analysis, as shown in figure 3, is used. From the underlying model it can be deduced that this is a link between a parameter and a time dependent function, and so the best viewpoint of the simulation is having the inside temperature shown for two increasing values of C_house, while the rest are kept unchanged. The scenario shows the active but unobtrusive behavior of the system; active in the sense that learner’s actions are evaluated thoroughly, unobtrusive in the sense that interventions only are generated at the time that everything else fails, leaving room for the learners to solve problems themselves. The scenario also shows the various roles of conflicts as learning opportunities.

5. Discussion In the current paper we describe a means of intervening in a collaborative discovery and modeling environment. This undertaking has its risks, as both collaborative learning environment and discovery environments are advocated because of the freedom they offer for self-directed learning; and (for collaborative environments) because of the support learners can give each other. However previous work on discovery learning has shown that further support for learners is needed (Van Joolingen and de Jong, 1997) and that intelligent support for discovery learning can deepen the processes of discovery (Veermans et al., 2000). These considerations justify the attempt for trying to design support interventions for collaborative modeling, as it becomes clear that support for learners is also needed in this area (Löhner et al., 2003). Many design decisions that led to the described system design involved the dilemma between an open learning environment versus a more structured one with regard to the interaction, communication and guidance. The guideline for weighing the tradeoffs was that the constraints we impose and the interventions we present should be meaningful in order to support reflection, communication and the understanding of both the subject matter and the task of modeling. The learning environment consists on artifacts that reify subject matter entities, modeling tools and communication entities. The layered


373

interventions enable reflection guidance that gradually turns (when and if needed) to more directive guidance interventions. In a discovery oriented learning environment it is impossible (and undesirable) to have a complete understanding and control of the learning processes. In a learning environment, which like ours is based on experiments to investigate simulated phenomena, several approaches were suggested for learner modeling and respective intervention. Veermans and van Joolingen (1998) concentrate on the discovery behavior and they model the induction and deduction mental activities of the learners and then generate the intervention accordingly. They describe a method for generating adaptive feedback that takes as input the current hypothesis of the learner (indicated by the learner) and the experiments performed by the learner to test this hypothesis. Koning et al. (2000) used qualitative modeling techniques for the phenomena in order to be able to map it to qualitative modeling mental activities in order to automatically diagnose learner behavior and generate the interventions accordingly. The above examples are suitable for the design of support interventions in individual learning, as it tracks some coherent rational of a learner. In a collaborative setting we encourage the exchange and reformulation of problem-solving ideas, so the tracking of an individual rational becomes impossible. When collaboration is desired, the support should be for having the learners support each other before any other intervention is generated and then generate an intervention that takes into account the content and the communication history and context. The goal of the support interventions should be to trigger processes that mutually enhance each other: a) Individual and group activities, b) Content-oriented and communication-oriented interactions. In our system the trigger for intervention is content oriented, so there is no fine analysis of the discourse, like, for example, in Soller’s system (Söller, 2001). The intervention itself is both content and communication oriented. The communication aspect is manifested both by the audience facet of the intervention (a specific learner or the whole group) and the type/layer of the intervention. Our approach for supporting collaborative learning in modeling tasks is directed more towards the modeling tasks than the discourse, and more towards individual learning processes within collaboration than towards a collaboration product. The idea is to support an iterative process similar to what Dekker and Elshout-Mohr (1998) suggested that is based on the following key activities: to show one’s work, to explain one’s work, to justify one’s work and to reconstruct one’s work. The methods described in this paper have the potential to be scaled up as more foreseen enhancements are added to the system, such as, more properties for the links, diagnosis of contradictions to common sense or required previous knowledge, and diagnosis of contradictions between what one did and what one said. These scaling up capabilities were a basic requirement as the design and methods described in the paper for supporting collaborative modeling are planned to be integrated in some other projects based on the SimQuest software (Van Joolingen and de Jong, 2003) and also in the Co-Lab project sponsored by the European commission. The computational methods we use are knowledge based, but not specific to the modeling problem or the subject matter for the learners modeling tasks.

374


Acknowledgements The research was conducted while the first author was with the Graduate School of Teaching and Learning, University of Amsterdam. The research was supported by a grant from NWO, the Dutch national science foundation, under number 411-211-13b.

References Ainsworth, S. (1999) The functions of multiple representations. Computers and Education, 33, 131–152. Baker, M. and Lund, K. (1996) Flexibly structuring the interaction in a CSCL environment. In Proceedings of the European Conference on Artificial Intelligence in Education (EuroAIED’96), pp. 401–407. Baker, M., de Vries, E., Lund, K., and Quignard, M. (2001) Computer-mediated epistemic interactions for coconstructing scientific notions: Lessons learned from 5-year research programme. In Proceedings of the European CSCL Conference, March, Maastricht. Constantino-Gonzalez, M. A. and Suthers, D. D. (2000) Using meta-cognitive conflicts to support group problem solving. In Intelligent Tutoring Systems: 5th International Conference, G. Gauthier, C. Frassson, and K. VanLehn (eds.). Springer, Berlin, pp. 232–242. Dekker, R. and Elshout-Mohr, M. (1998) A process model for interaction and mathematical level raising. Educational Studies in Mathematics, 36, 303–314. Dillenbourg, P. (1999) Introduction: What do you mean by “Collaborative Learning”. In Collaborative Learning, P. Dillenbourg (ed.). Pergamon, Oxford. Hoadley, C. M. and Enyedy, N. (1999) Between information and communication: Middle spaces in computer media for learning. In Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference, C. Hoadley and J. Roschelle (eds.). Lawrence Erlbaum Associates, NJ. Hoppe, U. and Ploetzner, R. (1999) Can analytic models support learning in groups? In Collaborative Learning, P. Dillenbourg (ed.). Pergamon, Oxford. Jackson, S., Stratford, S. J., Krajcik, J., and Soloway, E. (1995) Making system dynamics modeling accessible to pre-college science students. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA. Jermann, P., Soller, A., and Muehlenbrock, M. (2001) From mirroring to guiding: A review of state of the art technology for supporting collaborative learning. In Proceedings of the European CSCL Conference, March, Maastricht. Van Joolingen, W. R. and de Jong, T. (1997) An extended dual search space model of scientific discovery learning. Instructional Science, 25, 307–346. Van Joolingen, W. R. and Löhner, S. (2001) Representations in collaborative modeling tasks. In Proceedings of the External Representations in AIED Workshop of AIED 2001. Van Joolingen, W. R. and de Jong, T. (2003) SimQuest, authoring educational simulations. In Authoring Tools for Advanced Technology Learning Environments: Toward Cost-Effective Adaptive, Interactive, and Intelligent Educational Software, T. Murray, S. Blessing, and S. Ainsworth (eds.). Kluwer, Dordrecht, pp. 1–31. Katz, S. and O’Donnell, G. (1999) The cognitive skill of coaching collaboration. In Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference, C. Hoadley and J. Roschelle (eds.). Lawrence Erlbaum Associates, NJ. Koedinger, K. R., Suthers, D. D., and Forbus, K. D. (1999) Component-based construction of a science learning space. International Journal of Artificial Intelligence in Education, 10, 292–313. De Koning, K., Bredeweg, B., Breuker, J., and Wielinga, B. (2000) Model-based reasoning about learner behaviour. Artificial Intelligence, 117, 173–229. Löhner, S. and van Joolingen, W. R. (2001) Representations for model construction in collaborative inquiry environments. In European Perspectives on Computer Supported Collaborative Learning, P. Dillenbourg, A. Eurelings, and K. Hakkarainen (eds.). Maastricht McLuhan Institute, Maastricht, pp. 429–436.


375

Löhner, S., van Joolingen, W. R., and Savelsbergh, E. R. (2003) The effect of external representation on constructing computer models, Instructional Science, 31, 395–418. Van der Linden, J., Erkens, G., Schmidt, H., and Renshaw, P. (2001) Collaborative Learning. In New Learning, P. R. J. Simons, J. van der Linden, and T. Duffy (eds.). Kluwer, Dordrecht. Okada, T. and Simon, H. A. (1997) Collaborative discovery in scientific domain. Cognitive Science, 21, 109– 146. Or-Bach, R. and van Joolingen, W. R. (2001) Contradictions as an anchor for providing help during collaborative learning. In Proceedings of the Workshop on Help Provision and Help Seeking in Interactive Learning Environments, The 10th International Conference on Artificial Intelligence in Education, Texas. Saab, N. and van Joolingen, W. R. (2003) Effects of communication processes in collaborative discovery, paper to be presented at the AERA Conference, Chicago, April 21–25. Salomon, G. and Perkins, D. N. (1998) Individual and social aspects of learning. Review of Research in Education, 23. Singley, M. K., Fairweather, P. G., and Swerling, S. (1999) Team tutoring systems: Reifying roles in problem solving. In Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference, C. Hoadley and J. Roschelle (eds.). Lawrence Erlbaum Associates, NJ. Söller, A. L. (2001) Supporting social interaction in an intelligent collaborative learning system. International Journal of Artificial Intelligence in Education, 12. Springer, L., Stanne, M. E., and Donovan, S. (1999) Effects of small group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis. Review of Educational Research, 69, 21–51. Suthers, D. (1999) Representational support for collaborative inquiry. In Proceedings of the Hawai International Conference on System Sciences, January 1999, Maui, Hawai’i. Suthers, D. (2001) Towards a systematic study of representational guidance for collaborative learning discourse. Journal of Universal Computer Science, 7, 254–277. Tedesco, P. and Self, J. (2000) Using meta-cognitive conflicts to support group problem solving. In Intelligent Tutoring Systems: 5th International Conference, G. Gauthier, C. Frassson, and K. VanLehn (eds.). Springer, Berlin, pp. 232–242. Thomas, G. P. (2001) Toward effective computer use in high school science education: Where to from here? Education and Information Technologies, 6(1), 29–41. Veerman, A. (2000) Computer-supported collaborative learning through argumentation. Thesis, Utrecht University, Utrecht. Veermans, K. and van Joolingen, W. R. (1998) Using induction to generate feedback in simulation-based discovery learning environments. In Intelligent Tutoring Systems, 4th International Conference, B. P. Goetl, H. M. Halff, C. L. Redfield, and V. J. Shute (eds.), San Antonio, TX, USA. Springer, Berlin. Veermans, K., van Joolingen, W. R., and de Jong, T. (2000) Promoting self-directed learning in simulation based discovery learning environments through intelligent support. Interactive Learning Environments, 8, 257–277.