Examining factors affecting learning performance

Computers & Education 68 (2013) 429–439

Contents lists available at SciVerse ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

Enjoy and learn with educational games: Examining factors affecting learning performance Michail N. Giannakos 1 Department of Computer & Information Science, Norwegian University of Science and Technology (NTNU), Sem Sælands vei 7-9, NO-7491 Trondheim, Norway

a r t i c l e i n f o

a b s t r a c t

Article history: Received 1 March 2013 Received in revised form 27 April 2013 Accepted 6 June 2013

Educational games have enhanced the value of instruction procedures in institutions and business organizations. Factors that increase students’ adoption of learning games have been widely studied in past; however, the effect of these factors on learners’ performance is yet to be explored. In this study, factors of Enjoyment, Happiness, and Intention to Use were chosen as important attitudes in learning educational games and increasing learning performance. A two-step between group experiment was conducted: the first study compared game-based learning and traditional instruction in order to verify the value of the game. 41 Gymnasium (middle school) students were involved, and the control and experimental groups were formed based on a pretest method. The second study, involving 46 Gymnasium students, empirically evaluates whether and how certain attitudinal factors affect learners’ performance. The results of the two-part experiment showed that a) the game demonstrated good performance (as compared to traditional instruction) concerning the gain of knowledge, b) learners’ enjoyment of the game has a significant relation with their performance, and c) learners’ intention to use and happiness with the game do not have any relation with their performance. Our results suggest that there are attitudinal factors affecting knowledge acquisition gained by a game. Ó 2013 Elsevier Ltd. All rights reserved.

Keywords: Media in education Secondary education Learning communities Evaluation of CAL systems

1. Introduction The growth of educational games in the last years has largely impacted learning procedures (Shin, Sutherland, Norris, & Soloway, 2012; Virvou, Katsionis, & Manos, 2005). Studies indicate that playing video games gives learners a “mental workout” and the structure of activities embedded in computer games develops a number of cognitive skills. Players are faced with a stream of decisions, and must engage with problem solving strategies, which involve the engagement with a series of complex tasks and nested sub-tasks (Johnson, 2005). The four part cycle (Gee, 2003) defines certain periods where players engage/probe, hypothesize, re-probe, and rethink during the play time of a video game. Similarly, Garris, Ahlers, and Driskell (2002) indicate that educational games engage students in repeated judgment–behavior– feedback loops. In addition, McFarlane, Sparrowhawk, and Heald (2002) linked game-playing with the potential to develop skills in decision making, design, strategy, cooperation, and problem solving. On top of that, certain practices of educational games (e.g., Narration, Reward, Game Mechanics) have proved their value on the learning arena (Cohen Group Report, 2011). The emergence of educational games has further facilitated the wide adoption of learner-centered education and other changes in educational practices. Computer games have drawn significant attention from educational institutions and business organizations due to the potential educational and cost benefits; however, the introduction of games in teaching is often complex, and learners do not always use them as expected (Yi & Hwang, 2003), and/or learners do not have the expected performance when they are using them (Ketelhut & Schifter, 2011). In previous studies, attitudes (e.g., perceptions, beliefs, intentions) that led students to use games for learning have been thoroughly explored (Bourgonjon, Valcke, Soetaert, & Schellens, 2010; Papastergiou, 2009). Nevertheless, how learners’ attitudes (e.g., acceptance) for a game are connected with learning performance (hereinafter performance) has not been well addressed yet. For instance, Yusoff, Crowder, and Gilbert (2010) identified attitudes and combinations of attitudes that led the learner to accept and use the educational game. But the E-mail address: [email protected]. This work was carried out during the tenure of an ERCIM “Alain Bensoussan” Fellowship programme. The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no 246016. 1

0360-1315/$ – see front matter Ó 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.compedu.2013.06.005

430

M.N. Giannakos / Computers & Education 68 (2013) 429–439

question of how these attitudes and learners acceptance is connected with the actual performance of the students. Based on these concerns, this study aims to shed light in this uninvestigated area. The purpose of the empirical experiment is twofold: (1) to verify the value of the game used in our experiment (at least as good as traditional instruction) in order to be able (2) to empirically evaluate whether and how certain attitudes affect learners’ performance. For that reason, we used an educational math game, called “Gem-Game” (Giannakos, Chorianopoulos, Jaccheri, & Chrisochoides, 2012), which targets children that attend first and second class of Gymnasium (middle school). Afterward, we constructed a 2-study experimental procedure, one for each purpose. The experimental procedure included surveys with constructs regarding students’ attitudes and performance regarding the educational game. We had a baseline period with students in order to increase their familiarity with similar games in the school. Then students played the game during the school period, after which they completed the respective survey and performance test. Based on the aforementioned issues, the first step of this work is to prove that Gem-Game is at least as effective as traditional instruction (concerning the acquisition of knowledge). Next, we focus on the main research question of this study: RQ. Are there attitudes regarding the game which lead students to perform better? The rest of paper is structured as follows: in the next section, the literature review and the research hypotheses of the research are outlined; the third section presents the game we used for our experiment; the fourth section presents the methodology of both studies employed in this article and the empirical results; and the fifth section discusses the results derived, while the last section suggests the implications and limitations of the experiment and makes recommendations for future research. 2. Background work and research hypotheses 2.1. Educational games learning performance The performance of game-based learning in the curricular education (and elsewhere) has already been studied in the past (Hwang & Wu, 2012; Papasteriou, 2009). According to the cognitive approach of learning (Ertmer & Newby, 1993), educational games could be beneficial for students (Paraskeva, Mysirlaki, & Papagianni, 2010). First, knowledge or skills learned and practiced through gaming are more likely to transfer than when practiced on a single kind of problem. Once mastered, the knowledge and skills are practiced further to provide overlearning (Paraskeva, Mysirlaki, & Papagianni, 2010); this leads to the knowledge and skills becoming automatized and consolidated in memory, so that the learner can begin to focus consciously on comprehending and applying new information (Gentile & Gentile, 2008). Second, the individual’s construction of representations of the world, and therefore educational games, attempts to build intrinsic motivation by integrating learning and game experience (Egenfeldt-Nielsen, 2006). Third, educational games seem to put the learner in the role of decision-maker, pushing players through ever harder challenges, and they accomplish learning through trial and error procedures (Mikalef, Giannakos, Chorianopoulos, & Jaccheri, 2012). Games also have a significant advantage in that pupils receive immediate feedback on their actions and decisions, inviting exploration and experimentation (Kirriemuir, 2002). The Cognitive Theory of Multimedia Learning (CTML) may provide an explanation for the efficacy of educational games as a means to enhance student learning (Mayer, 2001). A key assumption in CTML is derived from Paivio’s (1971) Dual Coding Theory (DCT). Research in DCT suggests that a learner’s cognitive system (memory) includes two separate channels for pictorial and verbal. Presenting information that accommodates both of these channels allows a learner to construct better understanding of the material by integrating information from both channels (Mayer, 2001). Consistent with the assumptions from the CTML (Mayer, 2001) and taking into account that multimedia are facilitating meaningful contexts (Clegg, Gardner, & Kolodner, 2010), researchers advocate (Papastergiou, 2009) that games can enhance student performance by presenting well-designed instructional messages that support cognitive development. Overall, educational games have great potential for helping students to improve their learning performance (Huang, Huang, & Tschopp, 2010). On the other hand, traditional instruction is an emotional, social, and cognitive experience in which teachers use their knowledge, voice, and movement to address the learners with questions and stories; humor makes the students attentive and creates a comfortable learning atmosphere (Nordkvelle, Fritze, & Haugsbakk, 2009). When an educational game can help learners to acquire the knowledge they are gaining through traditional instruction learning atmosphere; or, in other words, when learning performance through a game is the same as that of traditional instruction; then we can consider it a “successful” educational game. As such, in our case we hypothesize that: H1. Students’ knowledge acquisition through Gem-Game learning has equal or higher performance when compared to knowledge acquisition during traditional instruction. 2.2. Students attitudes toward educational games The conceptual model of this study is depicted in Fig. 1. The model is composed of four main categories including: Entertainment, Acceptance, Emotions and Performance. As mentioned earlier, this research aims to examine if students’ attitudes toward the game affect their performance. To this end, we select to work with three attitudinal categories (Entertainment, Acceptance, Emotions), which have been recognized in the literature as of high importance (Bourgonjon et al., 2010; Cowley, Charles, Black, & Hickey, 2008; Pekrun, Goetz, Titz, & Perry, 2002). However, these categories are represented by numerous and diverse variables. Given that this study is one of the few so far regarding the exploration of students’ attitudes on their performance, we decide to select one of the most important and representative variable from each category (Fig. 1). In the following sub-sections we analyze in depth the selection of each variable and we formulate the hypotheses. 2.2.1. Students’ enjoyment Despite that serious gaming is one of the main categories of entertainment computing, limited research exists concerning the effect of entertainment characteristics of educational games on performance. Our work is grounded in existing theoretical basis of entertainment (Cowley et al., 2008) and information systems (Venkatesh, Speier, & Morris, 2002), which propose the factor of enjoyment (or named instruct motivation) for understanding the entertainment nature of a system.


431

Fig. 1. The conceptual model of the study.

Player enjoyment is the most important goal in gaming. Enjoyment is deemed as a most appropriate measure of motivation because enjoyment measures how the game helps achieve the task-related objectives. Moreover, contributions to entertainment theory (Vorderer, Klimmt, & Ritterfeld, 2004) suggest that media users actively ‘work’ on their enjoyment experience. As the learner’s evaluation establishes the link between performance and enjoyment, it is possible that players use the inherent complexity of the game to preserve a maximum enjoyment. Enjoyment is defined as the degree to which the activity of using technology is perceived to be enjoyable in its own right apart from any performance consequences that may be anticipated (Davis, Bagozzi, & Warshaw, 1992). Research has found that enjoyment plays an important role in user technology acceptance and that the correlation between enjoyment and ease of use is supported by research findings (Venkatesh et al., 2002; Yi & Hwang, 2003). Additionally, enjoyment was found to be significant in explaining intention to use applications (e.g., Hsu & Lin, 2008). Based on prior studies (e.g., Venkatesh & Speier, 2000), when comparing traditional instruction and game-based instruction, the latter results in higher enjoyment on students. In addition, Venkatesh (2000) found that as people used the system more, the effect of enjoyment on ease of use became stronger. These findings suggest that ease of use is influenced by the extent to which users perceive that using the system is enjoyable. On top of that, Davis et al. (1992) indicated that usefulness and enjoyment are significant determinants of behavioral intention and Venkatesh (2000) showed that enjoyment influenced usefulness via ease of use. Usefulness measures how people believe their performance can be improved by using the specific system. Teo, Lim, and Lai (1999) found that enjoyment has a significant effect on frequency and type of use. Thus, enjoyment may act to exert a significant influence on a students’ learning performance. When a player fails to pass one game task, his frustration increases as his self-esteem decreases and vice versa (Seery, Blascovich, Weisbuch, & Vick, 2004). Therefore, successful task resolution is theorized to level out self-esteem, and the increase of self-esteem is a highly enjoyable experience. Successful task resolution in an educational game means playing at a higher level of the game with more difficult competencies, resulting in better performance for the learner. In addition, the sense of enjoyment while the students learn through a game reduces anxiety and helps students feel confident about their success. We can then assume that higher levels of enjoyment facilitate higher levels of success (better performance). Hence we hypothesize that: H2. Students’ Enjoyment with the educational game has a positive relation with their Performance. 2.2.2. Students intention to use In recent decades, research has defined factors that affect acceptance of learning systems (Bourgonjon et al., 2010). Most of the studies have focused on which factors increased learners’ intention to use a learning system; however, the question of how the intention to use a learning system is connected with the learners’ performance has received minimal research attention. Technology Acceptance Model (TAM) has been a dominant tool for studying system’s acceptance (Davis, 1993). The main purpose of the development of the TAM was to offer a foundation for examination of the effects of external variables on internal beliefs, attitudes, and intentions. Even though TAM provides external variables that affect the use of a system, these variables may not be the best fit for every system. Even in learning systems, the variables that might influence the technology acceptance vary according to learners’ and systems’ needs and capabilities. TAM related theories have already been used in many studies to understand and predict game acceptance. Hsu and Lu (2004) were able to explain about 80% of the variance with an extended TAM model and ease of use appeared to be the key determinant to predict online game play instead of usefulness. In Ha, Yoon, and Choi (2007), it was found that enjoyment was a better predictor than usefulness in the case of mobile video game acceptance. The research of Ha et al. 2007, studied the acceptance of games with a TAM-based model that was enriched with variables as visibility, self-expression and innovativeness as determinants. All of these studies and most of the other TAMbased studies introduce new variables to better predict the game acceptance. In the current study, the focus is clearly different since game acceptance is examined as independent variable and not as dependent (which is the case in most of the studies). To this end, we select intention to use as it is considered the most commonly used variable and has also been the ultimate variable on most of the game acceptance studies (e.g., Ha et al., 2007; Hsu & Lu, 2004). Watching the use of educational games through the lens of TAM related theories (Davis, 1993; Venkatesh & Bala, 2008); we can clearly see that the practical use of a computer game system is determined by the users’ intentions to use the game (to play), which is determined by the users’ attitude toward playing and their intention to play that system. However, the intention to use the game still notably determines the performance. As a result, the following hypothesis regarding learners’ intention to use and their performance in the context of educational games is proposed: H3. Students’ Intention to Use the educational game has a positive relation to their Performance.

432


2.2.3. Students happiness Learning is an emotional and cognitive experience (Frijda, 1986). The emotion of learners can greatly contribute to the learning environment (Kay, 2008). Emotions negatively or positively affect students’ motivations (Pekrun et al., 2002), which, in turn, lead them to concentrate cognitive resources and, in some cases, pursue higher learning. Learners express different emotions, which can influence their attitudes toward a task (Pekrun et al., 2002), as well as their performance of the task. If educational technologies (particularly educational games) can yield positive emotional responses, then their use potentially leads to positive learning experiences and, by extension, improvement in learning outcomes (Antonio, Martin, & Stagg, 2012). A positive correlation between positive emotions (happiness) and the acquisition of computer knowledge has been found (Hay, 2008). Until now there is limited research on the effect of different emotional aspects that arise while a learner is using a learning game, and if these aspects affect performance. The focus of the existing research is to explore this issue. As a first step in our study, we examined the most well-used positive emotion named happiness, as the extent to which a person feels happy and has a warm feeling. Hence, the following hypothesis among learners’ happiness and their performance in the context of games is proposed: H4. Students’ Happiness with the educational game has a positive relation with their Performance. 3. The math game For the purpose of our study we used a math game named Gem-Game (Giannakos et al., 2012). The main purpose of Gem-Game is to improve the mathematical skills of players. The main character (Peter) moves up or down dependent on the operation executed by the player. So the students also get a visual idea of increasing quantity when adding and decreasing quantity when subtracting. The focus of the study is not to validate any sophisticated game technology. This is why Gem-Game was simple and based on the traditional game mechanics of Pong and Space Invaders, like reward, engage and rethink. Most notably, the game features a plot in which the player is assigned a mission. The story and the mission are used to stimulate the students’ interest and motivate them to play the game. The dialogue and the plot are funny, as to avoid resembling rigid book or common computer-based exercises. Our game’s design (Figs. 2–4) follows Vogler’s storytelling structure (1998), with each storytelling game consisting of some common stages. In the first stage (Fig. 2), the hero is situated in the ordinary world; in our game, the hero named Peter is in his bed-room and looking for his dog. Then the hero is presented with a problem or event that necessitates leaving the comfort of the ordinary world; Peter’s dog, Lucky has been kidnapped. Next, the hero meets a mentor or someone who may offer advice or guidance; the fairy guides Peter to collect 30 diamonds. Once the hero commits to the adventure, he begins the problem-solving process. During this process, the hero encounters various challenges that he must overcome in order to progress. In this stage Peter has to play and win the game in order to collect the necessary diamonds (Fig. 3). The ultimate goal of the player is to retrieve his dog by collecting diamonds. To achieve the ultimate goal, Peter must win the 3 stages (Fig. 3). Specifically, Peter must correctly add/subtract in order to earn diamonds. For example, if Peter is positioned on line 6,

Fig. 2. The first phase of Gem-Game storytelling structure.


433

Fig. 3. Video game stages air (left), sea bed (center) and helicopter (right).

and the diamond is on line 1, the player must write 5 in order to gain the diamond. The player completes each stage by collecting 10 diamonds. The first stage has only positive integers, the second stage has only negative integers, and the last stage has both positive and negative integers. Moreover, at each stage Peter wears a different uniform: a flyer uniform in the first stage; a diver uniform in second stage; and a helicopter uniform in the third stage. By completing each one of the first two stages, the player obtains a code in order to be able to continue from that point. When Peter passes all the stages and collects the diamonds, the fairy appears and calls the witch, who then takes the diamonds and releases the dog, thereby presenting the moral aspect of the game (Fig. 4). 4. Study 1: differences in learning performance among game based learning and traditional instruction In the first study, we will investigate the first research Hypothesis (H1) of that article; a positive (Gem-Game has better performance from traditional instruction) or even neutral (Gem-Game has the same Performance with traditional instruction) result on the first hypothesis will lead us to the second study and the other 3 hypotheses (H2,3,4) in order to specify which factors affect the performance of the students. 4.1. Research design In our effort to investigate the performance of Gem-Game, we conducted a study that compares two different learning types; the traditional instruction group corresponding to the control group and the game-based learning (using Gem-Game) group corresponding to the experimental group. The research design consists of four steps and will measure students’ performance in the two groups of students (Fig. 5). The two groups included identical learning objectives and content, and both groups were located in the same place (State Middle School of Greece). The learning objectives of the game were adding and subtracting integers. However, the difficulty increased throughout

Fig. 4. The second phase of Gem-Game storytelling structure.

434


Fig. 5. Graphical representation of the research design of the 1st study.

the game due to the speed increase of the Gems, as such students had to give faster their response with few chances for having “trial and error”. In addition to that, as the game progressed the student has to deal with adding and subtracting with both negative and positive numbers. In general, students have experience with adding and subtracting from prior years, however when speed increases and negative numbers appear to the game, it is clear that students of this grade still have certain problems on these objectives. Any differences in student learning outcomes and appeal to students should be attributed to the group type factor. 4.2. Sampling The first part of the experiment began with the selection of the two groups to participate in the experiment. The school has two classes with a total of 41 students. A pretest procedure was conducted. The pretest was developed with the assistance of the teachers and was the same in terms of size and question type with previous ordinary tests. The proposal of the teachers and the results obtained from the pretest procedure allowed us to form the two different groups (the game-experimental group and the traditional instruction-control group). Thus, the two groups consisted of 20 and 21 students respectively, with high similarity in their performance rate (first group: M1 ¼ 14.28, SD1 ¼ 3.86; second group: M2 ¼ 14.44, SD2 ¼ 4.10). The game group included 12 males and 8 females, while the traditional instruction group included 13 males and 8 females. The experiment was conducted in the first two weeks of December 2011. Student participants were 13 years old and attended the second grade of Gymnasium, learning the same course syllabus. They participated in educational games in the last 4 weeks before the experiment, in order to minimize the effect of students’ enthusiasm. During these 4 weeks students participated on games having exactly the same technology with the Gem-Game (flash), using exactly the same game-mechanics (Pong-based) but without having the aforementioned mathematic learning objectives. In this way we minimized the effect of students’ enthusiasm about the game and we did not provide them with the bias of the learning experience. 4.3. Measures and data analysis As aforementioned, 41 Gymnasium students were involved in the experiment. They were separated into two student groups with high similarity in their performance. One group was engaged in the Gem-Game-based learning, and the other group was engaged in traditional instruction. Afterward, a performance test was used to measure students’ performance. The test, developed with the assistance of the teachers, was the same in terms of length and question type as their regular tests. An independent samples t-test was conducted in order to compare the improvement in the performance for the respective groups. Especially in a small sample size of a study, it is necessary to validate the homogeneity of variance and the normality of the sample distribution (Nam & Smith-Jackson, 2007) before performing the ttest. Hence, we applied Levene’s test to evaluate the homogeneity of variance and the Shapiro–Wilk test to evaluate the normality criterion (Conover, 1998; Shapiro & Wilk, 1965). Both results revealed non-significant outcome (p > 0.05), suggesting that the samples had homogenous variances and normal distribution of data. This supports the usage of t-tests. The t-test method was appropriate as it applies to the problem of estimating means of a normally distributed population. 4.4. Research findings Using a t-test of the two groups, Game-Based Learning (M1 ¼ 14.99, SD1 ¼ 3.78) and Traditional Instruction (M2 ¼ 13.86, SD2 ¼ 5.66), the results showed no significant difference t(39) ¼ 0.75, p > 0.05. As a consequence, there was no difference between game-based learning and traditional instruction in their overall performance of the test, even though the latter is an interactive way of learning. This result is consistent with other studies (Papastergiou, 2009; Virvou et al., 2005) regarding game-based learning and provides us with a vehicle to the next study to identify which factors of the game play influences students’ performance. 5. Study 2: the effect of students attitudes on their learning performance As we have now proven the learning value of the game, which yields at least the same performance as traditional instruction. We will investigate the potential effect of factors on students’ performance. Such an investigation will lead us to the answer of the research Hypotheses (H2, H3, H4). 5.1. Research design In our effort to investigate the effect of 1) Enjoyment, 2) Intention to Use the Educational Game and 3) Happiness on students’ Performance, we conducted a study to measure these factors and test the potential effects. The sample of the study consisted of students who played the Gem-Game. The research design consists of four steps and will employ four factors (measures) as exhibited in the following figure (Fig. 6).


435

Fig. 6. Graphical representation of the research design of the 2nd study.

5.2. Sampling The sample of participants in this study consisted of 46 students. Of the 46 participants, 29 were boys and 17 girls. All of the students who participated in the experiment were around 13 years old, enrolled in the second grade of Gymnasium (middle school), and learning from the same syllabus on mathematics and informatics. The experiment took place in a Greek state Gymnasium. They played the game during a didactical hour, after which they completed the respective survey. The study was conducted over a weekly period from November 14–18, 2011. The game play was conducted on Windows desktops with a 17-inch screen using headsets. 5.3. Measures After the game play, students completed a paper-based survey. The surveys gathered feedback of students’ enrollment with the math game. The survey concerned factors adopted from prior studies: Enjoyment (ENJ), Intention to Use the Educational Game (IUEG), Happiness (HAP) and students’ Performance (PER). In Table 1, we summarize the operational definitions of these factors, the items, and their respective bibliographical sources. Students were asked to rate their experience with Gem-Game regarding their Enjoyment, Intention to Use it, Happiness, and Performance. In all measures excluding Performance, a 5-point Likert scale was applied (1 ¼ strongly disagree, 2 ¼ disagree, 3 ¼ neutral, 4 ¼ agree and 5 ¼ strongly agree). Table 1 clearly exhibits the questions made to the students. For Happiness, we used a single scale (“Please show us how you feel about using Gem-Game to study with?”) from very happy (5) to very unhappy (1) with Visual Analogue Scales (VAS) for students’ choices (see Fig. 7). VAS is a widely used question format; VAS uses pictorial representations that children use to identify their feelings or opinions. This approach has been adopted as an alternative to the traditional open-ended and closed question formats, researchers suggest that VAS can successfully being used with students’ aged 13 years old (Read & MacFarnale, 2006). Using only one item to measure the factor of happiness has certain limitations, however the use of VAS provides other benefits (e.g., the response reliability increased), which many times at this age are very important (Read & MacFarnale, 2006). For the case of Performance, we used a Performance test, which was developed with the assistance of the teachers and was the same in terms of length and question type as their regular tests. As we mentioned on Section 2.2, this study is one of the few so far regarding the exploration of students’ attitudes and their effect on learning performance. The selection of these three attitudinal factors and the respective measures were based on literature and prior studies. To this end, the focus of this study is to shed light in the interplay of students’ attitudes and their learning performance in the context of educational games. Follow up studies with larger sample and selection of more factors, covering better the area of these 3 major categories (Fig. 1), will definitely provide better understanding of the issue. 5.4. Data analysis As aforementioned, 46 Gymnasium students were involved in the evaluation. To test each hypotheses (H2, H3, H4) the data were separated into two groups by performing median split on Enjoyment, Intention to Use the Educational Game, and Happiness. Afterward, an

Table 1 The research factors and their respective items. Factors

Operational definition

Items

Source

Enjoyment (ENJ)

The degree to which the activity of using the educational game is perceived to be personally enjoyable.

Please indicate how much you agree or disagree with the following statements based on your experience with the Gem-Game: Studying is more interesting using Gem-Game (ENJ1). Using Gem-Game is fun (ENJ2). I like using Gem-Game (ENJ3). I enjoy those aspects of my studying that require me to use Gem-Game (ENJ4)

Venkatesh et al., 2002

Intention to Use the Educational Game (IUEG)

The degree of students’ willingness to play the educational game.

Please indicate how much you agree or disagree with the following statements regarding your intention to use Gem-Game: I plan to use Gem-Game for studying in the future (IUEG1). I intent to continue using Gem-Game for studying in the future (IUEG2). I expect my use of Gem-Game to continue in the future (IUEG3).

Lee, Yoon, & Lee, 2009

Happiness (HAP)

The degree to which a person feels about using storytelling game, regarding their happiness.

Please show us how you feel about using gem-game to study with? (See Fig. 7)

Read & MacFarnale, 2006

Performance (PER)

The level of knowledge acquisition students obtained using the game.

The Performance test.

436


Fig. 7. Emoticons used in the survey to measure students’ happiness using Gem-Game.

independent samples t-test was conducted in order to examine the effect of Enjoyment, Intention to Use the Educational Game, and Happiness on students’ Performance. The t-test method was appropriate as the samples had homogenous variances and normal distribution of data. Except for the data provided by the survey and the performance test, this study gathered information from a light conversation/ interview with students, as well as the observations of researchers/teachers. These data provide a means to interpret and validate the results. 5.5. Research findings Fornell and Larcker (1981) proposed three procedures to assess the convergent validity of any measure in a study: (1) Composite reliability of each construct, (2) Item reliability of the measure, (3) The Average Variance Extracted (AVE). First, we carried out an analysis of composite reliability and dimensionality to check the validity of the scale used in the questionnaire. Regarding the reliability of the scales, Cronbach’s a indicators was applied (Cronbach, 1951), in addition to inter-item correlations statistics for the items of the variable. As Table 2 demonstrates, the result of the test revealed acceptable indices of internal consistency in all the factors. In the next stage, we proceeded to evaluate the reliability of the measure. The reliability of an item was assessed by measuring its factor loading onto the underlying construct. Hair, Jr., Black, Babin, Anderson, and Tatham (2006) recommended a factor loading of 0.7 to be good indicator of validity at the item level. The factor analysis identified two distinct factors: 1) Enjoyment (ENJ); and 2) Intention to Use Educational Games (IUEG). Additionally, we used one single (visual) item factor: 3) Happiness (HAP); and another factor derived from the students’ knowledge acquisition, 4) Performance (PER) (Table 2). The third step for assessing the convergent validity is the average variance extracted (AVE). AVE measures the overall amount of variance that is attributed to the construct in relation to the amount of variance attributable to measurement error. Convergent validity is found to be adequate when the average variance extracted is equal or exceeds 0.50 (Segars, 1997). To examine the research hypotheses regarding the effect of ENJ, IUEG and HAP on students’ PER, we divided ENJ, IUEG, and HAP on high and low categories performing median split, then performing a t-test including students’ PER as a dependent variable and the other three factors (ENJ, IUEG, HAP) as independent variables. All statistical analyses reported were conducted with a significant level of 0.05. As we can see from the outcome data in Table 3, ENJ has indicated an impact on students’ PER while IUEG and HAP have not. Although the three independent variables exhibit the intentions and beliefs of the learner, they affect the Performance of the learner in different way. In order to identify the correlations among these three factors, we used Pearson’s correlation coefficient, which is about quantifying the strength of the relationship between the variables. Pearson’s test verified the relatively strong relation among the three factors as indicated in Table 4. Observing Table 3, we notice that only ENJ has the significant effect on learner PER, and from Fig. 8, we can observe that ENJ is the only one with a positive effect. In addition, IUEG and HAP have negative but insignificant effect. Although insignificant, we must acknowledge the negative effect of HAP; Fig. 7 exhibits that is much higher than IUEG. Overall, Fig. 8 clearly exhibits the positive and significant (based on the t-test) influence of ENJ on learners’ IUEG. To help the interpretation of the results we also used qualitative data. Although a structure interview could not be performed with the students due to legal constraints, they had a light interview/conversation with the researcher. In addition to that, observations were collected by the researcher and the teachers; after the treatment we had a detailed discussion with the teachers.

Table 2 Summary of measurement scales. Constructs

Items

Mean

S.D.

CR

Loadings

AVE

Enjoyment

ENJ1 ENJ2 ENJ3 ENJ4

4.05 3.55 3.40 3.90

2.39 2.28 2.54 2.20

0.91

0.89 0.71 0.91 0.84

0.71

Intention to Use the Educational Game

IUEG1 IUEG2 IUEG3

3.30 3.00 2.70

2.34 2.29 2.00

0.97

0.93 0.94 0.93

0.87

Happiness

HAP

3.4

1.27

–

–

–

Performance

PER

13.86

4.76

–

–

–


437

Table 3 Hypothesis testing using t-test. Dependent variable

Mean (S.D.)

T

Low Performance

High

Enjoyment (ENJ) 12.20 (6.26) 15.51 (5.12) Intention to Use the Educational Game (IUEG) 14.12 (5.12) 13.59 (5.68) Happiness (HAP) 14.62 (5.39) 13.09 (6.10)

2.35* 0.29 0.37

*p < 0.05.

Table 4 Pearson’s correlation coefficient between factors. Factors

ENJ

IUEG

HAP

ENJ IUEG HAP

1 0.587* 0.585*

1 0.705*

1

*Correlation is significant at the 0.01 level.

In the discussion with the students, the researcher guides the conversation in order to probe different aspects of students’ enjoyment, motivation and learning performance throughout the treatment. In the conversation, students supported that the continuation of that practice with similar games will be very positive. They also asked for the possibility to include similar activities in other courses. According to the informal data gathered from researcher’s observations, students seemed enthusiastic with the game-play during all the period of the experiment. It is quite interesting that their excitement did not reduce when they used the educational game (instead of the non-educational ones). It seems that as the game mechanics and the technology were similar they did not get frustrated from the educational content. During the intervention, they were relatively quiet, only some exclamations of satisfaction from students who had managed to get on the second and third level. Those exclamations came mostly from boys, whereas girls seemed more quiet. Sometimes, students walked around the lab to find out who else finished that stage and helped their friends who faced “difficulties”. Students, as observed, enjoyed viewing the number of collected diamonds and compared it to each other. They were engaged in challenging who will go first on the next level. In addition, it was observed that they skipped the storytelling in order to get faster on the next stage. Teachers who participated in the experiment discussed with the researcher for the educational benefits of the game and the attitude of their students while using it. Teachers were impressed by the effect of the game on students who otherwise would pay no attention in the class. In general, teachers thought that educational game use was highly beneficial for students that were considered undisciplined. In fact, teachers mentioned that students were as much immersed as with the prior non educational games. In general, teachers believe that the usage of similar games among the “boring” traditional instruction may valuably contribute to students’ attitude during a school day. Teachers’ opinion was in alignment with the results of the quantitative data. When we asked teachers about the factors used in our study and hypotheses, they said that based on their experience and observations, they believe that the most influential factor on students’ performance must be the enjoyment. They think that this is strongly connected with students’ engagement and persistence with the game and as a result with the knowledge they will acquire throughout the game. On the other hand, they believed that happiness and intention to use might be of less importance, since, on any case, students will be happy with the game and will want to participate again on similar activities. However, enjoyment clearly increased when students succeeded in the game. We believe that additional qualitative data (e.g., structured interviews with the students) is necessary to explore the relationship between the attitudes of the students and the knowledge acquisition. With more in depth qualitative-based studies we will be able to identify the exact features of the game, which help students increase their enjoyment, engagement and learning performance.

18

Performance

17 16 15 14 13 12 11 10 Low

Enjoyment

High

IUEG

Happiness

Fig. 8. The influence of students’ attitude in their actual performance with the educational game.

438


6. Discussion and conclusions This study evaluated the performance of a game targeted at the learning of mathematical competences within the Greek middle school math curriculum. In line with previous studies (Papastergiou, 2009; Virvou et al., 2005), this study indicates that Gem-Game successfully utilized in formal education to support students in Mathematics. The main objective of the presented study is to explore the relationship between students’ attitudes and learning performance regarding educational games. In particular, three hypotheses (H2–H4) were presented, which help in understanding how attitudinal factors contribute to game performance. The findings indicate that ENJ has a positive effect on learners’ performance. As such, our study results suggest that enjoyment could play a very influential role in determining the knowledge acquired by the learner. On the other hand, learners’ HAP and IUEG indicate an insignificant effect on performance. As such, the hypotheses H3 and H4 are rejected. This is quite curious, but can be connected to the fact that some high performance students’ have difficulties engaging in game playing (Cheng, Deng, Chang, & Chan, 2007), which results in low HAP and IUEG, while maintaining high performance. Such a case makes the connection between ENJ and PER even stronger as it is kept unaffected. Our study is also in line with many studies (Sumak, Hericko, & Pusnik, 2011) that also deal with the effect of learners’ attitudes on their intention to use educational games. Our study clearly points out (Table 4) the important roles of ENJ and HAP in influencing the decision of learners to use an educational, as well as subsequent actual use. The study has also produced a number of interesting qualitative findings that illustrate the impact of the enjoyment experienced through games on students’ behavior. At the end of the study, students suggested that they should play on their own laptop in the classroom, and they could even use it at home in order to be trained in this way. Another crucial aspect was the students’ sense during the experiment. Based on the researcher observation in the lab, it can be presumed that most of the students felt “excited”. Researchers’ opinions regarding the “excitement” of the students arise from the comments of the students, such as “awesome,” during and after the experiment. Teachers present were asked to observe students playing the game, and they responded that students who played the game seemed so immersed in playing that their behavior changed, as they appeared to be very interested in the game. In general, the results counter positive emotions and beliefs for the use of educational game, though we identify that all the positive emotions do not impact the knowledge acquired. The important role of enjoyment on the educational game is also agreed by the teachers. The research findings revealed that learners reflecting high enjoyment are more likely to acquire knowledge through the game. As such, instructors, higher education institutions, and schools should focus on the “entertainment” nature of educational games. To date, the majority of educational games is not focused on “entertainment” attributes. Collectively, the findings from the present study suggest that educators should provide a learning environment where enjoyment is supported and fostered in order to facilitate successful learning performance with the game. Overlooking students’ enjoyment could have detrimental effects on the learning performance. Although our study provides evidence for the learners’ performance and attitudes regarding educational games, there are also some limitations. First, the generalizability of the results must be carefully approached since the study was conducted in a specific context (e.g., educational level, course). Second, a self-report method (surveys) was used to measure the attitudinal research variables, so some of the results might have a common method biases. However, the main interest of our study is to explore the relationship among students’ attitudes and performance. This reduces the common method bias as the independent and dependent variables were measured with different methods. Nevertheless, other methods, such as in-depth interviews and observations, could provide a complimentary picture of the findings through data triangulation. Third, it could also be argued that our study was influenced by some kind of Hawthorne effect; this effect increases students’ performance produced by the psychological stimulus of being singled out, made to feel important and part of something new (Gillespie, 1991); or by a form of teacher expectancy effect, that is an increase which is generated simply because students are expected to do better (Rosenthal & Jacobson, 1968). However, we tried to reduce this effect with the 4 week introduction of the students with similar games. Finally, other demographic variables (i.e., age, educational level) may have a contingent effect on educational gaming. This study provides valuable insights and opens new avenues for future work. Future research that may include a larger sample, different educational levels, a wide variety of measures (i.e., observations, interviews) will shed valuable light on attitudes–performance relations. In addition, in-depth analyses (i.e., behavioral sequential) should be conducted to understand how gamers’ attitudes relate to their performance. For such follow-up research it would also be desirable to use diverse measures of emotion such as face expressions, EEG, pulse mouse and other biometric measures and interpret the results with the results of our study. Acknowledgments The author would like to thank all of the students and the school’s staff for their participation in the experiment. I would also like to thank T. Stamatoukou for developing the game, T. Plerou for assisting me on the data collection and I. Pappas for proofreading the article. I would also like to thank my colleagues K. Chorianopoulos and L. Jaccheri who helped in earlier phases of this project. References Antonio, A., Martin, N., & Stagg, A. (2012). Engaging higher education students via digital curation. In Proceedings of Ascilite’12. Retrieved from http://www.ascilite2012.org/ images/custom/antonio,_amy_-_engaging_higher_education.pdf. last visit 25/04/13. Bourgonjon, J., Valcke, M., Soetaert, R., & Schellens, T. (2010). Students’ perceptions about the use of video games in education. Computers & Education, 54(4), 1145–1156. Cheng, H. N. H., Deng, Y. C., Chang, S. B., & Chan, T. W. (2007). EduBingo: design of multi-level challenges of a digital classroom game. In T. W. Chan, A. Paiva, & D. W. Shaffer (Eds.), The first IEEE international workshop on digital game and intelligent toy enhanced learning (pp. 11–18). Los Alamitos, CA: IEEE Computer Society. Clegg, L. T., Gardner, C. M., & Kolodner, J. L. (2010). Playing with food: moving from interests and goals into scientifically meaningful experiences. In 9th Int. conf. of the learning sciences (pp. 1135–1142). Cohen Group Report. (2011). Young children, Apps & iPad. Retrieved from http://mcgrc.com/wp-content/uploads/2012/06/ipad-study-cover-page-report-mcg-info_newonline.pdf. last visit 25/04/13. Conover, W. J. (1998). Practical nonparametric statistics (3rd ed.). New Jersey: Wiley. Cowley, B., Charles, D., Black, M., & Hickey, R. (2008). Toward an understanding of flow in video games. Computers in Entertainment, 6, 2. Article 20. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.


439

Davis, F. D. (1993). User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. International Journal of Man–Machine Studies, 38, 475–487. Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1992). Extrinsic and intrinsic motivation to use computers in the workplace. Journal of Applied Social Psychology, 22, 1111–1132. Egenfeldt-Nielsen, S. (2006). Overview of research on the educational use of games. In Digital kompetanse: . http://www.itu.dk/wsen/papers/game-overview.pdf Accessed 20.10.12. Ertmer, P. A., & Newby, T. J. (1993). Behaviorism, cognitivism, constructivism: comparing critical features from an instructional design perspective. Performance Improvement Quarterly, 6(4), 50–72. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 48, 39–50. Frijda, N. H. (1986). The emotions. Cambridge, NY: Cambridge University Press. Garris, R., Ahlers, R., & Driskell, J. (2002). Games, motivation and learning: a research and practice model. Simulation and Gaming, 33(4), 441–467. Gee, J. P. (2003). What video games have to teach us about learning and literacy. New York, NY: Palgrave Macmillan. Gentile, D. A., & Gentile, J. R. (2008). Violent video games as exemplary teachers: a conceptual analysis. Journal of Youth and Adolescence, 37, 127–141. Giannakos, M. N., Chorianopoulos, K., Jaccheri, L., & Chrisochoides, N. (2012). This game is girly! perceived enjoyment and student acceptance of edutainment. In . LNCS 7516: Proc of the 7th international conference on e-learning and games (pp. 89–98). Springer. Gillespie, R. (1991). Manufacturing knowledge: A history of the Hawthorne experiments. Cambridge, UK: Cambridge University Press. Ha, I., Yoon, Y., & Choi, M. (2007). Determinants of adoption of mobile games under mobile broadband wireless access environment. Information & Management, 44(3), 276–286. Hair, J. F., Jr., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate data analysis (6th ed.). Upper saddle River, NJ: Prentice-Hall International. Hay, R. (2008). Exploring the relationship between emotions and the acquisition of computer knowledge. Computers and Education, 50(4), 1269–1283. Hsu, H. L., & Lin, J. C. C. (2008). Acceptance of blog usage: the roles of technology acceptance, social influence and knowledge sharing motivation. Information & Management, 45, 65–74. Hsu, C. L., & Lu, H. P. (2004). Why do people play on-line games? An extended TAM with social influences and flow experience. Information & Management, 41(7), 853–868. Huang, W. H., Huang, W. Y., & Tschopp, J. (2010). Sustaining iterative game playing processes in DGBL: the relationship between motivational processing and outcome processing. Computers & Education, 55(2), 789–797. Hwang, G.-J., & Wu, P.-H. (2012). Advancements and trends in digital game-based learning research: a review of publications in selected journals from 2001 to 2010. British Journal of Educational Technology, 43, E6–E10. http://dx.doi.org/10.1111/j.1467-8535.2011.01242.x. Johnson, S. (2005). Everything bad is good for you: How today’s popular culture is actually making us smarter. London: Allen Lane. Kay, R. H. (2008). Exploring the relationship between emotions and the acquisition of computer knowledge. Computers & Education, 50, 1269–1283. Ketelhut, D. J., & Schifter, C. C. (2011). Teachers and game-based learning: improving understanding of how to increase efficacy of adoption. Computers & Education, 56, 539–546. Kirriemuir, J. (2002). The relevance of games and gaming consoles to the higher and further education learning experience. Techwatch Report TSW 02.01 http://tecnologiaedu.us. es/nweb/htm/pdf/301.pdf Accessed 20.09.12. Lee, B.-C., Yoon, J.-O., & Lee, I. (2009). Learners’ acceptance of e-learning in South Korea: theories and results. Computers & Education, 53(4), 1–44. McFarlane, A., Sparrowhawk, A., & Heald, Y. (2002). Report on the educational use of computer games. In Teachers evaluating educational multimedia report: . Retrieved from http://www.teem.org.uk/resources/teem_gamesined_full.pdf. last visit 25/04/13. Mayer, R. E. (2001). Multi-media learning. Cambridge: Cambridge University Press. Mikalef, K., Giannakos, M. N., Chorianopoulos, K., & Jaccheri, L. (2012). “Do not touch the paintings!” The benefits of interactivity on learning and future visits in a museum. In Entertainment computing – ICEC 2012, Vol. 7522/2012 (pp. 553–561). http://dx.doi.org/10.1007/978-3-642-33542-6_73. Nam, C. S., & Smith-Jackson, T. L. (2007). Web-based learning environment: a theory-based design process for development and evaluation. Journal of Information Technology Education, 6, 23–43. Nordkvelle, Y. T., Fritze, Y., & Haugsbakk, G. (2009). The visual in teachingdfrom Bologna to YouTubiversity. Medien–Wissen–Bildung: University of Innsbruck. In: http://media. brainity.com/uibk2/amab2010/images/down/MWB09_Proceedings.pdf#page¼64 Accessed 21.09.12. Paivio, A. (1971). Imagery and verbal processes. New York: Holt, Rinehart & Winston. Papastergiou, M. (2009). Digital game-based learning in high-school computer science education: impact on educational effectiveness and student motivation. Computers & Education, 52(1), 1–12. Paraskeva, F., Mysirlaki, S., & Papagianni, A. (2010). Multiplayer online games as educational tools: facing new challenges in learning. Computers & Education, 54(2), 498–505. Pekrun, R., Goetz, T., Titz, W., & Perry, R. (2002). Academic emotions in students’ self-regulated learning and achievement: a program of qualitative and quantitative research. Educational Psychologist, 37, 91–105. Read, J., & MacFarlane, S. (2006). Using the fun toolkit and other survey methods to gather opinions in child computer interaction. In Proceedings of the conference on interaction design and children (pp. 81–88). NY, USA: ACM. Rosenthal, R., & Jacobson, L. (1968). Pygmalion in the classroom: Teacher expectations and pupils’ intellectual development. New York, NY, USA: Holt: Rinehart and Winston. Seery, M. D., Blascovich, J., Weisbuch, M., & Vick, S. B. (2004). The relationship between self-esteem level, self-esteem stability, and cardiovascular reactions to performance feedback. Journal of Personality and Social Psychology, 87, 133–145. Segars, A. H. (1997). Assessing the unidimensionality of measurement: a paradigm and illustration within the context of information systems research. Omega International Journal of Management Science, 25(1), 107–121. http://dx.doi.org/10.1016/S0305-0483(96)00051-5. Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality. Biometrika, 52(3), 591–611. Shin, N., Sutherland, L. M., Norris, C. A., & Soloway, E. (2012). Effects of game technology on elementary student learning in mathematics. British Journal of Educational Technology, 43, 540–560. http://dx.doi.org/10.1111/j.1467-8535.2011.01197.x. Sumak, B., Hericko, M., & Pusnik, M. (2011). A meta-analysis of e-learning technology acceptance: the role of user types and e-learning technology types. Computers in Human Behavior, 27(6), 2067–2077. Teo, T. S. H., Lim, V. K. G., & Lai, R. Y. C. (1999). Intrinsic and extrinsic motivation in internet usage. Omega, 27, 25–37. Venkatesh, V. (2000). Determinants of perceived ease of use: integrating control, intrinsic motivation, and emotion into the technology acceptance model. Information Systems Research, 11, 342–365. Venkatesh, V., & Bala, H. (2008). Technology acceptance model 3 and a research agenda on interventions. Decision Sciences, 39(2), 273–315. Venkatesh, V., Speier, C., & Morris, M. G. (2002). User acceptance enablers in individual decision making about technology: toward an integrated model. Decision Sciences, 33, 297–316. Virvou, M., Katsionis, G., & Manos, K. (2005). Combining software games with education: evaluation of its educational effectiveness. Educational Technology and Society, 8(2), 54–65. Vogler, C. (1998). The writer’s journey: Mythic structures for writers. Studio City, CA: Michael Wiese Productions. Vorderer, P., Klimmt, C., & Ritterfeld, U. (2004). Enjoyment: at the heart of media entertainment. Communication Theory, 14(4), 388–408. Yi, M. Y., & Hwang, Y. (2003). Predicting the use of web-based information systems: self-efficacy, enjoyment, learning goal orientation, and the technology acceptance model. International Journal of Human-Computer Studies, 59(4), 431–449. Yusoff, A., Crowder, R., & Gilbert, L. (2010). Validation of serious games attributes using the technology acceptance model. In Proceedings of 2010 second international conference on games and virtual worlds for serious applications (pp. 45–51). Braga, Portugal.