The effects of Lecture attendance and Lectopia

Student Engagement

471

More than one pathway to success: The effects of Lecture attendance and Lectopia viewing on exam performance in large Engineering classes. Dr J.E. McCredden and Dr T. Baldock University of Queensland, Brisbane, Australia

ABSTRACT This study aimed to create an understanding of how Engineering students use online lecture recordings (Lectopia) as a part of their learning toolkit, so as to better assist staff concerned about the possible negative consequences of introducing Lectopia into their course. Online surveys were given to third year civil and chemical Engineering students, twice throughout the semester, recording both quantitative and qualitative data regarding the frequency of lecture attendance, the reasons for non-attendance, the amount of Lectopia viewed, and the usefulness of Lectopia. The answers to these questions were related to exam marks for each student. The results showed that the effects of viewing Lectopia and Lecture attendance interact; i.e. that neither variable alone can explain student marks. Furthermore, in terms of exam results, Lectopia proved to be a successful substitute for lectures for some students, suggesting that Lectopia and lectures are comparatively effective forms of student engagement. However, lower grades were found for students who both attended most lectures and viewed over half of the Lectopia. Analysis of student comments suggests that this was due to factors such as ability to concentrate in class, clarity of the lecture, struggling to understand,and leaving study till the end of semester. Together with lecture attendance and Lectopia viewing, these factors are described within a ‘student engagement model’ of learning, which is presented in contrast to the implicit ‘student attendance model’ underpinning current beliefs regarding the importance of lecture attendance for student learning.

INTRODUCTION Within Australian universities, digital recording and streaming of lectures has become a standard component of the resources provided to students through

472

Engineering Education: an Australian Perspective

e-learning sites such as Blackboard. This study forms part of a larger project aimed at encouraging the uptake of web-based technologies and blended learning options within the School of Engineering at the University of Queensland. Teaching staff need evidence that lecture recordings (i.e. Lectopia) will assist or at least will not diminish student learning, and that major attrition in lectures will not occur because of Lectopia being used in Engineering classes (where Engineering students have a reputation for taking a very pragmatic approach to learning). However, the data that has been collected thus far is inconclusive. On the one hand, Lectopia has been shown to be well utilised and appreciated by students across Australia (McNeill, Woo, Gosper et al, 2007) including students at the University of Queensland (ITS project office, 2008). Results of many studies show that students use recorded lectures for checking over notes, reviewing difficult concepts, for exam revision and for listening to missed lectures (Gosper, Green, McNeill et al, 2008). On the other hand the evidence shows that in general, lecture recording (henceforth referred to using the term ‘Lectopia’) is tolerated rather than liked or embraced by lecturers. While Lectopia is seen as useful for ESL students, for external students and for exam revision, there is some skepticism in terms of its benefits (Birch & Burnett, 2009) e.g. I have no real indication of whether students learn just as well.. (Gosper, Green, McNeill et al, 2008, p. 23). Second, there is some fear that use of Lectopia may cause students to not bother attending lectures (Chang, 2007). Finally, there is some skepticism regarding the usefulness of modern teaching and learning initiatives in hard courses (Burnett & Meadmore, 2002). This is particularly true in Engineering and other STEM courses. As one lecturer in our study has stated we go to these workshops, learn about the latest method, bring it back here, try it out, and it fails. There seems to be a pervasive belief that technologies and methods that take students away from the lecture theatre will reduce the opportunities students have for putting in the mental effort that is required for understanding the complex concepts within the engineering disciplines. Given these fears, some concrete evidence is needed that will either confirm or alleviate them. If we turn to the literature, we find various pieces of the Lectopia-Attendance-Performance puzzle have been addressed. The effects of Lectopia on exam performance have been shown to be both neutral (Smeaton & Keogh, 1998) and positive (Ordoñez 2001; Young & Gibbings, 2007; Gosper, Green, McNeill et al. 2008). The effects of Lectopia on attendance have shown to be positive (Barker & Fothergill, 2005) neutral (Massingham & Herrington, 2006), and negative (Gosper, Green, McNeill et al., 2008). The effects of attendance on performance have been shown to be both neutral (Hyde and Flournoy, 1986) and negative (Massingham & Herrington, 2006). All of these studies have tested various combinations of the 3 important variables: lecture attendance, Lectopia viewing, and exam performance, some gathering perceptions and some collecting numerical data. The evidence thus far is inconclusive, and somewhat confusing. Furthermore, very little of the data is

Student Engagement

473

from Engineering; for example, in Gosper et al (2006), only 7% of the students came from maths, physics, engineering or IT. Some hard evidence is required from the Engineering disciplines, and in addition, the existing literature needs an organisational framework. In order to help to clear up the uncertain state of the evidence, we will first make explicit some of the implicit assumptions underpinning the evidence given in the aforementioned studies. We assert that a uni-dimensional sequential chain of events, which we will call a ‘Student Attendance Model’ (see Figure 1), is assumed by lecturers when considering the effects of Lectopia on their students. Proponents of this model would assert that the availability of Lectopia affects attendance, which affects student engagement, which affects students understanding, which effects student performance. This assumed sequential chain of events underpins the fear that increasing the availability of Lectopia will ultimately reduce exam performance.

Figure 1. The Student Attendance Model. We suggest that the Student Attendance Model is incomplete, and that it is based on false assumptions. First, the full sets of links within the model have not been established within a single study. For example, the amount of Lectopia viewed was not included in Massingham & Herrington’s (2006) study. Furthermore, the Student Attendance Model is based on causal links which have been questioned in the literature, such as the assumed causal chain between attendance at lectures and understanding. While the availability of Lectopia has been shown to reduce class attendance, viewing the lectures online rather than in class may have no detrimental affect on engagement and understanding, since The ‘traditional’ lecture ... is limited in promoting student learning (McKinlay, 2007). That is, the Student Attendance Model is based on the belief that sitting in class is more engaging than viewing lectures at home, which is often not the case in large didactic lectures. Finally, the assumption regarding the causal link between Lectopia availability and lecture attendance stems from the all or none thinking that Lectopia is used by students who don’t come, and not by those who do come. However, the way that students use the technology is not all or none. For example, many students use Lectopia in addition to lecture attendance, to help revise for exams, to review complex materials, and to work at their own pace (Gosper, Green, McNeill et al., 2008). To help to move the debate forward, the Student Attendance Model needs to be replaced with a more realistic model of how students learn, which we call the ‘Student Engagement Model’. This model puts lecture attendance in its place with several other routes to engagement available within the traditional course structures in Engineering, such as attending tutorials, discussing

474


content with peers, and viewing Lectopia (see Figure 2). This framework is consistent with the recent student engagement literature which suggests that engagement is paramount to learning and that there are many ways of engaging students in learning (Krasue, 2005). In the current literature on engagement, the focus is on general effort e.g. Student engagement represents both the time and energy students invest in educationally purposeful activities and the effort institutions devote to using effective educational practices (Kuh, Kinzie, Schuh et al, 2005, p2). However, the important aspect of engagement that is referred to in our Student Engagement Model, is the switching on and the operation of the cogs in the mental machine comprising a student’s effortful thinking processes. We assert that effortful thinking is the most important aspect of student engagement, without which, understanding cannot occur.

Figure 2. The Student Engagement Model. The emphasis in the Student Engagement Model on effortful thinking is consistent with the constructivist perspective that learning involves the construction of knowledge in a student’s mind. The challenge that constructivism gives to teaching and learning in Engineering, is that it is the job of teachers to help students create and incorporate the new concepts into their existing mental knowledge structures (Mayer, 2005). In order to do this, information needs to be given so that it is clear and manageable; i.e. so that it does not overload students working memory (Sweller, 1999) or exceed their cognitive capacity (Halford, Baker, McCredden & Bain, 2005). Unfortunately, in the difficult courses in Engineering, there is much new knowledge to be acquired, often based on equations with more than three interacting variables, thus pushing a student’s cognitive capacity to the limit (Halford et al, 2005). Moreover, such knowledge is often only presented once, and yet it is assumed that the student has acquired it (i) well and (ii) forever. While lecturers may believe that they have taught the information, students may not have had the mental resources or time to allow their working memory to soundly build all of the new concepts and integrate them into long-term memory. Thus many students leave each lecture with only partial and shaky

Student Engagement

475

conceptual constructions of what was delivered. To shore up these new mental constructions, it may be necessary for students to go over the material again more slowly, allowing each new concept to be placed more carefully. It is here where viewing or reviewing the lectures at a slower pace using Lectopia is able to help students to fill in the gaps and to strengthen their understanding, which can then be further refined by applying it to problem solving in tutorials. The Student Engagement Model, as illustrated in Figure 2, also incorporates the important variable of student motivation for fulfilling tasks required within the course (such as attending lectures and tutorials). This variable has been included in the model as it has been found that when student motivation is focused on course tasks, they process material more deeply than when their motivation is focused on performing well in exams, which promotes shallow processing (Pajares & Schunk, 2005). The Student Engagement Model highlights the multiple cyclical pathways of student experiences that lead to cognitive engagement, understanding, and subsequently to exam performance. The Student Engagement Model refutes the Student Attendance Model’s assumption that face-to-face attendance at lectures is vital for student engagement and for good exam performance, and instead, suggests that Lectopia viewing may play a comparable role. This assumption needs to be tested by collecting data that relates exam performance both to the amount of lectures attended and to the amount of Lectopia viewed. Such data will allow us to test the Student Engagement Model’s assertion that the analysis of student performance is at least a twodimensional rather than a one-dimensional issue. Furthermore, it will allow us to test whether Lectopia is a useful aid for student learning and a useful substitute for lectures within the Engineering cohort. If this is the case, we would expect that exam marks for students who rely on Lectopia will be no worse than marks for students who rely on attending lectures. Furthermore, we would expect that Lectopia will be well-utilised by students for revision and for improving their understanding of course content, rather than being used merely as a substitute for lecture attendance. This may significantly reduce the number of student enquiries experienced by instructors during exam revision periods. METHOD Students in a two large Engineering classes: a third-year Civil (fluid dynamics) course and a third-year Chemical (thermodynamics) course, were given Lectopia recordings in addition to normal lectures, to be used as study resources throughout the semester. The recordings for each lecture were made available the same day on the Blackboard site for each course. Online surveys were given midway through the semester and at the end of semester, asking about the amount of lectures attended, the reasons for non-attendance, the amount of Lectopia viewed, the reasons for viewing Lectopia, and students’

476


impressions of Lectopia. Marks from the mid-semester exam for the Chemical course and for the end of semester exam for both courses were related to survey responses. Finally, the number of students attending lectures was counted throughout the semester in both classes, to see whether attendance diminished due to the availability of Lectopia. RESULTS AND ANALYSIS Altogether, 142 students answered the mid-semester survey (75 Civil, 35 Chemical) and 91 answered the end of semester survey (67 Civil, 24 Chemical). Students who responded to the surveys had similar exam averages and standard deviations to their entire cohorts, and thus were adequately representative of their class groups. Since only 22 students repeated the survey, the mid-semester and end of semester survey respondents were treated as two different groups for the analyses that follow.. Attendance and Lectopia Viewing: According to the students’ self reports in these surveys, lecture attendance decreased (from 82% to 71%) and Lectopia viewing increased (from 41% to 49%) from the middle to the end of semester groups. According to our head-counts, attendance for the entire Civil class varied between 40% and 85% across the semester, showing more variance than the self reports of attendance in the surveys. However, we will accept the survey data as being sufficiently representative of the class for the purposes of the analysis. For both the Civil and Chemical class groups, Lectopia was used heavily throughout the semester by at least one quarter of the class, and this increased to half for the end of semester Civil group. Half of the Chemical class group hardly used Lectopia throughout the semester, relying on lectures alone. Uni-dimensional analysis The simple relationships (i.e. the bi-variate correlations) were investigated first, showing that these traditional types of analyses are incomplete. For example, for Civil Engineering students, there were significant negative correlations between Lectopia viewing and final exam marks for both the mid semester group (r = -.27 p=.01 N=83) and the end of semester group (r = -.27 p=.02 N=69). At first glance, these simple correlations seem to corroborate the beliefs underlying the Student Attendance Model, that greater use of Lectopia causes worse marks due to students not attending lectures. However, when the 8 top performing students, most of whom watched 0-10% of Lectopia were removed, then these correlations disappeared. Furthermore, inspection of the distribution of final marks for the Civil students showed that those who attended nearly all the lectures (90%) and those who attended few lectures ( 10%) all received the best marks (with group averages of 58%). On the other hand, students who attended no lectures, or between one quarter and three quarters of the lectures, or all of the lectures received lower marks (with group averages of aroung 50%), with the lowest marks being given to students who

Student Engagement

477

attended three quarters of the lectures (with a group average of 47%). This bimodal relationship between lecture attendance and marks suggests that something else besides lecture attendance was determining the marks that students received in exams. Altogether, the results of the uni-dimensional analyses show that, when used in isolation, neither Lectopia viewing not lecture attendance can predict exam performance. These results are similar to those that have been found in previous studies, in that they seem to contain conflicts in what conclusions can be drawn about the three main variables. Such results pointed to the need to look for interaction effects (ie the effects of two or more variables on a third variable) between Lectopia viewing and Lecture attendance on exam marks. Two-dimensional analysis The effects of both lecture attendance and of Lectopia viewing on marks were analysed together using a two-way analyses of covariance. Accordingly, students who responded to the surveys were categorised into one of four groups: 1. ‘Lecture Reliant’ students who used half or less of the Lectopia and came to 75% or more of the lectures, 2. ‘Lectopia Reliant’ students who used 75% or more of the Lectopia and came to half or less of the lectures, 3. ‘Both Reliant’ students who both used Lectopia and came to Lectures 75% or more of the time, and 4. ‘Low-engaged’ students who came to half or less of the lectures and who used half or less of the Lectopia. Exam results were compared between these four categories of students, with the effects due to overall student ability (i.e., the variance due to students’ grade-point-average) removed before the comparisons were made. Mid-semester results: For the Chemical engineering class, since all students attended 75% or more of the lectures, the mid-semester results only contained two groups: Both Reliant, and Lecture Reliant students.. Subsequently, this data was able to be used to compare the relationship between marks and Lectopia alone (while controlling for lecture attendance). The results showed that those who watched 75% or more of the Lectopia had a mid-semester mark about nine points lower (51%) than those who watched half or less of the Lectopia (60%). Given that the relationship between Lectopia viewing and exam performance is not necessarily causal in any direction, these results can lead to different conclusions: (i) That Lectopia viewing on top of lecture attendance was somehow decreasing understanding and ability, or (ii) that the students who struggled to understand were the ones who were using Lectopia more for revision and study so as to improve their understanding. Of these two possibilities, (ii) seems to be the most likely explanation; since it is unlikely that watching Lectopia in addition to lectures reduces understanding. Thus we can conclude that the Chemical Engineering students who used Lectopia more often received worse marks because they did not understand the Lecture, and that even after reviewing the lecture using Lectopia, their understanding did not increase enough to improve their exam performance to pass standards.

478


Thus the direction of the relationship is that ability causes Lectopia viewing rather than the other way around. These results also question the oft-held belief that poorer students do not study. End of semester results: The 91 students who responded to the end of semester survey were categorised into the four groups. For the analysis of the final exam marks, the Civil and Chemical Engineering final marks were equated by converting them to standardised Z-scores (described by a normal distribution with a mean of 0.0 and a standard deviation of 1.0). Marks for both classes were then able to be combined for the analyses of covariance. (When the covariance due to GPA was removed from the marks variable, the estimated exam means for each group became higher or lower by 5.02 marks. These were the means that were compared for the interaction analyses). The results are graphed in Figure 3, showing that there was an interaction between the effects of Lectopia viewing and the effects of Lecture attendance on exam marks. The analysis of covariance showed that the interaction between Lectopia and Lecture attendance was significant (F(1,86) = 6.32; p=.014).

Figure 3. The interaction effect between Lecture attendance and Lectopia Viewing. Figure 3 shows that the 43 Lecture Reliant students received the best marks (average Z-score = 0.1). These students would have used Lectopia to fill in for an occasional missed lecture, but mostly for reviewing attended lectures and for comprehension purposes. Furthermore, Figure 3 shows that the 18 Lectopia Reliant students also received fairly good marks (average Z-score = 0.08), just above the average class marks and close to the Lecture Reliant group. These students would have relied upon Lectopia for replacing many lectures. Furthermore, of the 25 students who attended half or less of the lectures, the 18 Lectopia Reliant students did much better than the 7 Low-engaged students,

Student Engagement

479

who received the lowest exam marks (average Z-score = -0.5). Altogether these results suggest that Lectopia viewing proved to be an adequate study resource for students who missed many lectures, bringing their exam performance up to a level comparable to the students who attended most lectures. However, Figure 3 also reveals an unexpected result for the 23 students who relied on both Lectures and Lectopia. These students, who would have used Lectopia to revise many lectures that they had already attended, were apparently putting in the most effort out of all the groups. However, they did worse on the exam than those who didn’t use Lectopia for revision (average Z-score = -.27; a quarter of a standard deviation below the class average exam marks). To understand what was going on for these students, the responses to the open ended questions included in the survey were analysed. Responses to the question: “If you have started using Lectopia, or are using it much more often than before, can you say why?” showed that the students who got poorer marks in this apparently harder working group were more likely to mention difficulty in seeing, hearing, understanding or paying attention to the lecture when they did attend, e.g. a student in this group who received 33% on the exam stated I re-watched the Lectopia and realised it was great because you could read what was being written on the board more clearly (its often hard to read, out of focus) and you could hear and not be distracted by other people in the audience talking. That is, although present, it seems these poorer achieving students were not able to concentrate or comprehend most of what was said in the lectures. This would be a likely outcome for students in large Engineering classes who sit in rows in the back half of large lecture theatres. Unfortunately, for the students who had not understood the lecture for whatever reasons, reviewing the lecture again did not seem to help them with comprehension. These findings are similar to the mid-semester results from the Chemical Engineering class which suggest that students who do not understand the lectures use Lectopia more. This result is for students who are attending most lectures, so that the availability of Lectopia causing students to skip lectures cannot be blamed for the lack of understanding. Neither can the blame be placed on the students’ abilities per se, since the marks had student GPA removed from the outcomes before the analyses were performed. Non-attendance at Lectures and Marks : The question remains as to whether the availability of Lectopia causes non-attendance at lectures. In order to investigate this more closely, the Civil students who answered the midway and end of semester surveys were divided into three groups: those who received good marks, those who received satisfactory marks and those who received poor marks. The percentages on the final exam that determined these groupings were decided by the lecturer for the course. For all of these groups, the numbers of students who circled each of the various reasons on the surveys for why they missed lectures are summarised in Table 1. In the table, reasons for non-attendance that were given by more than 20% of the students

480


within any given ability grouping are shown in boldface so as to highlight the prominent reasons for non-attendance. Table 1 shows that the availability of Lectopia was given as a reason for not attending lectures, mostly up to mid-way through the semester for the students who received good marks or towards the end of semester for students who received satisfactory marks. However, there was no pattern for Lectopia being used as a reason for non-attendance at lectures for the poorer students. The next major reason for non-attendance at lectures was the need to work on assignments etc. for other courses. The pattern was for satisfactory and poorer students to give this as a reason more often than good students. This result could have been because the poorer students had more assignments than the better students; however, a more likely interpretation is that time management was more of an issue for these students, and thus they perceived that there was more conflict with other courses than did the better students. The next most prominent set of reasons for non-attendance had to do with illness and other unknown issues. These reasons were given more by the good and satisfactory students than the poor students.

Table 1: The number of good, satisfactory and poor students in Civil Engineering who gave various reasons for not attending lectures for the mid-way survey and the end of semester survey. Percentages (in brackets) are calculated using column totals. Reason for Nonattendance

Good (>=50%)

Satisfactory (40-50% )

Poor (=50%)

Satisfactory (40-50% )

Poor (