Rapid Reaction and Response (R3): The in-class use of mobile technologies to support formative and diagnostic assessment The Pathfinder Journey May 2008 Dr T. K. Linsey, Dr A. Ooms and Andreas Panayiotidis
[email protected]
The Context In 1999 Kingston University (KU) as part of its Learning and Teaching Strategy made the decision to implement an institution wide virtual Learning environment (VLE) with the aim of enhancing student learning and the student experience of learning and teaching. An Educational Technology Units (ETU) was established to support the academic implementation of esupported learning and meet specified rollout targets. The VLE was implemented Universitywide in 2001 and integrated with its student record system. Over the following 3 years some form of online support through the VLE was implemented across the vast majority of modules. Back in 2001 the potential of the VLE, with the facility to access 'missed lecture' notes and to review material for further understanding or revision, was seen as important in supporting learning, as well as to enhance communication. This was a key selling point within the University community, as was the add-on rather than re-design approach. This was the most common approach at the time among universities with VLEs according to a survey for JISC. Adopters “indicated that usage was overwhelmingly supplementary to face-to-face delivery rather than being substantially integrated into course delivery” (JISC/UCISA, 2003). By the summer of 2004 the e-supported or Add-on support had been implemented across the majority of modules and focus of development had shifted to blended learning and the effective and appropriate integration of e-learning and face-to-face learning and teaching. The implementation of blended learning proved to be a significant challenge. To achieve the first part of the strategy staff did not need to change their teaching approaches but to implement blended learning staff needed to engage in a process of course redesign, including the potential re-design of the face-to-face elements which of course was a significant challenge. This process was supported by a range of staff development events and the use of e-learning developers to help faculties implement key blended learning developments. During this period (2002-2005) a research project funded by Atlantic Philanthropies was implemented to investigate the role of educational technologies in supporting students from diverse backgrounds, both prior to attending a higher education institution and on course. This clearly demonstrated the importance of flexibility in supporting students with various external constraints and commitments as well as their diversity in terms of approaches to learning. It also underlined the value that students placed on face-to-face learning and teaching events which was subsequently recognised in a number of other studies. This work partly stimulated a number of trials that were undertaken by the ETU investigating the role of classroom technologies in enhancing feedback, interaction and collaboration, particularly focussing on electronic voting systems (initially infrared based but then supplemented with wireless based systems). The findings from the Atlantic Philanthropies funded work along with classroom technologies trials and ongoing evaluations of our VLE infrastructure provided key inputs to the blended learning components of the institution’s Quality Enhancement Strategy (QES) published in Autumn 2006. The QES brought together all of the objectives of a Widening Participation strategy, Learning and Teaching Strategy and the academic staff developments elements of the Human Resources Strategy. At the time of developing the QES the University engaged with phase 1 of the HEA e-Learning Benchmarking exercise. It was seen that the outputs from the benchmarking exercise would be a crucial input in informing each faculty in developing their own blended learning strategies as required by the QES.
1
The subsequent HEA funded pathfinder project focussing on mobile classroom technologies was also seen as a key component in informing faculty blended learning strategy development, especially in terms of the re-design of face-to-face teaching and learning events. Also important was the University’s New University Project (NUP) to develop its estate. During the period of the pathfinder project over 10,000 square metres of new teaching space was opened including flexible learning spaces, pod based IT laboratories and a Collaborative Learning Zone. The new facilities provided an additional driver for pedagogic change and to reinforce this pathfinder workshops and dissemination events were run in the new spaces.
What was the problem the project set out to address Our initial premise was that the learning and teaching approaches in a classroom / lecture room environment could be significantly enhanced in terms of a number of criteria including student engagement and collaboration, formative and diagnostic assessment and feedback and overall student involvement in the learning and teaching process. Blended Learning through the effective and appropriate integration of online resources and activities can play an important role in achieving this enhancement. However there are a number of factors that can inhibit this including non-engagement with blended learning and student centred pedagogic approaches, the suboptimal design of physical spaces along with a range of other factors such as class size. Initial trials using electronic voting systems indicated that classroom technologies had the potential to overcome some of these constraints and enable enhancement. The Pathfinder project was designed to focus on a key element of this overall problem, specifically the use of a range of mobile classroom technologies to enhance in-class diagnostic and formative assessment and feedback. This recognises the fact that according to the National Student Survey (HEFCE, 2006), feedback on assessment is the weakest area for most universities in the UK It was also hoped that through the use of these technologies staff would be prompted to address the wider aspects of their learning and teaching models and approaches. Over the last 3 years, a range of new classroom technologies have been introduced at KU including:
Electronic Voting Systems (Wireless and infrared - 300 handsets) Interactive Wireless Graphics tablets Tablet PCs and Wireless data projectors Interactive podium screens
An example of their use at KU was recently published in ALT-J: Masikunas, G., Panayiotidis, A & Burke, L. 2007 The use of electronic voting systems in lectures within business and marketing: a case study of their impact on student learning, Association for Learning Technology Journal, 15(1), 3-20. Our experiences indicated that technology can support in-classroom formative assessment, where particular student’ areas of weakness and strengths can be identified immediately. The assessment data can also assist academic staff in identifying students’ misconceptions or naive conceptions. Students will receive immediate feedback on their knowledge and understanding, which will promote their learning and diminish misunderstanding of challenging concepts. The assessment data will also inform staff about needs to change teaching practices (i.e. revisit certain components, give students an opportunity to practise more, add or change pedagogical activities). Teaching practices could be adapted immediately or in future sessions because academic staff may reflect on their teaching practices and plan changes for the future. The project focussed on the examination of the effective use of in-class mobile technologies to enhance timely feedback and diagnostic/formative assessment and aim to answer the following research questions:
2
1. Under which conditions can each of the technologies be efficiently and effectively used for diagnostic / formative assessment in classroom settings? 2. What is the impact of the in-classroom use of mobile technologies for diagnostic / formative assessment on students’ attitudes toward the module? 3. What is the impact of the in-classroom use of mobile technologies for diagnostic / formative assessment on students’ conceptual understanding? 4. What is the impact of the in-classroom use of mobile technologies for diagnostic / formative assessment on students’ test results? 5. What is the impact of the R3 project on teaching practices? How likely is it that that impact, if there is any, will sustain? 6. What is the impact of the R3 project on assessment practices? How likely is it that that impact, if there is any, will sustain? 7. What is the impact of the R3 project on attitudes on in-classroom use of mobile technologies? How likely is it that that impact, if there is any, will sustain? 8. What indicators are there of institutional commitment to and subsequent uptake of inclassroom use of mobile technologies?
Intended Outcomes Our expected outcomes were as follows:
Improvement of students’ conceptual understanding (DiBattista et al, 2004) Improvement of test results (Cue, 1998; Elliott, 2003; Hake, 1998) Improvement of classroom interaction and discussion (Draper & Brown, 2004; Masikunas, Panayiotidis, & Burke, 2007) Improvement of student motivation (Boyle & Nicol, 2003) Improvement of students’ perceptions of assessment practices Improvement of academic staff’s perceptions of the value of diagnostic assessment Improved assessment practices
The Faculty Blended Learning Leaders along with the Faculty Learning and Teaching coordinators were asked to identify the project participants, two from each faculty. Each participant was required to implement one or more mobile technologies into their face-to-face teaching events. In addition staff were required to participate in a number of tasks and activities:
attend 4 workshops. These workshops were designed to cover the pedagogy of online assessment, the technologies and provide a forum to discuss experiences and planned implementations. The initial workshop was led by an invited expert on online assessment, Bob Rotheram, a national teaching fellow from Leeds Metropolitan University. This session focussed on the pedagogy of assessment and the writing of objective questions. During the afternoon participants had the opportunity to start developing assessment questions relevant to the modules within which they planned to introduce the classroom technologies. Complete a reflective journal. An online journal with supporting template was established with headings and questions prompting responses. Complete a project calendar. A shared project calendar was established which each participant updated with details of planned events and equipment being used. Attend ‘Camel’ support sessions with the two project mentors. Each participant was assigned to one of two mentors. The two mentors had significant experience in the use of classroom technologies and they organised meetings with the participants once a month to share practices and to report concerns to be addressed. The mentors have also supported the staff participants on an individual basis with follow-up training sessions on using the classroom technologies, assisting them in the preparation of the classroom sessions and also by attending some of the teaching sessions. The participants have clearly expressed the value they place on the support played by the mentors.
In terms of data collection participants completed a questionnaire at the beginning and end of the semester and also completed evaluations of the workshops. Additionally classroom 3
observations were undertaken during the semester and each participant was interviewed at the close of the semester. The mentors were also interviewed and questionnaires surveys distributed to students on the project modules.
What has the project achieved at this stage?
University wide implementation. Mobile educational technologies including electronic voting systems, Tablet PCs, audio devices and text messaging, were embedded in modules (class sizes for 15 to 500) across all 7 faculties of the University. This has produced a diversity of examples and has helped raise the awareness of the technologies. By the close of the second semester 12 participants had successfully used the technologies with one participant reporting an unsuccessful implementation. Electronic Voting systems were the most widely used technology followed by the Tablet PCs. Collection of a significant evidence base with which to evaluate the project including staff and student questionnaires, reflective journals and interviews. Preliminary analysis of the data indicates a very positive response from students. The following graph shows the percentage response to the statement ‘overall, the use of the new technologies was a positive experience for me as a student’.
Preliminary evidence that some staff participants have been adapting their learning and teaching approaches beyond the specific requirements of the project. Preliminary evidence indicates that the use of a staff mentor approach to introducing new approaches to learning and teaching has been successful. Internal dissemination. A regular project update has appeared in the University’s Academic Developments newsletter and additionally presentations at the University’s annual learning and teaching conference and as part of the Postgraduate Diploma in Learning and Teaching. External dissemination with presentations given at:
-
London SoTL 2008 Conference, London (May 2008)
- F-Team conference at the University of Wolverhampton (April 2008) - De Montfort University e-Learning Symposium (May 2008) Further presentations have been accepted at :
HE Academy Annual Conference (July 2008) EARLI/Northumbria Assessment Conference, Berlin (August 2008) ALT-C with the F-Team Cluster (September 2008) European E-Learning Conference, Cyprus (November 2008)
New Technologies introduced to overcome problems identified through staff participant experiences. One example was the use of Wireless data projectors. The original wireless data projector introduced at the beginning of the project was cumbersome and only provided one point of projection. This was replaced with a small lightweight adapter that could be used in conjunction any teaching podium across the university. This allowed a portable PC to be handed out to the audience allowing students to interact with the screen while being wirelessly projected via the existing fixed data projection system. As a result of the project further such units will made available for staff loan during the next academic year. Senior management (Deputy Vice Chancellor and the Director of Information Services) were involved with the project through participating on the project steering group. 4
Where Now?
Continuation of the ‘project’. As mentioned in the previous section our evidence to date shows the important role that the two staff mentors played in supporting the participants and it is hoped to continue with this model next academic year with a number of this year’s participants acting as mentors. With a minimum of 6 mentors supporting up to 30 staff participants next year it is expected that a significant impact can be made in each faculty in terms of enhancing face-to-face learning and overall blended learning models. Ongoing Technical support. The project funded technical support for the classroom technologies used and the University has agreed to continue funding this post for the next academic year. This will allow us to continue to provide a support infrastructure for the mobile technologies. During the period of the project the University has expanded the number of mobile technologies that are available for learning and teaching. In November 2007 the University opened a ‘Collaborative Learning Zone as a student centred learning space supporting group work and activities. This room has been equipped with both an electronic voting system and a set of 15 tablet PCs. It is hoped to continue to expand the technologies available. In September each faculty will begin the second year of their blended learning strategy implementation. Over the summer the project team will analyse the project data and findings and report back to the faculties to help inform this ongoing process and help shape faculty developments. Staff development events on the integration of mobile technologies into the classroom have been built into Academic Development’s staff development programme for the next academic year with a series of online resources either developed already as part of the project or are under development. We will continue to evaluate the role of new mobile technologies in learning and teaching and will apply for further project funding as appropriate. We are already investigating: the use of mobile phone technology to provide ubiquitous voting systems the use of low cost computing systems in combination with mobile broadband to provide networked computing in ‘any’ location such on fieldtrips etc.
Key Messages for the Sector. Embedding pedagogic change on an institutional scale is a complex task and it is recognised that the HEFCE eLearning strategy strand 1.4 ‘Produce and disseminate models of good e-learning practice including assessment’ is particularly relevant to this project. This project has demonstrated on evidence to date that a focussed limited scale project can be a significant catalyst for change especially where the project is closely linked to institutional strategy, is sustainable, the participants are well supported and that there are clear channels of dissemination. A number of focussed limited scale projects running in parallel may provide an effective approach to embedding institutional change. This approach will be continued at Kingston University during the next academic year as part of suite of targeted approaches to support the implementation of blended learning across the institution. In terms of specific points with regard project support the following points can be made: the role of staff mentors with recognised expertise in the relevant field was effective. The mentor groups also enabled staff to exchange experiences and advice on a regular basis. the provision of dedicated technical support for mobile technologies proved essential to maintain the equipment and provide support for staff. Unreliability can be a significant issue in terms of continued use by staff and acceptance by students. Due to the portable nature of the equipment it was subject to more wear and tear and the need, for example, to replace batteries on a regular basis. The ability for academic staff to call for advice / troubleshooting support while setting up the portable equipment in a pressurised situation with students present proved valuable. In terms of the in-class use of mobile technologies, based on the analysis of our evidence to date, they have been effective by: promoting greater interaction 5
enhancing feedback for both students and staff, and allowing staff to adapt their teaching based on this feedback. acting as catalysts for change in learning and teaching approaches.
It is obvious to state that technologies continue to evolve rapidly but it is important recognise that some technologies in use today could be considered relatively expensive interim solutions and this should be considered when assessing costs and benefits. Just over the period of the project the price of low cost mobile computers has dropped significantly as has that of mobile broadband. This has significant implications for ubiquitous networked mobile computing and its impact on learning and teaching. A key success of the overall Pathfinder programme has been the breadth of projects and institutions involved, and through this the impact on the academic community has been significant. One indicator of this has been the number of Special Interest Groups and external dissemination events that have been run. Two further elements of the programme that can be highlighted by the KU team are: The support ‘Camel’ clusters. From the Kingston University team’s perspective this proved to be a positive model. The ‘F-Team’ cluster held 3 meetings at different institutions, each of which made a positive contribution through the discussions held and the opportunity to learn from the experiences the other project teams in the cluster. This has already resulted in further discussions with regard future collaborations. The role of the Critical Friend. The critical friend has played an effective role with both the individual teams and the cluster. It is recommended that the HEA and other funding bodies consider the role of critical friends as part of future calls.
References Boyle, J. T., & Nicol, D. J. (2003). Using classroom communication systems to support interaction and discussion in large class settings. Association for Learning Technology Journal (ALT-J), 11 (3), 43-57. Cue, N. (1998). A universal learning tool for classrooms? Paper presented at the First Quality in Teaching and Learning Conference, Hong Kong. Dibattista, D., Mitterer, J.O., & Gosse, L. (2004). Acceptance by undergraduates of the immediate feedback assessment technique for multiple-choice testing. Teaching in Higher Education, 9 (1), 17–28. Draper, S. & Brown, M. (2004) Increasing Interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20, 81-94. Elliott, C. (2003). Using a Personal Response System in Economics Teaching. International Review of Economics Education, 1 (1), 80-86. Hake, R. R. (1998). Interactive-engagement versus traditional methods: a six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66 (1), 64–74. JISC/UCISA (2003) Management and implementation of Virtual Learning Environments. Retrieved from www.jisc.ac.uk/project_mle_activity.html 20 June 2005 Masikunas, G., Panayiotidis, A & Burke, L. 2007 The use of electronic voting systems in lectures within business and marketing: a case study of their impact on student learning, Association for Learning Technology Journal, 15(1), 3-20.
6
