A webbased system for teaching, assessment ... - Wiley Online Library

Anaesthesia, 2003, 58, pages 1079–1086 .....................................................................................................................................................................................................................

SPECIAL ARTICLE

A web-based system for teaching, assessment and examination of the undergraduate peri-operative medicine curriculum D. W. Wheeler,1 K. D. Whittlestone,5 H. L. Smith,2 A. K. Gupta4 and D. K. Menon3 on behalf of the East Anglian Peri-operative Medicine Undergraduate Teaching Forum* 1 Clinical Lecturer, University Department of Anaesthesia, 2 Peri-operative Medicine Undergraduate Attachment Director, School of Clinical Medicine, 3 Professor of Anaesthesia, University of Cambridge, Box 93, Addenbrooke’s Hospital, Cambridge CB2 2QQ, UK 4 Director of Postgraduate Medical Education, Box 93, Addenbrooke’s Hospital, Cambridge CB2 2QQ, UK 5 Assistant Director, Centre for Applied Research in Educational Technologies and Clinical and Biomedical Computing Unit, University of Cambridge, 1st Floor, 16 Mill Lane, Cambridge CB2 1SB, UK Summary

Today’s students are generally computer literate and have high expectations of university information technology resources. Most United Kingdom medical schools now provide networked computers for learning, research, communication and accessing the worldwide web. We have exploited these advances to augment and improve the teaching of peri-operative medicine and anaesthesia to medical students in our university, who are taught in several hospitals over a wide geographical area. Course material such as departmental induction information, lecture notes and assessment sheets can be accessed online, contributing to the smooth running of the course. Streamed videos and simulations allow students to familiarise themselves with common practical procedures in advance. Development of a web-based end of course assessment has resulted in substantially less administration and bureaucracy for course organisers and proved to be a valuable research tool. Students’ and teachers’ opinions of the new course structure have been overwhelmingly positive. Keywords

Anaesthesiology. Education; medical students. Teaching; educational technology. Computer Communication Networks; Internet.

. ......................................................................................................

Correspondence to: D. W. Wheeler E-mail: [email protected] *The East Anglian Peri-operative Medicine Undergraduate Teaching Forum consists of (all in United Kingdom): M. J. Garfield, Consultant Anaesthetist, Ipswich Hospital, Ipswich, J. T. McNamara, Consultant Anaesthetist, Bedford Hospital, Bedford, M. I. Palmer, Consultant Anaesthetist, West Suffolk Hospital, Bury St. Edmunds, P. J. Roberts, Consultant Anaesthetist, Hinchingbrooke Hospital, Huntingdon, C. A. Roud-Mayne, Consultant Anaesthetist, Luton and Dunstable Hospital, Luton, M. R. Stoker, Consultant Anaesthetist, Peterborough Hospital, Peterborough, B. J. Watson, Consultant Anaesthetist, Queen Elizabeth Hospital, King’s Lynn, M. Weisz, Consultant Anaesthetist, Peterborough Hospital, Peterborough, D. A. Zygun, Lecturer, University Department of Anaesthesia, University of Cambridge, Cambridge. Accepted: 29 June 2003

2003 Blackwell Publishing Ltd

1079

Æ

D. W. Wheeler et al. Web-based system for undergraduate peri-operative medicine curriculum Anaesthesia, 2003, 58, pages 1079–1086 . ....................................................................................................................................................................................................................

Falling costs of computer hardware and the advent of the worldwide web have led to an explosion in computer technology in universities. Medical schools have begun to realise the huge potential of communication and information technology (C&IT) to support learning, teaching and course administration. Rapid and convenient communication by e-mail across university computer networks has revolutionised interaction between students and faculty. Such networks often offer unlimited access to the worldwide web, which contains vast quantities of uncontrolled and unstructured information that could easily overwhelm students. In recognition of this problem, universities have begun to create ‘virtual campuses’, a structured local environment from which students can explore web-based learning resources [1]. Students of clinical and veterinary medicine spend long periods on attachments that may be far from their campus, and are prone to feel isolated from the teaching centre [2]. Our School of Clinical Medicine created the ‘ERWeb’ (Educational Resources Web) in 1998 to distribute teaching material, provide learning resources and improve the efficiency of administering many cohorts of students attending attachments in numerous locations. Collaboration between the University of Cambridge Department of Anaesthesia and the creators of the ERWeb has driven further development. The ERWeb now provides audiovisual material on practical procedures, the means to perform end of attachment assessments, a program to detect collusion in such assessments [3], and a streamlined and more efficient way of submitting student appraisals. These new resources have been created in a generic way so that they may be utilised by teachers in other specialties. Structure of the University of Cambridge ERWeb

The ERWeb is a secure, web-based, virtual learning and communication environment developed for the University of Cambridge School of Clinical Medicine. It was created to deliver online resources, monitor student activity and provide summative and formative student assessment. Its contact system, with student photographs, is used extensively for communication between staff and students. Online forms have replaced the paper-based system for administering many aspects of the course, for example arranging elective attachments and general practice placements. Each student is allocated a username and password for the ERWeb at the start of the clinical course. Their activity within the ERWeb environment is recorded in a database. This includes information such as the time of access to a resource, the computer used and any data entered by the student in response to questions. 1080

If required, resources can be made available for a specified period of time, to a specified group of students. This access can also be restricted to a specific group of computers so that, for example, an online examination is only available to specified students in a particular room, which can then be monitored (for example by a web camera). Further developments within ERWeb have investigated specific approaches to online learning and assessment, such as clinical skills demonstrations, online tutorial groups and the detection of collusion in online examinations [3]. Structure of the Peri-operative Medicine attachment

There are approximately 120 students in a year group at the University of Cambridge School of Clinical Medicine. In the final year, students are divided into 12 attachment groups and rotate through a number of specialties for 2 weeks at a time. One point on this rotation is the Peri-operative Medicine attachment. Therefore, cohorts of approximately 10 students attend every fortnight between May and October. The first day of the attachment comprises lectures and practical clinical skills ‘hands on’ sessions at the central teaching hospital. Thereafter, students are subdivided into smaller groups for the clinical phase of the attachment and are allocated to one of nine hospitals that lie within the National Health Service Eastern Deanery [4]. During the clinical phase, the students attend operating theatres, intensive care units and acute and sometimes chronic pain sessions. They are assessed on three criteria: attendance, performance in a case presentation, and performance in a multiple-choice questionnaire (MCQ) examination. The Clinical Lecturer (D.W.W.), Attachment Director (initially A.K.G. and latterly H.L.S.), and Professor of Anaesthesia (D.K.M.) administer the attachment. A designated Consultant Anaesthetist, the Undergraduate Specialty Tutor, supervises the students at each hospital. All those responsible for the undergraduate Peri-operative Medicine attachment are members of the East Anglian Peri-operative Medicine Undergraduate Teaching Forum, which meets regularly to discuss developments in the administration of the course and curriculum. Impact of ERWeb on the Peri-operative Medicine attachment

Course administration The Eastern Deanery covers a large area [5]. The ERWeb has substantially improved the ease of contacting students 2003 Blackwell Publishing Ltd

Æ

Anaesthesia, 2003, 58, pages 1079–1086 D. W. Wheeler et al. Web-based system for undergraduate peri-operative medicine curriculum . ....................................................................................................................................................................................................................

before, during and after the Peri-operative Medicine attachment. Prior to the course, the students may, as part of a previous attachment in another department, be posted to a hospital over 100 miles from the teaching hospital, or even on elective abroad. The ERWeb provides a straightforward means of contact by e-mail, providing details of the Peri-operative Medicine attachment structure, timetable and lecture programme (Fig. 1). The ERWeb also carries contact details and photographs of each student and many of the course organisers, with access restricted to appropriate individuals. This

greatly facilitates recognition of the students currently on attachment, especially in a busy theatre complex, and students are more likely to approach those responsible for their teaching and supervision if they recognise them. The e-mail system also facilitates contact with students after their 2-week attachment, when they may be dispersed across the region on their next placement. Those not satisfying the assessment criteria can be contacted easily to arrange remedial action. In all respects, the ERWeb saves time and resources in the administration of the course.

Figure 1 The welcome screen of the Peri-operative Medicine attachment, with links to online attachment instructions and educa-

tional resources.


1081

Æ


Provision of teaching resources One of the first virtual campuses was created at Johns Hopkins University, Baltimore, USA, when the syllabus, lecture notes and written teaching material were placed on the worldwide web [6]. The ERWeb began in a similar way. We have found that improved accessibility to teaching material has several benefits. Enthusiastic students may preview teaching material from forthcoming placements. Moreover, our teaching material is crossreferenced with that of other specialties, a valuable resource for the Peri-operative Medicine attachment that distils knowledge from other areas of the curriculum. In 2002, more than 500 000 journal articles were published in peer-reviewed biomedical journals. Perhaps three times as many were published in other journals and electronically without peer review [7]. Students can easily be overwhelmed by this plethora of information, some of which could be inaccurate or misleading. The ERWeb has collaborated with the medical library to establish on-line subscriptions to major journals, allowing access to students outside the University of Cambridge computing domain. The ERWeb also acts as a portal to websites or

online teaching materials that have been reviewed for quality and content by medical school staff and rated for usefulness by students. Assessment One of the assessment criteria for the Peri-operative Medicine and Anaesthesia attachment is a satisfactory performance in a negatively marked MCQ examination. Previously, this was held in the teaching hospital on the last afternoon of the attachment. Consequently, some students spent the final day travelling to take the examination, impinging upon their exposure to the specialty. There were six different MCQ examinations. An invigilator was required and the papers were marked manually. Using the ERWeb, the students can now take the MCQ examination online from any computer with Internet access at a convenient time on the last day of the attachment. As well as a reduction in paperwork, there is less travelling, less clinical time is lost, no invigilator is required, papers are marked automatically, and results are available immediately.

Figure 2 An example of a multiple-choice question used in the online Peri-operative Medicine assessment system.

1082


Æ


Collaboration, collusion or the use of crib sheets or books might be likely if students sit examinations unsupervised. This problem has been tackled in a number of ways. The MCQs (Fig. 2) are chosen from a large bank, so that each cohort of students sits a different examination. Each question and its answers are presented in random order. Students are pressed for time, with 25 min to answer 25 questions. The time that each student accesses and answers each question, and the Internet Protocol (IP) address from which they work, is recorded and these data are analysed to calculate the probability that any students have collaborated [3]. This combination of measures makes use of a textbook or crib sheet less inviting. Collaboration is more difficult to discourage, although as question order is random, students that might be collaborating are initially unaware

that they are sitting the same examination. Only one episode of collusion has occurred in 2 years. During the very first online assessment, the software detected two students sitting together, scrolling through the questions until they realised that their examinations were the same but presented in a different order, then answering in unison. These students were warned and required to re-sit the examination under invigilated conditions. There has been no further collusion. However, to further reduce the risk of such activity, the question selection system has been modified. Questions are now grouped according to category and difficulty. Now each student sits a different but equally challenging examination. On average, four questions overlap for any two students and the chance of two students receiving an identical question set is 1 in approximately 2.8 · 1019.

Figure 3 An example (using fictitious data) of the online student appraisal screen, collating results from different aspects of the

Peri-operative Medicine assessment.


1083

Æ


Recording the marks awarded for attendance, enthusiasm and ability, and performance in the case presentation in the regional hospitals has also been streamlined. Previously, reports were completed by hand and returned to the Medical School by post. Several weeks might elapse before students discovered their marks. The ERWeb collates the marks electronically, presents a summary for the Attachment Director and Clinical Lecturer (Fig. 3), and passes them to the students via e-mail within days. This also provides an opportunity for the students to complete an online feedback questionnaire about the course. Audit The data collected about students’ activities whilst using a virtual campus have previously been used to investigate the correlation between the time spent accessing online teaching material and success in an anatomy assessment [8]. Those using the resources least were most likely to perform poorly. However, the corollary was not true. Those accessing teaching material most scored below the

mean. In this study, students mostly viewed past examination papers rather than lecture notes, emphasizing that it is not sufficient simply to transfer teaching material from paper to computer. An audit of our students’ online habits also found that those using the ERWeb least are most likely to fail (unpublished observations). We have been able to use the ERWeb to audit the results of the online MCQ examination to address initial concerns that lack of invigilation might lead to collusion and ‘leakage’ of questions. Our hypothesis was that if students were copying questions down and passing them to members of future cohorts, the mean time taken to complete the examination would diminish and mean scores would gradually improve. This might be seen within a year group or over both years that the system had been in place. Figures 4 and 5 show graphs of the mean time taken to complete the examination and mean score for each cohort over 2 years. Data were analysed using bivariate scattergram plots with regression and Spearman rank correlation

Figure 4 a) Mean time taken for each cohort to complete the on-line MCQ examination. b) Bivariate plot with regression analysis of

time taken for each cohort. c) Bivariate plot with regression analysis of time taken for the cohorts sitting the examination in the first year. d) Bivariate plot with regression analysis of time taken for the cohorts sitting the examination in the second year. 1084


Æ


Figure 5 a) Mean score of each cohort sitting on-line MCQ examination between April 2001 and November 2002. b) Bivariate plot

with regression analysis of MCQ examination score of all cohorts. c) Bivariate plot with regression analysis of MCQ score of the cohorts sitting the examination in the first year. d) Bivariate plot with regression analysis of MCQ score of the cohorts sitting the examination in the second year.

(StatviewTM, SAS Institute Inc., Cary, NC). The results indicate that there has been no significant change in the time taken to complete the MCQ examination, whether all cohorts are analysed together (p = 0.06), or cohorts from the first (p = 0.15) or second (p = 0.08) years of the new system are analysed separately. When analysing the score of all cohorts together or cohorts from the first year of the new system, no significant improvement was detected (p = 0.12 and 0.81, respectively). The score of the cohorts from the second year of the new system was found to have significantly declined (p = 0.02). This latter finding may reflect the fact that cohorts towards the end of the academic year are distracted by impending Final examinations. Similar audits will be undertaken to ensure that future students do not communally acquire questions. Research The facilities of the ERWeb to contact students and set MCQ examinations have been used to conduct research 2003 Blackwell Publishing Ltd

into students’ comprehension of different expressions of the concentration of drugs in solution [9]. Generic programs used in the ERWeb have also been used to evaluate histopathology teaching. The future

The ERWeb will soon carry simulations and streaming video demonstrating intravenous cannulation and airway management, allowing students to familiarise themselves with practical skills and learn the indications, advantages and disadvantages of each before the attachment. This is not meant to substitute attendance in theatre and acquisition of ‘hands-on’ skills, which are some of the most important that are taught during the attachment. We believe that the best environment for teaching practical procedures is on the wards and in theatre, not virtual reality. The limited amount of research in this area reinforces this opinion. A virtual reality module to teach 1085

Æ


intravenous cannulation did not appreciably enhance teaching compared to traditional methods [10]. However, the students have reported that they feel more confident embarking on a procedure in theatre if they have seen an audiovisual tutorial beforehand, and the material can act as a useful reminder of a technique just learnt. Anaesthetists are increasingly working alone and with the modern emphasis on patient throughput, there is less time for teaching. Patients are less likely to consent to be treated by medical students. Anaesthetic simulators offer excellent opportunities for learning but are an expensive and sought-after resource [11], which is currently mainly confined to postgraduate training and performance assessment [12]. Under these circumstances, any means of boosting students’ confidence that might accelerate learning is a valuable resource. There is a great deal of potential to improve and develop computer-based assessment. Most virtual campuses that include assessment programs use the MCQ format. However, technology can be used not only to reduce staff workload but also to increase assessment objectivity and consistency [1]. Currently, there is a great deal of interest in developing computer-based clinical competency examinations centred on a series of evolving case scenarios, which adapt to take into account the answers already given, and in some ways simulate a viva voce examination [13, 14]. These methods of assessment are well received by students and require much less organisation and fewer personnel [15]. Conclusion

A move to a web-based Peri-operative Medicine attachment has proved popular with staff and students. Staff are pleased with the numerous labour-saving applications and ease of contacting students and other faculty members. On the whole, the students are more enthusiastic and the course is taken more seriously as it is better organised and has a challenging end of attachment assessment. The benefits of introducing technology into undergraduate teaching of Peri-operative Medicine undergraduates may also have relevance for other undergraduate specialties or the training and appraisal of postgraduate anaesthetists.

1086

References 1 Ward JP, Gordon J, Field MJ, Lehmann HP. Communication and information technology in medical education. Lancet 2001; 357: 792–6. 2 Hagdrup NA, Edwards M, Carter YH et al. Why? What? and How? IT provision for medical students in general practice. Medical Education 1999; 33: 537–41. 3 Ercole A, Whittlestone KD, Melvin DG, Rashbass J. Collusion detection in multiple choice examinations. Medical Education 2002; 36: 166–72. 4 NHS Eastern Deanery, Cambridge UK. Eastern Deanery – Postgraduate Centres and Trusts [online]. URL. http:// www.easternregion.org.uk/map_new/ghome.htm [accessed 2003, April 30]. 5 NHS Eastern Deanery, Cambridge UK. Map of NHS Eastern Deanery [online]. URL. http://www.easternregion.org.uk/map_new/map.htm [accessed 2003, April 30]. 6 Lehmann HP. Medical Informatics, Educational Technology, and the New Curriculum for the Twenty-First Century. Baltimore: Johns Hopkins University Press, 1998. 7 Hancock L. Physicians’ Guide to the Internet. Philadelphia: Lippincott-Raven, 1996. 8 Rizzolo LJ, Aden M, Stewart WB. Correlation of Web usage and exam performance in a human anatomy and development course. Clinical Anatomy 2002; 15: 351–5. 9 Wheeler DW, Remoundos DD, Whittlestone KD, House TP, Menon DK. Medical students are confused by different means of expressing the concentration of drugs in solution. Drug Safety 2003; in press. 10 Prystowsky JB, Regehr G, Rogers DA et al. A virtual reality module for intravenous catheter placement. American Journal of Surgery 1999; 177: 171–5. 11 Kurrek MM, Devitt JH. The cost for construction and operation of a simulation centre. Canadian Journal of Anaesthesia 1997; 44: 1191–5. 12 Weller JM, Bloch M, Young S et al. Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists. British Journal of Anaesthesia 2003; 90: 43–7. 13 Myers H. Clinical Competency Exam (CCX) (CD-ROM). Tampa, FL: Gold Standard Multimedia, 2000. 14 Chou C. Case-based teaching of perioperative medicine. Academic Medicine 2001; 76: 558–9. 15 Kreiter CD, Ferguson K, Gruppen LD. Evaluating the usefulness of computerized adaptive testing for medical in-course assessment. Academic Medicine 1999; 74: 1125–8.
