Evaluation of Educational Software - Semantic Scholar

Evaluation of Educational Software Titus K.L. Schleyer, D.M.D., Ph.D.; Lynn A. Johnson, Ph.D. Abstract: Evaluation is an important component of developing educational software. Ideally, such evaluation quantifies and qualifies the effects of a new educational intervention on the learning process and outcomes. Conducting meaningful and rigorous educational evaluation is difficult, however. Challenges include defining and measuring educational outcomes, accounting for media effects, coping with practical problems in designing studies, and asking the right research questions. Practical considerations that make the design of evaluation studies difficult include confounding, potentially small effect sizes, contamination effects, and ethics. Two distinct approaches to evaluation are objectivist and subjectivist. These two complement each other in describing the whole range of effects a new educational program can have. Objectivist demonstration studies should be preceded by measurement studies that assess the reliability and validity of the evaluation instrument(s) used. Many evaluation studies compare the performance of learners who are exposed to either the new program or a more traditional approach. However, this method is problematic because test or exam performance is often a weak indicator of competence and may fail to capture important nuances in outcomes. Subjectivist studies are more qualitative in nature and may provide insights complementary to those gained with objectivist studies. Several published examples are used in this article to illustrate different evaluation methods. Readers are encouraged to contemplate a wide range of evaluation study designs and explore increasingly complex questions when evaluating educational software. Dr. Schleyer is Associate Professor and Director, Center for Dental Informatics, School of Dental Medicine, University of Pittsburgh; Dr. Johnson is Associate Professor, Office of Dental Informatics, School of Dentistry, University of Michigan. Direct correspondence and requests for reprints to Dr. Titus Schleyer, Center for Dental Informatics, School of Dental Medicine, University of Pittsburgh, 3501 Terrace Street, Pittsburgh, PA 15261; 412-648-8886 phone; 412-648-9960 fax; [email protected]. Key words: education, evaluation, computers, educational software, methods, computer-assisted instruction Submitted for publication 5/5/03; accepted 9/5/03

E

ducational software (also called computeraided instruction [CAI], computer-assisted learning [CAL], instructional software, and computer-based training [CBT]) are often touted as a cure for various educational ills, such as high cost, low learner performance, and lack of faculty.1,2 Others perceive educational software as an “in” trend, illustrated by the growth of distance learning and educational technology initiatives worldwide.3-7 Universities, colleges, and schools often expend significant resources on educational software development, with much of the cost hidden rather than explicit. Whether that investment is worthwhile and what difference it makes, if it makes one at all, are questions not often pursued.8 As Laurillard said, “Research and development projects on educational media pay quantities of hard cash for development, lip service to evaluation, and no attention to implementation.”9 Evaluation is a labor- and resource-intensive undertaking. Developers who are happy to have finally burned their first CD-ROM or created their first website often lack the motivation, energy, and resources to conduct a full-scale evaluation.10 Yet evaluation is essential if we are to progress beyond the stage of merely experimenting with technology in health sciences education. We must determine

November 2003

■

Journal of Dental Education

whether the educational software we create achieves its goals and explore the reasons for success and failure. This article primarily addresses summative evaluation, which is used to assess the outcome or impact of a program after implementation. Summative evaluation determines the degree to which expected outcomes or goals have been achieved. These goals might include improvements in educational outcomes, such as better long-term retention of content by learners; economics, such as decreased cost of education per student; or efficiency, such as faster acquisition of learning content. Formative evaluation, on the other hand, evaluates the worth of a program while the program is under development and focuses primarily on the process.11 Formative evaluation, as the name implies, takes place during the development of an educational software program and consists of evaluations designed to help the developers plan the learning activity in the best way possible, including try-outs with users and reviews by experts. The purpose of this article is to help readers understand the important issues in designing summative evaluation studies. However, although it provides advice on study designs, it should not be

1221

viewed as a cookbook. We recommend that readers enlist the help of evaluation specialists (e.g., an individual with an advanced degree in instructional design) to determine goals, variables, and methods for specific evaluation studies and consult standard works in the field.12 Here, we will first examine barriers to and challenges for evaluation. Then, we will discuss objectivist and subjectivist approaches12 to evaluation; the two are philosophically different but can be complementary in nature. Finally, we will illustrate these approaches with case studies published in the literature.

Challenges for Evaluation There are five major challenges in evaluating educational software. Defining the outcome. Competencies are a primary outcome measure in dental education.13 They can range from the relatively basic, such as being able to identify a single tooth, to the very complicated, such as developing a comprehensive treatment plan for a difficult case. Anyone who has been involved in formulating competencies, learning goals, or course objectives is aware of the difficulty of defining educational outcomes precisely. Yet, defining educational outcomes clearly, unambiguously, and concisely is the first step toward assessing them. Measuring educational outcomes. Once educational outcomes have been defined, their attainment should be measured. The rich array of methods available for this purpose includes standardized tests, oral exams, assignments, direct observation, simulations, and competency exams on patients. Evaluating validity and reliability of assessments helps us develop, refine, or discard them. However, troubling observations regarding the measurement of educational outcomes continue to surface. Sometimes, two independent measures of the (supposedly) same educational outcome contradict each other. At other times, unexpected but important educational outcomes become apparent only by chance.8 Unfortunately, it is impossible to anticipate all educational outcomes, especially with novel educational interventions. Since we can measure only what we observe, important outcomes may often be missed. Sound educational assessment presupposes that 1) we have defined all relevant educational outcomes and 2) our measurement processes measure them in a valid and reliable manner. This is not always the case.

1222

Accounting for effects of the delivery medium. Many evaluation studies are flawed because they fail to separate the effects of the delivery method (that is, the medium) from those of the educational method(s) (defined by G Salomon, in Interaction of Media, Cognition, and Learning [1994], as “any way to shape information that activates, supplants, or compensates for the cognitive process necessary for achievement or motivation”). Distinguishing between the two is difficult, but essential. Let us assume that we compare three alternatives for teaching treatment planning to students. The first one is a traditional series of lectures about the didactic material, the second one a videotape of the same lectures, and the third one a case-based tutorial on the computer. In comparing the lecture with the videotape, we see that the instructional method (i.e., the externally paced, sequential presentation of didactic material) is the same and the delivery method (i.e., inperson lecture and video) is different. In comparing the lecture with the computer-based program, we see that both the instructional method and the medium are different. As opposed to the instructional method used in the lecture, the computer program is selfpaced and uses patient cases to teach the learning content. Richard Clark8,14 and others15 make the compelling argument that most observed differences are due to a change in the educational methods and are not due to the medium. Seventy years of educational research have produced no compelling evidence that media (such as film, video, computers, and various combinations of these techniques) cause learning increases under any conditions.14,16 However, media can affect the economics, logistics, and cognitive efficiency of learning, and studies quantifying these effects are entirely appropriate. Research has shown some counterintuitive effects of media on the motivation to learn. Surveys often find that students are excited about or positively inclined toward computerbased learning tools. However, there is evidence that their excitement about a new medium does not lead them to work harder.17,18 Practical problems in designing studies. Various study designs can be used depending on the circumstances. For instance, randomized experiments require that the intervention is well understood and that there is a hypothesis about causal effect.19 Observational or exploratory studies, on the other hand, may be appropriate when new methodologies or media are first evaluated and may help in formulating hypotheses to be tested in more rigorous experi-

Journal of Dental Education ■ Volume 67, Number 11

ments. Many practical impediments to designing optimal studies exist. Potential constraints include the size of available study groups, options for stratification, contamination effects, confounding factors, and ethics (such as withholding a potentially superior educational methodology from a group of students).15,20,21 Asking the wrong research questions. Much attention has been focused on this general question: Is computer-based instruction “better” than traditional instruction?22-25 Many studies that try to answer this question are randomized experiments with at least one intervention and one control group. The performance of the groups is measured with the same instrument, which in itself, as Friedman suggests,15 may invalidate the study design. Most randomized studies do not account for the effects of changes in the media and educational methods separately and are therefore confounded.8 Current evidence points to the fact that the question “Is CAI better than lecture?” might be the wrong question to begin with.22 Since most research designs appear inadequate for finding the answer to this question, it is not surprising that the literature provides only inconclusive evidence for the merits of educational software. These challenges, however, should not keep anyone from planning and conducting educational evaluation studies. They simply point out the need to define the evaluation objectives clearly; to use a study design that takes into account the evaluation objectives, the type, scale, and scope of the intervention, measurement issues, and practical limitations; and to be realistic about what can and cannot be inferred from the results of the study. In the next section, we discuss several concepts in evaluation and present different approaches to the design of evaluation studies.

Basic Concepts of Evaluation Evaluation requires well-planned, formal approaches to be successful. Teachers, researchers, external evaluators, and decisionmakers need evidence they can trust. Such evidence must be collected using systematic and scientifically rigorous approaches. Two fundamentally different approaches to evaluation are the objectivist and subjectivist approaches.12 Objectivist approaches attempt to measure variables of interest with the highest level of

November 2003

■


reliability and validity possible. They often use statistical methods to quantify the degree of and confidence in observed differences between variables. Subjectivist approaches, on the other hand, explore evaluation through qualitative methods. Subjectivist studies focus on “why,” while objectivist studies focus on “what” and “how much.” Subjectivist study designs tend to be radically different from objectivist designs, leading many researchers trained in objectivist methods to reject such designs out of hand. However, both approaches measure complementary aspects, and either method can uncover information hidden to the other. The methods described below are discussed in detail in Friedman and Wyatt.12

Objectivist Evaluation In general, there are two common approaches to objectivist studies: comparison-based and objectives-based. In comparison-based studies, two or more groups of subjects are compared on one or more dependent variables. (A dependent variable represents an outcome of interest that is a result of an intervention.) Usually, one group is exposed to the new educational resource, while the other receives the traditional method of instruction or a combination of the two. Many evaluation studies of educational software are comparison-based. As the discussion of the challenges of evaluation in education above has shown, this question may miss the point entirely. A test score may be a very weak proxy for the multitude of outcomes achieved in the learning process. The objectives-based approach addresses the degree to which a resource meets its designers’ objectives. This approach is useful when meaningful comparative experiences are not available or when the achievement of specific objectives is important. The main concern of this type of study is whether the resource is performing up to expectations, not whether the resource is outperforming what it replaced. Since objectivist studies attempt to measure one or more outcomes as precisely as possible, reliable and valid measures and measurement processes are important. We are implicitly comfortable with the results of many measurement instruments we use daily, such as a watch, an odometer, or a yardstick. However, many educational measurement instruments do not provide the same degree of validity and reliability. Many educational evaluations use instruments developed ad hoc (such as surveys and standardized tests) whose reliability and validity have

1223

not been assessed. The reliability and validity of the measurement process for educational outcomes should be assessed whenever possible. Such formal assessments are called measurement studies. A measurement study is typically followed by a demonstration study in which the actual outcomes are measured and analyzed. Friedman and Wyatt12 provide an excellent and in-depth discussion of measurement studies. Several measurement studies have been published recently in healthcare education.26-28 A general rule for all demonstration studies is that reliability and validity of the measurement process should be reported. This information allows readers to determine how confident they should be in the actual results. Ideally, the evaluation should be carried out by one or more individuals not involved with the design and development of the software. While this is often difficult to realize in practice, it minimizes potential bias and enhances the credibility of the study.29

Subjectivist Evaluation Subjectivist approaches fall into four categories: quasi-legal, art criticism, professional review, and responsive/illuminative.12 In the quasi-legal approach, proponents and opponents of a new program offer testimony in a mock trial format. A jury of independent experts then makes a decision about the merit of the program. The art criticism method uses a single, experienced critic to develop a formal evaluation of a program. Several critics might provide dissimilar critiques, but collectively may provide a very comprehensive evaluation of a program. The professional review approach employs a panel of experts who observe the users as they work with a program in the environment where it is installed. Because of the dynamics of interaction within the panel, new questions and issues may be discovered and pursued on the spot. Finally, the responsive/illuminative approach attempts to evaluate a program from the perspective of end users. Frequently, this method uses ethnographic or anthropological study designs.30 Importantly, research questions often evolve dynamically during such studies, as do requirements for the type and volume of information collected. In the next section, we review specific educational outcomes that we measure in dental education. We also present several sample evaluation studies that illustrate the concepts discussed above.

1224

Educational Evaluation Case Studies Typically, we evaluate two types of skills in dental education: cognitive and psychomotor. A competent or expert dentist integrates the two to render state-of-the-art patient care. Both are underpinned by knowledge of various types and depths—such as knowledge of basic concepts (such as anatomy, microbiology, and physiology), relationships, and problem-solving strategies. Current thinking in dental education focuses on competencies13 and emphasizes their evaluation. While a defined set of competencies is the end goal for practitioners,31 it is useful to evaluate the contribution of educational software to its foundations, namely knowledge, problem-solving skills, and psychomotor skills. Table 1 lists different types of educational software (including examples where available) and the learning outcomes they affect. In this section, we illustrate three study designs using published educational software evaluations from dentistry and medicine. These examples are not intended to provide a complete tutorial on evaluation study design. Readers interested in learning more about designing evaluation studies are referred to the seminal text by Friedman and Wyatt12 and to two comprehensive reviews of educational software in dentistry by Rosenberg et al. and Dacanay and Cohen.25,39

Combined Subjectivist Evaluation of a Computer-Based Histology Atlas Our first example illustrates a subjectivist study design. This study, which used the responsive/illuminative approach, evaluated the use of a computerbased histology atlas within a course called Organ Systems at Johns Hopkins University.40 This course integrated the teaching of histology and physiology. A computer-based histology atlas was used in one of four sections of a lecture/lab course on a rotating basis. One group of students at a time had use of the lecture, the microscope, and a computer. The other groups used the lecture and microscope only. A trained ethnographer observed both study groups during the eleven sessions of the course. The ethnographer recorded data about the physical lay-


Table 1. Types of educational software, the learning outcomes they affect, and examples published in the literature Type of Educational Software

Affected Learning Outcome

Examples

Tutorial

Knowledge

Drill Game Simulation (theoretical), case studies

Knowledge Knowledge/problem-solving skills Problem-solving skills

Simulation (practical)

Problem-solving/psychomotor skills

Oral anatomy,32 oral manifestions of HIV disease,33 mixed dentition analysis34 Not available Not available Geriatric patient simulations for dental hygiene,35 endodontic diagnosis,36 orthodontics37 DentSim38

out of people and objects; the working arrangements of the students; the language used by students; the activities of the students; and their use of the major learning aids, microscope, and computers. In addition, students and faculty completed a brief survey at the end of each session. Other outcome measures were test results, both theoretical and practical. Students who used the histology atlas reported increased satisfaction with the course. However, a core of students was opposed to using the computer. Students assigned the atlas also found their colleagues more helpful. Those students learned the material differently as evidenced by the complex relationship among the language necessary for communication, group size, interactivity, and learning environment. Instructors’ self-reports suggested that the atlas resulted in a mild decrease in the amount of time they spent helping at the microscope and a mild increase in the quality of questions students asked. No statistically significant differences in microscope skills of students were found after the introduction of the atlas. The study did not measure the effect of the atlas on learning and concluded with a discussion of implications for teachers. We chose to present this example first because it illustrates the richness of data to be gleaned with subjectivist methods. While many designs for evaluating the computer-based histology atlas are imaginable, it was the wide array of outcome measures, as well as the innovative approach of using an ethnographer to record most of the observations, that led to a comprehensive and rich appreciation of the effects of the atlas. The fact that the study did not include an objectivist assessment of the effect on learning is not necessarily a weakness. Designing such assessments is extremely difficult and may have been impossible in this case.

November 2003

■


Objectivist, Comparison-Based Evaluation of Computer-Based Simulations We now turn to an evaluation of a very different type: a comparison-based evaluation of patient simulations for dental hygiene.35 In this study, the investigators developed eighteen computer-based simulations of geriatric dental hygiene patients. Students progressed from the initial patient profile through a detailed investigation of the case to building a problem list. After consideration of behavioral modifiers, students developed a management plan. A final discussion compared the learner’s decisions with those of the authors. The program included decision trees to teach problem-solving skills rather than factual knowledge. In the summative evaluation, the program was used in associate and baccalaureate dental hygiene programs at four institutions. One group received no instruction on geriatric patients, another received a traditional course on geriatrics, and two groups completed ten computer-based patient simulations. An identical fifty-item pre- and post-test served as the evaluation instrument. The group that received no instruction on geriatric patients showed no significant improvement in the test scores, while the remaining groups did. Because of logistical constraints, study participants were not randomized. This use of intact groups was a major limitation of the study. While this study illustrates some of the practical limitations and constraints investigators face, it also highlights the larger problem of designing evaluations for higher-order learning objectives (such as treatment planning). A multiple-choice test may fail to capture many of the nuances and skills necessary for good clinical decisionmaking. One alternative

1225

would have been to evaluate students’ performance on a new set of simulated cases. However, in that case the confounding effect of the control group’s lack of exposure to the mechanics of the computerbased simulation would have had to be addressed. In addition, the study design not only entailed a change in media, but also in educational methods, and is therefore likely to be confounded.

Objectivist, Comparison-Based Evaluation of a Virtual RealityBased Simulator An evaluation of a virtual reality-based simulator for the diagnosis of prostate cancer illustrates a very different evaluation problem and strategy.41 The only simulation resembling this application in dentistry is DentSim by DenX Corp.38 Unfortunately, no studies have been published about DentSim to date. One technique used to help detect prostate cancer is the digital rectal exam (DRE). The DRE simulator included a haptic (relating to or based on the sense of touch) interface that provides feedback to the trainee’s index finger, a motion restricting board, and a Silicon Graphic, Inc., workstation that renders the patient’s anatomy. Four types of prostates were modeled—normal, enlarged with no tumor, incipient malignancy (single tumor), and advanced malignancy (tumor cluster). After only five minutes of training, nonmedical students had a 67 percent correct diagnosis rate of malignant versus nonmalignant cases. This compared with 56 percent for urology residents who also used the DRE simulator. A control group of urology residents performed the same trials on a different simulator. The control group scored significantly better (96 percent correct diagnosis of malignancies) than either intervention group. The study concluded that the virtual prostate palpation simulator, while promising, needed significant improvement in both model realism and haptic interface hardware. In this study, subjects performed a skill in a similar way as they would with a live patient. The urology residents who participated in the study drew upon relatively significant experience with patients and may have scored lower because the simulator did not feel “real” enough. In the subjective evaluation, however, they indicated that the virtual reality simulator may be an important tool in practicing DREs.

1226

These three studies present a small selection from a large universe of projects. Additional examples are listed in the reference section.32-37,42-60

Conclusion Educational software is here to stay, and evaluating its effectiveness is an important goal in advancing the state of the art in this field. Many schools invest heavily in educational technology, and such investments should be founded on a good understanding of real and expected effects. A progressively better understanding of what works and what does not will ultimately allow us to improve the outcomes, economics, and efficiency of learning. However, it seems that dental educators have, for the most part, been pursuing the question of whether educational software results in better learning outcomes than other methods in too general a manner. The effects of the media and the effects of the educational methods confound each other, and we therefore need to formulate research questions and apply study designs that clearly separate the two.8 That may not always be possible. Friedman15 suggests types of investigations that may bring about a better understanding of the use and effects of educational software. First, instead of comparing computer-based interventions with other media (and facing fundamental evaluation challenges), different approaches to the design of computer-based instruction could be evaluated. Second, relatively few studies have evaluated how learners use software. Today’s software tools open rich possibilities to track a user’s behavior within a program. Mining such data may lead to a better understanding of how learners receive and process information. Third, new instructional methods (such as threedimensional simulators) require new methods of testing, possibly in an identical environment. Last, Friedman recommends that we begin to focus attention on issues of integration of educational software with curricula and programs, not only courses. Computer technology, broadly applied throughout a curriculum, may have subtle effects that are not obvious from the outset. Computer science might provide an enlightening parallel to our quest to establish the merits of educational software. In the early history of artificial intelligence (AI), many expected AI systems to


perform many functions routinely associated with human intelligence. When those capabilities failed to materialize, the initial enthusiasm for AI systems quickly turned into disenchantment. But AI did not disappear. Instead, researchers developed a myriad of AI applications of limited scale and scope that nevertheless outperform the human mind—in relatively narrow functions. History may repeat itself, and educational software may simply become part of the fabric of education instead of replacing a whole swath of it.

Acknowledgments We are grateful for the assistance of Drs. C. Friedman and H. Spallek in the preparation of this article. We also would like to thank the reviewers for their detailed comments and Ms. Andrea Hyde for her assistance with the bibliography.

REFERENCES 1. The web-based education commission: the power of the Internet for learning. At: www.ed.gov/offices/AC/WBEC/ FinalReport/index.html. Accessed: October 24, 2003. 2. Alexander S, Hedberg J. Evaluating technology-based learning: which model? In: Beattie K, McNaught C, Wills S. Interactive multimedia in education: designing for change in teaching and learning. Amsterdam: Elsevier Science B.V., 1994. 3. Lord M. Getting a degree by e-mail. US News and World Report, October 30, 1005, 91-3. 4. Daniel JS. Why universities need technology strategies. Change, July/August 1997, 11-7. 5. Harasim L. A framework for online learning: the virtualU. Computer 1999;32(9):44-9. 6. Carnevale D. U of Maryland university college forms forprofit company for on-line courses. Chron Higher Educ December 7, 1999. 7. Loose C. Online education to be free. Washington Post Online 3-14-2000:1-4. At: n/.newsbank.com/n/search/we/ Archives?p_action-list&p_topdoc-21. Accessed: October 24, 2003. 8. Clark RE. Evaluating distance education: strategies and cautions. Quart J Distance Educ 2000;1(1):1-15. 9. Laurillard D. Rethinking university teaching: a framework for the effective use of educational technology. London: Routledge, 1993. 10. Geissinger H. Educational software: criteria for evaluation. At: www.ascilite.org.au/conferences/perth97/papers/ Geissinger/Geissinger.html. Accessed: October 24, 2003. 11. Bhola HS. Evaluating “Literacy for development” projects, programs and campaigns: evaluation planning, design and implementation, and utilization of evaluation results. Hamburg, Germany: UNESCO Institute for Education; DSE [German Foundation for International Developement], 1990.

November 2003

■


12. Friedman CP, Wyatt, JC, Evaluation methods in medical informatics. New York: Springer, 1996. 13. Chambers DW, Glassman Paul. A primer on competencybased evaluation. J Dent Educ 1997;61(8):651-66. 14. Clark RE. Media will never influence learning, 1993. At: www.usq.edu.au/material/unit/resource/clark/media.htm. Accessed: October 24, 2003. 15. Friedman CP. The research we should be doing. Acad Med 1994;69(6):455-7. 16. Kozma RB. Will media influence learning? reframing the debate. Educ Tech Res Dev 1994;42(2):7-19. 17. Kozma RB. Learning with media. Rev Educ Res 1991;61(2):179-212. 18. Salomon G. Television is easy and print is “tough”: the differential investment of mental effort in learning as a function of perceptions and attributions. J Educ Psych 1984;76(4):647-58. 19. Haertel G, Means B. Stronger designs for research on educational uses of technology: conclusion and implications. SRI International, October 17, 2000. 20. Clark RE. Dangers in the evaluation of instructional media. Acad Med 1992;67(12):819-20. 21. Clark RE. Confounding in educational computing research. J Educ Comput Res 1985;1:28-42. 22. Ehrmann SC. Asking the right question: what does research tell us about technology and higher learning. Change March/April 1995:20-7. 23. Cohen PA, Dacanay LS. A meta-analysis of computerbased instruction in nursing education. Comput Nurs 1994;12(2):89-97. 24. Twigg CA. Innovations in online learning: moving beyond no significant difference. Monograph 4. Troy, NY: Center for Academic Transformation, The Pew Learning and Technology Program, 2001. 25. Rosenberg H, Grad HA,Matear DW. The effectiveness of computer-aided, self-instructional programs in dental education: a systematic review of the literature. J Dent Educ 2003;67(5):524-32. 26. Cork RD, Detmer WM, Friedman CP. Development and initial validation of an instrument to measure physicians’ use of, knowledge about, and attitudes toward computers. J Am Med Inform Assoc 1998;5(2):164-76. 27. Copeland HL, Hewson MG. Developing and testing an instrument to measure the effectiveness of clinical teaching in an academic medical center. Acad Med 2000;75(2):161-6. 28. Schleyer TK, Humberto T, Gadd CS. Validation of an instrument to measure students’ use of, knowledge about, and attitudes towards computers. J Dent Educ 2001;65(1):50. 29. Cronbach LJ. Designing evaluations of educational and social programs. San Francisco: Jossey-Bass, 1982. 30. Bernard HR. Research methods in anthropology. 2nd ed. Walnut Creek, CA: Altamira Press, 1995. 31. AADS House of Delegates. Competencies for the new dentist. J Dent Educ 1997;61(7):556-8. 32. Bachman MW, Lua MJ, Clay DJ, Rudney JD. Comparing traditional lecture vs. computer-based instruction for oral anatomy. J Dent Educ 1998;62(8):587-91.

1227

33. Porter SR, Telford A, Chandler K, Furber S, Williams J, Price S, et al. Computer assisted learning (CAL) of oral manifestations of HIV disease. Br Dent J 1996;181(5):173-7. 34. Irvine NR,Moore RN. Computer-assisted instruction in mixed dentition analysis. J Dent Educ 1986;50(6):312-5. 35. Johnson LA, Cunningham MA, Finkelstein M., Hand JS. Geriatric patient simulations for dental hygiene. J Dent Educ 1997;61(8):667-77. 36. Fouad AF, Burleson JA. Effectiveness of an endodontic diagnosis computer simulation program. J Dent Educ 1997;61(3):289-95. 37. Luffingham JK. An assessment of computer-assisted learning in orthodontics. Br J Orthod 1984;11(4):205-8. 38. Dentsim. At: www.denx.com/dentsim_system_desc.html. Accessed: October 24, 2003. 39. Dacanay LS, Cohen PA. A meta-analysis of individualized instruction in dental education. J Dent Educ 1992;56(3):183-9. 40. Lehmann HP, Freedman JA, Massad J, Dintzis RZ. An ethnographic, controlled study of the use of a computerbased histology atlas during a laboratory course. J Am Med Inform Assoc 1999;6(1):38-52. 41. Burdea G, Patounakis G, Popescu V, Weiss RE. Virtual reality-based training for the diagnosis of prostate cancer. IEEE Trans Biomed Eng 1999;46(10):1253-60. 42. Long AF, Mercer PE, Stephens CD, Grigg P. The evaluation of three computer-assisted learning packages for general dental practitioners. Br Dent 1994;177(11-12):410-5. 43. Matthew IR, Pollard DJ, Frame JW. Development and evaluation of a computer-aided learning package for minor oral surgery teaching. Med Educ 1998;32(1):89-94. 44. Arene I, Ahmed W, Fox M, Barr CE, Fisher K. Evaluation of quick medical reference (QMR) as a teaching tool. MD Comput 1998;15(5):323-6. 45. Chan DH, Leclair K, Kaczorowski J. Problem-based small-group learning via the Internet among community family physicians: a randomized controlled trial. MD Comput 1999;16(3):54-8. 46. Lyon HC Jr, Healy JC, Bell JR, O’Donnell JF, Shultz EK, Moore-West M, et al. PlanAlyzer: an interactive computerassisted program to teach clinical problem solving in diagnosing anemia and coronary artery disease. Acad Med 1992;67(12):821-8. 47. Halloran L. A comparison of two methods of teaching: computer managed instruction and keypad questions versus traditional classroom lecture. Comput Nurs 1995;13(6):285-8.

1228

48. Lindquist TJ, Clancy JM, Johnson LA, Wiebelt FJ. Effectiveness of computer-aided removable partial denture design. J Prosthodont 1997;6(2):122-7. 49. Marsh CM, Hannum WH, Trotman CA, Proffit WR. Design and effectiveness of a computer-based continuing education program for orthodontists. Angle Orthod 2001;71(1):71-5. 50. Ohrn MA, van Oostrom JH, van Meurs WL. A comparison of traditional textbook and interactive computer learning of neuromuscular block. Anesth Analg 1997;84(3):657-61. 51. Desch LW, Esquivel MT, Anderson SK. Comparison of a computer tutorial with other methods for teaching wellnewborn care. Am J Dis Child 1991;145(11):1255-8. 52. Horn DL, Radhakrishnan J, Saini S, Pepper GM, Peterson SJ. Evaluation of a computer program for teaching laboratory diagnosis of acid-base disorders. Comput Biomed Res 1992;25(6):562-8. 53. Grundman JA, Wigton RS, Nickol D. A controlled trial of an interactive, web-based virtual reality program for teaching physical diagnosis skills to medical students. Acad Med 2000;75(10 Suppl):S47-S49. 54. Clark RD, Weekrakone S, Rock WP. A Hypertext tutorial for teaching cephalometrics. Br J Orthod 1997;24(4):3258. 55. Maki WS, Maki RH. Evaluation of a web-based introductory psychology course: II. contingency management to increase use of on-line study aids. Behav Res Methods Instrum Comput 2000;32(2):240-5. 56. Turner RE, Evers WD, Wood OB, Lehman JD, Peck LW. Computer-based simulations enhance clinical experience of dietetics interns. J Am Diet Assoc 2000;100(2):18390. 57. Carr MM, Reznick RK, Brown DH. Comparison of computer-assisted instruction and seminar instruction to acquire psychomotor and cognitive knowledge of epistaxis management. Otolaryngol Head Neck Surg 1999; 121(4):430-4. 58. Bauer MD, Huynh MV. Nursing students’ blood pressure measurement following CD-ROM and conventional classroom instruction: a pilot study. Int J Med Inf 1998;50(13):103-9. 59. Plasschaert AJ, Cailleteau JG, Verdonschot EH. The effect of a multimedia interactive tutorial on learning endodontic problem-solving. Eur J Dent Educ 1997;1(2):66-9. 60. Santer DM, Michaelsen VE, Erkonen WE, Winter RJ, Woodhead JC, Gilmer JS, et al. A comparison of educational interventions. multimedia textbook, standard lecture, and printed textbook. Arch Pediatr Adolesc Med 1995;149(3):297-302.
