ACAD EMERG MED
d
January 2005, Vol. 12, No. 1
d
45
www.aemj.org
EDUCATIONAL ADVANCES Online Bioterrorism Continuing Medical Education: Development and Preliminary Testing Thomas Terndrup, MD, Sarah Nafziger, MD, Norman Weissman, PhD, Linda Casebeer, PhD, Erica Pryor, RN, PhD Abstract Objective: Education to achieve awareness and competency in responding to incidents of bioterrorism is important for health care professionals, especially emergency physicians and nurses, who are likely first points of medical contact. The authors describe the development of a computer-based approach to initial education, incorporating a screensaver to promote awareness and a Web-based approach to provide initial content competency in the areas of smallpox and anthrax. Methods: Screensavers were developed and tested on emergency department rotating senior medical students and internal medicine interns. Conceptually, screensavers were designed as ‘‘billboards’’ for attracting attention to the educational domain. Five rotating images sequenced at fivesecond intervals incorporated a teaser question and an interactive toolbar. An interactive toolbar was linked to a Web site that provided content on smallpox and anthrax for hospital-based specialties (emergency physicians and
nurses, infection control practitioners, pathologists, and radiologists). The content included both summary and comprehensive content as well as free continuing education credits in an online, specialty-specific, case-scenario format with remediation pop-up boxes. Results: Formal testing indicated that the screensaver and Web site combination deployed on computers in the emergency department and the events of the fall of 2001 significantly increased the percentage of correct responses to five standardized bioterrorism questions. Formal evaluation with a randomized trial and long-term follow-up is ongoing. Conclusions: Screensavers and Web sites can be used to increase awareness of bioterrorism. Web-based education may provide an effective means of education for bioterrorism. Key words: education; Web site; bioterrorism; screensaver. ACADEMIC EMERGENCY MEDICINE 2005; 12:45–51.
Bioterrorism preparedness poses challenges associated with events that occur rarely but require immediate recognition and rapid action to prevent widespread outbreaks and to protect the health of the public. Before the events of the fall of 2001, only isolated cases of bioterrorism had occurred.1 These events have increased the awareness of bioterrorist threats and have highlighted the need for bioterrorism preparedness of health care providers.
Low knowledge and confidence survey scores among physicians and nurses on bioterrorism knowledge, awareness, and preparedness have indicated the need for continuing education and staff development in bioterrorism preparedness programs. A recent survey of family physicians demonstrated that while 95% believed that a bioterrorist attack is a real threat, only 26% thought they would know what to do and only 18% had received bioterrorism preparedness training.2 A recent survey of emergency physicians supported the development and implementation of comprehensive bioterrorism preparedness plans to resolve deficiencies, including physician education and training.1 A prospective, randomized trial of an educational Web site for educating physicians about bioterrorism demonstrated no increase in knowledge,3 but 30% of physicians randomized to the site did not access it despite weekly e-mail prompts. Of those who did access the site (total, 17), one third spent 20 minutes or less on the site. The purpose of this article is to discuss the development and preliminary evaluation of the effectiveness of an online continuing medical education (CME) course in improving recognition of potential bioterrorist agents among selected hospital-based specialties. Development of the impact of this approach was evaluated by testing correct responses on
From the Department of Emergency Medicine (TT, SN), Center for Emergency Care and Disaster Preparedness (TT), Department of Health Administration (NW), Division of Continuing Medical Education (LC), and School of Nursing (EP), University of Alabama at Birmingham, Birmingham, AL. Received April 19, 2004; revision received August 13, 2004; accepted August 17, 2004. Presented in part at the American Medical Informatics Association annual meeting, San Antonio, TX, November 2002; the SAEM annual meeting, St. Louis, MO, May 2002; the SAEM southeastern regional meeting, Jacksonville, FL, April, 2002; and the National Disaster Medical System national conference, Atlanta, GA, April, 2002. Supported by the Agency for Healthcare Research and Quality (contract no. 290-00-0022). Address for correspondence and reprints: Thomas Terndrup, MD, Department of Emergency Medicine, University of Alabama at Birmingham, JTN 266, 619 19th Street, Birmingham, AL 35249-7013. Fax: 205-975-4662; e-mail:
[email protected]. doi:10.1197/j.aem.2004.08.040
46
Terndrup et al.
a standardized examination given to short-term rotators in a university emergency department (ED).
METHODS Study Design. Our hypothesis was that awareness of bioterrorism could be improved in rotating first-year houseofficers and senior medical students by placement of a screensaver on hospital ED computers. In July 2001, we assembled a multidisciplinary team with expertise in educational methodology, distance education, continuing education, health services research, infection control, and the hospital-based medical specialties of radiology, pathology, and emergency medicine. We developed a screensaver to attract attention of providers using ED desktop computers in order to increase awareness of bioterrorism issues and developed a Web site with detailed content related to bioterrorism. These studies were approved by the university’s institutional review board for human use and were exempt from informed consent. Study Protocol. The screensavers were developed and modified by the multidisciplinary group. They were designed to incorporate a ‘‘billboard’’ effect with a captivating image or photomicrograph with a ‘‘teaser question’’ that might lead passersby to look more carefully (Figure 1). A series of five images was displayed with an interactive toolbar set to hold the image when the spacebar was depressed to allow more careful examination of the image. Otherwise, the screensaver images would turn on after an absence of keyboard or mouse activity for 30 seconds and were turned off by a user pressing the escape key. The screensaver uses five smallpox or anthrax images, in-
d
BIOTERRORISM WEB EDUCATION
cluding one image of a pathology slide and one radiology image, rotating at a frequency of five seconds. The software components developed were designed to be compatible with the Windows 95, 98, 2000, and NT operating systems. In addition, each software component complied with the Windows 2000 design guidelines. They were designed for computers with a minimum of 32 MB of RAM and 6 MB of free disk space on hard drives for program files. Files and source codes were provided to university personnel so that ongoing content modifications could be made following program development. All content is housed on the university’s continuing education Web server and may be viewed at www.bioterrorism.uab.edu. The screensaver incorporated an interactive toolbar that hyperlinked the user to a Web site featuring selected content on smallpox and anthrax (Figure 2). The site provided both a continuing education section with remediation (i.e., pop-up boxes were used for feedback and remediation on practice examinations) and a formal continuing education site where students and practitioners submitted answers to case-scenario questions for CME credit. Additional sections included answers to the ‘‘teaser questions’’ from the screensavers, summary and comprehensive information on smallpox and anthrax, a syndromics table for category A diseases, and a list of contributors page. For CME, users were asked to designate a specialty from one of the following: emergency nurse, emergency physician, pathologist, radiologist, and infection control. This Web site has been continuously available since October 18, 2001. Although the screensaver is available for download, it is not required for entry to the Web site alone. Modifications to meet federal
Figure 1. Screenshot of one of five rotating images for the screensaver tested. The question at the top is intended to generate further interest on the part of individuals who observe the image. The interactive toolbar allows the individual to hyperlink to the Web site for answers or more information as well as to pause the rotating images for more careful examination.
ACAD EMERG MED
d
January 2005, Vol. 12, No. 1
d
47
www.aemj.org
regulations for accessibility and content were added in the fall of 2002, and updates to content were delivered in August and October 2003 (www.bioterrorism-uab. ahrq.gov/). In meeting the federal restrictions to become a ‘‘.gov’’ Web site, limited information was then available on those who interact at the Web site. Beginning in the fall of 2003, additional specialties were added for providers specializing in pediatrics, family medicine, and internal medicine. In addition, a dermatology referral module for all primary care physicians was added in December 2003. Measures. Testing of the screensaver and Web site was undertaken in the ED of an urban tertiary care teaching hospital with around 50,000 ED visits each year. First-year housestaff in internal medicine and fourth-year medical students perform one-month rotations in the ED and are given a pretest before and an identical posttest at the completion of their rotation. Beginning in March 2001, the pretest and posttests given to these students were amended to include five additional test questions on bioterrorism, specifically on anthrax and smallpox. Test questions were in positively worded multiple-choice format, requiring the learner to choose the correct response from five possible answers. When appropriate, clinical case vignettes were used in the stem portion of the question. Correct responses were reviewed with rotators following completion of the posttest. We defined the ‘‘baseline’’ data collection phase as March 2001 to September 2001, the screensaver/Web ‘‘trial’’ phase as October 2001 to March 2002, and the ‘‘posttrial’’ phase as April 2002 to January 2004. During the screensaver/Web phase (October 2001 to March 2002), screensavers were installed at nine of 11 com-
puter workstations in common areas of the ED. Throughout the course of the rotation, the students and housestaff received no other formal training to recognize or treat diseases that could result from bioterrorism and were not aware of the purposes of the study. Screensavers were placed on ED desktop computers, distributed throughout the ED nursing station, and removed from ED computers at the completion of the screensaver/Web phase of the study. Most of the charting, patient discussions, and patient presentations are performed in this area. Rotators were physically present in the ED about 42 hours per week and received five to seven introductory lectures on emergency medicine topics during the rotation. Comparison of the percentage of correct responses on the standardized 50-question examination was performed in the three time phases of the study. Only one faculty member and a research assistant were aware of the purpose of the screensaver and Web site implementation. Data Analysis. For statistical analysis, the chi-square test was used for proportional data with significance assumed at p , 0.05. No formal power analysis was performed for hypothesis testing, because this was a series of developmental studies and no prior information was available on the impact of screensavers and Web sites from which to calculate estimated probabilities.
RESULTS Data on performance of standardized testing of ED rotators are presented (Table 1). During the baseline, trial, and posttrial phases, 53, 55, and 49 students and interns, respectively, rotated through the department
Figure 2. The entry page to the Web site, showing an answer to one of the screensaver questions (pop-up box, right-hand side). The menu of Web site options is shown on the left.
48
Terndrup et al.
d
BIOTERRORISM WEB EDUCATION
TABLE 1. Comparison of Correct Responses for Study Phases Pre-rotation Examination Phase (dates), number tested No. Correct/No. Possible EM questions Baseline (3/01–9/01), n=53 Trial (10/01–3/02), n=55 Posttrial (4/02–1/04), n=49 BT questions Baseline (3/01–9/01), n=53 Trial (10/01–3/02), n=55 Posttrial (4/02–1/04), n=49 Trial vs. baseline Posttrial vs. trial
Post-rotation Examination %
No. Correct/No. Possible
%
% Change
1,516/2,650 1,955/2,750 1,751/2,450
57.2 71.1 71.4
1,682/2,650 2,141/2,750 1,858/2,450
63.5 77.9 75.8
6.3 6.8 4.4
108/265 165/275 146/245
40.8 60.0 59.6
137/265 205/275 156/245
51.7 74.6 63.7
11.0 14.6 4.1 14.6 210.4
95% CI (%)
p-value
3.6, 8.9 4.5, 9.1 1.9, 6.8
,0.0001 ,0.0001 0.0005
2.5, 19.4 0.0115 6.8, 22.3 ,0.001 24.5, 12.7 0.35 22.1, 9.2 0.21 215.3, 25.6 ,0.0001
Frequencies are indicated as correct responses/total possible responses. EM = general emergency medicine questions; BT = bioterrorism questions on smallpox or anthrax.
and completed pretesting and posttesting. During the baseline period, rotators on average answered 28.60 emergency medicine questions and 2.04 bioterrorism questions correctly on the pretest and 31.74 emergency medicine questions and 2.58 bioterrorism questions correctly on the posttest. During the trial period, rotators on average answered 42.50 emergency medicine questions and 3.59 bioterrorism questions correctly on the pretest and 46.54 emergency medicine questions and 4.46 bioterrorism questions correctly on the posttest. During the posttrial period, rotators on average answered 33.04 emergency medicine questions and 2.75 bioterrorism questions correctly on the pretest and 35.06 emergency medicine questions and 2.94 bioterrorism questions correctly on the posttest. When comparing prerotation with postrotation testing, during all phases of the study, rotators demonstrated significant improvements in the raw number and percentage of correct responses to general emergency medicine questions (range, 4.4%–6.8%; p , 0.05 for all phases of the prerotation/postrotation comparisons). When comparing prerotation with postrotation testing, during the baseline and screensaver/Web phases, rotators demonstrated significant improvements in the percentage of correct responses to the bioterrorism questions (range, 4.1%–14.6%; p , 0.05). A non–statistically significant improvement in performance on the bioterrorism questions was also observed in the posttrial period (4.1%; p = 0.35). The largest percentage increase of correct responses on the bioterrorism questions occurred during the trial period, with the second largest percentage increase occurring during the baseline period, the latter aspects of which coincided with the national events of September 2001. The trial phase did not achieve significant difference compared with baseline but did so during the posttrial phase.
DISCUSSION These developmental data indicate that a rotating screensaver linked to Web-based education may enhance awareness and knowledge of medical students
and houseofficers who are exposed to the screensaver as measured by standardized bioterrorismoriented questions on smallpox and anthrax. Also, a four-week ED rotation results in significantly improved performance on a standard question set in senior medical students and first-year internal medicine houseofficers. Compared with other studies of computer-based education for bioterrorism, this study provides useful information on the impact of screensavers. Almost no prior data are available on the impact of these inexpensive educational tools, although they are widely used by many industries. While ‘‘pop-up’’ images on computer screens are widely disseminated by advertisers, there is almost no formal testing of their educational translation. The only publication on screensavers4 utilized a CD-ROM for providing interactive content on domestic violence, and it is not clear from the article whether an actual screensaver was utilized for teaching. Our data indicate improved performance on five standardized examination questions on smallpox and anthrax, even though no additional formal training beyond the screensavers and Web site was offered. We have no information on what caused the rotators to have an enhanced performance, whether they accessed the Web site itself for more information, might have been stimulated to look up the correct responses to the five questions on bioterrorism, or some other explanation. Possible explanations for improvement include the ED rotation itself, informal discussions after seeing the screensavers, outside reading, national events, or other mechanisms. Instructional methods using the Internet have produced variable results. A recent meta-analysis of Webbased medical education identified 206 papers since the first report in 1992.5 The investigators concluded that the evidence suggests Web-based learning is comparable but not superior to traditional educational methods. Learners report high satisfaction, as long as modem speed is adequate. But admittedly, most studies introduced significant selection bias, because participants are already Web users. Importantly, there
ACAD EMERG MED
d
January 2005, Vol. 12, No. 1
d
49
www.aemj.org
may be greater efficiency of learning with similar amounts of information but more rapidly with Internet approaches.5 Greater learning efficiency was also reported by Bell et al. in their randomized, controlled trial of printed materials compared with a Web-based tutorial system among 162 resident physicians.6 We think of this approach as using technology that is simple, inexpensive, and ubiquitous. Costs remain minimal to develop and distribute, because none of these capabilities require sophisticated software or lengthy periods of development. Compared with live educational sessions, the cost-effectiveness and reach of this educational strategy becomes even more apparent when the direct costs and opportunity costs of participants are factored in. With Web access available for virtually all clinicians, traditional educational delivery systems of live instruction, videotape, audiotape, disk-based computer-assisted instruction, and CD-ROMs have depended on producing an educational product and mass mailing or creating other delivery systems for the instruction itself. The Internet itself has become a very efficient educational delivery system. The tools proposed for this project will travel easily over either modem or broadband for automatic installation without any need for the user to take actions to access them. Clinicians have identified credibility of medical information on the Internet as their single largest concern. The most important barrier for primary care physicians to finding what they are looking for in medical information on the Internet is too much information to scan and not having adequate search skills to sort through all of the information. Our Web site provided comprehensive, compact summaries on the disease, diagnosis, and treatment of both anthrax and smallpox. It was designed and created for busy health care professionals to provide the user with quick one- or two-page summary information on anthrax and smallpox, easy access when connected to the Web, and accurate, updated information. If the user had additional time for review, the site provides in-depth information on these diseases, including a differential diagnosis table, a syndromics chart for diagnosing categories of rare infections, and the opportunity to complete a case-based learning module and receive one hour of Category 1 CME credit at no cost. Compared with other studies of Web-based education, this study demonstrates developmental data and processes that may help to guide future medical education using the Internet.
LIMITATIONS The most significant limitation to this study is that we have no mechanism to measure the impact of the events of September and October 2001 on testing performance. While the trial period combining screensaver and Web access demonstrated the highest
percentage of correct responses to bioterrorism questions among the participants, we are unable to separate out the effects of these very significant events. Further, the medical student population that chooses to perform an emergency medicine elective is self-selected in this situation, and they may have been highly motivated to acquire information on potential bioterrorist agents. However, we would not have anticipated such a large difference in overall performance as was observed during the trial period among the student population. While we were able to verify that the Web site was heavily utilized during the trial period, we have no data regarding actual Web site usage specifically by ED rotators and therefore cannot conclude that improvements in performance are related solely to implementation of screensavers linking to our Web site.
CONCLUSIONS Future expansion of the Web site and screensaver will continue to focus on the dual-use applications regarding public health and bioterrorism. As our modules reflect, we believe that the more prepared our health care professionals are in recognizing and responding to a natural public health disaster, the more efficient and effective their response will be during an actual act of bioterrorism (e.g., in distinguishing chickenpox vs. smallpox, influenza vs. inhalation anthrax, etc.). Thus, if resources permit, we plan to expand upon our current content information and add tularemia, botulism, plague, and viral hemorrhagic fevers—the four additional category A diseases as defined by the Centers for Disease Control and Prevention—and category B agents, such as ricin toxin and Brucella. We also intend to add information on emerging infectious diseases such as sudden acute respiratory syndrome, monkeypox, and West Nile virus, which are presently of importance. The authors wish to acknowledge the absolutely incredible contributions of Ms. Emily Heck and Ms. Margaret Tresler, who have been outstanding in every detail of this project. We also express our sincere appreciation to Drs. H. Michael Maetz, David R. Franz, Kevin Moye, Steve Baldwin, Bonnie Elewski, and Michael Jacobs for their assistance with subject matter expertise, and to Dr. Shimin Zheng for his assistance with statistical analysis. Finally, we are indebted to individuals of the Health Services Research and Development Deep South Center on Effectiveness at the Birmingham Veterans Administration Medical Center, VHA Clinicians and Bioterror Events: Interactive Web-Based Learning, Project #BTI 02-0292.
References 1. Crupi RS, Asnis DS, Lee CC, Santucci T, Marino MJ, Flanz BJ. Meeting the challenge of bioterrorism: lessons from West Nile virus and anthrax. Am J Emerg Med. 2003; 21:77–9. 2. Chen FM, Hickner J, Fink KS, Galliher JM, Burstin H. On the front lines: family physicians preparedness for bioterrorism. J Fam Pract. 2002; 51:745–50.
50
Terndrup et al.
3. Chung S, Mandl KD, Shannon M, Fleisher GR. Efficacy of an educational web site for educating physicians about bioterrorism. Acad Emerg Med. 2004; 11:143–8. 4. Hoffman E. Domestic violence screen saver. Hospitals and Health Networks. 1998; Feb: 37–9.
d
BIOTERRORISM WEB EDUCATION
5. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? A review of the evaluation literature. Acad Med. 2002; 77:S86–93. 6. Bell DS, Fonarow GC, Hays RD, Mangione CM. Self-study from web based and printed guideline materials. Ann Intern Med. 2000; 132:938–46.
d
REFLECTIONS Mentor in a Foreign Land In June 2004, I took a trip with the chairman of my department, Dr. Lewis R. Goldfrank, to lecture to various departments of emergency medicine in Korea. I was not invited to lecture, but Dr. Goldfrank kindly included me since my heritage is Korean and he thought that I would be able to assist in ideas to develop their research programs. I have known Dr. Goldfrank approximately ten years. The first time I met him, he gave me a life-changing opportunity. I was rotating through the emergency department (ED) as a surgical intern who had not matched in surgery. At the time, I was told that I was too quiet and mild-mannered to survive as a surgeon. Some equated their image of me as an Asian woman who is short in stature with a personality of someone who is meek, shy, and unassertive, even though this assumption is not accurate. I was considering my options for the following year. I had not considered emergency medicine until I rotated through the ED. Dr. Goldfrank took the time to meet with me during my rotation. During our meeting, his eyes glanced up from my CV and he simply stated, ‘‘Well, your CV tells me that perhaps general surgery is not what you were meant for. I see many items here related to public health and volunteering. Why don’t you think about joining us?’’ The following year, I was one of his residents at Bellevue. Now, ten years later, I was lecturing overseas with him. We visited a few different hospitals, and at each of the programs it was clear that emergency medicine in Korea was lagging behind the United States. We observed a lack of poison control centers, overcrowded EDs, lack of communication between hospitals, and overworked physicians. Women entering the specialty were few because it was considered ‘‘too difficult.’’ As we were introduced to various faculty and to audiences before lectures, my chairman was introduced as ‘‘the esteemed and famous toxicologist, author, physician, and chairman, Dr. Lewis Goldfrank,’’ an opening that he certainly deserved. I was uniformly introduced as ‘‘Miss Nancy Kwon,’’ ‘‘Nancy Kwon,’’ and occasionally just ‘‘Nancy.’’ Sometimes they did not introduce me at all, and Dr. Goldfrank would then step in to introduce me. I certainly do not expect an introduction of the same caliber as Dr. Goldfrank’s, but I am still a physician, an assistant clinical professor, and one of the co-directors of research in my program. Dr. Goldfrank easily recognized what was so glaring to me but not so apparent to the Korean physicians. He was always cognizant of introducing me and addressing my role in his department. The introductory etiquette of the host physicians did not anger me, as I understand the cultural differences between Koreans and Americans. Discrimination cannot be eliminated in a revolutionary manner, but attitudes can often change gradually through exposure and example. In some ways, the more obvious discrimination I experienced in the Korean culture was easier to manage than the more subtle discrimination that I face daily within the culture in which I live. Nonetheless, I greatly appreciated my mentor’s actions. I have heard Dr. Goldfrank speak numerous times over the years and I always listened intently, knowing the vast experience, knowledge, and passion for emergency medicine, toxicology, and public health that make him a leader. In Korea, he lectured in the same manner, but words cannot replicate the extent to which he inspired this new audience. He spoke from his heart on the immense effort it takes to create and develop a specialty in emergency medicine. Over the years he has struggled to create and develop emergency medicine and toxicology as recognized fields in medicine. The number of times his efforts have been rejected would have been sufficient excuse for anyone to quit, but Dr. Goldfrank only persisted more. It was only this past year, after decades of trying to create a Department of Emergency Medicine at Bellevue Hospital, that we officially became one. Now these emergency physicians were faced with similar challenges and were looking to Dr. Goldfrank for direction. They found the hope and direction they were seeking. In one of his first lectures in the new academic year that he gave to our residents, he quoted Theodore Roosevelt: The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who knows the great enthusiasms, the great devotions, and spends himself in a worthy cause: who at best, if he wins, knows the thrills of high achievement, and, if he fails at least fails daring greatly, so that his place will never be with those cold and timid souls who know neither victory nor defeat. Dr. Goldfrank is the embodiment of these words. He relayed this message using different words to the Korean physicians, but has always lived and acted by these words. Recognizing my own struggles with being a woman and a minority in medicine
