Informatics and Knowledge Translation Michael J. Bullard, MD, Stephen D. Emond, MD, Tim A. D. Graham, MD, Kendall Ho, MD, Brian R. Holroyd, MD
Abstract To ensure that the benefits of knowledge translation synthesis are accessible to care providers at the point of decision-making, fast, efficient, usable clinical information systems are required. Medical informatics appears to hold the greatest promise to be able to create systems with the necessary capacity and functionality. Emergency medicine needs to be actively engaged at all levels of the process. This includes driving the development and filtering of emergency-specific synopses and summaries. It requires advocating for hardware and software that suit the needs of the emergency department environment. It is increasingly important to educate and participate on committees with funders and policy-makers to ensure they support this growing evolution. To determine the outcome of these initiatives, careful evaluation is required to inform the discussion. End-users need to be actively involved in the development and usability testing of clinical information retrieval technology and clinical decision-support systems and make certain relevant best evidence is readily accessible and formatted to meet the needs of the working emergency physician. The integration of knowledge translation into clinical practice, and the impact of delivering electronic clinical decision-support, requires methodologically sound studies to confirm or refute its benefits and guide future development of medical informatics. ACADEMIC EMERGENCY MEDICINE 2007; 14:996–1002 ª 2007 by the Society for Academic Emergency Medicine Keywords: information dissemination, informatics, emergency medicine, clinical decision-support systems, information systems, user-computer interface
T
his proceedings paper concentrates on the top tier of Hayne’s ‘‘5S pyramid’’ of organization of evidence.1 It will address the global question, ‘‘What are the characteristics and conditions required From the Department of Emergency Medicine, University of Alberta (MJB, TADG, BRH), Edmonton, Alberta, Canada; Stanford-Kaiser Emergency Medicine Residency Program and Department of Emergency Medicine, Kaiser Foundation Hospital (SDE), Santa Clara, CA; and Division of Emergency Medicine, University of British Columbia (KH), Vancouver, British Columbia, Canada. Received June 9, 2007; revision received June 17, 2007; accepted June 18, 2007. This is a proceeding from a workshop session of the 2007 Academic Emergency Medicine Consensus Conference, ‘‘Knowledge Translation in Emergency Medicine: Establishing a Research Agenda and Guide Map for Evidence Uptake,’’ Chicago, IL, May 15, 2007. Workshop participants included Michael J. Bullard, Stephen D. Emond, Tim A. D. Graham, Kendall Ho, Brian R. Holroyd, Jon Handler, Peter Pang, Roland Grad, and Todd Taylor. Google Group members included Carey Chisholm, Marcel Emond, Stephen Emond, V. Jane Findlater, Brad Gordon, Jen Greenier, Taj Hassan, Brian Holroyd, David Howse, Sam Keim, Gloria Kuhn, Jim McClay, Diane Muench, Ben Munger, Peter Pang, Shirley Ooi, Todd Taylor, and Ellen Weber. Contact for correspondence and reprints: Michael J. Bullard, MD; e-mail:
[email protected].
996
ISSN 1069-6563 PII ISSN 1069-6563583
for informatics to best support knowledge translation (KT) in the emergency department (ED)?’’ Internationally, there is a paradigm shift in health care toward a paperless clinical environment with an integrated electronic health record and electronic decision support as part of the overall strategy. Policy makers have determined that health care should be following the lead of other industries and that informatics is essential to the improvement of patient safety and quality of care delivered.2,3 Computer capacity for information storage far exceeds even the best human mind and can allow us access to the ever-expanding wealth of new knowledge in a way that no individual could ever hope to achieve. The challenge is for informaticians to create applications that understand and support the unique work environments of clinical care providers, allowing clinicians to focus more on becoming expert information interpreters and managers, rather than expert repositories of information. This article will explore what is needed to operationalize these challenges. At the 2004 Academic Emergency Medicine Consensus Conference, Handler et al.4 made a series of recommendations regarding computerized physician order entry and online decision-support and made recommendations on system design, potential workflow delays and introduction of new medical errors, and basic areas of decisionsupport. Key areas included access to best evidence and guidelines through Internet search access, computer application alerts identifying critical data such as drug
ª 2007 by the Society for Academic Emergency Medicine doi: 10.1197/j.aem.2007.06.032
ACAD EMERG MED
November 2007, Vol. 14, No. 11
www.aemj.org
interactions and critical laboratory values, and drug dosing calculators. In this article, a summary of the KT and informatics workshop session at the 2007 Academic Emergency Medicine Consensus Conference, we review potential impediments to informatics in the ED, including ED working conditions, clinical application design, computer knowledge and facility among health care providers, and a paucity of emergency medicine (EM)-specific synopses and evidence summaries. We make recommendations on accessibility and the value of reference libraries and guideline repositories within the clinical work environment and on the design, implementation, and support for evidence-based electronic decision support tools in the ED. Haynes et al.5 proposed that optimal application of evidence-based medicine should involve four main domains. 1. Clinical state and circumstance. Point-of-care knowledge of a patient’s relevant past history is supported by an accessible electronic health record. Helping limit the impact of geographic location on the availability of medical resources will require informatics, telemedicine, robotics, and other virtual technologies. 2. Patient preferences and actions. Studies on patientcentered management preferences, compliance, and impacts of noncompliance are electronically accessible and should be available for real-time guidance of patient care decisions. 3. Research evidence. Few clinicians can accurately recall number needed to treat or harm for most interventions; easily accessible and usable best research evidence should be integrated into ED clinical decision support-systems (CDSSs) to enhance informed decision making. 4. Clinical expertise. Although currently limited to the domain of clinician experience, cataloging a practitioner’s entire clinical experience is feasible and theoretically could inform a neural network, which may enhance a practitioner’s ‘‘clinical memory’’ by incorporating lessons learned into the management of future patients. RECOMMENDATION 1 Emergency department computer access and use continue to increase. Clinical users expressed concerns that they often lack necessary computer literacy, have inadequate input into the development and implementation of clinical applications, and have no easy access to filtered, best evidence synopses, which are believed to be key to informing clinical decision-making. A more complete understanding of user characteristics and concerns is urgently needed. Data on computer literacy and implementation of electronic health care systems are sparse. For example, a MEDLINE search combining ‘‘attitude to computers’’ and ‘‘emergency medicine’’ identified only two articles: one comparing personal digital assistants with paper texts and a second assessing a computer-aided instructional program for the pediatric ED.6,7 Similarly, a second search combining ‘‘computer literacy’’ and ‘‘emergency medicine’’ also brought up two articles. A
997
1993 survey reported that medical school computer training was variable and recommended implementing a residency-based computer curriculum.8 A 2002 study of 124 EM residents compared the results of a selfreported survey of computer proficiency with their performance on a computer skills test.9 The authors concluded that while EM residents had more access to computers and possessed some computer literacy, they were not facile with a commonly-used word processing application and could not conduct a comprehensive literature search. No data on computer literacy and use in the nonacademic setting were identified. Regarding access to high-quality evidence to guide medical decisions, a 2002 review of the Cochrane Database of Systematic Reviews found that one third had some relevance to EM and 12% had direct relevance; the authors considered the format of a Cochrane review difficult to use in the hectic ED environment.10 Confounding the issue, evaluations of the scientific quality of systematic reviews have often reported them as being low.11 Currently, no comprehensive, EM-specific, quality-filtered resource of synopses and summaries is easily available. Focus for Future Research and Development 1. What is the current level of computer literacy among emergency care providers? 2. How many emergency physicians have Web access in their ED, how efficiently can they use search strategies to answer clinical questions, and how frequently do they perform such searches during clinical shifts? 3. What are the perceived needs and expectations of medical informatics by front-line providers? 4. What is the current online availability of filtered, best evidence, weighted, EM-specific knowledge? 5. How does the current status of accessible EM-relevant synthesized evidence compare with user needs, and how can we develop a strategy to close that gap? RECOMMENDATION 2 While most physicians would agree in principle that having clinical information retrieval technology (CIRT) accessible at the point of care to rapidly provide best evidence answers to key clinical questions is beneficial, many current electronic applications fall short of this expectation, and even fewer are designed to improve physician efficiency in locating information. Ease of access, speed of use, clinical relevance, user-friendly formatting, and accuracy will be paramount to the successful integration of CIRT into practice within the often chaotic environment of the ED. While many ED patient encounters stimulate questions that demand answers, time pressure often leads to them being left unanswered. CIRT encompasses systems such as Internet-based digital libraries containing reference information that clinicians can consult. Although widely used, there have been few studies on the impact of CIRT on outcomes. Nearly one third of observational studies of the impact of CIRT reported a positive impact on clinical decision-making.12 In a 1996 article on accessing clinical information needs, the author suggested, ‘‘The tool must be able to answer highly complex
998
questions and so will have to be connected to a large valid database. Inevitably, it will be electronic, but it must be portable, fast, and easy to use.‘‘13 Internal medicine has conducted the majority of studies around the need for real-time answers to clinical questions. Surveys suggest that the need is increasing, but barriers exist: time pressure, cost, poor organization, lack of sources, and variable reliability of existing sources.14 In another study of attitudes toward real-time information use, physicians believed having that information would have influenced patient care 40% of the time.15 An early study on digital access reported that evidence searches taking longer than 25.4 seconds were not useful in a busy clinical environment.16 Recent research evaluated the use of a CIRT portal, VIVIDESK, a single sign-on comprehensive information environment including clinical, knowledge, teaching, safety, administrative, and communication resources with embedded ‘‘quick searches.’’ In the study, 123 internists and 90 residents accessed information resources 20,813 times while accruing 4,210 hours with knowledge-based applications. Residents were twice as likely as faculty to use the application at the point of care. Overall, 55% of the application episodes were completed within 5 minutes and 44% within 3 minutes.17 Attention needs to be paid to accessibility, speed, ease of use, and clinical utility for CIRT to become part of everyday practice in the ED setting. Focus for Future Research 1. What kind of clinical questions arise at the point of care? 2. What features and organization of CIRT are best suited to the ED environment to optimize usage? 3. Can CIRT actually improve quality of care? RECOMMENDATION 3 There is some evidence that CDSSs can improve delivery of evidence-based care and support patient safety. To support the development and maintenance of CDSSs, structured resources need to be available and regularly updated in a user- and informatics-friendly format. For the purpose of this consensus recommendation, CDSSs refer to electronic systems that provide access to health knowledge (including patient-specific and best evidence) as integrated decision-support tools to assist clinicians in making the best health care decisions for their patients at the point of care. Many studies report a variance between published best evidence and actual care delivered by health providers, indicating a need for clinician-friendly improvements in knowledge dissemination and implementation. A retrospective ED study of patients with severe sepsis and septic shock found compliance with ‘‘goal-directed therapy’’ guidelines to be poor. Only 12.5% of patients in shock had blood lactate levels measured, 32% received their first dose of antibiotics within the first three hours, 46.6% received aggressive fluid therapy, 43.3% of patients with an indication for vasoactive drugs received them, and no patient had central venous pressure monitoring in the ED.18 A study by Grad et al. evaluating the effectiveness of personal digital assistant–assisted evidence-based med-
Bullard et al.
KT AND INFORMATICS
icine access found that residents’ ability to estimate disease probability improved when they accessed and used clinical prediction rules. This finding was derived from a subgroup analysis and therefore requires further study.19 A systematic review looking at CDSSs concluded that several features were closely correlated with improved patient care. These included decision-support provided automatically, support delivered at the time and location of decision-making, actionable recommendations provided not just an assessment, and that the system was computer-based.20 CDSSs utilize patient-specific information and apply evidence or rules to help direct clinical care, but if they are not updated to reflect new knowledge, they will become outdated. A subclass of CDSSs has been labeled ‘‘evidence-adaptive CDSSs,’’ with clinical knowledge bases reflecting the most up-to-date evidence.21 The Cochrane Collaboration is a unique global organization that creates and maintains a large number of systematic reviews available to KT practitioners.22 An additional initiative, the Trial Bank Project proof of concept, was undertaken in collaboration with JAMA and Annals of Internal Medicine to enter the design, methods, and results of randomized controlled trials into standardized ‘‘trial banks’’ (http://rctbank.ucsf.edu/); a similar Global Trial Bank is under development with the support of the American Medical Informatics Association (http://www.amia.org/gtb/).23 Without better processes to synthesize new knowledge from clinical research, ensure it is relevant to the EM community, and then transfer it to clinicians and ED informaticians in a timely and usable format, the impact of CDSSs on care delivery will remain limited. Focus for Future Research 1. What is the role of CDSSs in improved safety and health outcomes? 2. Is it possible to decrease errors without decreasing clinician efficiency? 3. What is the impact of permitting application customization based on local practices/resources versus best evidence? RECOMMENDATION 4 Knowledge translation endeavors in EM should shed light on the overall system of practice and delivery of emergency medical care, as well as how individual emergency physicians are influenced by this system and vice versa. Making this a dynamic interaction between individuals and the system will illuminate innovative pathways and evaluative efforts to help synergy at the two levels to promote and nurture an effective and perpetual culture toward iteratively incorporating best evidence into routine clinical practice. How does research evidence get incorporated into mainstream medical practices? If today we have definitely established a new therapy to eradicate hepatitis C and, through research, identify no significant adverse effects (i.e., increased viral resistance, loss of herd immunity, and so on), how long will it take to entrench this practice to become the recognized standard of care? Then how long will patients need to wait before routine
ACAD EMERG MED
November 2007, Vol. 14, No. 11
www.aemj.org
application by all clinicians ensures consistent access to the new treatment? Reaching this KT goal (adopting knowledge into routine health practice) will take more than just educating health professionals and trainees to change their personal testing and prescribing behavior. Within the health system, government agencies need to evaluate and approve the therapy and health care budgets, and insurers need to determine whether or not to fund further testing and provide the medications. Finally, diffusion of this knowledge regarding safety and cost efficacy to the patient population are just some of the necessary steps before any broad implementation of this new knowledge into practice. Regular monitoring and surveillance of this new therapy for continuing efficacy, drug safety, and disease mutation as this therapy is introduced are key feedback mechanisms required to validate the original research and stimulate additional questions for future investigations.24 A 2003 conference on technology-enabled KT pointed out the importance of knowledge synthesis and the need to develop knowledge brokers to assist in obtaining and updating research into medical informatics.25 Focus for Future Research 1. What are the impacts and behavioral changes following access to KT, not only on individual clinicians but also on health systems, and what is the optimal interface between them? 2. Are the policies, practice, culture, and individual perceptions within medical organizations synchronized? 3. How do these components work together in the larger picture of EM practice? Lasting KT cannot occur without the synchronization of the system’s policy, practice, and culture with the behaviors and perceptions of the individuals within that system. Therefore, KT research should shed light on each of these components and the totality of how the components work together in the larger picture of EM practice. RECOMMENDATION 5 The ED environment is chaotic and often frenetic, differing in both the type and clinical approach to patients from other areas of the health care system. For informatics to be able to provide the best filtered information where and when it is needed to support clinical decision-making, system usability must be a priority. Clinical applications have often been designed with limited input from clinicians and inadequate testing in real-world settings, a practice analogous to piloting an airplane full of passengers without first ensuring the plane can fly. Koppel et al. evaluated the impact of a widely used computer physician order entry system and identified 22 unexpected system facilitated errors, most of which were the result of suboptimal design.26 There is a growing body of knowledge that highlights some important issues that affect system usability. Garg et al., in their systematic review of computerized CDSS, found that practitioner performance improved 64% of the time, most frequently in systems that automatically prompted users rather than requiring users to activate the system (73% vs. 47%; p = 0.02),27 and in
999
studies where the authors were also the CDSS developers (74% vs. 28%; p = 0.001). Bates et al. offered ‘‘ten commandments for effective clinical decision-support,’’ which included speed; real-time delivery; fitting user work flow; changing direction is better than stopping; and monitoring impact, getting feedback, and responding as being some of the key requirements to encourage user acceptance.28 For a medical CDSS to be adopted by end-users, it needs to be not only factually accurate but also designed in a way that clinicians will find pleasant to use; if people cannot or will not use a feature, it may as well not exist.29 Usability is a quality attribute relating to ease of use. More specifically, it refers to how quickly people can learn to use something, how efficient they are while using it, how memorable it is, how error prone it is, and how much it is liked. A structured set of methods developed in the field of usability engineering is achieving greater recognition and holds the promise to improve future clinical application design. Kushniruk et al. applied such methods to a handheld prescribing tool to predict aspects of system design that could lead to medical error.30 The testing of ten subjects identified 37 errors; 50% were slips (errors recognized and corrected before submitting) and 50% mistakes (unrecognized errors), with more than 90% of the errors due to interface problems. Other work has shown that layout and screen organization can have significant impact on physician behavior.31,32 It is paramount that CDSSs used in the ED (and elsewhere in the health care system) be subjected to rigorous testing early in the design phase and during iterative stages of software design and modification before implementation. Problems that can lead to adverse health outcomes and affect optimization of workflow in the ED should be identified while they are still easy and cost effective to fix, rather than postimplementation when costs are higher and new user comfort levels are fragile. Emergency department CDSSs must be easy to use and should be time saving or at least be time neutral. The goal needs to be a clear and intuitive user interface, and the focus should be on patient care, not cost savings. Clinicians should demand these features and processes when selecting a decision-support system. Focus for Future Research and Development 1. Is it possible to apply usability testing to identify and prevent technology-induced errors before implementation, while facilitating clinical efficiency? 2. What are the key education needs for clinical informatics users? 3. How can administrators and information technology leaders be engaged and educated on the importance of usability before purchasing or implementing clinical applications? RECOMMENDATION 6 Although technology is rapidly changing, current mobile hardware options are generally not designed to support ED work at the point of care, including insufficient battery technology able to power a computing device for a full shift. Wireless computing should be encouraged, while recognizing the importance of data security. Developing
1000
a strategy to encourage industry to work to meet EM needs is important for future development. The identification of the best technology to meet individual or departmental KT needs is demanding, given the number of options in a rapidly changing field. The use and self-reported cognitive impact of CDSS tools and CIRT were recently evaluated in a cohort of family medicine residents given one electronic knowledge resource on handheld computer. This study showed that 1) electronic knowledge resources can be systematically evaluated for their impact on the physician and 2) the impact of CIRT differs from CDSS tools in that CDSS tools were associated with reports of practice improvement, whereas CIRT was associated with reports of learning.33 In a study comparing the performance of a wireless laptop computer on wheels with hard-wired desktop computers using a seven-point Likert scale, physicians reported the mobile computer to be as fast (5.04 vs. 4.54; p = 0.13) and convenient (5.08 vs. 4.14; p = 0.07) and encouraged the use of decision support tools. Unfortunately, the wheeled cart proved cumbersome and was deemed less efficient (3.18 vs. 4.30; p = 0.02), leading to an ongoing search for a more usable technology.34 Given the need for mobility and instant information access, advances in CDSSs will continue to fall short of the goal of providing best evidence at the point of care unless hardware solutions are able to meet the requirements of the ED environment. Focus for Future Research and Development 1. What are the key features of the human–computer interface in the clinical environment, and how can that information be used by industry to create hardware upgrades to better support EM needs? RECOMMENDATION 7 Leadership from the EM community should drive KTrelated ED information system software enhancements. This leadership needs to come in many forms, including serving on informatics-related committees and interest groups, participating in the regulatory process, working with payers and malpractice insurers to provide incentives for KT-enhanced software, and working with industry to identify EM needs. The major EM societies should collaborate to stimulate these efforts and investigate the most effective approaches to dissemination and ongoing application development. Messaging from medical policy-makers, funders, and industry would lead us to believe that computerization is inevitable and will enable us to provide improved care. Frequently, decisions about funding and how clinical systems are implemented are made by administrators and informaticians without adequate input or authority from the end-users. While some of the more successful ED information system implementations have been locally developed, the majority of medical institutions rely on commercial vendors to meet their informatics needs. Experience has shown that commercial willingness to deliver on suggested programming changes is limited, because income is primarily derived from sales and implementations, not enhancements. Gillam et al., in their
Bullard et al.
KT AND INFORMATICS
2004 consensus recommendations on information technology principles for management recommended to the Society for Academic Emergency Medicine, encouraged members with expertise to become involved with the National Health Information Infrastructure (Department of Health and Human Services), the Unified Medical Language System (National Library of Medicine), the Integrated Advanced Information Management Systems Project (National Library of Medicine), industry standards development projects such as Health Level 7 (HL7), and national biosurveillance initiatives.35 The American College of Emergency Physicians has an active EM informatics interest group, and the Society for Academic Emergency Medicine has an academic informatics group focusing more on education. In Canada, the Canadian Association of Emergency Physicians has a Canadian emergency medicine information systems committee focusing on optimizing the key data elements and reports of value to the EM community. Changes in government or payer regulations, such as the Joint Commission or Medicare in the United States, or national or provincial health care policy in Canada, would have a significant impact on market forces. The American College of Emergency Physicians EM informatics interest group previously developed a sample ED information system request for proposal that anyone could access (http://www.ncemi.org/library/index. htm) and use as a template to develop an application. To enhance its effectiveness, there needs to be a process for updating and also a means of evaluating the applications of the major vendors to further assist EDs looking to introduce or upgrade their ED information systems. This would either reduce the field of vendors able to compete for the bids or challenge them to make intuitively reasonable programs that functionally meet the request for proposal specifications. In this way, the field of EM may be able to induce a change in the way vendors operate. If this capability became commonplace, then the major competition between vendors would move away from content and focus more on speed, look, and usability. By defining their information needs, emergency health care providers could support system development and revision of applications better adapted to support ED workflow and culture. Focus for Future Research and Development 1. What mandatory data elements and ED-specific data standards are required to support emergency informatics? 2. As new data elements and reports are identified for EM, how can policy makers and national and large database managers be engaged to support the new standards? 3. With improved data standards, it will be important to create consortiums amalgamating information from multiple CDSS databases to be able to perform costeffective quality of care research. CONCLUSIONS Knowledge translation should help support patient care by efficiently providing the necessary information when
ACAD EMERG MED
November 2007, Vol. 14, No. 11
www.aemj.org
and where it is needed. In the face of the continued exponential growth of medical knowledge and its synthesis, and the limitless capacity of data storage, the ability to program applications to organize, search, and retrieve information in near real time is critical. Barriers to these applications must be eliminated. There is an urgent need to develop filtered EM-relevant synopses that can be rapidly searched using CIRT or incorporated into CDSSs. There must be reliable processes to keep these resources current. Convenient portable computing tools with extended battery life are important to support pointof-care KT. Most importantly, clinical applications need to be user-friendly, because the ED environment cannot tolerate inefficiency. Speed is essential if electronic decision support is to meet its promise. To achieve success in reaching these targets, it is critically important that EM be involved at all levels: promoting the necessary data standards, collaborating with industry to ensure they understand EM needs, and developing connections with key policy makers to ensure that the unique EM requirements are recognized and supported. Equally importantly, emergency physicians must take on the key role of local champions, without whom the evolution of KT and informatics is destined to fail.
References 1. Haynes BR. Of studies, syntheses, synopses, summaries, and systems: the ‘‘5S’’ evolution of information services for evidence-based healthcare decisions. Evid Based Med. 2006; 11:162–4. 2. U.S. Department of Health and Human Services. National Committee on Vital and Health Statistics (NCVHS) Workgroup report on the National Health Information Infrastructure (NHII). Information for Health: A Strategy for Building a National Health Information Infrastructure. Available at: http://www .ncvhs.hhs.gov/. Accessed Jul 1, 2007. 3. United Kingdom Department of Health. The NHS in England: the Operating Framework for 2007/08. Guidance on Preparation of Local IM&T Plans. Available at: http://www.dh.gov.uk/en/Publicationsandstatistics/ Publications/PublicationsPolicyAndGuidance/DH_ 063443. Accessed Jul 1, 2007. 4. Handler JA, Feied CF, Coonan K, et al. Computerized physician order entry and online decision support. Acad Emerg Med. 2004; 11:1135–41. 5. Haynes BR, Devereaux PJ, Guyatt GH. Clinical expertise in the era of evidence-based medicine and patient choice. Evid Based Med. 2002; 7:36–8. 6. Rudkin SE, Langdorf MI, Macias D, Oman JA, Kazzi AA. Personal digital assistants change management more often than paper texts and foster patient confidence. Eur J Emerg Med. 2006; 13:92–6. 7. Adler MD, Duggan A, Ogborn CJ, Johnson KB. Assessment of a computer-aided instructional program for the pediatric emergency department. AMIA Annu Symp Proc. 2003:6–10. 8. Debehnke DJ, Valley VT. Assessment of the current computer literacy and future computer needs of emergency medicine residents and faculty. Am J Emerg Med. 1993; 11:371–3.
1001
9. Jwayyed S, Park TK, Blanda M, et al. Assessment of emergency medicine residents’ computer knowledge and computer skills: time for an upgrade? Acad Emerg Med. 2002; 9:138–45. 10. Emond SD, Wyer PC, Brown MD, Cordell WH, Spooner CH, Rowe BH. How relevant are the systematic reviews in the Cochrane library to emergency medical practice? Ann Emerg Med. 2002; 39: 153–8. 11. Kelly KD, Travers A, Dorgan M, Slater L, Rowe BH. Evaluating the quality of systematic reviews in the emergency medicine literature. Ann Emerg Med. 2001; 38:518–26. 12. Pluye P, Grad RB, Dunikowski LG, Stephenson R. Impact of clinical information-retrieval technology on physicians: a literature review of quantitative, qualitative, and mixed methods studies. Int J Med Inform. 2005; 74:745–68. 13. Smith R. What clinical information do doctors need? BMJ. 1996; 313:1062–8. 14. Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med. 1985; 103:596–9. 15. Gorman PN, Helfand M. Information seeking in primary care: how physicians choose which clinical questions to pursue and which to leave unanswered. Med Decis Making. 1995; 15:113–9. 16. Sackett DL, Strauss SE. Finding and applying evidence during clinical rounds. JAMA. 1998; 280: 1336–8. 17. Hayward RS, El-Hajj M, Voth TK, Deis K. Patterns of use of decision support tools by clinicians. AMIA Annu Symp Proc. 2006:329–33. 18. De Miguel-Yanes JM, Andueza-Lillo JA, GonzalezRamallo VJ, Pastor L, Munoz J. Failure to implement evidence-based clinical guidelines for sepsis at the ED. Am J Emerg Med. 2006; 24:553–9. 19. Grad RM, Meng Y, Bartlett G, et al. Effect of a PDA-assisted evidence-based medicine course on knowledge of common clinical problems. Fam Med. 2005; 37:734–40. 20. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005; 330: 765–73. 21. Sim I, Gorman P, Greenes RA, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001; 8:527–34. 22. Grimshaw JG, Santesso N, Cumston M, Mayhew A, McGowan J. Knowledge for knowledge translation: the role of the Cochrane collaboration. J Contin Educ Health Prof. 2006; 26:55–62. 23. Sim I, Owens DK, Lavori PW, Rennels GD. Electronic trial banks: a complementary method of reporting randomized controlled trials. Med Decis Making. 2000; 20:440–50. 24. Ho K, Bloch R, Gondocz T, et al. Technology-enabled knowledge translation: frameworks to promote research and practice. J Cont Health Prof. 2004; 24:90–9. 25. Ho K, Chockalingam A, Best A, Walsh G, JarvisSelinger S. Technology enabled knowledge translation: current best practices in innovation, evaluation,
1002
26.
27.
28.
29. 30.
and future directions. Available at: http://www.cme .med.ubc.ca/Home.htm. Accessed Jul 1, 2007. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005; 293:1197–203. Garg AX, Adhikar NKJ, McDonld H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes. JAMA. 2005; 293:1223–38. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003; 10:523–30. Neilsen J, Loranger H. Prioritizing Web Usability. Berkeley, CA: New Riders, 2006. Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform. 2005; 74:519–26.
Bullard et al.
KT AND INFORMATICS
31. Kushniruk AW, Patel V, Cimino JJ, Barrows RA. Cognitive evaluation of the user interface and vocabulary of an outpatient information system. Proc AMIA Annu Fall Symp. 1996:22–6. 32. Patel VL, Kushniruk AW, Yang S, Yale JF. Impact of a computer-based patient record system on data collection, knowledge organization, and reasoning. JAMIA. 2000; 7:569–85. 33. Grad RM, Pluye P, Meng Y, Segal B, Tamblyn R. Assessing the impact of clinical information-retrieval technology in a family practice residency. J Eval Clin Pract. 2005; 11:576–86. 34. Bullard MJ, Meurer DP, Colman I, Holroyd BR, Rowe BH. Supporting clinical practice at the bedside using wireless technology. Acad Emerg Med. 2004; 11: 1186–92. 35. Gillam M, Rothenhaus T, Smith V, Kanhouwa M. Information technology principles for management, reporting, and research. Acad Emerg Med. 2004; 11:1155–61.
