for Continuing Medical Education, the Society for Medical Education, the Society for ... Rkjean Laprise, PhD, Laure Perrier, MEd, MLIS, David Ryan, PhD,.
The Journal of Continuing Educution in the Health Professions, Volume 24, pp. 90-99. Printed in the U.S.A.Copyright 0 2004 The Alliance for Continuing Medical Education, the Society for Medical Education, the Society for Academic Continuing Medical Education,and the Council on CME, Association for Hospital Medical Education. All rights reserved.
Innovations in Continuing Education
Technology-Enabled Knowledge Translation: Frameworks to Promote Research and Practice Kendall Ho, MD, FRCPC, Ralph Bloch, MD, PhD, FRCPC, Tunde Gondocz, MSc, Rkjean Laprise, PhD, Laure Perrier, MEd, MLIS, David Ryan, PhD, Robert Thivierge, MD, FRCPC, FAAP, and Elizabeth Wenghofer, BSc, MSc, PhD Abstract Knowledge translation articulates how new scientific insights can be implemented efficiently into clinical practice to reap maximal health bentfits. Modern information and communication technologies can be effective tools to help in the collection, processing, and targeted distribution of informationfrom which clinicians, researchers, administrators, policy makers in health, and the public can benefit. Effective implementation of knowledge translation through the use of information and communication technologies, or technology-enabled knowledge translation (TEKT),would benefit both the individual health professional and the health system. Successful TEKT in health requires cultivation an,d acceptance in thefollowing key domains: Perceiving types of knowledge and ways in which clinicians acquire and apply knowledge in practice Understanding the conceptual and contextual ,frameworks of information and communication technologies applied to health systems, particularly the push, pull, and exchange communication models Comprehending essentiul issues in implementationof information and communicationtechnologies and strategies to take advantage of emerging opportunities and overcome existing barriers Establishing a common and widely acceptable evaluation framework in order that researchers can compare various methodologies in their rightful contexts in TEKT research and adoption Achieving harmony and common understanding in these areas will go a long way in fostering a fertile and innovative environment to encourage research and advance understanding in this exciting domain of TEKT Key Words: Continuing medical education, continuing professional development, information and communication technology, knowledge translation
in Continuing Education and Knowledge Translation, Faculty of Medicine, University of Toronto, Toronto, Ontario; Dr: Thivierge: Vice Dean, Continuing Professional Development, Faculty of Medicine, UniversitC de MontrCal, Associate Professor of Pediatrics, Department of Pediatrics, UniversitC de MontrCal, Montreal, Quebec; D K Wenghofer: Candidate, Senior Epidemiologist and Manager, Research and Evaluation Department, Quality Management Division, College of Physicians and Surgeons of Ontario, Toronto, Ontario. Reprint requests: Kendall Ho, MD, FRCPC, Division of Continuing Medical Education, Faculty of Medicine, University of British Columbia, 104-740 Nicola Street, ‘Vancouver, BC V6P 5G8.
D KHo: Assistant Professor, Department of Surgery, Associate Dean, Division of Continuing Medical Education, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia; Dr: Bloch: Institute for Medical Education, Faculty of Medicine, University of Bern, Bern, Switzerland; Tunde Gondocz: Manager, Centre for Learning in Practice, The Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario; Dr: Laprise: Manager, Research, Development and Innovations in Health Promotion, Department of Professional Education, Aventis Canada, Laval, Quebec; Laure Perrier: Information Specialist, Continuing Education, Faculty of Medicine, University of Toronto, Toronto, Ontario; Dr: Ryun: Assistant Professor/Consultant
90
Ho et al. Introduction
address issues of suitability and accessibility of information.’ Information and communication technologies can substantially relieve this tension by capturing and archiving the volume of information and through rapid searching capabilities to draw out relevant information for discussion, decision support, and use on demand. Universities, health care institutions, and health regulatory bodies look toward information and communication technologies to broaden the dissemination of educational materials, just-in-time decision support, and convenience and accuracy of practice audits.8 Through telemedicine, governments expect information and communication technologies to improve access to care, especially in geographically isolated cornmunitie~.~ These perspectives illustrate the tremendous opportunities that information and communication technologies offer in TEKT. They also present the importance of setting up appropriate frameworks, implementation strategies, and evaluation structures to rigorously analyze how information and communication technologies can contribute to health care delivery solutions.
Knowledge translation (KT) is defined as “the exchange, synthesis and ethically-sound application of researcher findings within a complex system of relationships among researchers and knowledge users.”’ It “potentially includes all sectors of society and all activities from creation of knowledge to its application to yield positive health outcomes.”’ KT articulates how new scientific insights can be implemented efficiently into clinical practice to reap maximal health benefits. Modern information and communication technologies provide a panoply of tools for the collection, processing, and targeted distribution of information from which clinicians, researchers, administrators, policy makers in health, and the public can all benefit.2 Witness the popularity of the Internet for information dissemination, pervasiveness of computers in everyday life, the use of personal digital assistants for daily decision support, and wireless devices for instant messaging and communications. To date, in health care, neither KT nor the extensive use of information and communication technologies has made its full impact in health care research and delivery. The reason lies neither in the insufficiency of available new information nor in the inadequacy of information technology but primarily in the lack of appropriate integration of the two. In this article, we explore how the potential synergy of KT and information technology, or technology-enabled knowledge translation (TEKT),2%3 methodically can be exploited.
The Nature of Knowledge and Its Adoption in Practice Before considering the translation of knowledge, it is important to recognize the nature of knowledge as it applies to health care practices. Knowledge is the fundamental blueprint to guide health professionals’ choices in patient management. Derived from the literature of knowledge management, education, and business,l0-I2 manifest knowledge in health may be seen in three dimensions:
Potential Benefits of TEKT A paradox exists within the health care environment. It involves managing the polar tensions of information overload4 alongside the information gaps experienced by physician^.^ Similarly, public access to health information has increased, but the quantity available requires time and expertise to assess the appropriateness of the information found.6 Practitioners and patients alike must
Explicit knowledge, or formal knowledge, is generated through scientific studies, captured in peer-reviewedjournals and medical textbooks, and commonly expressed in clinical practice guidelines. It is perceived by health professionals as the ideal standard in disease management.
91
Technology-Enabled Knowledge Translation portfolio management, using virtual patient simulators for application of newly acquired knowledge into practice, and capturing clinical indicators for health outcome measurement and performance i m p r ~ v e m e n t . ’ ~ . ’ ~ - ’ ~ ~ ~ ~
Clinical knowledge is gathered by health professionals as they question, examine, obtain laboratory tests, and interpret them during clinical encounters. Tacit knowledge is generated through the sum of past experiences of the health professionals through encounters with different patients. Often called clinical judgment or practical wisdom, it influences profoundly the clinician’s decision-making process in each new clinical situation.
Knowledge Translation Frameworks
As producers of knowledge, researchers work in communities of researchers essential for the maintenance of their creativity. Health professionals, as consumers of knowledge, operate in communities of practice that encourage the distribution How do health professionals acquire new of tacit wisdom. Patients, as beneficiaries of clinknowledge in all of its dimensions and apply it in ical knowledge, form their own communities, practice? Although formal continuing health edufacilitating the diffusion of popular lore. Intercation activities, such as conferences, workshops, acting with all three communities, partially faciland rounds, provide opportunities for acquisition itating and partially inhibiting the KT process is of explicit knowledge, this type of learning alone the “system” including formally organized govfalls far short in helping health professionals in the ernmental and health care industry organizations translation of this knowledge into clinical prac(Figure 1). tice.I3 Practice-based learning that occurs during Although one important direction of KT flows patient encounters is highly powerful in knowledge from producers to beneficiaries to lead to incorattainment and behavioral change. In “ h ~ n t i n g ” ’ ~ poration of knowledge into practice, an informaor “reflection in a~tion,’’’~,’~ health professionals tion flow in reverse is equally necessary to have generate clinical questions during patient encounpractice inform research to keep the search for ters and search for their corresponding answers. innovation relevant. Practitioners play a central role Outside clinical encounters, health professionals in this process. Gain in health status can occur only “reflect on a ~ t i o n ” ’through ~ ~ ’ ~ recalling their clinif KT responds to their explicit needs yet, at the ical experiences and search for information through same time, to the implicit needs of patients. Effecinteraction with teammates, perform “foraging” of tive KT requires concerted efforts of knowledge the medical literat~re,’~ participate in self-assessproducers, consumers, and beneficiaries. If the ment programs or self-directed learning, and take system does not actively support and encourage part in performance improvement programs such this process, at least it should not obstruct it (see as Plan-Do-Study-Act and SIX SIGMAI7 or sysFigure 1).*’ tems-based reflection. In all of these knowledge Factors influencing the process of KT can be acquisition and performance improvement initiaconceptualized in a matrix model. KT is initiated tives, information and communication technoloeither by knowledge producers or the system in a gies can play a substantial and supportive role, such “push” configuration or by knowledge consumers as on-the-fly capturing of clinical questions durand beneficiaries in a “pull” operation. In the foring patient encounters, accessing medical database mer case, practitioners remain in a passive role as and knowledge repositories (e.g., Cochrane information receivers. In the latter case, the pracLibrary, MEDLINE, ACP Journal Club) on titioner is actively seeking specific information, demand through the Internet, documenting cliniwhich can be packaged in several ways. Inforcal questions and responses into personal learning mation can be broadcast without specifically being
92
Ho et al. tant for patients and practitioners that systems changes occur.
I
Knowledge producer
I
I I
Knowledgeconsumer
I
1
Knowledge beneficiary
I
Implementation Issues Although the use of information and communication technologies to improve KT in health care holds great promise to enhance patient care, implementation often results in “unforeseen costs, unintended organizational consequences, and even failure.”22Successful implementation depends on developers’ ability to provoke a compelling vision about the goals to be achieved, address cultural and political issues that may arise within and between organizations, deal with conflicts, and negotiate implementation trade-offs with users and other stakeholders while taking into account available human and financial resources as well as legal, ethical, and technological limitations (Table l).23 To create an inspiring vision that will motivate the organization to invest the time and resources needed to achieve the change, it is necessary to understand the environment in which the potential solution is to fit. The environment includes organizational values, goals, and culture. The culture involves people, their potential, their needs, their motivations, and the ways in which organizations choose to work collaboratively. Many organizations face the tension of achieving goals doing what is right versus reaching for a compromised solution that will minimize negative impacts for the organization. The problem with the compromised solution in knowledge-intensive organizations is that the resulting negotiations may reduce the ability to provide solutions that continuously adapt vision with strategy and resource capacity.23 One way in which an organization can deal with the tension to guide good decision making is by focusing on meaning. The purpose or meaning of what the organization is set to accomplish can create a vision that sets in motion the alignment process.23 Once the climate is right, the implementation strategy becomes one of analyzing what is required,
Figure 1 The inter-related dynamics between researchers, practitioners, patients, and governance in the knowledge translation cycle: a contextual framework.
pushed toward any particular consumer or consumer group. In cases of practice-specific transmission, information may be directed at selected groups of consumers or may deal directly with information relevant for individual cases and practitioners. From the consumers’ vantage point, under favorable conditions, they can pull and convert innovative information into knowledge by integrating it with preexisting explicit and tacit knowledge, such as through the “hunting” and “foraging” proce~ses’~ described earlier in this review. KT efforts must attend not only to “what” issues are being addressed and “how” these problems can be solved but also to “why” it is impor-
93
Technology-Enabled Knowledge Translation records, Web sites, videoconferences) and a means of evaluation (e.g., on-line quizzes, feedback discussion forum). As a KT medium, TEKT can stand alone or, perhaps ideally, be blended with other approaches. As a means of evaluation, it provides an opportunity to extend more traditional approaches, such as multiple-choice questionnaires, and to provide unique evaluation opportunities, such as Web site use statistics. As well, it can provide opportunities for both summative evaluation (an outcome-based, retrospective analysis of the success of a certain technology-enabled methodology in the end) and formative evaluation (the prospective evaluation of a methodology to measure progress toward a certain goal). Indeed, the speed of technology-mediated communications can add substantial value to the formative stages of the development of KT methods. Finally, innovative evaluation technologies themselves should also be subject to TEKT evaluation. In this context, no single evaluation method is sufficient. Table 2 provides a framework for selecting a blend of evaluation methods to meet the specific needs, contexts, and concepts driving TEKT deployment. The subjective, cognitive, and behavioral elements of the framework and their evaluation methods-surveys, questionnaires, standardized patients, and objective structured clinical examination-are familiar in education. Elements of structural and systemic evaluation such as usabilityz4and return on assets2smight be less familiar and need a little explanation. Structural evaluation refers to the usability of learning materials and applies to both TEKT and non‘TEKT modules or learning objects. Structural evaluation of a paper-based module might ask, “Are sections easily identified?” “Do student instructions work?” “Is the information accurate and up to date?” Structural evaluation of TEKT asks additional questions such as “Do materials download quickly?’ “Does the utility work on different kinds of computers?’ and “Is the Web site easy to navigate?’24 Standardized checklists and “walk-through” procedures are commonly used structural evaluation methods.
where the barriers are, and how to implement what is required while preventing, reducing, or dealing with the barriers using standard project management principles. To maximize the likelihood of success, developers must consider potential implementation issues early in the creation stage of the project and adopt appropriate tactics to overcome barriers. Although not exhaustive, Table 1 provides an extensive list of the major factors to consider. Acknowledging implementation issues and being aware of some of the tactics to increase the likelihood of overcoming implementation barriers increase the chances of success. Implementation can be achieved using feedback loops to understand what is working and what needs to be adjusted to identify the best next steps. The context from one organization to another adds a level of complexity that can be countered only by a customized solution. Nevertheless, the list of factors in Table 1 offers a high-level decision-making template that can act as a checklist to cover the basics. The table identifies issues related to organizational values, goals related to the solution, and an understanding of the culture in which the solution is to be introduced. It is a combination of factors that allows decision-makers to negotiate smartly by knowing where the trade-offs can or cannot happen. After implementation, consideration of cultural factors is vital to ensure a smooth transition by knowing where resources exist or where they are lacking and where attitudes of fear and anxiety related to change may impose barriers and to establish a comfortable pace for the transition period to take.
Evaluation Issues To further the understanding of the impact of TEKT and to promote further learning and development about it, a strong evaluative foundation is essential because the convergence of technology and evaluation in TEKT is layered and complex. Technology can be both a medium for KT (e.g., delivering reminders within electronic medical
94
Ho et al. Table 1 Example of Factors to Consider during the Development of a TEKT Project Category
Issues
Tactics
Organizational factors
Institutional policies28 Effect on organization”’ Finan~ing~~,~’ Pr~fitability~l S~stainability~’ Professional c~mpensation~~
Top management support and Participative decision making29 Accurate and timely comm~nication~~ Investment by organizations and governments2x Professional incentives and rewards29
Legal issues
Liability3I Copyright infringement32 Jurisdiction3’ Electronic signature3’
Interjurisdiction reciprocity agreements3’ Registration process for out-of-jurisdiction practitioners3’
Ethical issues
Interregional variability in political, economic, and health care contexts34 Patient c o n ~ e n t ~ ’ ~ ’ ~ C~nfidentiality~~~~~~~~
Full control over data provided32 Informed consent3’ Information about privacy and confidentiality standards’l
Professional issues
National licensure systems3I Regional differences in standards of de~elopment~~ practice, certification, and l i c e n ~ u r e ~ ’Multidisciplinary .~~ Lack of standards in communication Inclusion of IT into professionals’ role28 and documentation of care within and Integrated clinical workstation^^^ between professions and across settingsx6 Professional access rights36 Interference with patient-physician encounter3’
Users factors
Authentication32 Access32 Under~tanding~~ Interface design3” Workflow30 Cultural impediments to acceptance’* Cultural variation in languages34
Practitioner and patient-centered design principle^'^.^^ Preprocess resources to standardize presentation to the user3* Training39 Virtual reality4”
Logistical issues
System performance, portability, and ~ompatibility~’,~~) Data archival and retrieval standards3* Multiplication of users’ interfaces and soft ware^^^.^^ Installation, maintenance, and user s u ~ p o r t ~ * , ~ ~
Accommodates multiple platforms (e.g., PC, PDA, ~ e l l p h o n e ) ~ ~ Tools working around the clock, recovery procedures, redundant h a r d ~ a r e ’ ~ Integrated health IT systems2* Health information-sharing cons or ti urn^^^
Cognitive issues
Lack of a general metaknowledge to structure Internet resources32 Validity of medical concept repre~entation~~ Avoid “~verlearning”~~
Meaningful organization of the medical domain3* Ergonomics42 Tailored information4I Learning by doing with simulators4” Diagnostic support systems3’
Financial issues
95
Technology-Enabled Knowledge Translation Table 1 (continued)Example of Factors to Consider during the Development of a TEKT Project Category
Issues
Tactics
Content issues
Author credentials and re~ponsibility~~Quality seal of approval standards4' Reliability3".31 Knowledge must be documented; Accuracy30*3i date of publication, releases, authors, sources, references32 Peer-review system32 Repurposing data by sharing30
IT = information technology; PC = personal computer; PDA = personal digital assistant; TEKT = technology-enabled knowledge translation. A full description of implementation issues and tactics to increase the likelihood of implementation success can be found in the cited articles.
Systemic evaluation of TEKT can include evaluations of health outcomes and economic viability. Two traditional accounting principles are suggested for the latter. Return on investment is a ratio of earnings to costs of construction, usually within one fiscal year. Return on assets (ROA)
directs evaluators to view TEKT within a mix of assets whose value extends over time to include estimates of future earnings, hard and soft profits, and depreciations. Although both provide general estimates of economic viability or profitability, ROA may be more realistic for TEKT evaluation.
Table 2 Evaluation Dimensions and Methodologies for TEKT Elements of Evaluation Examples of What Is Evaluated Structural
Educational design
Subjective
Happiness index Perceived usefulness User friendliness Knowledge acquisition Knowledge retention Reasoning, judgment Decision making
Cognitive
Behavioral
Core competencies Practice change Evidence-based practice
Systemic
Population health Service use Cost-benefit analysis
Examples of Evaluation Methods Standardized structural usability analysis Walk-through usability testing Surveys, walk-through Simulation focus group Integrated site statistics data (e.g., time on task) Multiple-choice questions On-line quizzes Self-assessment Reflection tools Web diary OSCE Standardized patients EMR data Practice reviews Health economics Epidemiologic data Return on investments Return on assets
EMR = electronic medical record; OSCE = objective structured clinical examination; TEKT = technology-enabled knowledge translation.
96
Ho et al. Although any one group of TEKT developers is not expected to implement the entire evaluation framework, it would be highly productive if each could share the vision of the same common framework to carry out its particular piece of evaluation. This would facilitate use of common indicators and complementary characterization of best practices and lessons learned, leading to the development and maturation of a consistent body of knowledge on TEKT.
Lessons for Practice Thoughfful deployment of modern information and communication technologies can help individuals and organizations accelerate the incorporation of research and new knowledge into dinical practice: technology-enabled knowledge translation (TEKT).
Examples
Before technology-driven solutions are adopted, review solutions by assessing advantages against bamers to systematic implementation.
TEKT processes will be “wrapped around” practicing health professionals and their patients in the office of the future. TEKT will support traditional continuing education, provide formal on-line education, and deliver education embedded within integrated information systems to support more informal practice-based KT. The MOXXI project,26whose effectiveness was demonstrated in a randomized control trial, provides a good example. MOXXI used an electronic chart and prescribing system and allowed physicians in the province of Quebec to download information about all drugs that were prescribed to their patients by any physician. When the software identified any of 159 potential prescribing problems, physicians were alerted, consequences were explained, and alternative therapies were suggested. The Ontario Medical association'^^^ guideline repository provides a second example of embedded TEKT. Because the number of guidelines available for common conditions can be an obstacle to guideline adoption, this guideline service supports a distributed team of practitioners who use electronic tools to submit guideline evaluations so that a single guideline can be recommended for each condition and be made available via a Web interface in a structured format that provides a point-of-care synopsis and the full guideline for more leisurely study. A parallel site for consumers is developing, and the guideline utility integrates with a personalized continuing education (CE) portal that includes an on-line self-directed chart
The adoption of a common, multidimensional evaluation framework in TEKT should help researchers compare successful models and characterize best practices.
audit utility and CE credit manager similar to the Web-diary platform developed by the Royal College of Physicians and Surgeons of Canada. Future TEKT will adapt to the learning needs and contexts of the practitioners and their communities of practice, bringing information to the point of care in tailored formats that promote reflection and practice change. Whether supporting traditional CE or embedded within integrated systems, TEKT will wrap information resources around practitioners and their patients to support best practices and maximize population health.
Conclusion Technologies can make significantcontributions to the acceleration of KT. To maximize the benefits, it is important to recognize the contextual and conceptual frameworks from which information and communication technologies can be perceived as beneficial to KT processes. Appreciating the chal-
97
Technology-Enabled Knowledge Translation 10. Baird L, Henderson JC. The knowledge engine. San Francisco: Berret-Koehler Publishers Inc., 2001.
lenges of embedding technologies into the work flow of health delivery, effective strategies can be formulated for successfully introducing technologies that accelerate KT in the workplace. With the accept a n ce of a c o m m o n e va l ua t ion f r a m e w o r k , researchers can comDare assorted methods in their rightful contexts, thereby leading to a rich and innovative environment to foster research and advance understanding in this exciting domain of TEKT.
11. Liebowitz J. Knowledge management: learning from knowledge engineering. Boca Raton, FL: CRC Press, 2001. 12. Perrenoud P. Developper la pratique reflexive dans le mktier d’enseignant. Paris: ESF editeurs, 200 1. 13. Davis DA, O’Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal CME: do conferences, workshops, rounds and other CE activities change physician behavior or health care outcomes? JAMA 1999; 2829677874.
References 1. The Canadian Institutes for Health Research Knowledge Translation Framework. Draft definition of Knowledge Translation. Available at: http://www.cihrirsc.gc.ca/e/about/7518.shtml. Accessed February 15, 2004.
14. Ebell MH, Shaughnessy A. Information mastery: integrating continuing medical education with the information needs of clinicians. J Contin Educ Health Prof 2003; 23(Suppl):S53-S62.
2. Ho K, Chockalingam A, Best A, Walsh G. Technology-enabled knowledge translation: building a framework for collaboration. Can Med Assoc J 2003; 168(6):710-711.
15. Slotnik HB. How doctors learn: education and learning across the medical-school-to-practice trajectory. Acad Med 2001; 76:1013-1016.
3. Howell JD. The physician’s role in a world of technology. Acad Med 1999; 74(3):244-247.
16. Schon DA. Educating the reflective practitioner: toward a new design for teaching and learning in the professions. San Francisco: Jossey-Bass, 1987.
4. Wilson TD. Information overload: implications for healthcare services. Health Inform J 200 1; 7: 112-1 17.
17. Staker LV. Teaching performance improvement: an opportunity for continuing medical education. J Contin Educ Health Prof 2003; 23(Suppl):S34-S52.
5. Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med 1985; 103(4):596-599.
18. Frankford DM, Patterson MA, Konrad TR. Transforming practice organizations to foster lifelong learning and commitment to medical professionalism. Acad Med 2000; 75(7):708-717.
6. Eysenbach G, Kohler C. How do consumers search for and appraise health information on the World Wide Web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002; 324(7337):573-577.
19. Caudell TP, Summers KL, Holten J IV, et al. Virtual patient simulator for distributed collaborative medical education. Anat Rec 2003; 270B( 1):23-29.
7. Gawande AA, Bates DW. The use of information technology in improving medical performance. Part 111. Patient-support tools. Med Gen Med 2000; 2(1):E12.
20. Crowley SD, Owens TA, Schardt CM, Wardell SI, Peterson J, Garrison S, Keitz SA. A Webbased compendium of clinical questions and medical evidence to educate internal medicine residents. Acad Med 2003; 78(3):270-274.
8. Whitten P, Adams I, Davis S. A success model: Marquette General Health System. Telemed J E Health 2003; 9( 1):41-48. 9. Yellowlees P. Government relations, government regulations: jumping through the hoops. J Telemed Telecare 2002; 8(Suppl 3):83-85.
21. Merenstein D. Winners and losers. JAMA 2004; 291(1):15-16.
98
Ho et al. 22. Ash J. Organizational factors that influence information technology diffusion in academic health sciences centers. J Am Med Inform ASSOC1997; 4(2):102-111.
32. Lovis C, Baud RH, Scherrer JR. Internet integrated in the daily medical practice within an electronic patient record. Comput Biol Med 1998; 28(5):567-579.
23. McGee JV, Prusak L. Managing information strategically (Emst and Young Information Management Series). New York: John Wiley and Sons, Inc, 1993.
33. Campbell SG, Gibby GL, Collingwood S. The Internet and electronic transmission of medical records. J Clin Monit 1997; 13(5):325-334, 34. Rodriguez JC. Legal, ethical, and professional issues to consider when communicating via the Internet: a suggested response model and policy. J Am Diet Assoc 1999; 99( 1l): 1428-1432.
24. Hix D, Harston HR. Developing user interfaces: ensuring usability through product and process. New York: Wiley, 1993. 25. Lopez J. Return on enterprise architecture: measure IT in asset productivity. The Gartner Report, July 19, 2002. Available at: http://gartnerg2.com/rpt/rpt-0702-0119.asp. Accessed February 15, 2004.
3.5. Shortliffe EH. The evolution of electronic medical records. Acad Med 1999; 74(4):414-419.
36. Jastremski CA. Nursing informatics. Issues for critical care medicine. Crit Care Clin 1999; 15(3):563-576.
26. Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, McLeod P, Laprise R. The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary car. Can Med Assoc J 2003: 169(6):549-556.
37. Ridderikhoff J, vanHerk E. A diagnostic support system in general practice: is it feasible? Int J Med Inform 1997; 45(3):133-143. 38. More DG, Clarke PA. The General Practice computer system project: a doctor’s desktop for Australia. Int J Med Inform 1999; 55(1):65-75. 39. Liaw ST, Ugoni AM, Cairns C. Computer education. Don’t forget the older GPs. Aust Fam Physician 2000; 29(8):802-805.
27. Rosser W, Davis D, Gilbart E, Guideline Advisory Committee. Promoting effective guideline use in Ontario. Can Med Assoc J 2001; 16S(2): 181-1 82.
40. Kneebone R. Simulation in surgical training: educational issues and practical implications. Med Educ 2003; 37(3):267-277.
28. Hughes RA. Clinical practice in a computer world: considering the issues. J Adv Nurs 2003; 42(4):340-346.
41. D’Alessandro DM, Dosa NP. Empowering children and families with information technology. Arch Pediatr Adolesc Med 2001; 1S5( 10):113 1-1 136.
29. Ash J. Organizational factors that influence information technology diffusion in academic health sciences centers. J Am Med Inform ASSOC1997; 4(2):102-111. 30. Kuhn KA, Giuse DA. From hospital information systems to health information systems. Problems, challenges, perspectives. Methods Inf Med 2001; 40(4):275-287.
42. Kelly CS. Perinatal computerized patient record and archiving systems: pitfalls and enhancements €or implementing a successful computerized medical record. J Perinat Neonat Nurs 1999; 12(4):1-14.
3 1. Silverman RD. E-health and e-medicine. Hematol Oncol Clin North Am 2002; 16(6):1495-1507.
43. Moehr JR. Informatics in the service of health, a look to the future. Methods Inform Med 1998; 37(2): 165-170.
99
