**Department of Medical Informatics. Eighth Avenue and C Street. Salt Lake City, UT 84143. Abstract. Protocols were created to direct the management of ...
Performance Evaluation of Computerized Clinical Protocols for Management of Arterial Hypoxemia in ARDS Patients Susan Henderson, BA*, Thomas D. East, PhD.* **, Alan H. Morris, M.D.*, Reed M. Gardner, Ph.D.**
LDS Hospital, and University of Utah *Pulmonary Division and Department of Medicine **Department of Medical Informatics Eighth Avenue and C Street Salt Lake City, UT 84143 Abstract Protocols were created to direct the management of arterial hypoxemia in critically ill patients with adult respiratory distress syndrome (ARDS). The protocols used clinical information to generate suggestions for therapy. They were initially created on paper and then implemented on a computer. We measured how often the clinical staff followed protocol suggestions in 16 ARDS patients. We also compared computer generated suggestions with those from the paper protocols. The study included 5130 hours of patient care and 3553 computer generated suggestions. The clinical staff followed the protocol suggestions 76.9% of the time and compliance increased with time (63.9% compliance for the first 8 patients and 91.8% compliance with the last 8 patients). Differences between the paper and computer protocol versions were primarily due to software errors and inaccurate and untimely entry of data into the computer. Both software errors and data entry problems decreased with time but the data entry problems were more persistent (From the first 8 to the last 8 patients, software errors decreased from 7.2 to 0.8% while incomplete data base and data entry problems decreased from 6.8 to 4.5%). The major problem in creating the protocols was obtaining clinician agreement on protocol logic and their willingness to utilize it clinically around the clock. The major problem in implementing protocol directed care was obtaining accurate and timely data entry. We conclude that computerized protocols can direct the clinical care of critically ill patients in a manner that is acceptable to experienced clinicians.
process and to provide educational feedback to health care providers. Miller and Menn developed separate computerized systems to assist in the management of ventilation and oxygenation problems (9-10). Both were thought to be clinically useful but were never well accepted by physicians because they required manual entry of data. Manual entry made the systems cumbersome, time consuming and prone to error. Protocols for the management of acutely ill patients with adult respiratory distress syndrome (ARDS), were developed at the LDS Hospital by a physician-nurse team as part of a randomized clinical trial comparing traditional positive pressure treatment of ARDS with extra-corporeal C02 removal (11). The protocols were initially developed on paper as flow diagrams. These paper versions have been used and evaluated in a clinical setting for a total of 12,500 patient care hours, representing about 12,000 decisions. Protocol logic successfully directed patient care decision making 87% of the time. The paper protocol decision logic was computerized taking advantage of a large computerized patient data base at the LDS Hospital. The information needed for protocol execution is available from this data base, so that manual entry of data is essentially eliminated, and protocol instructions are automatically generated and available at bedside computer terminals. Sittig et al (12-14) has described these preliminary computer protocols and their performance in detail. The initial studies demonstrated the feasibility of computerized protocols but also identified deficiencies in the system. Revised computerized protocols have now supplanted the paper protocols as the primary source of therapy
Introduction The management of acutely ill ICU patients is an extremely complex activity. Sisson (1) found that a physician with numerous different pieces of interacting data may be unable to comprehend their aggregate meaning. Shulman's (2) study on the incidence of errors in the ICU concluded that human errors are frequent and, in fact, much more numerous than equipment malfunction. Grossman (3) showed that senior physicians made "unacceptable" ventilator adjustments 8% of the time and respiratory therapists made "unacceptable" adjustments 14% of the time. Twenty-eight percent of the "unacceptable" respiratory therapist changes in ventilator therapy were thought to be potentially harmful. Several studies (4-8) have shown that the use of.computer protocols reduces errors, thereby improving the performance of physicians and other health care providers. Cannon (6) listed the potential benefits of computerized clinical care protocols as being flexibility, accuracy, versatility, consistency, and the ability to perform detailed analysis. Grimm (4) reported that patient care protocols provided an excellent standard to identify deficiencies, to monitor the medical care
suggestions. We tested these protocols in a clinical setting where they were applied 24 hours a day by the routine clinical staff. We measured how often the clinical staff followed protocol suggestions for therapy in patients with ARDS and compared computer protocol suggestions with those derived from the paper based protocols. Methods The HELP system The HELP information system at LDS Hospital runs on a network of 10 Tandem fault tolerant computer processors using the Guardian Operating System with 3.4 gigabytes of disk storage distributed over 14 disk drives (15-17). The 8 drives handling clinical data are mirrored to reduce the possibility of data loss. Eighteen Charles River Data Systems (CRDS) minicomputers are interfaced to the Tandem serving as multiplexers and pre-processors. The system currently supports more than 450 terminals and 100 printers. About half of each of these are connected to
588
0195-4210/89/0000/0588$01.00
1989 SCAMC, Inc.
(vital signs, respiratory care parameters, laboratory data, etc.) stored in the HELP System integrated data base. This entire process operates in a multiuser, multitasking environment. The protocol logic operates in background mode and does not interfere with use of the bedside terminal for other tasks such as nurse and respiratory care charting. Protocol suggestions are reviewed at the bedside terminal through the use of menus. These menus also allow the user to manually drive the protocols and generate new therapy suggestions using the current patient data base, to review data, and to suspend protocol execution when medical problems not addressed by the protocols demand attention. Performance Evaluation The computerized protocols were used simultaneously with the paper based protocols for the management of arterial hypoxemia in 16 ARDS patients between September 1988 and July 1989. There were 5130 patient care hours and 3553 computer generated therapy suggestions. Attending physicians at the bedside were allowed to over-ride the protocol suggestions when the medical problem was not covered by the protocol, when the patient was unstable and required immediate intervention, and when it was thought to be medically justified to challenge the protocol suggestion. If time allowed, physicians at the bedside were required to discuss a protocol challenge with a member of the research staff. These challenges were always reviewed later by the entire team and were the basis for many of the protocol logic modifications. Computer and paper based protocol suggestions were compared for each therapy decision. If the two did not agree, the clinical staff deferred to the paper based protocol suggestions in the first 7 patients while the computer protocol problems were being solved, and to the computer based suggestions in the last 9 patients. Paper-based and computer protocol suggestion disagreement and all clinical decisions which differed from the protocol suggestions were examined and the reason for the difference classified into one of the following categories: Clinical staff logic differences (CS LOGIC): Protocol suggestions that were medically challenged, interpreted incorrectly or ignored. Software error (SW ERROR): Error in the software code. Incomplete data base (INCOM DB): Computer entry of data was delayed or absent at the time the protocol was activated. Data drive (DATA DRIVE): The mechanism for triggering execution of the protocols failed or the Tandem computer was inoperative. Undefined protocol logic (UNDEF LOGIC): Decisions made in sections of the protocols that were not finalized. Protocol not suspended appropriately (SUSPEND): Operation of the protocols was suspended during procedures or events not covered by protocols (e.g. During transport of the patient to the radiology department or during surgery). This error occurred when protocol operation was incorrectly suspended or not suspended. Data entry errors (DATA ENTRY): Incorrect data was entered in the computer by either the clinical or laboratory staff. Patient unstable (UNSTBL PT): The data in the computer data base at the time of evaluation was not representative of the patient's steady state (e.g. The
the Tandem directly and the remainder are connected via the CRDS computers. All nursing units have terminals located at the nursing station and at various hallway sites. The four intensive care units and one 48 bed acute care unit have terminals at each bedside as well as at the nursing stations. All clinical and laboratory information on each patient is stored in the integrated data base and is, therefore available for decision making. The data dictionary of the HELP system is a hierarchial representation of data elements known as PTXT. Patient demographic and clinical data is stored in coded form in a variety of active and archived files. Most of the programs which manipulate the data base are written in PTXT Application Language (PAL) (16), a structured programming language similar to Pascal. A few of the programs that require access to more fundamental operating systems functions (such as interprocess communication) are written in the Tandem Application Language (TAL), a structured programming language similar to C. Protocol Implementation Paper based protocols were developed by a team of 15 physicians and nurses from the departments of anesthesia, pulmonary, and critical care medicine. They used clinical information to generate suggestions for medical therapy. The protocols were designed as a means of providing uniform care for patients participating in a clinical trial comparing conventional positive pressure ventilation and extra-corporeal C02 removal (11,13). The preliminary computerized versions of the protocols have been extensively modified to incorporate logic changes, and additions, and to correct computer and bedside problems identified during the clinical trial. These changes included abandoning the use of blackboard control architecture, altering the mechanism which initiates execution of the protocols when an arterial blood gas analysis was entered into the computer (referred to as the "data drive"), and improving the user interface. The current protocols cover about 25 pages of flow diagrams and the computerized version has approximately 10,000 lines of PAL code. Figure 1 illustrates the basic organization of the computerized protocols. COMPUTERIZED RESPIRATORY CARE PROTOCOLS
Figure 1
Arterial blood gas laboratory data (or, alternatively, bedside pulse oximetry data) trigger protocol execution, resulting in the generation of a protocol decision and a suggestion for the.-apy. The generated decisions are based on the patient's medical data
589
protocol may have been activated during a period of transient hypoxemia associated with the suctioning of a patient's endotracheal tube or with a ventilator tube disconnection). Unknown (UNKN): A reason for digression from the protocol suggestions or for differences between the computer and paper protocols could not be identified. The paper based protocols, and consequently the computer based protocols were constantly evolving. The protocol logic used in subject 1 was, therefore, not identical to that used in subject 16. The differences included refinement of the decision logic to correspond more closely to the physician's intent, the addition of previously undefined logic, and correction of errors. Mantel Haenszel Chi-Square statistical analysis was used to evaluate the number of protocol suggestions followed as a function of time with a p value of less than 0.05 indicating significance (BMDP, BMDP Statistical Software, Inc., Los Angeles, CA).
Distribution of Differences Between Computer and Paper Protocols 2% 1%
7%
20}% z~~~~~~~~~~.% IV
89{u
13% 13/
I f
.
CS LOGIC SW ERROR INCOM DB DATA DRIVE UNDEF LOGIC SUSPEND DATA ENTRY UNSTBL PT UNKNOWN
..
18%
18
Figure 2
Results The number of protocol suggestions, the number of deviations from the protocol, the number of differences between the paper and computerized versions of the protocol, and the reasons for these differences are shown in table 1. The reasons, expressed as a percent of the total differences between the paper and computer protocols, are illustrated graphically in figure 2.
To examine the effects of the modifications in the computer protocols and changes in the paper protocols, the categories CS Logic, SW Errors, and Number Follow (the number of suggestions followed) were plotted against the patient's sequence number (Figure 3). Mantel Haenszel Chi-Square analysis showed a highly significant increase in adherence to protocol suggestions with TABLE 1
PATIENT NUMBER
TOTAL NUMBER NUMBER SlUtG FOLLOW. DIFF.
SW INCOM DATA UNDEF SUSPEND DATA UNSTBL CS DB DRIVE LOGIC ENTRY PT. LOGIC EFDR
5 2 14 8 5 13 4 8
0 0 0 4 0 9 1 26
0 0 2 6 7 0 1 0
0 0 0 5 3 0 1 0
0 0 0 1 8 0 8 0
1 0
0 0 0 0 0 0 0 0
110 3.10
107 3.01
68 1.91
57 1.60
16 0.45
5 0.14
89 4.70
108 5.71
91 4.81
59 3.12
40 2.11
12 0.63
5 0.26
58 3.49
2 0.12
16 0.96
9 0.54
17 1.02
4 0.24
0 0.00
10
366 1 44 592
56 46 81 87 98 246 106 488
81 89 87 79 86 120 38 1 04
13 27 8 10 18 56 1 11
27 10 17 28 21 5 6 22
5 11 8 20 5 12 15 13
21 39 40 0 0 0 1 7
0 0 7 36 23 8 7
9 10 11 12 13 14 15 16
25 40 82 1 00 703 5 498 208
24 39 78 84 637 4 464 194
1 1 4 16 66 1 34 14
0 0 0 0 15 0
0 0 1 0 5 0 6
0 0 1 4 25 1 16
1
11
0 1 0 0 1 0 0 0
TOTrALS
3553
2732 76.89
821 23.11
162 4.56
149 4.19
147 4.14
FIRST 8 PTS. 1 892
1208 63.85
684 36.15
144 7.61
136 7.19
1524 91.75
137 8.25
18 1.08
13 0.78
2 3 4 5 6 7 8
LAST 8 PTS.
1 37 1 35 1 68 1 66 1 84
1 661
1 2
590
UNKN
0 0 0 0 1 2 2 7
1 0 0 0 2 0
0 0 0 2 0 0 0 3
ICU, patient care must retain top priority, and data entry of patient parameters will be accomplished when the opportunity to do so exists. Automated data collection and recording is currently being tested in the ICU, using a Medical Information Bus (MIB) system. Implementation of the MIB should minimize this category of error. Data drive and computer operating system errors were the next most frequent cause of discrepancy between the two protocol forms. The data drive is a system tool which initiates a particular process whenever a specific data item is stored in the data base. This tool was originally used to activate the computer protocols whenever arterial blood gas data was stored for a protocol controlled patient. The tool proved to be unreliable and an alternate method of protocol initiation was developed and implemented after patient 3. The errors subsequently recorded under this category were due only to computer down time (0.3% of the time). The differences which occurred as a result of undefined protocol logic were more prevalent in the first 8 patients, but continued to occur in the later patients as well. These differences were a result of areas of the protocol logic that were not explicitly defined. The process of recording specific steps in the clinical treatment of an illness requires careful examination of not only the medical therapy, but also of the assumptions involved in instituting procedures and treatments in the clinical environment. For example, the current protocol logic contains the simple clinical question, "Is paralysis needed?". As yet, the patient parameters used and the clinical assumptions involved in answering this question have not been explicitly defined. Therefore, we have been unable to develop logic that would allow the computer to determine a patient's need for paralysis. Computerization of protocol logic forces the medical care provider to discard subjective decisions, and carefully examine the basis for decisions and is therefore, an excellent method of clarifying the process of medical thought, and identifying assumptions and deficiencies. Differences categorized as "Suspend Errors" are a result of incorrectly suspending or terminating suspension of the computer protocols. Protocol control is suspended for situations or processes which are out of the scope of protocol logic, such as patient transport, surgical procedures, dialysis, etc. The differences attributable to the "Data Entry" category were generally due to typographical errors which occurred during computer charting. The incidence of this type of discrepancy varied with the individual entering the data. An interesting side effect of computer protocol use was an improvement in the accuracy of the patient's medical record. Once an individual was familiar with protocol logic, he/she was able to recognize that a computer protocol suggestion was based on erroneous data. It became common for the clinical staff to return to the patient's medical record, edit the bad data, and generate a new protocol suggestion using the corrected data. The "Unstable Patient" category, although not frequent, continues to be a problem. Occasionally a computer protocol suggestion would be generated during a period when the information in the computer data base was not representative of the patient's true steady state. This occurred when minor manipulations of the patient, such as turning the patient, or changing the bed linen, caused temporary arterial oxygen desaturation. Protocol suggestions based on the transient data were to be ignored, and a new therapy suggestion generated by bedside activation of the protocol once the patient had stabilized.
time (p < 0.00001).
an
t6O
Z 0 5,:
jaQ
:E1
:.
1
:D
-ir-Ia i9 -. I3 4 C. 6II
*:. I.1 .-
7
I
3
n
M
eJ 1
1
Figure 3 Discussion Protocol therapy suggestions were followed 76.9% of the time in the sixteen patients (Table 1). The most common reasons for digression from the protocol were challenges to protocol logic by the physicians attending the patient, misinterpretation of or failure to follow the protocol (Clinical staff logic differences, Table 1). This category also includes instances in which protocol logic was ignored without a valid reason. Failure to follow the protocol without a valid reason was considered to be a function of the attending physicians clinical style, and such instances decreased as confidence in the protocols grew. The frequency of clinical staff logic differences decreased with time as a result of training and as experience with, and clinical confidence in the protocol increased. CS logic differences occurred with 7.6% of the protocol suggestions for the first 8 patients, and with 1.1% of the suggestions in the last 8 patients (Table 1). However, we expect logic differences to continue to be a problem as new clinical staff members rotate into the ICU and are introduced to protocol controlled patient care. Differences between the paper and the computer versions of the protocols were found for 23.1% of the therapy suggestions (Table 1). Software errors explained 7.2% of the differences in the first 8 patients and 0.8% of the differences in the last & patients. During the care of the first 7 patients the software was being updated to correspond with the current versions of the paper based protocols, and at the same time was being debugged. The paper version of the protocol was, therefore, used to direct clinical care for these patients. As new decision logic was added to the existing protocols, this process of testing and
debugging was repeated. The differences recorded as incomplete data base errors primarily a result of delayed data entry by the clinical staff, but also includes missing data or data never entered in the computer. These differences are distributed fairly evenly over the 16 patients (Table 1). Some improvement can be expected through training programs. However, we expect the effect of training will be limited. In the complex and stressful setting of
were
591
On five occasions, in the first 8 patients, a computer protocol suggestion was generated which could not be traced or reproduced. We believe that the most significant implication of this study is that protocol controlled care of critically ill patients is possible in spite of the complexity of the environment and the differing clinical styles of the medical staff. The clinical staff's adherence to protocol suggestions increased dramatically over time. After the debugging process, software errors in the computerized protocols proved to be insignificant. The major
1. 2.
3. 4.
5. 6.
7.
8.
9.
problems confronting use of computerized protocols are logistical problems. Accurate and timely data entry of the patient's clinical parameters and an understanding by the clinical staff of the scope of therapy covered by the protocols would eliminate practically all of the remaining differences. In summary, the computer based protocol performance and the comparison of computer based and paper based protocol suggestions establishes the feasibility of protocol controlled care of critically ill ICU patients.
10.
References Sisson JC, Schoomaker EB, Ross JC. Clinical decision analysis: The hazard of using additional data. JAMA 1976; 236:1259-1263. Shulman D, Donchin Y, Dekel S, Heller 0, Gopher D, Cotev S. Human errors in an intensive care unit: A pilot study. Crit Care Med 1987; 15:371. Grossman R, Hew E, Aberman A. Assessment of the ability to manage patients on mechanical ventilators using a computer model. Acute Care 1984; 10:95-102. Grimm RH, Shimoni K, Harlan WR, Estes EH: Evaluation of patient care protocol use by various providers. N Engl J Med 1975; 292:507-511. McDonald CJ: Protocol-based computer reminders: The quality of care and the non-perfectability of man. N Engl J Med 1976; 295:1351-1355. Cannon SR, Gardner RM. Experience with a computerized interactive protocol system using HELP. Comp Biom Res 1980; 13:399-409. Wirtschafter DD, Scalise M, Henke C, Gams RA. Do information systems improve the quality of clinical research? Results of a randomized trial in a cooperative multi-institutional cancer group. Comp Biom Res 1981; 14:78-90. McDonald CJ, Hui SL, Smith DM, Teirney WM, Cohen SJ, Weinberger M, McCabe GP. Reminders to physicians from an introspective computer medical record. Ann Int Med 1984; 100:130-138. Miller PL Goal oriented critiquing by computer: ventilatory management. Comp Biom Res 1985; 18:422438.
11.
12.
13. 14.
15.
16. 17.
Menn SJ, Barnett GO, Schnechel D, Owen WD, Pontoppidan H. A computer program to assist in the care of acute respiratory failure. JAMA 1973; 2323:308312. Morris AH, Menlove R, Rollins RJ, Wallace CJ, Beck E. A controlled clinical trial of a new 3-step therapy that included extracorporeal C02 removal for ARDS. ASAIO 1988; 11:48-53. Sittig DF, Pace NL, Gardner RM, Beck E, Morris AH. Implementation of a computerized patient advice system using the HELP clinical information system (Submitted to Com Biom Res, 6 December 1988). Sittig DF. Computerized management of patient care in a complex, controlled clinical trial in the intensive care unit. SCAMC 1987a; 11:225-232. Sittig DF, Gardner RM. Summary of the 9th annual conference Computers in Critical Care and Pulmonary Medicine, San Diego, CA, June 15-17, 1987b. Int J Clin Monit and Comp (in press). Gardner RM. Computerized management of intensive care patients. MD computing 1986; 3(1):36-51. Pryor TA. The HELP medical research system. MD Computing 1988; 5(5):22-23. Pryor TA, Gardner RM, Clayton PD, Warner HR. The HELP system. J Med Systems 1983; 7:87-102.
Supported by NIH Grant HL36787.
592
