An integrated software system for quality ... - Wiley Online Library

Clin. Cardiol. 16,745-752 (1 993)

This section edited by Marek Malik, M.D., Ph.D.

An Integrated Software System for Quality Assurance-Related Kappa Coefficient Analysis of Intraoperative Transesophageal Echocardiography Interpretive Skills T.- ~ , M . D . , *

B.EDWARDS,B.S.,$ J. JuDD,B.s.,$ V. SWMY,B.S.,$ R. WORKMAN,M.S.,$ H. LIPPMANN,R.N.,* S. WIS,M.D.,* I. COHEN, M.D.,? E. PROKOP, M.D.,$ M.EZEKOWrrZ, M.D.?

Departments of *Anesthesiology,tcardiology, and $Radiology,Yale University School of Medicine, and $Department of Computer Science, Southern Connecticut State University, New Haven, Connecticut, USA

Summary: This report describes the development of a quality assurance-oriented integrated software system designed for an anesthesiology-basedintraoperativetransesophagealechocardiography service.Entry data include patient and operation demographics, two-dimensionalechocardiographic,saline-contrast, and color flow/pulsed Doppler assessmentsof the heart and great vessels, presented in a defined sequence.A statistical analysis component (kappa coefficient analysis) allows for comparison of inhaoperativereal-time interpretationswith lab oratory interpretationsmade by experiencedfull-time echocardiographerson a field-by-fieldbasis. This provides a means of quantifying expertise in each individual aspect of the patient examination sequence. We believe that such self-appraisal data are essential for delineatingthe status and tracking the progress of service being provided. Key words: echocardiography, transesophageal,intraoperative, quality assurance, interpretation,kappa coefficient

Presented in part at Computers in Cardiology 1992, IEEE Engineering in Medicine and Biology Society Annual Meeting, Durham, North Carolina, October 1992 The publication of this paper has been facilitated by Computers in Cardiology, Inc. Address for reprints: Terence Rafferty, M.D. Department of Anesthesiology 333 Cedar Street, P.O. Box 3333 New Haven, CT 06510, USA Received: February 8. 1993 Accepted with revision: May 14, 1993

Introduction Transesophagealechocardiography (TEE) is being applied increasingly to intraoperative care of critically ill patients.', Rightly, opinions have been expressed that injudicious use of TEE might result in misdiagnoses and institution of inappropriate therapy.= Accordingly,it would seem prudent that any anesthesiology-basedintraoperativeTEE service develop formal mechanisms for regular systematic review of the validity of echocardiographic interpretations. We have previously described our quality assurance data in a report involving assessment of patient complications, quantification of the adequacy of procedure documentation,and quantitativeevaluation of the technical skills of the involved physician^.'.^ The quality assurance criteria employed in that study were chosen on the basis of logistic considerations. Specifically, it was necessary to construct a practical initial screening procedure which would lend itself to facile data extraction.8The evaluation was rudimentary in that there was no provision for analysis of interpretive skills. Since that initial evaluation, we have designed an integrated software system to facilitate data entry and analysis for a quality assuranceprogram. This system compares on-line intraoperative echocardiographic interpretations made by anesthesiologists with off-line interpretationsmade by experienced full-time echocardiographers. This report represents a systematic analysis specific to the involved issues.

Background Experience Our existing intraoperativeTEE monitoring program presently involves the p e r f o m c e of more than 800 cases per year. Organizational structureand quality aqsurance mechanisms include the following: (1) medical record documentation of the TEE examination; (2) intraopemtive completion of an examination databaseentry form; (3)provision of a videotape for each case, to be subsequently assessed for technical quality by external reviewers and stored in an archival filing system; (4)documenta-

146

Clin. Cardiol. Vol. 16, October 1993

tion of patient complications; (5)technician identification of the physician responsible for requisition of the TEE endoscope; (6) data entry into a permanent computer database, with notation of lesions suitable for education and research; (7) weekly conferences featuring review of recent cases; and (8) a monthly didactic seminarAecture series on basic imaging principles.* Our previous database was maintained using three commercially available software packages with major and minor revisions. Initially, data were stored within a StatPac Gold@system (Walonick Associates, Inc., Minneapolis, Minnesota). Our use of this software program, primarily a statistical analysis package, was related to the limited existence of commercially available true database programs. Subsequently,the most extensive data were contained within a Paradox@system (Borland International, Inc., Scott's Valley, California),for which prototypical software was developed for manipulating the database entry and for editing and reporting procedures.' The information in this Paradox@system consisted of patient and surgical procedure demographicdata, echocardiographicdata, and external cardiology review evaluations. Data pertaining to adequacy of medical record documentation, TEE database form completion, and provision of procedure videotape recordings of the intraoperative examination were maintained with the Paradox@-basedsystem and within a Wordperfect@document (WordPerfect Corporation,Orem, Utah). The Paradox@package allows for storing of data in layers, ideal for a neophyte intraoperativeTEE service. This program structurefacilitatedready expansion and revisions of the database which could not have been anticipated at the inception of the service. The actual database consisted of a single file containing all the information deemed to be pertinent at that time. Such a monolithic structure was consistent with maintenance of an archive and with storage of research data, the primary purpose of the database. An optimal arrangement would have been the creationof smaller, more manageable database tables oriented toward specific findings or areas of interest,for example, demographics,cardiac findings,quality assurance,administration, and so on. This framework would have facilitated data extraction. Disadvantages also included the paucity of userfriendly and erroneous input limiting interfaces.These are key features for timely and accurate input of information by clerical personnel. At the inception of the monitoring service, the absence of these features was not a major issue. However, as the clinical service grew, so too did the workload involved in data entry and retrieval, causing skilled resource personnel to become increasingly diverted from other activities.Because of this lack of accessibility,data requiring regular and timely retrieval, such as quality assurance reports, were also maintained in tandem with a more user-friendly program (Wordperfect@). This required separate data entry-a wasteful duplication of services.

se1s.l". I 1 These scans are acquired by the anesthesiologist of record for the operative procedure. Specifically,the study consists of a 10-step sequence of two-dimensional echocardiographic views (Figs. 1-10], salinecontrastinterrogationof the interatrial septum, a 5-step sequence of color flow Doppler views (Figs. 1 1-15), together with color M-mode assessment of the left ventricular outflow tract and pulsed Doppler evaluation of pulmonary vein flow. The previously developed TEE database entry form, completed intraopemtively, consists of 155

TEE Protocol

FIG.2 Two-dimensional echocardiographic basal short-axis view of the great vessels. The cross-section is such that the aorta (Ao) and superior vena cava (SVC) are concentric.The Ao lies anteromedial to the bifurcation of the main pulmonary artery (MPA).The SVC lies directly anterior to the right pulmonary artery (RPA).

The TEE data consist of recorded sequences of standard transverse plane cross-sections of the heart and great ves-

FIG.1 Two-dimensional echocardiographic view of the transverse and descending thoracic aorta. (A) The stippled area represents that portion of the transverse and descending thoracic aorta accessible to ttilnsverse plane imaging. (B and C) The transverse aorta appears as a linear structure, while the descending portion is concentric.

T. Rafferty et ul.: Software system for analysis of TEE interpretive skills

FIG.3 lXvo-dimensional echocardiographic basal short-axis view at the level of the aortic valve. The aortic valve (AV) is represented during diastole. The level of this cross-section is such that the right atrialyRA) area is normally less than that of the left atrium (LA). It should be noted that the AV and right ventricular outflow tract (RVOT) be at the same level, the pulmonic valve being superior to both structures. LCC = left coronary cusp, RCC = right coronary cusp, NCC = noncoronary cusp.

747

FIG. 5 Two-dimensional echocardiographic long-axis view of the mitral valve and left ventricle. As illustrated, the anterior mitral leaflet (AML) is normally disproportionately longer than the posterior leaflet (PML).It should be noted that the true apex of the left ventricle (LV) may not be visualized. LA = left atrium, RV = right ventricle.

fields, encompassing patient and procedure demographic data in addition to the listing of the echocardiographic and Doppler interpretations. I 2 Intraoperative interpretations are expressed either in a dichotomous fashion (yesno or present:absent), for example, thrombus, vegetation, and so on, or as values on a 4-point ordinate scale (norma1:mild:moderate:severeor normal:hypokinetic:dyskinetic:akinetic),specific to valvular re-

FIG.6 Two-dimensional echocardiographicbasal short-axis view of the left atrial appendage. As illustrated, the left atrial appendage (LAA) lies anterior to the descending thoracic aorta (Ao) and the left upper pulmonary vein (LUPV). LA = left atrium.

FIG.4 Two-dimensional echocardiographiclong-axis view of the left ventricularoutflow tract. This cross-section is identified by such landmark structures as the aortic valve (AV), with its characteristic “Mercedes-Benz”emb1em morphology, and the anterior m i d leaflet. LVOT= left ventricularoutflowtract, LV = left ventricle, RV =right ventricle, LA = left atrium, IVS = interventricularseptum.

FIG.7 Saline-contrasttwo-dimensional echocardiographicimaging of the interatrial septum. The interrogation should include several views of the interatrial septum. The ultrasound-opaque cavitations represent intravenously administered agitated saline. As illustrated, a right-to-left shunt is evidenced by the presence of saline in the left atrium (LA). RA = right atrium, RV = right ventricle, AV = aortic valve.

148


FIG.8 Wo-dimensional echocardiographic long-axis view of the right ventricle at the level of the anterior niitral leaflet. In this crosssection, the normal tricuspid valve (TV) is usually not visualized in its entirety. As with the long-axis view of the left ventricle, the true apex of the right ventricle may not be visualized. RA = right atrium, AML = anterior mitral leaflet, RV = right ventricle, LV = left ventricle, LA = left atrium.

gurgitation and ventricular regional wall motion, respectively. In cases where image quality is deemed to be inadequate for valid diagnosis, the field is categorized as “not well seen” ( N W S ) . Similarly, in cases where image quality is adequatebut the observer is undecided as to the diagnosis, the field is classified a,,“uncertain of diagnosis” (UNC). Finally, “not done” (ND) defines that no evaluation was performed in a particular field. The cardiologist reviewer form is completed off-line using the

FIG.10 Ibo-dimensional echocardiographic transgastric short-axis view of the right and left ventricle. ‘Thisview rcptcsents a mid-papillary muscle level cross-section. Asymmetry of the papilliuy muscles usually indicatesan improper oblique section.The left ventricular (LV)image is divided into four segments, namely, posterior, lateral, anterior, and septum. The motion of each of the segments is graded separately.RV = right ventricle.

FIG.9 Two-dimensional echocardiographiclong-axis view of the right ventricle at the level of the coronary sinus. The cross-section is identified by imaging the orifice of the coronary sinus. This structure lies proximal to the septal leaflet of the tricuspid valve. The right ventricular (RV) chamber is invariably foreshortened at this level. RA = right atrium, LA = left atrium, LV =left ventricle.

recorded videotapes, utilizing an identical format to allow for direct comparison of interpretations. The resulting spectrum of total possible interpretations can be represented in the form of a rdw data grid (Figs. 16, 17).For dichotomous variables, the anesthesiologist has four interpretation choices, that is, positive, negative, N W S , or UNC. The cardiologist, likewise, has the same four interpretation choices.Accordingly, the total possible combinations arising from the grid are represented by 16data intercomparison combinations. For variables on a 4-point ordinate scale, the anesthesiologisthas six interpretation choices, that is, normal, mild/hypokinetic, moderatddyskinetic, severe/akinetic, N W S , or UNC. Again, the cardiology reviewer has the same six interpretation choices, giving rise to a total of 36 possible data intercomparison combinations.

FIG.1 1 Color-flow Doppler view of the aortic valve. (A) Two-dimensional echocadiogmphicview of the left ventricular outflow tract (LVOT) is presented for orientation. (B) Color-flow Doppler-defined aortic regurgitation (AR), the discrete brightly speckled area of backflow into the LVOT (arrow).AV = aortic valve, AML = anterior mitral leaflet, IVS = interventricularseptum.

T. Rafferty et al.: Software system for analysis of TEE interpretive skills

749

FIG.12 Color-flow Doppler view of the mitral valve. The bright speckled area within the left atrium (LA) illustrates color-flow Doppler-defined mitral regurgitation (MR). MV = mitral valve, LV = left ventricle, RA = right atrium, RV = right ventricle, TV = tricuspid valve.

FIG.15 Color-flow Doppler view of the interventricular septum. The bright speckled area within the right ventricle (arrow) represents color-flow Doppler-defined left-to-right shunt flow through a ventricular septal defect. LA = left atrium, RA = right atrium, LA = left atrium, TV = tricuspid valve, RV =right ventricle, VSD = ventricular septal defect, LV = left ventricle.

Cardiology interpretation

FIG. 13 Color-flow Doppler view of the interatrial septum. The elongated jet extending across the right atrium (arrow) represents colorflow Doppler-defined left-to-right shunt flow through a patent foramen ovale. RA = right atrium, LA = left atrium, IAS = interatrial septum, RV = right ventricle.

C.-

Eiii

+ '

NWS 3 3 UNC

Cardiology interpretation 0 1 2 3 NWSUNC

CUC

Ez

.2

0

85

sp

0

; 3

3dNWS

=.& UNC

FIG. 14 Color-flow Doppler view of the tricuspid valve. The bright speckled area within the right atrium (RA) represents color-flow Doppler-definedtricuspid regurgitation (TR). RV = right ventricle.

FIG.17 Interpretationpossibilitiesfor fields involvingordinate variables. The grid illustratesthe spectrum of total possible interpretation choices. Ordinate measurements were restricted to valvular regurgitation (normal; mild; moderate; severe) and left ventricular regional wall motion (normal; hypokinetic;dyskinetic;akinetic).The 4-point scale was expressed as follows: 0 = normal; 1 = mild or hypokinetic; 2 = moderate or dyskinetic; 3 = severe or akinetic. N W S = not well seen, UNC = uncertain of diagnosis.

750


TEE Database System

Conclusion

The data entry program was created using a “Microsoft Windows” version of Object Vision 2.1 (Borland International, Inc., Scott’sValley, CA). The computer display data entry SCreen was designed to closely resemble the actualdata collection document. The encoding of fields was by alphanumeric, numeric, and “check box” methods, with intentional minimal provision for free-text entry. Check box fields which were mutually exclusionary were programmed as such. This provision of a frontend interface event tree filtering mechanism decreased the necessity for specific line-by-line coding in the data analysis section of the propun, with the added benefit of allowing facile and rapid data input, guarding against impper data enhies,and minimizing archival storage space requirements. Initial data conversion consisted of programmed segmentation of related items into file structures designed to separate specific functions or areas of interest, such as demographics,cardiac findings, and so on. The file-linking features of the softwareare used to mate Paradox@’ 4.0 (Borland International, Inc., Scott’s Valley, California) data tables. Within the Paradox@’program, these data were organized to create summary reports and the echocardiography interpretive fields were translated for input to an S A P statistical analysis program (SAS Institute, Inc., Cay, North Carolina).

This report describes the developmentof a prototypical integrated software system for evaluation of echocardiography interpretive skills, customized for use by an anesthesiologybased intraoperative TEE service. This program was created for the purpose of organizing data for direct automated input into a standard statistical analysis package for quantification of interobserver variability (real-timeintraoperativeinterpretations vs. cardiologistlaboratory interpretations)on a field-byfield basis. Such a database allows for assessmentof expertise in each individual aspect of the TEE examination.We believe that the quality assurance data obtained are a sine qua non for delineating the status and tracking the progress of service being provided. This is particularly essential in a complex and rapidly evolvingfield such as TEE. This software program involves several innovative concepts. The performance of quality assurance for echocardiography interpretive skills, in essence, involves assessmentof interobserver variability, with one group of observers designated as the “gold standard.”Measurementvalidation by evaluation of interobserver variability is, of course, a normal component of any echocardiography or Doppler research endeavor. Superficially, it would seem reasonable to suppose that standard research methods could be directly applied to a transesophageal quality assurancesituation without modification or specialprovisions. Such is not the case for the following reasons. First, in research protocols, poor quality images are automaticallyeliminated fromthe study material. In contrast, it is essential that thesefields be analyzed in quality assurancesituations. For instance, an intraoperative observer might well feel that a particular view was uninterpretable because of poor image quality, whereas the external reviewer might differ and believe that an interpretation was certainlypossible and should have been made despite the quality of the image. On the other hand, there might be concurrence of judgment or, indeed, a reverse situation.The necessity to quantify the degree of concordance or discordance in this category led to the creation of the N W S (“not well seen”) variable in the interpretation grids. Similarly, provision was made for situations in which an adequate image was obtained but the intraoperativeinterpreter had no idea as to the nature of the involved pathology. Again, this UNC (“unclear”)category would not arise in a research situation as the reviewers are assumed to be recognized experts in the field. Second, in this software program, every variable, be it dichotomousor ordinate, is coded for kappa coefficientanalysis.The use of a single statistical analysistechnique allows for facile interpretation of findings. In addition, the units in which the correlation values are reported (range 1.O-1 .O) can be readily related to by persons who do not necessarily possess a research background. Moreover, this statistical analysis method is epidemiologically sound and appropriate to the issue being studied. Any survey of echocardiography and Doppler literature reveals a different situation. Dichotomous variables have been reported as sensitivity, specificity, and positivdnegativepredictive values and kappa coefficient analysis. Ordinatevariables have been analyzedby Spearmanrank correlation coefficientanalysis,Bland-Altman

Statistical Analysis Component A wide spectrum of statisticalmethods can be employed to evaluate the validity and reproducibility of interpretations of echocardiographicdata. The analysis component in this particular system involvesthe use of the kappa coefficient of agreement. The kappa coefficient has been stated to be the preferred single index for quantifyingthe magnitude of agreement and for comparing studies of observer variability.1s16 The kappa coefficient measures agwment of categorical measures above that of chance agreement. Specifically,the kappa coefficient depicts the ratio of the proportion of agreement to the maximum possible proportional agwment, corrected for chance.17Perfect agreement is represented by a kappa coefficientof unity. Decreased values represent degrees of disagreement(range 1.O1.O).A value of 20.75 has been proposed as indicating acceptable concurrence in this particular context.18This standard analytic technique allows for quantification of the degree of concordancddiscordancebetween intraoperative interpretations and those of the cardiology reviewers. By intent,all measurement sets in the fields of the databaseentry form were expressed as either dichotomous or ordinate variables (vide supra), a method consistent with the use of this parameter. The correlations (kappa statistic)between intraoperative interpretations and the “gold standard”(cardiologist reviewer interpretations) are calculated for each field of the orderedcomplete examination sequence (descendingthoracic aorta through cardiac long-axis and transgastricshort-axis scans).Values are listed both for all fields (overall kappa) and by individual field, allowing for self-appraisal on each aspect of the examination.

T. Rafferty et al.: Software system for analysis of TEE interpretive skills

analysis, kappa coefficientanalysis, or mean variability or error. Mean variability or errorhas been defined as (a) estimate 1 minute estimate2, with the averageof the differencesreported in absoluteterms without a frame of reference;(b) with calculations performed individually for each matching measurement set, with subsequent averaging and expressed as percentage error; (c) as mean percentage error, asjust described, with the major modificationthat entered values were restricted to absolute differences; and (d) where the precise method of calculation of the mean percentage error was not specified.Finally, interobserver variability has also been evaluated by listing either the standard error of the estimate of each of the data sets, by the standard deviation of the coefficient of variation of each data set, or by expressingthe standard deviation of the first data set as a function of the averageof the mean values of the two data sets. While most of these multiple reporting methods are not necessarily inappropriate in the given situations,at least some are difficult to interpret, even given a solid foundation in statistical analysis techniques. In addition, assessment of different variables in differing fashions makes interpretation of any final report confusing. Sincethe purpose of the quality assurance reports is to maintain or improve the quality of service being p~ vided, it is crucial that the administratorof the quality assurance program be able to communicate the findings in terms that are simple and easy to interpret. Since unsuspected cardiovasculardisease can be detected by TEE,valid assessment of intraoperative interpretive skills must involve fields of interrogation representing the essential elements of a routine diagnostic study. The TEE data in this particular program are derived from sequential transverse plane scans of the heart and great vessels, defined unequivocally by presentation of the images and matching diagrams. It is our aim that the ordeml sequence might aid the beginning practitioner in performing and interpretinga nominally complete intraoperative study. Notwithstanding the fact that the interpretations and videotape recordings are furnished by practitioners whose primary responsibility is the delivery of anesthesia care,it is the authors’ experience that these data can be provided repmducibly on a mutine basis without interfering with clinical management. The images are acquired and interpreted during periods of hernodynamic stability. This patient examination and interpretation sequence takes approximately 5-7 min. This duration is somewhat shorter than that generally cited in the literature, where Erbel eta1.19have reported durations“less than 15minutes’’for a standard TEE examination. Schlutereral.” have stated that “the investigation rarely takes longer than 10minutes.” T i e savings arise from the fact that (1) the endoscope is placed immediately following endotrachealintubation and therefore is already in place from the beginning of the case; (2) the patients are undergoing general anesthesiaand patient toleration is not an issue; (3) virtually all of the patients, being either cardiac surgery cases or undergoing other high-risk procedures, have flow-directed pulmonary artery cathetersin place, allowing immediate access to a right atrial injectate port for highquality saline-contrastimaging; and (4)none of the interpretations involve timeconsumingplanhetry. An inability to perform the TEE examination sequence accurately and expedi-

75 1

tiously may be due to a need for continued “hands-on” experience, coupled with a lack of educational motivation, rather than a need for further one-on-one training. Further, it might reasonably be proposed that repeated failure to obtain these data by experiencedpersonnel represents dilettante behavior and substandard care. It should be emphasized that completion of the intraoperative TEE database does not eliminatethe necessity for tracking TEE-defined changesin cardiovasculardynamicson the anesthesia operative record, the standard mechanism for relating changes in functionalstatus to intraoperative events and therapeutic interventions.As with any other measured hemodynarnic variable, we recommend that the anesthesia recordshould contain a chronologic time line of the entire surgical procedurewith reference to specific relevant fields, such as regional wall motion, with periodic functional status updates. The American Society of Echocardiography and the American Heart Association have defined minimum levels of training and the cognitive skills required to perform a competent echocardiographicstudy, and have noted that training equivalent to that of a fellowshipcan be acquired through preceptor21 ships and ongoing case review with recognized experts.20* The purpose of this software system is consistent with these recommended guidelines. This report represents a tentative definition of a practical intraoperativeTEE examination broken into component parts for ease of analysis. TEE videotapes will be provided by practitioners at variable points on a learning curve. It must be expected that, in many instances, a lack of skill and facility with the equipment will manifest itself in prolonged and incompletely edited recordings, making the review process a tedious and timeconsuming task? Accordingly, it is our intent that 25% of examinations be selected for external review, with the selection process guided by a random number generator to eliminate bias. Reviewed v i b tapes will also be randomly selected for repeat evaluation to determine reviewer variability. . It should be emphasized that there are numerous equivalent ways in which this softwareprogram might have been written, including the use of other languagessuch as C language. It is not to be expected that the program reported offers unique advantages.In this project, an Object Vision@program was created to closely resemble a previously developed database form (vide supra). A major feature of Object Vision@is that it is both a form design and a program developmentproduct, structured so that the data entry form design and the programming process can be combined into a single step. When this technique is employed,the created data entry form is also the data entry program, thus eliminatinglaborious and time-consuming programming.This feature should prove useful in the development of similar software for biplane and omniplane imaging. Other anticipatedareas of investigation includeexplorationof methods for direct data input from the operating room to remote location hard drives using voice and pen-based data entry cellular communicationsystems. In summary, we have chronicled the development of an integrated software system for quantitativeanalysisof real-time intraoperative TEE interpretive skills based on a defined se-


152

quence of transverse plane views of the heart and great vessels. This program is intended to serve as a guide for completion of an inmperative study in additionto providing a mechanism for collective self-appraisal. It is hoped that this system will enhance data acquisition, storage,and retrieval, and that it will imp v e patient cace by facilitatingready feedback of performance of intraoperativeTEE examinations.

References 1. Rafferty T Transesophagealtwo-dimensionalechocardiography in

2.

3. 4. 5. 6. 7.

8.

9.

the critically ill-is the Swan-Ganz catheter redundant? Yale J Biol Med 64,375-384 (1991) RafTerty T Intraoperative monitoring of ischemia and systolic cardiac function. In TransesophagealEchocardiography: Clinicaland InrraoperariveApplications (Ed. Miseri I).Churchill Livingstone, New York(1992) 181-227 Feigenbaum H: Growing interest in transesophageal e c h d o g raphy (editorial).JAm Soc Echo 3,347 (1990) Goldman M, Mindich B, Nanda N: Intraoperativeechocardiography: Who monitors the flood once the flood gates are opened? (Editorial).J A m Coll Cardiol 11,1362-1364 (1988) Kaplan J: Monitoring technology: Advances and restraints (editorial). J Cardiorhorac VascAnesrh 3.257-259 (1989) Conti C: Invasive"non-invasive" cardiology(editorial). Clin Cardwl 14,545-546(1991) Rafferty T, LaMantia K. Davis E, Phillips D, Gaal D, Harris S, Carter J, McCloskey G , Muller J, Godek H, Rimar S,Kraker P, Jaeger D, Kosarussavadi B, Hines R, Kopriva C, O'Connor T Quality assurance in intraoperativetransesophageale c h d o g r a p h y a critical analysis of a two-year experience (abstr). J Clin Monitor 8,170(1992) RafTerty T. W a n t i a K, Davis E, Phillips D, Harris S,Carter J. Ezekowitz M, McCloskey G,Godek H, Kraker P, Jaeger D, Kopriva D, Barash P: Quality assurance for intraoperative transesophageal echocardiography: A report of 846 procedures. Anesth Analg 76, 228-232 (1993) LaMantia K, Davis E, Phillips D: Anatomy of a transesophageal monitoring service. Scientific Exhibit, American Society of Anesthesiologists Annual Meeting, Las Vegas, Nevada, 1990

10. Seward JB,Khandheria BK, Oh JK, Abel MD, Hughes RW, Edwards WD,Nichols BA, Freeman WK, Tajik AJ: Transesophageal echocardiography:Technique, anatomic correlations,implementation, and clinical applications. Mayo Clin P m 63,649-680 (1988) 11. Schluter M, Hinrichs A, Thier W, Kremer P, Schroder S,Cahalan M, Hanrath P Transesophageal two-dimensional echocardiography: Comparison of ultrasonic and anatomic sections.Am J Cadi01 53,1173-1178(1984) 12. RaEerty T, Davis E, Lippmann H, Penino A, Harris S,Carter J, Prokop E, Cohen I A clinical computer database entry form for an intraoperative anesthesiology-basedtransesophageal echocardiography monitoring service. J Clin Monit Comput (in press) 13. Feinstein A, Cicchetti D: High agreement but low kappa. I: The problem of two paradoxes. J Clin Epidemiol43,435-550 (1 990) 14. Cicchetti D, Feinstein A: High agreement but low kappa, II: Resolving the paradoxes. J Clin Epidemiol43.551-558 (1990) 15. Goldman R The reliability of peer assessmentsof quality of care.J Am Med Assoc 267.958-960 (1992) 16. Goldman R: The K statistic-in reply. J Am Med Assoc 268, 25 13-25 14 ( 1992) 17. Non-Paramerric Sratisricsfor the Behavioral Sciences (2nd ed.) (Eds. Siege1 S. Castellan I). McCraw-Hill, New York (1988) 2 8 4 919 18. Maurer G, Czer L, Chaux A, Bolger A, DeRobertis M, Resser K, Kass R, Lee M, Matloff J: IntraoperativeDoppler color flow map ping for assessment of valve repair for mitral regurgitation. Am J Cardiol60,333-337 (1987) 19. Erbel R, Daniel W, Visser C, Engberding R. Roelandt J, Rennollet H: Echocardiography in diagnosis of aortic dissection. Lancer 1, 456461 (1989) 20. Popp R, Winters W Clinical competence in adult echocardiography-a statement for physicians from the ACP/ACC/AHA Task Force on Clinical Privileges in Cardiology, Cirrulation 81,20322035 (1990) 21. Pearlman A, Gardm J, Martin R, Parisi A, Popp R, Quinones M, StevensonG,SchillerN, Sewad J, Stewart W Guidelines for physician training in transesophagealechocardiography: Recommendations of the American Society of Echocardiography Committee for Physician Training in Echocardiography.JAm Soc Echocadwg 5,187-194 (1992)