Procedural Performance in Gastrointestinal Endoscopy ... - Springer Link

3 downloads 19685 Views 276KB Size Report
May 4, 2010 - compare the use of a human and a computer-based tech- nical skills assessment ... oscopy) were performed by 18 consultants and 37 trainees.
World J Surg (2010) 34:1764–1770 DOI 10.1007/s00268-010-0579-0

Procedural Performance in Gastrointestinal Endoscopy: Live and Simulated Sudip K. Sarker • Tark Albrani • Atiquaz Zaman Isis Kumar



Published online: 4 May 2010 Ó Socie´te´ Internationale de Chirurgie 2010

Abstract Background The purpose of the present study was to compare the use of a human and a computer-based technical skills assessment tool in live and simulated gastrointestinal endoscopies performed by consultants and trainees. Methods Validated human-based Likert scales were used individually for generic and specific technical skills for two procedures. Two observers assessed each procedure independently and blindly. Computer-based assessment tools were also used, assessing aspects of the mucosa as well as various procedural times. The two tools were then compared. Results A total of 210 live and simulated endoscopies (107 esophagogastroduodenoscopy [OGD], 103 sigmoidoscopy) were performed by 18 consultants and 37 trainees. Mean inter-rater reliability using Cronbach alpha was good for the human-based tool (range: p = 0.62–0.75). Construct validity was good for the human-based tool (analysis of variance [ANOVA] range: p = 0.000–0.002) but not for the computer-based tool (ANOVA range: p = 0.263– 0.701). Conclusions This human-based technical skills assessment tool of gastrointestinal endoscopies seems to have face, content, concurrent, and construct validities compared to computer-based assessment parameters. The tool has the

S. K. Sarker (&)  I. Kumar Colorectal Surgical Unit, Whittington Hospital, Magdala Avenue, London, UK e-mail: [email protected] T. Albrani  A. Zaman Academic Department of Upper GI Surgery, Queen Mary’s College London, London, UK

123

possibility of being used in training, self-appraisal, and revalidation.

Introduction In the last few years, significant developments have been made in the objective assessment of technical skills, and a range of methods have been developed [1]. The assessment of technical skills during training has been considered to be a shape of quality assurance for the future [2].The assessment of technical ability is the job of the trainers, and their assessment is mainly subjective [3]. Objective assessment is necessary because deficiencies in training and performance are hard to correct without objective feedback [4]. Significant progress has occurred in the advance of methods that objectively evaluate surgical technique [5]. Objective and unbiased data on procedural performance are essential during the prospective assessment in training [6, 7]. Generic and procedure-specific technical skills are different kinds of skills that are acquired at different times during endoscopy training. Generic technical skills are mainly acquired during basic training—e.g., handling of the scope, handling of controls, hand–eye and monitor co-ordination, etc. These basic skills are required by the endoscopist to perform a complete procedure. They alone do not form a task or sub-task but they allow the endoscopist to perform the task. Specific technical skills are learned throughout basic and higher endoscopy training—e.g., luminal vision, strategy for progression, and identifying end landmarks. These specific technical skills are essential to the quality and outcome of the procedure. Virtual reality is defined as a compilation of technologies that allow people to cooperate competently with

World J Surg (2010) 34:1764–1770

1765

score 1–3, body mass index (BMI) \30, and [18 years or \80 years old. Two high-volume live and simulated elective endoscopic procedures were chosen, esophagogastroduodenoscopy (OGD) and flexible sigmoidoscopy. They were chosen because they are done in large numbers in the UK and worldwide by both consultant/attending and trainees. The entire procedure (live and simulated) was recorded and converted to DVD. Each full-length procedure was then assessed prospectively by at least two experienced observers, who assessed each procedure independently and blindly with the human-based assessment tool. For the computer-based assessment tool, only simulated endoscopies were assessed.

three-dimensional computerized databases in real time by using their natural senses and skills [8]. Medical virtual reality systems allow interaction to occur through an interface, by producing three-dimensional images of figures, which users can operate as virtual reality simulators are computer-based systems that generate output data, which are called metrics [1]. Computer-based tools for assessing technical skills target, for example, economy of movement, length of path, and instrument errors [9]. One of the main advantages of virtual reality systems, in comparison to dexterity analysis systems, is that they provide real time feedback about skill-based errors [1]. Recently, the virtual reality simulator has become popular as a tool for teaching and assessment of surgical and endoscopic skills [8, 10], and it permits quantitative and entirely objective evaluation of operator performance [11]. However, there has not been a comparison of human-based or computer-based assessment tools in endoscopy procedures. If we are to use virtual reality simulators for teaching and assessment, we must have robust and valid assessment tools in place before we embark on a curriculum of acquiring skills in the simulated environment. In the present study we sought to compare the usefulness of human and computer-based assessment tools in assessing endoscopic procedures. The end point of the study was to determine which assessment tool would be more effective and realistic in a training and appraisal environment.

Human-based assessment tool Generic technical skills To assess generic endoscopic technical skills, a hierarchical task analysis was done as previously described [12, 13]. From this a validated Likert generic scoring system was used [devised by first author (SKS)] [14] (Table 1). Specific technical skills A hierarchical task analysis was constructed on OGD and flexible sigmoidoscopy as previously described [12, 13]. From this a validated weighted Likert scale was constructed focusing on key points of the procedure [devised by first author (SKS)] [14]. The scoring system was scored double of the generic skills, as this section is more clinically relevant to the technical outcome of the procedure. Table 2 is the procedure specific technical skills assessment tool.

Methodology Setting The procedures were performed at the West and East London Hospitals, London, UK between 2004 and 2007. Consultants/attending (had previously performed [500 gastrointestinal endoscopies) and trainees (had previously performed \100 gastrointestinal endoscopies). All live patients were between American Society of Anesthesiologists (ASA)

Computer-based assessment tool The GI Mentor II (Simbionix Ltd, Israel) virtual reality endoscopic simulator was used in the study. Endoscopic

Table 1 Generic endoscopic technical skills Generic skill

1

2

3

4

5

Handling of scope

Awkward

Competent

Expert

Handling of controls Force used

Many unnecessary moves Rough and excessive force

Some unnecessary moves Gentle most of the time

All purposeful movements Gentle at all times

Hand, eye, and monitor coordination

Poor

Good

Expert

Patient discomfort and reaction

Severe discomfort; ignores discomfort and proceeds

Some discomfort; stops and proceeds when safe

Minimal discomfort; stops and proceeds when safe

Flow of procedure

Unsure

Confident with occasional hesitation

Confident at all times

123

1766

World J Surg (2010) 34:1764–1770

Table 2 Specific endoscopic technical skills Specific skill

2

4

6

8

10

Luminal vision

Inadequate; always on mucosal wall

Adequate and occasionally on mucosal wall

Almost all of mucosa visualized

Therapeutic procedure (biopsy or polypectomy)

Clumsily done; inadequate specimen

Competently done; adequate specimen

Expertly done; good specimen

Strategy for progression (torque/patient positioning/irrigation/scope withdrawal)

Has no strategy when technical difficulty arises

Uses appropriate strategy and competently progresses

Uses appropriate strategy and progresses expertly

Identifies end landmarks (sigmoidoscopy– transverse colon) (OGD–duodenum D2)

Not clearly identified

Adequately identified

Clearly and expertly identified

OGD esophagogastroduodenoscopy

procedures on the GI Mentor were performed on a humansized mannequin with a modified Pentax ECS-3840F endoscope. As in real life, steering and torque of the endoscope are possible; suction and inflation buttons are also available. Based on the characteristics of the GI tract and the motion of the scope, the simulator gives the sensation of force feedback. The computer registers objective performance parameters, and gives you a summary of the performance at the end of the procedure. The virtual endoscopy (free hand) task includes different modules: upper GI and sigmoidoscopy modules. Each module has 10 virtual patients complete with case histories, biological test results, and X-ray films. The cases in the two modules are arranged from more simple diagnostic procedures (case 1) to difficult cases involving therapeutic procedures in patients with major pathologies (case 10). For this study, cases 1 and 2 were chosen for both procedures. Doctors were shown the GI Mentor II virtual reality endoscopic simulator and introduced to its various built-in tools. The instrument was placed in good daylight, with no direct path of wind or blowing air. The room was centrally air-conditioned. There was no disturbance from noise or any interruptions. All the doctors used the same machine. They all were shown the simulator machine and were guided regarding its various handles in a standard format by a brief demonstration of the function of the endoscopy. No guidance, instruction, or feedback was given to the doctors regarding the technique for performing the gastrointestinal endoscopy before, during, or after the attempt. Various parameters were calculated and registered by the computer system (see below). These were used to assess the simulated procedures also. Mucosal score The percentages of mucosa surface examined and efficiency of screening of the mucosa were added to together to form a mucosa score. The higher the score, the better the endoscopist was evaluating the mucosa.

123

Time score The total time in seconds of the procedure, time the patient was in pain, time to reach end landmarks (the second duodenum in OGD and the transverse colon in flexible sigmoidoscopy) were added up in seconds. The lower the score, the quicker the endoscopist was in doing the procedure and the shorter the time the patient was in pain. Statistical analysis Data were collated in an Excel database (Microsoft, Redmond, WA). Statistical analysis was carried out with SPSS software statistical package (SPSS, Chicago, IL). For reliability Cronbach alpha was used, and a value [0.61 was deemed significant for reliability. For the non-parametric data the Repeated Measures analysis of variance (ANOVA) test was used for comparison between the two groups of endoscopists for validity analysis. A value of p \ 0.05 was regarded as statistically significant.

Results There were no major procedural complications or technical difficulties. A total of 210 endoscopies were assessed (103 flexible sigmoidscopies, 107 OGD). There were 105 live and 105 simulated procedures. There were 18 consultant/attending and 37 trainee (postgraduate year 4–8) participants recruited to the study. There were 44 male endoscopists and 11 female endoscopists, 51 right-handed and four left-handed endoscopists. All of the live trainee procedures were supervised by a consultant/attending, although that person may not have been present throughout the entire procedure. The simulated procedures were unsupervised. Inter-rater reliability between independent observers using Cronbach alpha were as follows: flexible sigmoidoscopy Generic Skills Likert, Cronbach alpha = 0.73, p \ 0.05; flexible sigmoidoscopy Specific Skills Likert,

World J Surg (2010) 34:1764–1770

Cronbach alpha = 0.75, p \ 0.05. For OGD the inter-rater reliability was as follows: Generic Skills Likert, Cronbach alpha = 0.69, p \ 0.05; Specific Skills Likert, Cronbach alpha = 0.62, p \ 0.05. Construct validity in live and simulated procedures between consultants and trainees using the human-based assessment tool were as follows: OGD Generic Skills Likert (Fig. 1), ANOVA p = 0.000, OGD Specific Skills Likert (Fig. 2), ANOVA p = 0.002, flexible sigmoidoscopy Generic Skills Likert (Fig. 3), ANOVA p = 0.000, and flexible sigmoidoscopy Specific Skills Likert (Fig. 4), ANOVA p = 0.000. Construct validity between consultants and trainees using the computer-based assessment parameters were as follows: OGD mucosal score (Fig. 5), ANOVA p = 0.701; OGD Time Score (Fig. 6), ANOVA p = 0.278; flexible sigmoidoscopy mucosal score (Fig. 7), ANOVA p = 0.695; and flexible sigmoidoscopy time score (Fig. 8), ANOVA p = 0.263.

1767

Fig. 3 Sigmoidoscopy generic technical skills

Fig. 4 Sigmoidoscopy specific technical skills Fig. 1 Esophagogastroduodenoscopy (OGD) generic technical skills

Fig. 2 Esophagogastroduodenoscopy (OGD) specific technical skills

Fig. 5 Esophagogastroduodenoscopy (OGD) mucosal score

123

1768

World J Surg (2010) 34:1764–1770

Discussion

Fig. 6 Sigmoidoscopy mucosal score

Fig. 7 Esophagogastroduodenoscopy (OGD) time score

Fig. 8 Sigmoidoscopy time score

123

In the present study 55 subjects with different endoscopic experience doing two different procedures (flexible sigmoidoscopy and OGD) on real live and simulated patients were assessed with two performance assessment tools. We found that there was a good reliability and good construct validity for generic and specific Likert scores of the human-based performance assessment tool, which means that the assessment tool can differentiate between consultants and trainees with different endoscopy experience. The computer-based parameters of mucosal and time aspects of the procedures did not show any construct validity. No cost-effectiveness analysis was done between human and computer-based assessment in the present study. However, one may speculate in the long term that human cost in hours may overweigh the initial purchase and servicing computer costs. This aspect should be addressed in a future larger study, as the use of a reliable assessment tool should be the paramount goal in endoscopic training, rather than cost. The parameters in the computer assessment were different from the human-based assessment tool. These differences contributed significantly to the construct validity of the computer-based assessment tool. These parameters are not as accurate in deciphering different grades of endoscopists compared to the human-based assessment tool. A previous study was done to evaluate the ability of an upper gastrointestinal virtual reality simulator to assess skills in endoscopy [15], and to validate its metrics using a video-endoscopic (VES) technique. The VES assessment was able to discriminate performance across the groups, but that was only on the simulators not on live procedures. Another study [16] for objective assessment of technical skills in lower gastrointestinal endoscopy using a Likert score assessment tool was also done. The Likert score was found to discriminate the level of skills across the groups, providing good construct validity. However, in both of these studies the assessment criteria were mainly on generic technical skills in endoscopy. It has been shown that endoscopic simulation training can improve novice endoscopists in early acquisition of skills [17, 18]. Using the computer parameters of assessment of skills has a limited capability for distinguishing varying levels of endoscopic competence [19, 20]. A recent study [21] confirmed this fact by showing that the time to reach the cecum in colonoscopy for novices (no experience) was approximately 30 min and the mean time for the other groups (intermediate, experienced, expert) was approximately 5 min (range 4:19–5:45 min:s). This is not a realistic or effective way to assess endoscopic competency, as time of procedure does not quantify the quality of the procedure being performed.

World J Surg (2010) 34:1764–1770

Two recent studies have demonstrated the benefit of a structured curriculum using virtual reality in acquiring technical skills [22, 23]. Therefore it seems the optimal basis of teaching and training the future medical trainees is combining a structured curriculum, using real patients, virtual realty simulators, and other simulation devices, with an assessment/self-appraisal tool that is able to assess and decipher the various technical (generic and specific) skills required to complete a particular procedure or operation. In our study, we have done the assessment of the 10 subjects on the GI Mentor II (Simbionix) simulator by taking the parameters (mucosal surface examined, efficiency of screening, time spent with clear view, operating time, time to reach the duodenum/transverse colon, and time the patient was in pain), which are calculated by the simulator. We found that there was no construct validity for these parameters. Ferltitsch et al. [24] described the first study designed to validate GI Mentor flexible endoscopy construct. Their study included 13 beginners and 11 experts in gastrointestinal endoscopy. Parameters that were calculated, such as insertion time, correctly identified pathologies, adverse events during colonoscopy, retroflection during (OGD), and skill test performance, showed significant differences across the groups. The conclusion was that virtual endoscopy simulator is capable of identifying differences between beginners and experts in gastrointestinal endoscopy. Another study was carried out by Felsher et al. [25] to determine whether the GI Mentor (Simbionix, Ltd, Israel) flexible endoscopy simulator construct could distinguish experienced endoscopists from beginners. Seventy-five surgical attendings, fellows, and residents were recruited for participation in the study. Two cohorts were used, and these groups were selected from two separate scientific sessions. Participants completed a standardized questionnaire documenting their endoscopic training and experience. Physicians subsequently were designated as experienced or beginner after their endoscopic training and experience were evaluated. All participants completed 1 of 2 colonoscopic simulations. The GI Mentor objectively evaluated performance on the basis of programmed data points, including the time to reach the cecum, the percentage of mucosa visualized, the completed polypectomy rate, the percentage of time spent in clear view through the lumen, the percentage of time that the patient was in pain, and overall efficiency. The conclusion was that The GI Mentor VR colonoscopy construct appears valid. Significant performance differences were shown between the experienced and beginner cohorts. However, these studies have limitations as they can only differentiate between novices and experts. Also, analyzing the p values between each parameter, one would expect them to be very significant i.e., p \ 0.01, rather than merely significant, i.e.,

1769

p = 0.01–0.05, in view of the vast difference in experience in each of the groups.

Conclusions The human-based performance assessment tool in the present study was able to differentiate between subjects with different endoscopic generic and specific technical skills. Thus the performance assessment tool evaluates skills relevant for gastrointestinal endoscopy and it could be used in training programs as an assessment tool as well as for ongoing appraisal. Further studies are required to validate the use of the performance assessment tool for evaluating trainees with different endoscopic experience and to determine the validity of this tool in training that may be achieved by the junior endoscopists concentrating on the technical skills aspects they may lack in the first assessment, and to determine if there is any improvement after the second assessment. Also, the tool could be used to evaluate the amount of virtual training needed to acquire adequate basic endoscopic skills from simulator training, and whether there is a benefit from regular repetition, or whether the simulator can be used for updating skills after longer periods during which endoscopy has not been practised.

References 1. Moorthy K, Muntz Y, Sarker SK et al (2003) Objective assessment of technical skills in surgery. BMJ 327:1032–1037 2. Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165:358–361 3. Kopta JA (1971) An approach to the evaluation of operative skills. Surgery 70:297–303 4. Bridges M, Diamond DL (1999) The financial impact of teaching surgical residents in the operating room. Am J Surg 177:28–32 5. Ross DG, Harris CA, Jones DJ (2002) A comparison of operative experience for basic surgical trainees in 1992 and 2000. Br J Surg 89(Suppl 1):60 6. Skidmore FD (1997) Junior surgeons are becoming deskilled as result of Calman proposals. BMJ 314:1281 7. Darzi A, Smith S, Taffinder N (1999) Assessing operative skill needs to become more objective. BMJ 318:887–888 8. McCloy R, Stone R (2001) Science, medicine, and the future. Virtual reality in surgery. BMJ 323:912–915 9. Satava RM, Cuschieri A, Hamdorf J (2003) Metrics for objective assessment. Surg Endosc 17:220–226 10. Grantcharov TP, Bardram L, Funch-Jensen PM et al (2001) Virtual reality computer simulation as a tool for training and assessment of skills in laparoscopic surgery. Ugeskr Laeger 163:3651–3653 11. Grantcharov TP, Rosenberg J, Pahle E et al (2001) Virtual reality computer simulation—an objective method for evaluation of laparoscopic surgical skills. Surg Endosc 15:242–244

123

1770 12. Sarker SK, Chang A, Albrani T et al (2008) Constructing a hierarchical task analysis in surgery. Surg Endosc 22:107–111 13. Sarker SK, Chang A, Vincent C et al (2006) Development of assessing generic & specific technical skills in laparoscopic surgery. Am J Surg 191:238–244 14. Albrani T, Zaman A, Patel B et al (2007) Self appraisal and assessment of technical skills in endoscopic procedures. Gut 56:A86 15. Moorthy K, Munz Y, Jiwanji M et al (2004) Validity and reliability of a virtual reality upper gastrointestinal simulator and cross validation using structured assessment of individual performance with video playback. Surg Endosc 18:328–333 16. Moorthy K, Munz Y, Orchard TR et al (2004) An innovative method for the assessment of skills in lower gastrointestinal endoscopy. Surg Endosc 18:1613–1619 17. Sedlack RE, Kolars JC, Alexander JA (2004) Computer simulation training enhances patient comfort during endoscopy. Clin Gastroenterol Hepatol 2:348–352 18. Sedlack RE, Kolars JC (2004) Computer simulator training enhances the competency of gastroenterology fellows at colonoscopy: results of a pilot study. Am J Gastroenterol 99:33–37

123

World J Surg (2010) 34:1764–1770 19. Sedlack RE, Kolars JC (2003) Validation of a computer-based colonoscopy simulator. Gastrointest Endosc 57:214–218 20. Sedlack RE (2007) Validation of computer simulation training for esophagogastroduodenoscopy: pilot study. J Gastroenterol Hepatol 22:1214–1219 21. Koch AD, Buzink SN, Heemskerk J et al (2008) Expert and construct validity of the Simbionix GI Mentor II endoscopy simulator for colonoscopy. Surg Endosc 22:158–162 22. Immenroth M, Burger T, Brenner J et al (2007) Mental training in surgical education: a randomized controlled trial. Ann Surg 245:385–391 23. Aggarwal R, Grantcharov TP, Eriksen JR et al (2006) An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 244:310–314 24. Ferlitsch A, Glauninger P, Gupper A et al (2002) Evaluation of a virtual endoscopy simulator for training in gastrointestinal endoscopy. Endoscopy 34:698–702 25. Felsher JJ, Olesevich M, Farres H et al (2005) Validation of a flexible endoscopy simulator. Am J Surg 189:497–500