Gastrointestinal Endoscopy Competency Assessment Tool ...

ORIGINAL ARTICLE: Clinical Endoscopy

Gastrointestinal Endoscopy Competency Assessment Tool: development of a procedure-specific assessment tool for colonoscopy Catharine M. Walsh, MD, MEd, FRCPC,1,8,10 Simon C. Ling, MB, ChB, MRCP (UK),1,8 Nitin Khanna, MD, FRCPC,2 Mary Anne Cooper, MD, MSc, MEd, FRCPC,3,9 Samir C. Grover, MD, MEd, FRCPC,4,9 Gary May, MD, FRCPC, FASGE,4,9 Thomas D. Walters, MD, MBBChir, MRCP,1,8 Linda Rabeneck, MD, MPH, FRCPC,5,9 Richard Reznick, MD, MEd, FRCSC, FACS,6 Heather Carnahan, PhD7,10 Toronto, Ontario, Canada

Background: Ensuring competence remains a seminal objective of endoscopy training programs, professional organizations, and accreditation bodies; however, no widely accepted measure of endoscopic competence currently exists. Objective: By using Delphi methodology, we aimed to develop and establish the content validity of the Gastrointestinal Endoscopy Competency Assessment Tool for colonoscopy. Design: An international panel of endoscopy experts rated potential checklist and global rating items for their importance as indicators of the competence of trainees learning to perform colonoscopy. After each round, responses were analyzed and sent back to the experts for further ratings until consensus was reached. Main Outcome Measurements: Consensus was defined a priori as R80% of experts, in a given round, scoring R4 of 5 on all remaining items. Results: Fifty-five experts agreed to be part of the Delphi panel: 43 gastroenterologists, 10 surgeons, and 2 endoscopy managers. Seventy-three checklist and 34 global rating items were generated through a systematic literature review and survey of committee members. An additional 2 checklist and 4 global rating items were added by Delphi panelists. Five rounds of surveys were completed before consensus was achieved, with response rates ranging from 67% to 100%. Seven global ratings and 19 checklist items reached consensus as good indicators of the competence of clinicians performing colonoscopy. Limitations: Further validation required. Conclusion: Delphi methodology allowed for the rigorous development and content validation of a new measure of endoscopic competence, reflective of practice across institutions. Although further evaluation is required, it is a promising step toward the objective assessment of competency for use in colonoscopy training, practice, and research. (Gastrointest Endosc 2014;79:798-807.) (footnotes appear on last page of article)

Over the last 2 decades, there has been a movement in medical education, both in North America and around the globe, toward an approach based on competencies.1-3 The goal of competency-based education is to ensure that

Use your mobile device to scan this QR code and watch the author interview. Download a free QR code scanner by searching “QR Scanner” in your mobile device’s app store.

trainees attain the knowledge, skills, values, and attitudes required to practice as competent, independent physicians.4 Potential benefits to this approach include increased public accountability, promotion of learner centeredness, transparent and consistent standards, and individualized flexible training.4,5 In response to this movement, there has been an effort to create evaluation tools that allow for objective, valid, and reliable assessment of clinical performance throughout the learning cycle. The integration of objective and reproducible assessment tools into training is essential because they can serve to monitor skill acquisition, provide a basis for structured evaluations and

798 GASTROINTESTINAL ENDOSCOPY Volume 79, No. 5 : 2014

www.giejournal.org

Walsh et al

constructive feedback, aid with promotion and credentialing decisions, and afford a form of quality assurance for the future.6 Competence in performing GI endoscopy requires demonstrated proficiency in 3 domains: (1) technical (psychomotor); (2) cognitive; and (3) integrative competencies required for safe, intelligent performance in varied contexts (eg, communication, judgment, clinical reasoning, and ethical integrity). Traditionally, the number of colonoscopy procedures completed has been used to assess competence on completion of training and subsequently to document maintenance of competence in practice.7-9 Although adequate volume is necessary to achieve competence, performance of a set number of procedures alone does not provide an indicator of level of ability, because there is wide variation in skill among endoscopists with similar levels of experience.10 Another possible marker of competence is adverse event data; however, adverse events are too rare to track as a meaningful indicator of quality and are influenced by patient characteristics.11 Training programs have typically relied on supervising staff to provide ongoing formative feedback and global impressions of trainees’ competence toward the end of training, without the use of predefined criteria. However, this type of non-criterion–based rating is largely subjective and unreliable12,13; therefore, it cannot be considered an optimal means by which to assess competency. There is a growing appreciation in the field of medical education that the addition of structure to components of the assessment process makes the process more objective, valid, and reliable.14 Similarly, there has been an augmented focus on evaluation of real-world events, such as procedures, through direct observation.15 By use of Delphi methodology,29 this study aimed to develop and establish the content validity of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT), a structured multiple-item measure of endoscopic performance designed to assess the full breadth of technical, cognitive, and integrative competencies required to perform colonoscopy safely and proficiently. Although measures of clinical ability in performing colonoscopy have previously been produced,16-28 our instrument adds to the existing literature in that it is developed in a comprehensive and systematic manner by using an international panel of endoscopy experts, thus reflecting clinical practice across institutions.

METHODS Study design Delphi methodology was used to achieve consensus among a panel of endoscopy experts regarding standardized criteria for the assessment of competence of clinicians performing colonoscopy. The Delphi method is a research technique that draws on the collective intelligence of a www.giejournal.org

Gastrointestinal Endoscopy Competency Assessment Tool

Take-home Message Use of the Delphi consensus technique allowed for development of the Gastrointestinal Endoscopy Competency Assessment Tool, a structured, multipleitem measure of endoscopic performance designed to objectively assess the full breadth of technical, cognitive, and integrative competencies required to perform colonoscopy safely and proficiently. The comprehensive and systematic approach to tool development that the authors used provides evidence of content validity of the resultant measure.

panel of experts to achieve consensus on a specific topic through the use of iterative rounds of anonymous questionnaires.29 Content validity is “the degree to which elements of an assessment instrument are relative to and representative of the target construct for a particular assessment purpose.30” Delphi methodology, through the provision of expert professional judgment, can be used to generate content-related validity evidence.31 Approval for this study was obtained from the institutional review board at the University of Toronto.

Delphi panel recruitment and sample In order to finalize item generation and aid with item reduction and gradation, a Delphi group of international endoscopy experts was established. To ensure that appropriate experts were invited to participate, we used purposive and criterion sampling, selecting Delphi panelists according to the nature of our study question. To help establish content validity of the resultant instrument, panelists were identified based on predefined criteria and were selected to represent a wide geographic area including North America and Europe. First, we identified individuals with a strong publication record in the field of endoscopy assessment and/or performance. Second, some panelists were identified as experts as evidenced by their role as opinion leaders within organizations such as the American Society for Gastrointestinal Endoscopy, the Society of Gastrointestinal and Endoscopic Surgeons, and other gastroenterological and surgical societies. In order to increase the reliability of the Delphi group’s composite judgment, our goal was to include a multidisciplinary (eg, gastroenterologists, surgeons, endoscopy nurses), international sample of approximately 30 experts with broad expertise reflective of current knowledge and perceptions in areas related to endoscopy assessment.32,33 Sixty-eight prospective panelists were sent e-mail invitations explaining the study purpose and methodology. The membership of the Delphi panel was kept anonymous.

Item generation A list of potential checklist and global rating items was generated by the authors based on (1) a systematic literature review and (2) an open-ended survey of steering Volume 79, No. 5 : 2014 GASTROINTESTINAL ENDOSCOPY 799


group members. The published literature on the topic of endoscopic performance and assessment was systematically reviewed. Relevant items were extracted and integrated into the initial item pool. In addition, members of the steering group, comprised of 5 individuals selected from diverse backgrounds related to endoscopy; including persons with expertise in educational methodology, clinical and therapeutic endoscopy, and motor learning; were asked to indicate, independently, which variables (checklist and global rating items) they rely on to assess the competence of clinicians performing colonoscopy. Furthermore, during round 1 of the Delphi process, expert panelists were asked to comment on the list of potential indicators and identify any additional items.

Elimination of data redundancy The list of potential indicators was initially reduced by combining redundant items. This process was completed by two investigators (C.M.W., S.C.L.), who reviewed the checklist and global rating items independently for redundancy and then met on two occasions to compare items and establish consensus.

Item reduction Iterative rounds of Delphi surveys were used to further reduce the item pool. In order to maximize response rates, the surveys were designed and distributed in accordance with the principles outlined by Dillman’s tailored design method: the surveys were respondent friendly, using clear and easy-to-understand language; all correspondence was personalized; up to 4 contacts were made by e-mail for each Delphi round; and a token of appreciation, in recognition of respondent’s efforts, was offered but not meant to remunerate for time spent.34 During each round, panelists were provided with a link to the online survey. In addition, a printable paper-based version of the survey was provided. During the first round, participants individually rated, on a 1 (somewhat important) to 5 (extremely important) ordinal scale, the importance of each checklist and global rating item as an indicator of the competence of trainees learning to perform colonoscopy. In addition, panelists were asked to identify up to 10 global rating and checklist items that they viewed to be critical indicators of endoscopic competence. Participants had the opportunity to provide reasons for their choices, add additional items, and/or suggest modified wording of items. During the subsequent rounds, participants once again rated, by using the same 5-point ordinal scale, the importance of each of the remaining checklist and global rating items and identify any items that they viewed to be critical. Panelists were informed of what the group mean score and number of critical ratings were for each item in the preceding round. Panelists were then given the opportunity to change their scores in view of the group’s response to the previous round but were told that they need not 800 GASTROINTESTINAL ENDOSCOPY Volume 79, No. 5 : 2014

Walsh et al

conform to the panel’s view. Once again, participants were given the opportunity to provide open-ended comments. Iterations of Delphi surveys were conducted until consensus among the expert panel was achieved by using the criteria described in the following.

Analysis of Delphi data After each round, 3 authors (C.M.W., H.C., S.C.L.), blinded to the sources of the data, discussed participants’ qualitative and quantitative answers. Quantitative data were analyzed by using standard descriptive statistics. We calculated the mean rating and 95% confidence interval and the proportion of panelists rating an item within each category (1 to 5), for the data from the questionnaire of each Delphi round. By using criteria set a priori, we removed a checklist and global rating item from subsequent rounds if its mean rating was!4 and if it received!5 critical ratings. Based on panelists’ comments, redundant items were combined, items were modified or expanded, and additional items proposed by participants were added. The views of all participating experts were given equal weight. Consensus, a condition of homogeneity of opinion of the expert panelists, was predefined as 80% or more of respondents, in any given round, scoring 4 and above on the 5-point scale on all remaining checklist and global rating items. The Delphi process continued until consensus was achieved.

Item gradations Gradations used in similar instruments were analyzed in order to generate an appropriate grading frameworkdan essential element necessary to minimize the interobserver variability. An expert judgment-based approach to item weighting was used. Delphi group members were asked to assign weights to items within the checklist and global rating scale. In addition, proposed definitions and anchors for each global rating item were distributed to the Delphi panel for feedback. All comments were reviewed with the final instrument reflecting consensus opinion.

RESULTS Panel of participants Of the 68 endoscopy experts initially contacted, 55 responded. Experts were from 44 centers internationally and had performed an average of 612 (range 0-2600) colonoscopies over the preceding year. The characteristics of the panel of participants are described in Table 1.

Delphi process Of the 55 panelists, 55 (100%) completed the round-1 survey, 43 (78.18%) round 2, 38 (69.09%) round 3, 36 (65.45%) round 4 and 37 (67.27%) round 5. There were no significant differences in demographic characteristics www.giejournal.org

Walsh et al

between panelists who completed all 5 rounds of Delphi surveys, as compared with those who did not, including sex, number of colonoscopies performed over the preceding year, job profile, and years of experience in performing colonoscopy (P O .05). Across all 5 rounds of surveys, 3.65% of items had missing ratings. The 274 checklist items (202 steering committee, 72 systematic literature review) and 82 global rating items (56 steering committee, 26 systematic literature review) generated initially were collapsed into 73 non-redundant checklist and 34 non-redundant global rating items by combining like items. Two additional checklist items and 4 global rating items were added by the expert panel during round 1 of the Delphi process. The flow through each round of the Delphi consensus process is outlined in Figure 1. Items with a mean score of R4 and/or R5 critical ratings were retained after each round, and the survey was revised to reflect these changes for each subsequent round (Fig. 1). It took 5 iterations of surveys to achieve consensus and complete the Delphi process. At conclusion of the Delphi process, 19 checklist items, categorized into 5 domains, and 7 global rating items reached consensus as good indicators of the competence of clinicians performing colonoscopy (Tables 2 and 3). The final selection of checklist and global rating items was endorsed by 97.22% and 94.44% of panelists, respectively. Based on the opinion of the majority of Delphi panelists, the global rating item professionalism was dropped from the final instrument, to be replaced by a question asking proctors to indicate if a lapse in professionalism was observed.


TABLE 1. Profile of the panel of experts participating in the Delphi consensus method to develop the GiECAT, designed to assess the competence of clinicians performing colonoscopy Characteristic

No. (%)

Male

44 (80)

Job profile Gastroenterologist

43 (78)

Surgeon

10 (18)

Endoscopy manager

2 (4)

Country Canada United Kingdom United States

15 (27) 1 (2) 39 (71)

Experience in performing colonoscopy, y O10

32 (58)

6-10

12 (22)

!5

11 (20)

GiECAT, Gastrointestinal Endoscopy Competency Assessment Tool.

DISCUSSION

The final gradation scheme of the GiECAT checklist reflects a consensus opinion of the Delphi panel and is based on a review of the gradations used previously in similar instruments. Checklist items are graded on a dichotomous scale: (1) done correctly or (2) not done or done incorrectly. The Delphi panel proposed that the mean weighting for each of the 5 domains of the checklist scale should be as follows: (1) before procedure, 13.6% (median 12.5, range 5%-25%); (2) procedure, technical, 34.3% (median 30, range 20%-50%); (3) procedure, cognitive, 25.6% (median 25, range 10%-50%); (4) procedure, non-technical, 14.2% (median 15, range 5%20%); and (5) after procedure, 12.5% (median 10, range 5%-30%). Similarly, the weighting of the global rating scale reflects consensus opinion, with 92.3% of panelists agreeing that the 7 items within the global rating scale should be weighted equally, with the total scale score being the sum of the scores for all global rating items. Wording of the anchor points for the global rating items is outlined in Table 4 and reflects consensus opinion of the Delphi panel.

Recent changes in the medical training environment, including a greater emphasis on patient safety and efficiency, an increased complexity of cases, and work hour restrictions, have resulted in decreased trainee independence and time for hands-on clinical training.35 These changes, along with the variable learning curve for colonoscopy,10 emphasize the need for assessment tools capable of objectively and reproducibly documenting trainee progress over time. The GiECAT was developed specifically to address this need. We used Delphi consensus methodology to determine the importance of potential indicators of endoscopic competence that were derived from a systematic literature review and a survey of 55 endoscopy experts from 44 centers internationally. After 5 rounds of Delphi surveys, 19 checklist and 7 global rating items reached consensus as being most important to assess the competence of trainees learning to perform colonoscopy. It is well-recognized that a comprehensive approach to instrument development, including an a priori conceptual framework, a thorough literature review, incorporation of existing instruments, and use of expert opinion, helps to ensure that a measure adequately captures the concept of interest, thus increasing its content validity and generalizability.36 The systematic approach we used to generate and select pertinent performance indicators strengthens the content validity of this assessment tool.37 First, a

www.giejournal.org

Volume 79, No. 5 : 2014 GASTROINTESTINAL ENDOSCOPY 801

Item gradation


Walsh et al

Figure 1. Overview of the Delphi process to develop the Gastrointestinal Endoscopy Competency Assessment Tool. Ratings were on a 1 (somewhat important) to 5 (extremely important) ordinal scale, and checklist and global rating items with a mean rating of!4 and critical ratings of!5 were omitted from subsequent rounds. GiECAT, Gastrointestinal Endoscopy Competency Assessment Tool.

systematic search of the published literature was used to establish a domain of content on the topic of endoscopic competence. Relevant items were extracted from the literature and integrated into the initial pool of items. In this way, knowledge embedded within existing measures was not discarded but was subjected to a more rigorous evaluation process. Then, the Delphi panel, through iterative rounds of voting and discussion, confirmed the content validity of the construct of endoscopic competence and of each item in the final GiECAT. We chose to use an assessment approach that combines a checklist and global rating scale. Checklists have the ability to provide feedback that is focused, formative, and easily interpretable. Checklists also can be used as a framework for teaching. However, checklists that use binary ratings often neglect higher level competencies such as clinical reasoning and communication and can, therefore, be limited in their ability to discern nuances of expertise.38,39 In addition, checklists tend to penalize experts unfairly because they reward thoroughness as opposed

to clinical competence.39 Global ratings, which have comparable psychometric properties, can be used to augment the assessment process because they have been found to be more sensitive to level of expertise.38,39 In addition, there is some evidence that global ratings are better at capturing more holistic components of clinical competence and competencies related to patient safety.40 Many of the endoscopy assessment tools developed to date are limited by their focus on assessing the technical (psychomotor) aspects of colonoscopy. Although the acquisition of psychomotor skills is essential, competency in performing technical skills should be conceptualized more broadly to include cognitive and integrative competencies, such as clinical judgment and communication skills, which are required for safe, intelligent performance in varied contexts.41 These facets of competency must not be overlooked because there is growing evidence that context-specific non-technical skills contribute significantly to patient safety and clinical outcomes.42 In reflecting on the relationship of the final GiECAT items to the


www.giejournal.org

Walsh et al


TABLE 2. Checklist items reaching consensus for inclusion in the GiECAT

Competency domains

Checklist item

Round 5, mean (SD) (max score [ 5)

Round 5, consensus level (% rating item R4)

Before procedure 1

Reviews relevant patient information (health records, relevant investigations) and obtains history as appropriate (indications, contraindications, medical history, medications, allergies)

Cognitive and integrative

4.56 (0.60)

94.6%

2

Takes action in response to patient history and investigations where appropriate (eg, prophylactic antibiotics, anesthetic risk factors)

Integrative

4.56 (0.65)

91.9%

3

Demonstrates a sound knowledge of the indications and contraindications to colonoscopy, its benefits and risks, potential alternative investigations and/or therapies, and an awareness of the sequelae of endoscopic or non-endoscopic management

Cognitive

4.67 (0.53)

97.3%

4

Explains to the patient and/or caregivers the perioperative process and procedure (likely outcome, time to recovery, benefits, potential risks/adverse events and rates), checks for understanding and addresses concerns and questions

Integrative

4.59 (0.55)

97.3%

Procedure, technical 5

Recognizes loop formation and avoids or reduces appropriately during the procedure (by using pull-back, torque, external pressure, patient position change)

Technical

4.70 (0.46)

100.0%

6

Uses rotation and/or torque appropriately

Technical

4.38 (0.72)

91.9%

7

Uses withdrawal (as an advancement strategy) appropriately

Technical

4.39 (0.68)

94.6%

8

Uses abdominal pressure and changes in patient position appropriately to aid endoscope advancement

Technical

4.35 (0.63)

91.9%

9

Advances to cecum (in an appropriate time)

Technical

4.49 (0.61)

94.6%

10

Withdraws from cecum/terminal ileum to rectum in an appropriate time (O6 min)

Technical

4.32 (0.85)

89.2%

11

Withdraws endoscope in a controlled manner

Technical

4.59 (0.64)

91.9%

12

Performs therapeutic maneuvers (biopsy and/or polypectomy) independently, appropriately, and safely

Technical

4.68 (0.67)

94.6%

Procedure, cognitive 13

Demonstrates recognition of anatomical landmarks (rectum, flexures, ileocecal valve, appendiceal orifice, etc) and/or incomplete examination

Cognitive

4.73 (0.67)

97.3%

14

Demonstrates recognition of pathologic and anatomic abnormalities

Cognitive

4.86 (0.35)

100.0%

15

Describes findings accurately, interprets abnormalities in the context of the patient, and selects the appropriate strategy/technique to deal with them

Integrative

4.73 (0.45)

100.0%

Cognitive and integrative

4.54 (0.65)

91.9%

Procedure, non-technical aspects 16

Administers sedation appropriately (type, dose), monitors the patient’s vital signs and comfort level throughout the procedure, and responds appropriately and/or demonstrates appropriate interaction with the anesthetist to ensure appropriate sedation and monitoring throughout the procedure

(continued on next page)

www.giejournal.org



Walsh et al

TABLE 2. Continued Round 5, mean (SD) (max score [ 5)


Checklist item

Competency domains

17

Integrative

4.35 (0.63)

91.9%

Demonstrates appropriate interaction and communication with the procedure nurses and/or assistants throughout the procedure

After procedure 18

Educates the patient and/or caregiver about the colonoscopic findings (explanation, significance) and follow-up plan and provides advice regarding potential postprocedure adverse events, recommended course of action, etc

Integrative

4.46 (0.69)

94.6%

19

Appropriate and timely documentation of procedure (written/ dictated/electronic medical records)

Integrative

4.57 (0.55)

97.3%


3 broad domains of endoscopic competence, 8 checklist and 5 global rating items are reflective of technical competencies, 5 checklist and 1 global rating item are reflective of cognitive competencies, and 8 checklist and 2 global rating items are reflective of integrative competencies required to perform endoscopy safely and proficiently. A further limitation of existing measures is that, aside from the Mayo Colonoscopy Skills Assessment Tool25 and the UK Joint Advisory Groups Colonoscopy Direct Observation of Procedural Skills22 assessment form, most have been designed primarily as summative assessment tools, intended to make an overall judgment about competence, fitness to practice, or qualification for advancement.6 The move toward outcomes-based education, centered on the achievement of core training competencies or milestones, and a growing emphasis on patient safety, has had a significant impact on how trainees are taught and assessed. Integration of regular evaluations of competency development is essential in order to set objectives for the next stage of training.3 In addition, incorporation of formative assessment across the learning trajectory provides ongoing evidence of safe practice and affords trainees with timely, focused, constructive feedback to improve performance, thus contributing to improved learning outcomes.43 As such, the GiECAT was designed for use as both a summative and formative assessment tool, capable of assessing the full breadth of technical, cognitive, and integrative competencies. The capacity of the Delphi technique to capture those areas of collective knowledge that are held within professions but not often verbalized makes it enormously useful as a means of tackling a complex problem such as how best to assess endoscopic competence.44 Involvement of a heterogeneous Delphi panel comprised of members from diverse backgrounds related to endoscopy (gastroenterologists, surgeons, and managers) and from a wide

geographic area helps to ensure that the final GiECAT instrument measures the intended concept of endoscopic competence despite variations in clinical practice across institutions. In addition, the anonymous nature of the Delphi process serves to deter self-censorship, gives participants the flexibility to modify their views without social pressure, and helps to ensure that the opinion of the group is not inadvertently swayed by a dominating member.45 The Delphi technique is considered one of the most ideal methods of harnessing expert opinion to arrive at informed group consensus on complex issues32; however, it is not without its limitations. First, there are no universal guidelines regarding panel size and method of expert selection. It has been observed that few new ideas are generated once panel size exceeds 3032; therefore, we aimed to recruit a minimum of 40 panelists to allow for a non-response rate of up to 40%46 across the 5 rounds of surveys. Our response rate ranged from 67% to 100%, which is consistent with other published survey response rates in the health professions literature.47 It must be recognized that the variable response rate could bias the results; however, there were no significant differences in demographic characteristics between panelists who completed all 5 rounds of surveys, as compared with those who did not. Furthermore, in order to minimize potential bias in panel composition, we defined a priori selection criteria to help ensure a breadth of expertise and wide geographic representation. Although we had an international array of Delphi panelists, they were predominantly from North America, and their opinions may differ from endoscopists in other jurisdictions. Critics of Delphi methodology also suggest that the process may be subject to bias because the scope of the problem under study is in part controlled by the research team.48 In order to ensure a broad


www.giejournal.org

Walsh et al


TABLE 3. Global rating items reaching consensus for inclusion in the GiECAT

Global rating item

Competency domain(s)

Definition

Round 5, mean (SD) (max score [ 5)


1

Technical skill

Demonstrates an ability to manipulate the endoscope by using angulation control knobs, advancement/withdrawal, and torque steering for smooth navigation

Technical

4.67 (0.47)

100.0%

2

Strategies for endoscope advancement

Demonstrates an ability to use insufflation, pull-back, suction, loop-reduction, external pressure, and patient position change to advance the endoscope independently, expediently, and safely

Technical

4.76 (0.43)

100.0%

3

Visualization of mucosa

Demonstrates an ability to maintain a clear luminal view required for safe endoscope navigation and complete mucosal evaluation

Technical

4.70 (0.46)

100.0%

4

Independent procedure completion (need for assistance)

Demonstrates an ability to complete the procedure expediently and safely without verbal and/or manual guidance

Technical

4.54 (0.55)

97.3%

5

Knowledge of procedure

Demonstrates general procedural knowledge including procedural sequence, endoscopy techniques, equipment maintenance and trouble-shooting, indications and contraindications, and potential adverse events

Cognitive

4.65 (0.58)

94.6%

6

Interpretation and management of findings

Demonstrates an ability to accurately identify, interpret, and appropriately manage pathology and/or adverse events

Integrative

4.78 (0.48)

97.3%

7

Patient safety

Demonstrates an ability to perform the procedure in a manner that minimizes patient risk (atraumatic technique, minimal force, minimal red-out, recognition of personal and procedural limitations, safe sedation practices)

Technical and Integrative

4.84 (0.37)

100.0%


conceptualization of endoscopic competence, items were generated from multiple sources, including a survey of steering committee members, a systematic review of the literature, and input from Delphi panelists. Finally, there has been little evidence of the reliability of the method. However, we carried out an independent Delphi process with pediatric endoscopy experts by using the same initial indicators derived from our systematic literature review and survey of steering committee members. The global rating scale items that achieved consensus were the same, providing some evidence of reliability of the process.49 In addition, by using a large, heterogeneous sample of endoscopy experts, we increased the reliability and generalizability of the Delphi panel. In conclusion, the Delphi technique allowed for the establishment of expert consensus regarding the items

to be included in the GiECAT, an assessment tool designed to assess competence in performing colonoscopy in a standardized and reproducible manner. Contentrelated validity evidence of the resultant tool was established through the provision of expert professional judgment. In addition, the involvement of an international panel of experts from diverse backgrounds related to endoscopy will facilitate widespread implementation of the resultant tool and ensure that it is reflective of practice across institutions. The next phase of the project is to assess the reliability, validity, and responsiveness of the GiECAT in both the clinical and simulated settings, its relationship to other performance indicators such as completion rates, and ease of use. Research of this nature is critical to the advancement of high quality endoscopic training, practice, and research.

www.giejournal.org



TABLE 4. Global rating scale anchors 1

Unable to achieve tasks despite significant verbal and/or hands-on guidance

2

Achieves some of the tasks but requires significant verbal and/or hands-on guidance

3

Achieves most of the tasks independently, with minimal verbal and/or manual guidance

4

Competent for independent performance of all tasks without the need for any guidance

5

Highly skilled advanced performance of all tasks

ACKNOWLEDGMENTS The authors thank the expert panelists who participated in the Delphi process. The authors would like to thank Drs Brian Hodges and Dorcas Beaton for their insightful comments. REFERENCES 1. Frank JR, Langer B. Collaboration, communication, management, and advocacy: teaching surgeons new skills through the CanMEDS project. World J Surg 2003;27:972-8. 2. Nasca TJ, Philibert I, Brigham T, et al. The next GME accreditation systemdrationale and benefits. N Engl J Med 2012;366:1051-6. 3. ten Cate O, Scheele F. Competency-based postgraduate training: Can we bridge the gap between theory and clinical practice? Acad Med 2007;82:542-7. 4. Frank JR, Snell LS, ten Cate O, et al. Competency-based medical education: theory to practice. Med Teach 2010;32:638-45. 5. Leung WC. Competency based medical training: Review. BMJ 2002;325:693-6. 6. Epstein RM. Assessment in medical education. N Engl J Med 2007; 356:387-96. 7. Romagnuolo J, Enns R, Ponich T, et al. Canadian credentialing guidelines for colonoscopy. Can J Gastroenterol 2008;22:17-22. 8. Wexner SD, Litwin D, Cohen J, et al. Principles of privileging and credentialing for endoscopy and colonoscopy. Gastrointest Endosc 2002;55:145-8. 9. Faigel DO, Baron TH, Lewis B, et al. Ensuring competence in endoscopy: ASGE Press; 2006. Available at: http://www.asge.org/uploaded Files/Publications_and_Products/Practice_Guidelines/competence.pdf. Accessed November 8, 2013. 10. Dafnis G, Granath F, Pahlman L, et al. The impact of endoscopists’ experience and learning curves and interendoscopist variation on colonoscopy completion rates. Endoscopy 2001;33:511-7. 11. Bjorkman DJ, Popp JW, Jr. Measuring the quality of endoscopy. Am J Gastroenterol 2006;101:864-5. 12. Reznick RK. Teaching and testing technical skills. Am J Surg 1993;165: 358-61. 13. Streiner DL. Global rating scales. In: Neufeld VR, Norman GR, editors. Assessing clinical competence. Springer Series on Medical Education. New York: Springer; 1985. p. 119-41. 14. Moorthy K, Munz Y, Sarker SK, et al. Objective assessment of technical skills in surgery. BMJ 2003;327:1032-7. 15. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA 2002;287:226-35. 16. ASGE Standards of Practice Committee, Lee KK, Anderson MA, Baron TH, et al. Modifications in endoscopic practice for pediatric patients. Gastrointest Endosc 2008;67:1-9.


Walsh et al 17. Boyle E, Al-Akash M, Patchett S, et al. Towards continuous improvement of endoscopy standards: validation of a colonoscopy assessment form. Colorect Dis 2012;14:1126-31. 18. Cass OW, Freeman ML, Peine CJ, et al. Objective evaluation of endoscopy skills during training. Ann Intern Med 1993;118:40-4. 19. Chak A, Cooper GS, Blades EW, et al. Prospective assessment of colonoscopic intubation skills in trainees. Gastrointest Endosc 1996;44: 54-7. 20. Moorthy K, Munz Y, Orchard TR, et al. An innovative method for the assessment of skills in lower gastrointestinal endoscopy. Surg Endosc 2004;18:1613-9. 21. Neumann M, Hahn C, Horbach T, et al. Score card endoscopy: a multicenter study to evaluate learning curves in 1-week courses using the Erlangen endo-trainer. Endoscopy 2003;35:515-20. 22. Joint Advisory Group on GI Endoscopy (UK). Formative DOPS assessment formdcolonoscopy & flexible sigmoidoscopy. Available from: http://www.thejag.org.uk/downloads/DOPS%20Forms%20For% 20International%20and%20reference%20use%20only/Formative% 20DOPS%20Assessment%20Form%20-%20Colonoscopy%20and% 20FS.pdf. Accessed July 31, 2013. 23. Park J, MacRae H, Musselman LJ, et al. Randomized controlled trial of virtual reality simulator training: transfer to live patients. Am J Surg 2007;194:205-11. 24. Sarker SK, Albrani T, Zaman A, et al. Procedural performance in gastrointestinal endoscopy: an assessment and self-appraisal tool. Am J Surg 2008;196:450-5. 25. Sedlack RE. The Mayo Colonoscopy Skills Assessment Tool: validation of a unique instrument to assess colonoscopy skills in trainees. Gastrointest Endosc 2010;72:1125-33; 1133.e1-3. 26. Vassiliou MC, Kaneva PA, Poulose BK, et al. Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc 2010;24: 1834-41. 27. Koch AD, Haringsma J, Schoon EJ, et al. Competence measurement during colonoscopy training: the use of self-assessment of performance measures. Am J Gastroenterol 2012;107:971-5. 28. Shah SG, Thomas-Gibson S, Brooker JC, et al. Use of video and magnetic endoscope imaging for rating competence at colonoscopy: validation of a measurement tool. Gastrointest Endosc 2002;56: 568-73. 29. de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach 2005;27:639-43. 30. Haynes SN, Richard DCS, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and methods. Psychol Assess 1995;63:797-801. 31. Keeney S, Hasson F, McKenna H, editors. The Delphi technique. In: The delphi technique in nursing and health research. Oxford, UK: Wiley-Blackwell; 2011. 32. Delbecq AL, Van De Ven AH, Gustafson H. Group techniques for program planning. Glenview (Ill): Scott, Foresman; 1975. 33. Murphy MK, Black NA, Lamping DL, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess 1998;2:i,iv, 1-88. 34. Dillman DA. Mail and internet surveys: the tailored design method, 2nd ed. New York: John Wiley and Sons; 2007. 35. Kramer W, Alman B, Reznick R. It’s the quality of time and not the quantity that matters. Canadian Orthopedic Association Bulletin 2009 Sept 8; Issue 85. Available from: http://www.coa-aco.org/coabulletin/issue-85/forum-resident-training-in-2009-its-the-quality-of-timeand-not-the-quantity-that-matters.html. 36. Guyatt GH, Bombardier C, Tugwell PX. Measuring disease-specific quality of life in clinical trials. CMAJ 1986;134:889-95. 37. Goodman CM. The Delphi technique: a critique. J Adv Nurs 1987;12: 729-34. 38. Gerard JM, Kessler DO, Braun C, et al. Validation of global rating scale and checklist instruments for the infant lumbar puncture procedure. Simul Healthcare 2013;8:148-54.

www.giejournal.org

Walsh et al


39. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ 2003;37:1012-6. 40. Ma IW, Zalunardo N, Pachev G, et al. Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ Theory Pract 2012;17:457-70. 41. Thomas WE. Teaching and assessing surgical competence. Ann R Coll Surg Engl 2006;88:429-32. 42. Sharma B, Mishra A, Aggarwal R, et al. Non-technical skills assessment in surgery. Surg Oncol 2011;20:169-77. 43. Vincent C, Moorthy K, Sarker SK, et al. Systems approaches to surgical quality and safety: from concept to measurement. Ann Surg 2004;239:475-82. 44. Stewart J. Is the Delphi technique a qualitative method? Med Educ 2001;35:922-3. 45. Keeney S, Hasson F, McKenna HP. A critical review of the delphi technique as a research methodology for nursing. Int J Nurs Stud 2001;38:195-200. 46. Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. JAMA 2012;307:1805-6. 47. Burns KEA, Durreet M, Kho ME, et al. A guide for the design and conduct of self-administered surveys of clinicians. CMAJ 2008;179 245-25. 48. Graham B, Regehr G, Wright JG. Delphi as a method to establish consensus for diagnostic criteria. J Clin Epidemiol 2003;56: 1150-6. 49. Ling SC, Walters T, Carnahan H. Development of an objective skill assessment tool for pediatric colonoscopy: a Delphi approach. J Pediatr Gastroenterol Nutr 2011;53(Suppl 1):E83.

C. Walsh is a doctoral fellow of the CIHR Canadian Child Health Clinician Scientist Training Program and is the recipient of a Department of Paediatrics Research Fellowship (Hospital for Sick Children) award and a Postgraduate Medical Education Award, University of Toronto. H. Carnahan is supported by the BMO Chair in Health Professions Education Research. No funding organization had any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript. All other authors disclosed no financial relationships relevant to this publication. Copyright ª 2014 by the American Society for Gastrointestinal Endoscopy 0016-5107/$36.00 http://dx.doi.org/10.1016/j.gie.2013.10.035 Received August 1, 2013. Accepted October 17, 2013. Current affiliations: Division of Gastroenterology, Hepatology and Nutrition, Hospital for Sick Children, University of Toronto (1), Division of Gastroenterology, St. Joseph’s Health Centre, University of Western Ontario (2), Division of Gastroenterology, Sunnybrook Health Sciences Centre, University of Toronto (3), Division of Gastroenterology, St. Michael’s Hospital, University of Toronto (4), Cancer Care Ontario (5), Faculty of Health Sciences, Queen’s University (6), Centre for Ambulatory Care Education, Women’s College Hospital (7), Department of Paediatrics (8), Department of Medicine (9), The Wilson Centre (10), University of Toronto, Toronto, Ontario, Canada. Presented at Canadian Digestive Disease Week, February 24-27, 2012, Montreal, Quebec, Canada (Can J Gastroenterol 2012;26(Suppl A):74A).

Abbreviation: GiECAT, Gastrointestinal Endoscopy Competency Assessment Tool..

Reprint requests: Dr Catharine M. Walsh, Hospital for Sick Children, Division of Gastroenterology, Hepatology and Nutrition, 555 University Ave, Room 8417, Black Wing, Toronto, ON, Canada M5G 1X8.

DISCLOSURE: This study was supported by an American Society for Gastrointestinal Endoscopy Quality in Endoscopic Research Award.

If you would like to chat with an author of this article, you may contact Dr Walsh at [email protected].

Receive tables of content by e-mail To receive tables of content by e-mail, sign up through our Web site at www.giejournal.org. Instructions Log on and click “Register” in the upper right-hand corner. After completing the registration process, click on “My Alerts” then “Add Table of Contents Alert.” Select the specialty category “Gastroenterology” or type Gastrointestinal Endoscopy in the search field and click on the Journal title. The title will then appear in your “Table of Contents Alerts” list. Alternatively, if you are logged in and have already completed the Registration process, you may add tables of contents alerts by accessing an issue of the Journal and clicking on the “Add TOC Alert” link. You will receive an e-mail message confirming that you have been added to the mailing list. Note that tables of content e-mails will be sent when a new issue is posted to the Web.

www.giejournal.org



Walsh et al

APPENDIX 1A. Outcome of all global rating items (n [ 38) rated by Delphi panelists. Global Rating Item

Outcome*

1

Technical skill

Item in final GiECAT tool

2

Atraumatic technique

Eliminated – Round 3

3

Force used


4

Instrument handling


5

Use/manipulation of instrument controls


6

Tip/steering control


7

Movement of hands on controls


8

Grip of instrument head


9

Manipulation of instrument insertion tube


10

Time motion efficiency


11

Colonoscope advancement


12

Smoothness of scope passage


13

Use of torque and rotation


14

Flow of procedure


15

Respect for tissue


16

Visualization of mucosa


17

Ability to maintain a clear field


18

Time spent in red-out


19

Reaction to red-out


20

Strategies for scope advancement


21

Independent procedure completion (need for assistance)


22

Use of assistants


23

Knowledge of instruments


24

Use and knowledge of accessories


25

Knowledge of procedure


26

Interpretation and management of findings


27

Communication - verbal


28

Communication - non-verbal


29

Reaction to patient comfort


30

Patient comfort


31

Patient safety


32

Clinical safety


33

Professionalism


34

Documentation

Eliminated – Round 3 (continued on next page)

807.e1 GASTROINTESTINAL ENDOSCOPY Volume 79, No. 5 : 2014

www.giejournal.org

Walsh et al


APPENDIX 1A. Continued Global Rating Item

Outcome*

35

Maximum depth of scope insertion (Item added by Delphi panel in Round 1)


36

Amount of instruction required during procedure (Item added by Delphi panel in Round 1)


37

Limits of aborts procedure when required (Item added by Delphi panel in Round 1)


38

Seeks assistance when required (Item added by Delphi panel in Round 1)


*A global rating item was removed from subsequent rounds if its mean rating assigned by the Delphi panel was less than 4, and if it received fewer than 5 critical ratings. Items were otherwise retained.

www.giejournal.org

Volume 79, No. 5 : 2014 GASTROINTESTINAL ENDOSCOPY 807.e2


Walsh et al

APPENDIX 1B. Outcome of all checklist items (n [ 75) rated by Delphi panelists. Checklist Item

Outcome*

1

Demonstrates understanding of different colonic preparations (indications, adverse effects, etc.)


2

Reviews which colonic preparation was taken by the patient and its success


3

Introduces themselves and establishes rapport with patient and/or caregivers


4

Checks patient’s health records


5

Reviews patient’s investigations (bloodwork, imaging, etc.)

Eliminated – Round 3 (incorporated under item 4)

6

Obtains a brief patient history (indications for endoscopy, rules out contraindications, significant medical history, complicating medical factors, medications, allergies, etc.)


7

Takes action in response to patient history and investigations where appropriate (prophylactic antibiotics, anesthetic risk factors, etc.)


8

Provides an opportunity for the patient and/or caregiver to ask questions


9

Demonstrates a sound knowledge of the indications and contraindications to colonoscopy, its benefits and risks, potential alternative investigations and/or therapies, and an awareness of the sequelae of endoscopic or non-endoscopic management


10

Explains to the patient and/or caregivers the perioperative process and procedure (likely outcome, time to recovery, benefits, potential risks/ complications and rates)


11

Explains to the patient and/or caregivers potential alternatives to colonoscopy (Barium enema, flexible sigmoidoscopy) and their advantages/disadvantages


12

Checks for understanding and addresses patient’s and/or caregivers’ concerns and questions


13

Sets up equipment and ensures all necessary accessories are available (biopsy forceps, sample containers, etc.)


14

Checks to ensure the equipment is functioning appropriately (suction, water, air, light, image) prior to intubation


15

Ensures appropriate monitoring is in place


16

Positions and drapes patient appropriately


17

Ensures safe and secure IV access


18

Inspects perianal area and performs a digital rectal examination


19

Proper insertion of scope (lubricates, passes into rectum, inserts in luminal direction)


20

Smooth passage of scope with minimal tissue damage


21

Proceeds at appropriate pace with economy of movement


22

Advances to rectosigmoid junction (in an appropriate time)


23

Advances to splenic flexure (in an appropriate time)


24

Advances to hepatic flexure flexure (in an appropriate time)


25

Advances to cecum (in an appropriate time)


26

Advances to terminal ileum (in an appropriate time)

Eliminated – Round 2 (continued on next page)


www.giejournal.org

Walsh et al


APPENDIX 1B. Continued Checklist Item

Outcome*

27

Withdraws from terminal ileum to rectum in appropriate time (O 6 min)


28

Withdraws endoscope in a controlled manner


29

Removal of air on withdrawal


30

Rectal retroflexion in rectum with 360 degree visualization


31

Completes procedure in reasonable time


32

Appropriate posture


33

Appropriate two-handedness (minimizes removal of right hand, manipulates dials & buttons with left hand)


34

Proper handling while advancing and withdrawing scope


35

Uses rotation and/or torque appropriately


36

Uses withdrawal (as an advancement strategy) appropriately


37

Recognizes loop formation and avoids or reduces appropriately during the procedure (using pull-back, torque, external pressure, patient position change)


38

Uses torque steering


39

Uses abdominal pressure and changes in patient position appropriately to aid scope advancement.


40

Uses insufflation of air (distension) and suction appropriately


41

Uses lens washing appropriately


42

Uses irrigation appropriately


43

Adequate mucosal visualization

Eliminated – Round 2 (92.3% panel felt better represented as single global rating of visualization)

44

Adequate visualization of mucosa of the cecum


45

Adequate visualization of mucosa of the ascending colon


46

Adequate visualization of mucosa of the transverse colon


47

Adequate visualization of mucosa of the descending colon


48

Adequate visualization of mucosa of the sigmoid colon


49

Adequate visualization of the TI (Item added by Delphi panel in Round 1)


50

Adequate visualization of mucosa of the rectum

Eliminated – Round 2 (92.3% panel felt better represented as single global rating of visualization) (continued on next page)

www.giejournal.org

Volume 79, No. 5 : 2014 GASTROINTESTINAL ENDOSCOPY 807.e4


Walsh et al

APPENDIX 1B. Continued Checklist Item

Outcome*

51

Takes biopsies from appropriate places


52

Appropriate technique with biopsy forceps


53

Uses therapeutic techniques appropriately, independently and safely


54

Demonstrates recognition of anatomical landmarks (rectum, flexures, ileo-cecal valve, appendiceal orifice, etc.) and/or incomplete examination


55

Demonstrates recognition of normal mucosa


56

Demonstrates recognition of pathological abnormalities


57

Demonstrates recognition of anatomical abnormalities

Eliminated – Round 3 (under item 56)

58

Interprets anatomical and pathological abnormalities in the context of the patient and selects the appropriate strategy/technique to deal with them


59

Recognizes & manages complications appropriately


60

Describes findings accurately (Item added by Delphi panel in Round 1)


61

Clinical Safety (washing hands, gloving, etc.)


62

Pays appropriate attention to patient monitoring and condition throughout the procedure and utilizes supplemental oxygen appropriately to ensure adequate oxygenation and/or demonstrates appropriate interaction and communication with the anesthetist throughout the procedure to ensure appropriate monitoring


63

Administers sedation appropriately (type, dose) AND/OR demonstrates appropriate interaction with the anesthetist to ensure appropriate sedation throughout the procedure


64

Pays appropriate attention to the patient’s comfort and needs throughout the procedure


65

Communicates clearly with the patient throughout the procedure


66

Demonstrates respect for patient’s views and modesty during the procedure


67

Demonstrates appropriate interaction and communication with the procedure nurses and/or assistants throughout the procedure


68

Handling of scope at end of procedure (rinsing, etc.) and changeover of equipment if necessary


69

Insertion of rectal tube if necessary


70

Proper handling of specimens (labels correctly, requisitions, etc.)


71

Attention to recovery of patient and monitoring, checks patient for complications and treats/investigates appropriately (e.g. administration of additional sedation)


72

Education of patient about colonoscopic findings (explanation, significance)


73

Informs patient of follow-up plan


74

Advice given to patient at discharge regarding potential post-procedure complications, recommended course of action, etc.

Eliminate – Round 3 (incorporated under item 72)

75

Appropriate and timely documentation of procedure (written/dictated/EMR)


*A Checklist items was removed from subsequent rounds if its mean rating assigned by the Delphi panel was less than 4, and if it received fewer than 5 critical ratings. Items were otherwise retained. Some items were combined with others upon suggestion of the Delphi panel.


www.giejournal.org