How Do Examiners and Examinees Think About Role ... - Springer Link

Original Article

How Do Examiners and Examinees Think About RolePlaying of Standardized Patients in an OSCE Setting? Majid Sadeghi, M.D., Arsia Taghva, M.D. Gholamreza Mirsepassi, M.D., Mehdi Hassanzadeh, M.D. Objective: The use of standardized patients in Objective Structured Clinical Examinations in the assessment of psychiatric residents has increased in recent years. The aim of this study is to investigate the experience of psychiatry residents and examiners with standardized patients in Iran. Method: Final-year residents in psychiatry participated in this study. Experienced examiners were asked to complete a questionnaire concerning the ability of standardized patients to realistically portray psychiatric patients. Results: Standardized patients can convincingly portray psychiatric disorders and act according to the requested complex scenarios. Conclusions: According to these findings, the authors recommend the use of standardized patients in OSCEs for psychiatric board certification exams. Academic Psychiatry 2007; 31:358–362

Received July 1, 2006; revised February 5, 2007; accepted March 22, 2007. Dr. Sadeghi is affiliated with the Department of Psychiatry, School of Medicine, Medical Sciences/University of Tehran, Iran. Dr. Taghva is affiliated with the Medical University of the Military, 506 Military Hospital, Tehran, Iran. Dr. Mirsepassi is affiliated with the Iranian Psychiatric Association, Tehran, Iran. Dr. Hassanzadeh is affiliated with the Psychiatric Institute of Iran, Mental Health Research Center, Tehran, Iran. Address correspondence to Dr. Sadeghi, Department of Psychiatry, School of Medicine, Medical Sciences/University of Tehran, Roozbeh Hospital, South Tehran 13334, Iran; [email protected] (e-mail). Copyright 䊚 2007 Academic Psychiatry

358

http://ap.psychiatryonline.org

T

he assessment of residents’ interviewing and examination skills is as important as an evaluation of their factual knowledge. Medical educators have strived to achieve reliable, valid, and fair methods of assessment (1). The validity and reliability of real patients in traditional psychiatric oral examination are under debate, primarily because the number of real patients in traditional psychiatric oral examination is limited to one or two patients. Another question that arises is whether real patients behave and respond to questions in the same manner in different circumstances. These sources of variability raise questions with regard to fairness to candidates (1). Objective structured clinical examinations (OSCEs) are a method of assessment in which the examinee passes through several stations. He or she performs different tasks at each station and then is rated by an examiner using a standardized rating form (2). An increasing number of articles have confirmed the validity, reliability, and acceptability of OSCEs in psychiatric literature (3–5). Simulated patients or standardized patients have been used for several years as a highly effective instrument of assessment in medical education (6). The standardized patient is a simulation of a patient by an actor or other lay person who is rigorously trained to present the specified history and physical finding in a particular manner (7). This performance is repeated uniformly and consistently (8, 9). Norman et al. (10) report that the use of standardized patients in evaluations has the advantage of presenting different students with a similar challenge, thus reducing one important source of variability. Evidence supports the high credibility of standardized patients, who are often indistinguishable from real patients. Standardized patients can simulate patients so realistically that they can even deceive experienced physicians; however, can standardized patients convincingly portray psychiatric disorders? Although data on this issue are limited, the preliminary findings are promising. In a study by Hodges et al. (11, 18) Academic Psychiatry, 31:5, September-October 2007

SADEGHI ET AL.

residents in psychiatry participated in an OSCE consisting of eight 12-minute standardized patient interviews. Ninetythree percent of those who completed a postexamination survey rated the scenarios as very realistic, and 80% considered the scenarios reflective of clinical situations they had experienced (11). Retan and Von Boven (12) reported that physicians’ performances with standardized patients were more accurate reflections of actual practice than written examinations. The psychiatry residency training course is 3 years long in Iran. After successfully finishing the course, the residents should pass a written examination delivered by the Iranian Board of Psychiatry and Ministry of Health and Medical Education. Those who acquire the minimum pass scores are evaluated by an oral examination, which is a clinical interview with a real patient. The candidate is evaluated during this 20-minute interview by two or three examiners. As this method of evaluation appeared not to provide equal and fair circumstances for all candidates, the Iranian Board of Psychiatry decided to substitute OSCEs in the oral examination. The Iranian Board of Psychiatry started using OSCEs as an evaluation tool 2 years ago. At that time, the board decided to perform a pilot study to observe the practicality of this evaluation tool in the Iranian cultural setting. The aim of this study was to investigate the experience of psychiatry residents and examiners with standardized patients in an OSCE setting. The study evaluated residents’ and examiners’ views considering standardized patients’ competence in simulating psychiatric disorders. The study also evaluated the extent of accuracy of standardized patients’ performances.

month period prior to this study. Finally, eight stations with eight standardized patients were chosen for the study. Four standardized patients were students of Rehabilitation College of Iran University and four were employees of Tehran Institute of Psychiatry. None of them was an actor. They were paid acceptable wages for their time. They were trained during three sessions. The training sessions consisted of studying guidelines, role-playing under the supervision of a psychiatrist and a clinical psychologist, and observing hospitalized psychiatric patients. The OSCE consisted of eight 12-minute stations in which residents interacted with standardized patients who tried to portray psychiatric disorders. One day before the examination the examiners rechecked and verified the stations and the standardized patients’ performances of the scenarios. The scenarios were chosen in a way to cover a wide spectrum of psychiatric disorders. The eight stations according to the scenarios assigned to them were as the following: 1. 2. 3. 4. 5. 6. 7. 8.

Depressed male patient Schizophrenic female patient Evaluation of a demented patient Male malingerer Bipolar II female patient Mother of a schizophrenic patient Female patient with borderline personality disorder Female patient with obsessive-compulsive disorder

The subjects of the study were 21 final-year residents from psychiatry training programs through out Iran. The residents were invited to take part in this study, which took place 2 weeks prior to their psychiatry board examination. Agreeing to competence-based goals and designing stations took nearly 9 months. This consisted of several sessions of debate over the design of the stations and setting of competence goals by expert examiners. The examiners were 24 board-certified psychiatrists who were all faculty members with an average of 17 years of academic experience. Nearly two-thirds of them were members of the national Iranian Psychiatric exam team. They had been trained through three workshops on OSCEs during a 9-

An academic psychiatrist and an academic psychologist trained the standardized patients. Thereafter, the standardized patients were reevaluated by experienced examiners who were staff psychiatrists. The standardized patients were trained by relevant readings, observing real patients, and supervised practice. Each standardized patient was pilot tested with examiners assigned to other stations. One hour before the start of the examination, the residents were quarantined and oriented to the OSCE and the process of the examination. A written instruction, including the content, aim, length, and method of evaluation, was attached at the entrance of each station. The examinees were blind to the standardized patients’ diagnoses unless the aim of the station was not to make a diagnosis, so they were expected to act as they would for a regular exam. The residents’ interviewing and management skills were assessed in all stations, and they were told that none of their individual results would be released to the board examiners.

Academic Psychiatry, 31:5, September-October 2007


Method

359

360


Good Excellent Excellent Excellent Good Excellent Excellent Good Excellent Excellent Good Good Excellent Excellent Excellent Excellent Good Good Good Good SP⳱Standardized patient

7 6 5 4 3 2

Good Moderate Excellent Excellent Moderate Moderate Excellent Excellent Good Good Examiner 1 Good Examiner 2 Good

The examiners’ and residents’ evaluations of standardized patients were not totally concordant but most of them

Acting Acting Acting Acting Acting Acting Acting Acting Role- according Role- according Role- according Role- according Role- according Role- according Role- according Role- according Examiner playing to playing to playing to playing to playing to playing to playing to playing to Rating SP scenarios SP scenarios SP scenarios SP scenarios SP scenarios SP scenarios SP scenarios SP scenarios

Discussion

1

The mean age of standardized patients was 27 years old. One was a high school graduate, one a junior college graduate, three had bachelor’s degrees and three had master’s degrees. Three were men and five were women. The opinions of examiners about how realistically the standardized patients portrayed their illnesses and how closely they followed the scripted objectives are shown in Table 1. The table demonstrates that the examiners’ opinions about role-playing of standardized patients in seven stations were identical (87.5%). As none of the examiners rated standardized patients as “weak” or “very weak,” they were not included in the table. The examiners’ opinions about how standardized patients acted according to the objectives of stations were not concordant in three stations, but the difference ranged from good to excellent. The residents’ opinions about standardized patients’ performances and how well they acted according to the defined objectives of the stations are shown in Table 2.

Station

Results

TABLE 1. The Examiners’ Opinions About Standardized Patients’ Role-Playing

There were two examiners per station. The mark sheet used by the examiners to rate the residents’ performance had a 4-point global scale (1⳱weak, 2⳱moderate, 3⳱good, 4⳱excellent). Each examiner also completed the resident assessment mark sheet independently. The pass rate of this examination was the same with the standardized patients as with real patients in the usual oral board examinations. After completing the mark sheets by the examiners and at the end of the examination, when the residents left the stations, the examiners asked for the standardized patients’ opinions about the residents’ performances on the same 4point scale. The examiners also evaluated the performance of the standardized patients using a 5-point global rating scale (0⳱very weak, 1⳱weak, 2⳱moderate, 3⳱good, 4⳱excellent). Following the examination, the residents were asked to fill out a form consisting of a detailed quantitative and qualitative assessment of their experience. The questionnaire included nine questions about the realism of the examinations’ scenarios and standardized patients’ performances, communications and reactions, clearness of the guidelines and duration of each station. Each question consisted of a 5-point global rating scale (0⳱very weak, 1⳱weak, 2⳱moderate, 3⳱good, 4⳱excellent).

8

USING STANDARDIZED PATIENTS IN OSCE SETTING



6 (28.6) 8 (38.1) 4 (19) 3 (14.3) 0 (0.0) 21 (100) 4 (19) 4 (19) 11 (57.9) 2 (10.5) 0 (0.0) 21 (100) 1 (4.8) 15 (71.4) 5 (32.8) 0 (0.0) 0 (0.0) 21 (100) 2 (9.5) 11 (52.4) 8 (38.1) 0 (0.0) 0 (0.0) 21 (100) 10 (47.6) 9 (42.9) 2 (9.5) 0 (0.0) 0 (0.0) 21 (100) 11 (52.4) 9 (42.9) 1 (4.8) 0 (0.0) 0 (0.0) 21 (100) 8 (38.1) 8 (38.1) 3 (14.3) 2 (9.5) 0 (0.0) 21 (100) 5 (23.8) 12 (57.1) 2 (9.5) 2 (9.5) 0 (0.0) 21 (100) 2 (9.5) 16 (76.2) 3 (14.3) 0 (0.0) 0 (0.0) 21 (100) 2 (9.5) 14 (66.7) 5 (23.8) 0 (0.0) 0 (0.0) 21 (100) 9 (42.9) 7 (33.3) 4 (19) 1 (4.8) 0 (0.0) 21 (100) 6 (28.6) 9 (42.9) 5 (23.8) 1 (4.8) 0 (0.0) 21 (100)


SP⳱Standardized patient

Very weak Total

Weak

Moderate

7 6 5 4 3 2

5 (23.8) 11 (52.4) 5 (23.8) 0 (0.0) 0 (0.0) 21 (100) 2 (9.5) 8 (38.1) 10 (47.6) 1 (4.8) 0 (0.0) 21 (100) 9 (42.9) 9 (42.9) 3 (14.3) 0 (0.0) 0 (0.0) 21 (100) 8 (38.1) 7 (33.3) 3 (14.3) 3 (14.3) 0 (0.0) 21 (100) Good

Rating

Excellent

1. Yudkowsky R: Should we uses standardized patients instead of real patients for high stake exams in psychiatry? Acad Psychiatry 2002; 26:187–191

Station

References

Acting Acting Acting Acting Acting Acting Acting Acting according Role- according Role- according Role- according Role- according Role- according Role- according Role- according Roleto playing to playing to playing to playing to playing to playing to playing to playing scenarios SP scenarios SP scenarios SP scenarios SP scenarios SP scenarios SP scenarios SP scenarios SP N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%)

These findings show that since the use of real patients is not practical and feasible in OSCEs, the use of standardized patients may be a rational and a more practical alternative method for board certification exam.

1

Conclusions

TABLE 2. The Residents’ Opinions About Standardized Patients’ Role-Playing

rated standardized patients as good or excellent. The residents evaluated more standardized patients as “moderate” than examiners. This may be due to the fact that the examiners had already contributed to the design and final preparation of the stations and standardized patients. Most examiners evaluated standardized patients’ performances higher than average, which may indicate their potential ability to fulfill the demands of high stakes exams, such as board examinations. The station which portrayed the mother of a schizophrenia patient was evaluated by the residents as the most realistic, whereas examiners rated the stations where the residents interviewed with the patients as the best. This might be due to the fact that the mother of a schizophrenic patient did not have to show symptoms of a severe mental disorder and only was supposed to convey some information to the resident. In the second station where the examinee interviewed an acute psychotic patient, the role-playing by the standardized patients was rated as less realistic. This could be due to the complexity of simulating psychotic symptoms. Contrary to the study by Hodges et al. (11), our residents preferred standardized patients to real patients in board examinations. Besides, in our study, the standardized patients with higher education could act more effectively. Our study showed that both psychiatry residents and examiners found standardized patients satisfactory. Both rated the standardized patients competent to portray complex scenarios, with the examiners’ ratings showing higher satisfaction. This may be due to the examiners having been involved in the writing of the case scenarios. Finally, both the residents and the examiners rated the standardized patients’ performances as accurately portraying psychiatric illness. This was a preliminary study and the results may not be accurate.

8

SADEGHI ET AL.

361

USING STANDARDIZED PATIENTS IN OSCE SETTING 2. Harden RM, Gleeson FA: Assessment of clinical competence using an observed structured clinical examination. Med Educ 1979; 13:41–47 3. Stillman PL, Swanson DB, Stillman AE, Ebert TH, Emmel VS: Assessing clinical skills of residents with standardized patients. Ann Intern Med 1986; 105:762–771 4. Vander Vleuten CPM, Swanson D: Assessment of clinical skills with standardized patients: state of the art. Teach Learn Med 1990; 2:58–76 5. Hodges B, Regehr G, and Hanson M: Validation of an objective structured clinical examination in psychiatry. Acad Med 1998; 73:910–912 6. Adamo G: Simulated and standardized patients in OSCE: achievements and challenges 1992–2003. Med Teacher 2003; 25:262–270

362


7. Barrows HS, Abrahamson S: The programmed patient a technique for appraising student performance. Clin Neurol J Med Educ 1964; 39:802–805 8. Kane MT: The assessment of professional competence. Eval Health Profession 1992; 15:163–182 9. Colliver JA, Williams RG: Technical issues: test application. Acad Med 1993; 68:443–451 10. Norman GR, Neufeld VR, Walsh A, et al: Measuring physician performance by using simulated patients. J Med Educ 1985; 60:925–934 11. Hodges B, Hanson M, McNaughton N, et al: what do psychiatric residents think of an objective structured clinical examination? Acad Psychiatry 1999; 23:198–204 12. Retan J, Van Bovens C: Simulated patients in general practice: a different look at the consultation. BMJ 1987; 294:809– 812
