The relationship between non-functioning distractors

[Downloaded free from http://www.thejhs.org on Tuesday, March 15, 2016, IP: 2.88.88.0]

Original Article

The relationship between non-functioning distractors and item difficulty of multiple choice questions: A descriptive analysis Hamza Mohammad Abdulghani1, Farah Ahmad1, Gominda G. Ponnamperuma3, Mahmoud S. Khalil1, Abdulmajeed Aldrees1,2 Departments of Medical Education, and 2Physiology, College of Medicine, King Saud University, Riyadh, Saudi Arabia, 3Department of Medical Education, Faculty of Medicine, University of Colombo, Colombo, Sri Lanka 1

ABSTRACT Objectives: This study investigates the relationship between the presence of non-functioning distractors (NFDs) and the difficulty index of multiple choice questions (MCQs). Materials and Methods: The number of NFDs and difficulty index were correlated for each MCQ of Family Medicine Examination, College of Medicine, King Saud University (KSU) in three consecutive years (2010, 2011 and 2012). Results: The MCQs with more NFDs showed high difficulty index as compared to MCQs having less NFDs. The pattern of increasing difficulty index was 1NFD < 2NFD < 3NFD. However, the MCQs with a lesser number of NFDs would not necessarily be always difficult. Conclusion: MCQs with a higher number of NFDs are easier than those with lower number of NFDs. Keywords: Difficulty index, MCQ, non-functioning distractors

INTRODUCTION Multiple choice questions (MCQs) item analysis is the process of collecting, summarising and using information from students’ responses to evaluate the quality of test items.[1,2] It is a valuable, yet relatively easy process executed after the examination. It also gives information about the reliability and validity of a test item. The main advantage of item analysis is to identify the items that need improvement or deletion before item banking.[3] Access this article online Quick Response Code:

Website: www.thejhs.org DOI: 10.4103/1658-600X.142784

Address for correspondence: Dr. Hamza Mohammad Abdulghani, Department of Medical Education, Assessment and Evaluation Centre, College of Medicine, King Saud University, Riyadh - 11321, Kingdom of Saudi Arabia. E-mail: [email protected] 148

Difficulty index, also called facility index or P value, is one of the key parameters of item analysis. It describes the percentage of students who correctly answered a given test item. This index ranges from 0 to 100% or 0 to 1. An easy item has a higher difficulty index.[4] A single best answer type MCQ should have a stem, a clear lead-in and a set of options.[5] The options should contain one most correct answer (the key) and the rest are called distractors. Writing plausible distractors is time consuming and possibly the most difficult part of constructing MCQs.[5] Analysis of distractors can function as a valuable indicator of item difficulty. A good quality distractor should be selected by those who perform poorly and should be ignored by those who perform well.[6-8] If a distractor is chosen more often than the correct answer, this may indicate poor instructions or a misleading question. [9] As such, analysis of distractors can function as a valuable indicator of the item difficulty. To assess the students’ capacity properly, the distractors must generate a reasonable misconception about the correct option, at least in the average examinee’s mind.[10] Hence, the examinee performance in MCQ test depends upon how the distractors are planned.[11] Journal of Health Specialties / October 2014 / Vol 2 | Issue 4

[Downloaded free from http://www.thejhs.org on Tuesday, March 15, 2016, IP: 2.88.88.0] Abdulghani, et al.: Non-functioning distractors and MCQ item difficulty

Non-functioning distractors (NFDs) are those options that are selected by less than 5% of the examinees.[12] These NFDs may have no connection or having some clues, which are not directly related to the correct answer. [13] Distractors that are not chosen or are consistently chosen by only less than 5% of the examinee are obviously ineffective and should be omitted or replaced.[8,9] Many studies have been conducted on the quality of MCQs and item-writing flaws.[14] However, studies that investigated the relationship between NFDs and item difficulty are scarce. Extensive electronic literature search found, perhaps the only study that explicitly investigated this relationship is a Dental College, Karachi, Pakistan, which revealed that MCQs with 2 NFDs and no NFDs showed nearly equal difficulty indices (0.365 and 0.351 respectively), although items with 3NFDs showed a relatively high difficulty index.[4] Hence, this study was initiated to investigate the relationship of the NFDs and the difficulty of an MCQ item.

MATERIALS AND METHODS Study context The Family Medicine course is an essential course in the undergraduate curriculum at the College of Medicine, which runs for a period of six weeks duration and conducted in the fourth academic year. At the end of the course, the candidates sit for the final examination of 100 single best MCQs along with other parts of the clinical assessment including, Objective Structured Clinical Examination. The 100-item MCQ paper has to be completed in 2 hours 30 minutes. All 300 MCQs of the undergraduate Family Medicine examination in three consecutive years (2010, 2011 and 2012) were analysed for their difficulty indices. Construction of MCQs All the test items were created using a test blueprint that align each test item to the corresponding Family Medicine main outcomes. The blueprint would then be reviewed by a panel of examination committee members of the college, prior to conducting the exam to check the test content validity. Structure and scoring of MCQs Each MCQ item comprised a lead-in and four options. A correct response to an item was awarded ‘one’ mark, while an incorrect response and no attempt or blank response was given ‘zero’. There was no negative marking for the wrong answers. Journal of Health Specialties / October 2014 / Vol 2 | Issue 4

Data collection All the MCQs of the three examinations (300 items) had 300 correct answers and 900 distractors. The test was analysed for the reliability index by Kuder — Richardson Formula 20 (KR20), distractor analysis, difficulty index and discrimination index by the output of the Optical Response Reader using ‘Question Marks Perception’ software. After retrieving the basic information about the NFDs and functioning distractors, the MCQs were categorised based on the number of NFDs; i.e. 0 NFD, 1 NFD, 2 NFDs, and 3 NFDs. Item analysis The item difficulty index (P), the ratio between the number of examinees who correctly answered a given MCQ (R) to the total number of examinees (T), was calculated by the formula P = R/T (T included correct, incorrect and blank responses). Based on the value of the difficulty index, the MCQs were then categorised as very difficult (< 0.2), moderately difficult (0.21 - 0.69), easy (0.7 - 0.9) and very easy (> 0.9).[15] Data analysis The Pearson’s correlation coefficient between the number of NFDs and the difficulty indices of the 300 MCQs was calculated, using Statistical Package for Social Sciences (SPSS) version 19. A P value of less than 0.05 was considered as statistically significant. The study was approved by the College Board Ethical Committee, College of Medicine, King Saud University, Riyadh, Saudi Arabia.

RESULTS The number of students (appeared, passed, failed) and their scores (mean, maximum and minimum) for each of the three years are given in Table 1. The mean (± standard deviation) difficulty index was 0.76 (± 0.19) for all three examinations (i.e. 300 MCQs). Figure 1 showed that 41% of the MCQs, which were classified as easy and very easy questions, had three NFDs. The Pearson’s correlation between the difficulty index and the number of NFDs for the 300 MCQs was found significant (r, 0.448; P < 0.0001). Table 1: Characteristic of examinations Year Number Number of students (%) of MCQs Appeared Passed Failed 2010 100 70 51 (73) 19 (27) 2011 100 85 74 (87) 11 (13) 2012 100 104 77 (74) 27 (26)

Score (Out of 100) Mean Maximum Minimum 74.07 88.00 54.00 78.98 92.00 61.00 73.71 85.00 50.00

149


Figure 2: The relationship between non-functioning distractors and difficulty index of MCQs of Family Medicine Final (2010, 2011 and 2012) Examinations

Figure 1: Classification of MCQs according to the level of difficulty and the number of NFDs in the Family Medicine Exams (2010, 2011 and 2012)

Figure 2 shows that the MCQs which have more NFDs are easier. However, as expected and can be seen by the range of difficulty indices for the items with no NFDs [Figure 1], it could be concluded that having no NFDs, although is clearly related to the difficulty, does not always guarantee a low difficulty index; i.e. a difficult item. As shown in Figure 2, however, the general trend is that as the number of NFDs increases the easiness of the MCQs (i.e. the difficulty index) also increases.

DISCUSSION AND CONCLUSION This study has confirmed the general understanding in the literature that NFDs can affect the examination quality by decreasing the difficulty of the MCQ.[16] This study also confirmed the results of a similar study by Hingorjo and Jaleel,[4] which showed that the MCQs with more NFDs are easier than the MCQs with a lesser number of NFDs. Other studies had shown that easy MCQs may decrease the discriminating power of the MCQ items, as medium difficulty items, are having better discriminating indices.[15,17] However, this issue was not investigated in the current study. 150

The number of NFDs significantly and inversely affects the difficulty of MCQs. However, as seen in this study, NFDs are not the only determinant that contributes to the difficulty of an MCQ item. Other item-writing flaws like writing questions that include unfocussed stems, negative stems, gratuitous and unnecessary information, implausible distractors and clues to students can affect student performance on MCQs making the items either more or less difficult to answer.[12] By analysing the quality of distractors using NFDs analysis, the item writers could improve the difficulty and hence the quality of an MCQ.[18] This could be recommended as an important quality assurance activity before banking items for future use. Strength and limitation To our knowledge, it is the first local study, which has investigated the relation of the MCQ items difficulty and NFDs in a medical school assessment. The findings of this study is consistent with similar to other international studies. However, this study has investigated only one medical course and looked only at one aspect of the item analysis. To assess the quality of the MCQ items, further studies are needed at a larger scale, as this study did not show the other parameters of the MCQs item analysis for quality assurance process.

REFERENCES 1. 2. 3.

Carroll RG. Evaluation of vignette-type examination items for testing medical physiology. Am J Physiol 1993;264:S11-5. Fowell SL, Southgate LJ, Bligh JG. Evaluating assessment: The missing link? Med Educ 1999;33:276-81. Considine J, Botti M, Thomas S. Design, format, validity and reliability of multiple choice questions for use in nursing research and education. Collegian 2005;12:19-24. Journal of Health Specialties / October 2014 / Vol 2 | Issue 4


4.

Hingorjo MR, Jaleel F. Analysis of one-best MCQs: The difficulty index, discrimination index and distractor efficiency. J Pak Med Assoc 2012;62:142-7. 5. Vyas R, Supe A. Multiple choice questions: A literature review on the optimal number of options. Natl Med J India 2008;21:130-3. 6. Gronlund N. How to construct achievement tests. 1 st ed. Engelwood Cliff (NJ): Prentice-Hall Publisher; 1988. 7. Haladyna TM. Developing and validating multiple-choice test items. 2nd ed. Mahwah (NJ): Erlbaum Associates; 1999. 8. Linn R, Gronlund N. Measurement and assessment in teaching. 8th ed. Upper Saddle River (NJ): Merrill Co; 2000. 9. Nunnally JC, Bernstein IH. Psychometric theory. 3rd ed. New York: McGraw-Hill; 1994. 10. Haladyna TM, Downing SM. How many options is enough for a multiple-choice test item? Educ Psychol Meas 1993; 53:999-1010. 11. Dufresne RJ, Leonard WJ, Gerace WJ. Making sense of student’s answers to multiple-choice questions. Phys Teach 2002;40:174-80. 12. Tarrant M, Ware J, Mohammed AM. An assessment of functioning and non-functioning distractors in multiple-choice questions: A descriptive analysis. BMC Med Educ 2009;9:40.

13. Tarrant M, Knierim A, Hayes SK, Ware J. The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. BMC Med Educ 2006;26:662-71. 14. Haladyna TM, Downing SM. A taxonomy of multiple-choice item-writing rules. Appl Meas Educ 1989;2:37-50. 15. Abdulghani HM, Ponnamperuma GG, Ahmad F, Amin Z. A comprehensive, multi-modal evaluation of the assessment system of an undergraduate research methodology course: Translating theory into practice. J Pak Med Assoc 2014; 30:227-32. 16. Gronlund NE, Linn RL. Measurement and evaluation in teaching. 6th ed. New York: Macmillan; 1990. 17. Sim SM, Rashid RI. Relationship between item difficulty and discrimination indices in true/false- type multiple choice questions of a para-clinical multidisciplinary paper. Ann Acad Med Singapore 2006;35:67-71. 18. Tarrant M, Ware J. A framework for improving the quality of multiple-choice assessments. Nurse Educ 2012;37:98-104. How to cite this article: Abdulghani HM, Ahmad F, Ponnamperuma GG, Khalil MS, Aldrees A. The relationship between non-functioning distractors and item difficulty of multiple choice questions: A descriptive analysis. J Health Spec 2014;2:148-51. Source of Support: Nil. Conflict of Interest: None.

Author Help: Reference checking facility The manuscript system (www.journalonweb.com) allows the authors to check and verify the accuracy and style of references. The tool checks the references with PubMed as per a predefined style. Authors are encouraged to use this facility, before submitting articles to the journal. • The style as well as bibliographic elements should be 100% accurate, to help get the references verified from the system. Even a single spelling error or addition of issue number/month of publication will lead to an error when verifying the reference. • Example of a correct style Sheahan P, O’leary G, Lee G, Fitzgibbon J. Cystic cervical metastases: Incidence and diagnosis using fine needle aspiration biopsy. Otolaryngol Head Neck Surg 2002;127:294-8. • Only the references from journals indexed in PubMed will be checked. • Enter each reference in new line, without a serial number. • Add up to a maximum of 15 references at a time. • If the reference is correct for its bibliographic elements and punctuations, it will be shown as CORRECT and a link to the correct article in PubMed will be given. • If any of the bibliographic elements are missing, incorrect or extra (such as issue number), it will be shown as INCORRECT and link to possible articles in PubMed will be given.

Journal of Health Specialties / October 2014 / Vol 2 | Issue 4

151