Examining the validity of the Model of Human ... - SAGE Journals

Research PAN.qxd:Research PAN

14/1/11

15:47

Page 34

Research

Examining the validity of the Model of Human Occupation Screening Tool: using classical test theory and item response theory Ay-Woan Pan,1,2 Chia-Wei Fan,3 LyInn Chung,4 Tsyr-Jang Chen,5 Gary Kielhofner,6 Ming-Yi Wu 7 and Yun-Ling Chen 8 Key words: Validity, Rasch measurement model, classical test theory.

1 Associate

Professor, School of Occupational Therapy, National Taiwan University, Taipei, Taiwan. 2 Occupational Therapist, National Taiwan University Hospital, Department of Psychiatry, Taipei, Taiwan. 3 Doctoral Student, Department of Occupational Therapy, University of Illinois at Chicago, Chicago, Illinois, USA. 4 Associate Professor, Department of Statistics, National Taipei University, Taipei, Taiwan. 5 Associate Professor, Department of Mechanical Engineering, LungHwa University of Science and Technology, Taoyuan County, Taiwan. 6 Formerly Professor, Department of Occupational Therapy, University of Illinois at Chicago, Chicago, Illinois, USA. 7 Assistant Professor, Graduate Institute of Rehabilitation Counselling, National Kaohsiung Normal University, Kaohsiung, Taiwan. 8 Doctoral Student, School of Occupational Therapy, National Taiwan University, Taipei, Taiwan. Corresponding author: Dr Ay-Woan Pan, Associate Professor, School of Occupational Therapy, College of Medicine, National Taiwan University, Room 407, No. 17, Xu-Zhou Road, Taipei, Taiwan 100. Email: [email protected] Reference: Pan A-W, Fan C-W, Chung L, Chen T-J, Kielhofner G,Wu M-Y, ChenY-L (2011) Examining the validity of the Model of Human Occupation Screening Tool: using classical test theory and item response theory. British Journal of Occupational Therapy, 74(1), 34-40. DOI: 10.4276/030802211X12947686093648 © The College of Occupational Therapists Ltd. Submitted: 9 December 2009. Accepted: 29 November 2010.

34

Introduction: This study examined the psychometric properties of the Model of Human Occupation Screening Tool, using both item response theory and classical test theory. Method: One hundred and one people with mental health problems, aged 18-65 years, were recruited. The Chinese version of the Model of Human Occupation Screening Tool, the National Taiwan University Hospital Symptom Checklist, the Volitional Questionnaire, the Assessment of Communication and Interaction Skills, and the Mini Mental State Examination were administered. Rasch analysis and correlational analysis were used to examine the construct, convergent, divergent validity and known group validity. Results: Rasch analysis confirmed that there were six subscales within the Chinese version of Model of Human Occupation Screening Tool. The Volitional Questionnaire strongly correlated with the volition subscale (r = 0.583). The Assessment of Communication and Interaction Skills strongly correlated with the communication and interaction subscale (r = 0.815). The Mini Mental State Examination was moderately correlated with the process subscale (ρ = 0.334) and the symptomatology was not associated with any of the subscales as expected. There were significant differences on selected subscale scores across four known groups of participants. Conclusion: The Chinese version of the Model of Human Occupation Screening Tool was valid when applied to people with mental health problems.

Introduction One of the major goals of occupational therapy in psychiatry is to assist people with mental health problems to return to a productive lifestyle (Pratt and Jacobs 1997, Fan et al 2007, Blank and Hayward 2009, Waghorn et al 2009). In order to provide relevant services to clients who need vocational rehabilitation, there is a need for evaluation tools to identify the strengths and limitations of those clients (Fan and Pan 2009, McFadden et al 2010). The purpose of the study was to examine the validity of one of the proposed instruments, the Model of Human Occupation Screening Tool (MOHOST), as applied in a group of people with mental health problems receiving vocational training. People with mental health problems experience a range of challenges, which contribute to unemployment (Marwaha and Johnson 2004). The unemployment rate for people with mental health problems was high in studies over the past two decades in western countries such as the United Kingdom, the United States, Germany and France (Bond and McDonel 1991, MacDonald-Wilson et al 2001, Marwaha et al 2007). These challenges include symptoms, comorbidity of other impairments and medication side effects,

British Journal of Occupational Therapy January 2011 74(1)


14/1/11

15:47

Page 35

Ay-Woan Pan, Chia-Wei Fan, LyInn Chung, Tsyr-Jang Chen, Gary Kielhofner, Ming-Yi Wu and Yun-Ling Chen

such as the impact of sedation on concentration (Razzano et al 2005). They also include difficulties with social skills (Honey 2003, Donnell et al 2004, Mueser et al 2005) and with attitudes and motivation towards work (Henry and Lucca 2004). Environmental challenges include difficulties of acquiring competitive employment due to social stigma (Lehman et al 2002, Cook et al 2005). Since numerous factors influence the work participation of people with mental health problems, practitioners require a multifaceted assessment and intervention approach.

Literature review

Study design The present study combines an item response theory approach via the Rasch measurement model (Rasch 1960, Wright and Stone 1979, Rasch 1980) with classical test theory approach by examining criterion validity. Combining the Rasch measurement approach with classical test theory enables a more comprehensive assessment of an instrument (Kyngdon 2004, Mok 2004) and has been applied to other MOHO assessments (Asgari and Kramer 2008, Kramer et al 2009b). Validity refers to the capacity of an instrument to measure the intended construct (Benson and Schell 1997). Both item response theory and classical test theory are concerned with validity. However, they approach the problem in different ways, as noted in the following sections.

The Model of Human Occupation The Model of Human Occupation (MOHO) (Kielhofner 2008) is one of the most frequently used models in occupational therapy (Lee et al 2008). Its application in the area of work is supported by a substantial body of evidence (Baron and Littleton 1999, Braveman 1999, Kielhofner et al 1999a, Mentrup et al 1999, Paul-Ward et al 2005, Braveman et al 2006, Kåhlin and Haglund 2009, Lee and Kielhofner 2009). This model postulates that success in work is a function of volition (sense of capacity and efficacy, values and interests), habituation (roles and habits), performance capacity and environmental factors. Although a number of MOHO assessments have been developed for application in work settings (that is, the Worker Role Interview, the Work Environment Impact Scale and the Assessment of Work Performance) (Fisher 1999, Kielhofner et al 1999b, Sandqvist et al 2006, Lee and Kielhofner 2009), they require time and client resources that are not always available or can easily be used by occupational therapists in current clinical settings. The Model of Human Occupation Screening Tool (MOHOST) (Parkinson et al 2004), based on MOHO, was developed to screen for the strengths and weaknesses of the clients who may need further assessment or intervention. The MOHOST aims to give a broad overview of occupational participation. It consists of 24 items, four for each of the following six sections: volition (or ‘motivation for occupation’), habituation (or ‘pattern for occupation’), communication and interaction skills, process skills, motor skills and environment. These items are rated using a four-point rating scale, which indicates whether the factor reflected in the item facilitates, allows, interferes or restricts the client’s occupational participation. Some settings have adapted the MOHOST as a vocational screening assessment (Kielhofner 2008). A previous study of 166 participants employed confirmatory factor analysis to examine whether the MOHOST items represent six factors influencing participation (Kielhofner et al 2009). The results showed that there were six underlying factors for the MOHOST. A second study also indicated that the subscales were able to detect change in clients from initial assessment to discharge (Kramer et al 2009a). The purpose of this study was to examine further the construct validity using both item response theory and classical test theory.

Rasch model approach Item response theory (Rasch analysis) approaches the problem of validity by generating evidence that describes the underlying characteristic, or ‘latent trait’, that is intended to be captured by the scale (Wolfe and Smith 2007). The MOHOST seeks to capture the following latent traits: motivation for occupation (volition), pattern for occupation (habituation), communication and interaction skills, motor skills, process skills and environment. Rasch analysis can provide evidence of the validity of the MOHOST subscales. The most important type of evidence is whether each of the items captures effectively the intended latent trait. To assess this, each item is analysed as to whether it ‘fits’ with the rest of the items in targeting the trait. An item that does not fit is considered a threat to validity. For a scale to be considered valid, all items should fit.

Classical test theory approach: convergent, divergent and known group validity When examining validity from the perspective of classical test theory, one examines relationships based on theoretical assumptions about the targeted construct and its relationship to other variables. Convergent validity refers to the expectation that measures intended to capture related traits should be correlated. Divergent validity refers to the expectation that measures of unrelated traits should not be correlated (Campbell and Fiske 1959, Kielhofner 2006). In this study, we asked if selected subscales of the MOHOST would correlate with other variables as expected. First, we examined whether the volition subscale would correlate with the Volitional Questionnaire (VQ) (Chern et al 1996, Yang et al 2007), since both are intended to capture volition. Next, we asked whether the communication and interaction skills subscale would correlate with the Assessment of Communication and Interaction Skills (ACIS) (Forsyth et al 1999, Hsu et al 2008). We expected a strong correlation in both instances (that is, above 0.50). We also examined the relationship between the process skills subscale and the Mini Mental State Examination (MMSE) (Folstein et al 1975, Guo et al 1988), which we used as a screening tool to detect the cognitve competence of the participants. In this instance, we expected a moderate correlation since cognition


35


14/1/11

15:47

Page 36

Examining the validity of the Model of Human Occupation Screening Tool: using classical test theory and item response theory

is related, but not identical, to process skills. We also correlated all the MOHOST subscales with the National Taiwan University Hospital Symptom Checklist (NTUHSC) (Tsai et al 1978); we did not expect the MOHOST subscales to be associated with the symptom checklist since the former targets traits that are conceptually distinct from psychiatric symptomatology. Finally, known group validity refers to the ability of the scale to discriminate among groups with different levels of disability. In order to examine this, we compared the MOHOST subscale scores across the four types of job training groups of participants, who were categorised based on their level of functioning. We expected that the scales related to the person would show at least some differences across groups. There was no reason to suspect differences in the environment scores, since participants were assigned to the groups based solely on their personal characteristics and not on environmental support.

Method Participants The participants in this study were people with mental health problems in northern Taiwan. They were all diagnosed with a psychiatric disability by a psychiatrist according to the DSM-IV criteria (American Psychiatric Association 2000). All participants were involved in one of four types of vocational programmes (that is, task-oriented groups, vocational training programmes, sheltered employment and supported employment), based on their functional performance and needs. The four vocational programmes differed in that each one offered increasing levels of challenge for the attendees. For example, the task-oriented groups provided the least challenge for the participants and the supported employment offered the most challenge because the participants needed to work independently in the community. The inclusion criteria of the study were (1) age ranging from 18 to 65 years and (2) cognitively competent, as indicated by the scores of the Mini Mental State Examination (MMSE) to be greater than 24 (Folstein et al 1975). The study protocol was reviewed and approved by the ethical committees of the National Taiwan University Hospital (NTUH-REC No.: 200712023R).

The Volitional Questionnaire (VQ) The VQ, based on MOHO, is an observational tool with 14 items that reflect behavioural expressions of volition. Studies of the VQ have provided evidence of the instrument’s reliability and validity (Chern et al 1996, Li and Kielhofner 2004). The VQ has been translated into Chinese (VQ-C). Research on the Chinese version has provided evidence of good test-retest reliability and validity (Yang et al 2007).

The Assessment of Communication and Interaction Skills (ACIS) The ACIS is an observational measure of the interactional skills of participants. This assessment is also based on MOHO and includes 20 items that capture physicality, information exchange and relations in social interaction. Several studies have provided evidence of the reliability and validity of the ACIS (Forsyth et al 1999, Kjellberg et al 2003). The ACIS has been translated into Chinese (ACIS-C) and there was evidence that the translated version has good reliability and validity (Hsu et al 2008).

The Mini Mental State Examination (MMSE) The MMSE was used to assess the participants’ mental status. It measures cognitive function in five subscales: orientation, registration, attention and calculation, recall and language (Folstein et al 1975). The MMSE has been validated extensively in both clinical practice and research (Foreman et al 1996). There are 11 items in the MMSE; the maximum score is 30. It was translated into Chinese and a study of 441 normal adults with high, low or no education developed norms for Taiwanese (Guo et al 1988). The results of the study indicated that participants between 2 and 10 years of education should attain minimum scores of 23 to 24. Thus, the MMSE score 24 was set as the minimum score for the person to be included in this study.

The National Taiwan University Hospital Symptom Checklist (NTUHSC) The NTUHSC was used to examine the severity of psychiatric symptoms. It was originally developed based on the Symptom Checklist – 90 (SCL-90) (Derogatis et al 1973). There are 50 items within 10 subscales of the NTUHSC (somatisation, obsession, interpersonal sensitivity, depression, anxiety, hostility, phobia, paranoid, psychoticism and other symptoms) (Tsai et al 1978).

Instruments Data collection The Model of Human Occupation Screening Tool – Chinese version (MOHOST-C) The MOHOST was translated into Chinese (MOHOST-C) and it was validated through a two-stage back translation procedure and a pilot study. The back translation was performed independently by two occupational therapists who were familiar with MOHO. A pilot study was conducted with 12 people with mental health problems. Following this pilot study, the MOHOST-C was further revised. The resulting instrument was used in this study.

36


The second author scored the MOHOST-C based on observation of participants who took part in work or taskoriented groups for 20-30 minutes, and then she interviewed the participants individually. She was also responsible for the collection of other measures. All data were gathered within a month.

Statistical analysis Rasch analysis was performed using FACETS 3.62 (Linacre 2007). Item fit statistics were used to determine the extent


14/1/11

15:47

Page 37

Ay-Woan Pan, Chia-Wei Fan, LyInn Chung, Tsyr-Jang Chen, Gary Kielhofner, Ming-Yi Wu and Yun-Ling Chen

Table 1. Characteristics of participants (n = 101) Characteristic Number Percentage Gender Male..............................................................52......................51.5............. Female ........................................................49......................48.5............. Diagnosis Schizophrenia..............................................74......................73.3............. Schizoaffective .................................................4........................4................ Bipolar ...........................................................18......................17.8............. Depression ....................................................5........................5................ Education Illiteracy...........................................................0........................0................ Elementary school ...........................................5........................5.0............. Junior high school .........................................22......................21.8............. Senior high school .......................................43......................42.6............. University ....................................................29......................28.7............. Graduate school ............................................2........................2.0............. Marital status Single..........................................................80......................79.2............. Married .........................................................9........................8.9............. Divorced......................................................12......................11.9............. Widow ..........................................................0........................0................ Living status With family..................................................60......................59.4............. In institution................................................35......................34.7............. Alone ............................................................6........................5.9.............

to which four items in each subscale captured the trait that the items are supposed to target. In keeping with the recommended standards (Wright and Linacre 1994, Velozo et al 1999, Bray et al 2001), fit statistics of MnSq >1.4 associated with Zstd >2.0 were taken as an indication of an item misfit. The convergent and divergent validity of selected sections of the MOHOST-C with related or unrelated variables were also examined. Pearson’s correlation coefficient was used when the data were interval. Spearman’s correlation coefficient was used when the data were ordinal. Analysis of Variance (ANOVA) and a Tukey HSD post-hoc procedure were used to identify if there were differences in the measures for six subscales of the MOHOST-C among four groups of participants who were at various levels of job training. Bonferonni correction was used to address the problem of multiple significance comparisons; thus, the alpha was set to 0.008 (0.05/6 tests).

Results Demographic characteristics One hundred and one people with mental health problems were included in this study. Most participants were diagnosed as having schizophrenia (n = 74, 73.3%). The mean age of the participants was 40.5 years old (SD = 9.6) and the mean age of onset of mental health problems was 24.5 years (SD = 9.5). Additional demographic characteristics

Table 2. Fit statistics of six subscales of the MOHOST-C Items Infit MnSq Zstd Subscale 1: Volition Appraisal of abilities......................................1.11....................1.1............ Expectation of success...................................0.88...................-1.2............ Interest .........................................................1.06....................0.6............ Commitment.................................................0.91...................-0.9............ Subscale 2: Habituation Routine.........................................................1.03....................0.3............ Adaptability ..................................................1.12....................1.2............ Responsibility................................................0.98...................-0.1............ Roles..............................................................0.86...................-1.5............ Subscale 3: Communication and interaction skills Non-verbal skills .............................................1.02....................0.2............ Conversation.................................................1.18....................1.8............ Vocal expression .............................................0.88...................-1.2............ Relationships ................................................0.86...................-1.5............ Subscale 4: Process skills Knowledge.....................................................1.05....................0.5............ Planning .......................................................0.90...................-1.0............ Organisation .................................................0.99....................0.0............ Problem-solving ............................................1.03....................0.3............ Subscale 5: Motor skills Posture and mobility......................................1.11....................1.1............ Coordination.................................................0.83...................-1.8............ Strength and effort........................................0.92...................-0.8............ Energy ..........................................................1.12....................1.1............ Subscale 6: Environment Physical space...............................................1.15....................1.4............ Physical resources ..........................................0.94...................-0.5............ Social groups ................................................1.05....................0.5............ Occupational demands..................................0.91...................-0.9............

of the participants are shown in Table 1. There were 28 (27.7%) participants from the task-oriented occupational therapy groups; 41 (40.6%) were from job training programmes, 20 (19.8%) were from sheltered employment and 12 (11.9%) were from supported employment.

Item fit Table 2 shows the results of the Rasch analysis for each subscale of the MOHOST-C. None of MnSq for the items exceeded 1.4. In fact, most items were very near an ideal MnSq of 1.0. These findings indicated that the items within each subscale were targeting the intended construct.

Convergent validity The Pearson’s correlation coefficient of the volition subscale of the MOHOST-C and the VQ-C was r = 0.583 (p = 0.000). The Pearson’s correlation coefficient between the communication and interaction subscale of the MOHOST-C and the ACIS-C was r = 0.815 (p = 0.000). The Spearman’s correlation coefficient between the MMSE and process subscale of the MOHOST-C was ρ = 0.334 (p = 0.001). British Journal of Occupational Therapy January 2011 74(1)

37


14/1/11

15:47

Page 38

Examining the validity of the Model of Human Occupation Screening Tool: using classical test theory and item response theory

Table 3. The Spearman’s correlation coefficient of the NTUHSC and six subscales of the MOHOST-C Volition Habituation Communication Process Motor Environment Correlation coefficient ....................-0.048...................0.046............................-0.104 ......................0.005...............0.042 ...................-0.013.................... p value ............................................0.633...................0.645 .............................0.302 ......................0.961...............0.675......................0.895.................... Number ...........................................101.......................101..............................101..........................101..................101.......................101......................

Table 4. ANOVA results for each of the six subscales of the MOHOST-C among four groups of participants Sum of df Mean F Sig. squares square Volition Between groups............58.173..........3 ......19.391.....6.439.......0.001*.... Within groups.............292.137........97 ........3.012.................................... Total...........................350.311......100 ..................................................... Habituation Between groups............66.855..........3 ......22.285.....5.702.......0.001*.... Within groups.............379.074........97 ........3.908.................................... Total...........................445.929......100 ..................................................... Communication and interaction Between groups............61.787..........3 ......20.596.....6.058.......0.001*.... Within groups.............329.763........97 ........3.400.................................... Total...........................391.550......100 ..................................................... Process Between groups............87.796..........3 ......29.265.....5.964.......0.001*.... Within groups.............475.944........97 ........4.907.................................... Total...........................563.740......100 ..................................................... Motor Between groups............47.672..........3 ......15.891.....3.715.......0.014...... Within groups.............414.881........97 ........4.277.................................... Total...........................462.554......100 ..................................................... Environment Between groups............11.575..........3 ........3.858.....2.360.......0.076...... Within groups.............158.593........97 ........1.635.................................... Total...........................170.168......100 ..................................................... *p