A Multidimensional Evaluation - Springer Link

4 downloads 0 Views 122KB Size Report
called Procovery in selected inpatient and community mental health centers. Procovery is a ... This evaluation sought to determine both consumer outcomes and.
Statewide Implementation of Recovery Support Groups for People with Serious Mental Illness: A Multidimensional Evaluation Michael A. Mancini, PhD Donald M. Linhorst, PhD Anthony A. Menditto, PhD James Coleman, PhD Abstract This study evaluated a statewide demonstration project to implement a group-based intervention called Procovery in selected inpatient and community mental health centers. Procovery is a facilitated mutual support group designed to build hope and a sense of social inclusion by raising consciousness and helping people develop an understanding of the ways one can move toward recovery in their own lives. This evaluation sought to determine both consumer outcomes and perceptions of the program and implementation efforts held by consumers and the facilitators of the intervention. A multidimensional approach was used, including a quasi-experimental design with consumers, questionnaires and focus groups with the intervention facilitators, and individual interviews with administrators. The Procovery model was shown to have a positive impact on consumers’ recoveries and was viewed favorably by consumers, facilitators, and administrators. Several barriers to effective implementation were identified. These findings and their implications for future practice and research are discussed.

Recovery from serious psychiatric disabilities is a complex process of change contingent on the development of an empowered sense of self and hopefulness for the future.1–9 It also requires

Address correspondence to Michael A. Mancini, PhD, School of Social Work, Saint Louis University, 3550 Lindell Blvd., Saint Louis, MO 63103, USA. Phone: +1-314-9772736; Fax: +1-314-9772731; Email: [email protected]. Donald M. Linhorst, PhD, School of Social Work, Saint Louis University, 3550 Lindell Blvd., Saint Louis, MO 63103, USA. Phone: +1-314-9772745; Fax: +1-314-9772731; Email: [email protected] Anthony A. Menditto, PhD, Fulton State Hospital, 600 East 5th St., Fulton, MO 65251, USA. Phone: +1-573-5923405; Fax: +1-573-5923000; Email: [email protected] James Coleman, PhD, Central Texas Veterans Healthcare System, 2101 Interstate Highway 35, South Austin, TX 78741, USA. Phone: +1-512-4332020; Fax: +1-512-4332078; Email: [email protected] This research was supported in part by the Missouri Department of Mental Health.

)

Journal of Behavioral Health Services & Research, 2013. 391–403. c 2013 National Council for Community Behavioral Healthcare.

Statewide Implementation of Recovery Support Groups

MANCINI et al.

391

access to concrete resources such as adequate housing, meaningful activities, quality treatment, opportunities for skill enhancement, and the development of personal relationships.2,3,7,8 The process of recovery can be hindered by the development of an internalized sense of stigma that can lead to identities dominated by illness and patienthood.2,10–13 Recovery-focused programs have become recognized as an important component in mental health service systems throughout the USA because of the continued recognition that persons with serious mental illness are difficult to engage in treatment and, as a result, experience a range of negative social outcomes including premature death.14–17 It also is recognized that participation in comprehensive social rehabilitative interventions offers the best hope for recovery.18–22 The President’s New Freedom Commission on Mental Health explicitly recommended involving consumers and family members more in service delivery.23 Consumerfocused programs place an emphasis on the values of self-determination, choice, empowerment, and respect and have shown to deliver promising outcomes.24–28 Consumer-focused programs have also shown to be an integral part of a mental health policy that is shifting toward more of an integration of peer and non-peer programing to help people achieve their recovery goals.29 A new, and as yet unstudied, approach to providing peer services is called Procovery. 30 Procovery is a 14-week consumer-developed and focused approach based on providing education and developing hope through a facilitated, structured mutual support group. The word, Procovery, is designed to foster a sense of looking forward. It is defined as “attaining a productive and fulfilling life regardless of the level of health assumed attainable [vs. recovery, returning to a prior state of health].” 30 (p 4) The groups are referred to as Procovery Circles and are comprised of voluntary members and a facilitator. All participants are considered equal members of the Circle, and the facilitator remains a neutral coordinator of the Circle’s activities, rather than functioning as an instructor or group leader. Facilitators can be peer-specialists, non-peer professionals, or family members trained in the Procovery model. Procovery Circle sessions are highly structured. Each session begins with a 5minutes introduction of group members, followed by 15 minutes during which participants read out loud short selections from the book, The Power of Procovery in Healing Mental Illness, and discuss the reading as it pertains to them.30 Next, 15 minutes is devoted to participants selecting a “Procovery Card,” which they can read aloud affirmations and other positive statements to the group and discuss. Discussion topics range from developing hope and insight to addressing practical matters such as managing medications, developing support networks, engaging in meaningful activities, self-care, and getting a job. Each session ends with a 15-minutes period of refreshments and socialization. Procovery exhibits many of the critical ingredients inherent in consumer-run services as it is voluntary, consumer controlled, and focused on consumer choice and respect.25 The department of mental health of a Midwestern state conducted a statewide demonstration project in which the Procovery Model was implemented at approximately 40 inpatient and outpatient sites in five geographic regions across the state. A multidimensional evaluation of the project was conducted to help determine the impact of the program on consumer and facilitator outcomes. This study reports on a portion of that evaluation and includes three components: (1) recovery outcomes reported by 43 consumers who attended at least four Procovery Circles at a large state psychiatric hospital using a quasi-experimental design; (2) perceptions of the program reported via survey from 71 Procovery Circle facilitators and six regional focus groups conducted with 60 facilitators; and (3) implementation issues identified through interviews with 13 of the administrators of Procovery demonstration sites.

Methods Consumer outcome evaluation Design A quasi-experimental design was used to measure the subjective recovery experiences of consumers at a large, secure state psychiatric hospital serving persons with serious mental illness.

392

The Journal of Behavioral Health Services & Research

40:4

October 2013

All participants were inpatients residing in the maximum, intermediate, minimum, or open campus security units. The program group consisted of Procovery Circle participants who were administered an initial questionnaire prior to Circles starting and then again at 6 months after the Circles started. A non-randomly assigned comparison group also completed the questionnaires at these two time periods. The comparison group consisted of individuals who had the opportunity to sign up for Procovery Circles, but chose not to. Participants were invited by staff to participate in a research study to evaluate the effects of Procovery. Informed consent was obtained from those who agreed to participate in the research and from their legal guardians when applicable. Identical procedures were employed to recruit members of the comparison group. Participants’ information remained confidential throughout the study; no identifying information was required on the questionnaire. The initial questionnaire data were matched with 6-month follow-up data by having participants follow specific instructions to develop a unique individualized code on each questionnaire. Of the 110 participants completing the initial questionnaire, a successful match between preintervention and post-intervention questionnaires was accomplished for 72 of these individuals, or 65 % of the number of initial participants. The most common reasons that individuals did not complete a follow-up questionnaire were that they were no longer interested in participating, or had achieved discharge by the time of the follow-up. Data from the 38 participants who completed only the initial questionnaire were excluded from the analyses. To ensure protection of human subjects, the reasons for voluntary dropout were not pursued. Individuals completing the post-intervention questionnaire self-selected whether they were members of the program group or the comparison group according to how many Procovery Circle sessions they reported attending in the last 6 months at the time of the follow-up questionnaire. The data from four participants whom reported attending only one to three Circles were excluded from the analyses. Questionnaire data were analyzed for 43 participants whom reported attending at least four Procovery Circles (program/experimental group), and the data from the remaining 25 individuals who never attended a Procovery Circle (comparison group). Data collection The Procovery Evaluation Instrument (PEI) is a self-report measure that was used to evaluate each participant’s subjective experience on several dimensions of recovery. It was derived from the Peer Outcomes Protocol (POP) with slight modifications made to the language of some items to make them apply to hospitalized participants.31,32 The Peer Outcome Protocol is a 241-item questionnaire designed to measure a range of recovery-oriented outcomes identified as important in the self-help and professional literature.31,32 The POP demonstrated modest to good internal consistency with Cronbach’s alpha scores on 12 scales ranging from .71 to .95. Test–retest reliability scores were also modest to good and ranged from r=.47 to r=.88.32 The PEI contained subscales targeting participant’s sense of well-being, perceived attitudes towards recovery held by the mental health staff providing services to them, their own quality of life, the quality of their social relationships, and their perceived sense of recovery. These scales will be discussed in the next sections. Social Satisfaction and Social Acceptance Scales The Social Satisfaction (SS) Scale was used to measure degree of loneliness, intimacy, and satisfaction with relationships with family and friends. The Social Acceptance (SA) Scale was used to measure perceived attitudes towards recovery held by the mental health staff providing services to respondents.33 These scales have demonstrated modest test–retest reliability (SS, r=.54 and SA, r=.66) and good internal consistency (SS, alpha=.76 and SA, alpha=.93).32 Quality of life The PEI used one question from the Lehman Quality of Life Interview (QOLI) to measure overall quality of life.34 This question asked participants to rate how they felt about their

Statewide Implementation of Recovery Support Groups

MANCINI et al.

393

life overall on a seven-point scale ranging from 1—terrible to 7—delighted. The Lehman QOLI has shown good reliability and validity.34The PEI also used seven items from the Subjective Quality of Life Scale from the POP measuring satisfaction with physical and emotional health, daily activities, and general life satisfaction. This scale demonstrated modest test–retest reliability (r=.63) and internal consistency (alpha=.74).32 Well-being The concept of well-being was measured through items from the POP’s Personhood and Empowerment Scales measuring respondents’ personal views of themselves and the amount of participation they had in their daily routines. These scales have demonstrated modest test–retest reliability (r=.73 and r=.63, respectively) and very good internal consistency scores (alpha=.83 and alpha=.89, respectively).32 The Personhood Scale correlated well with the Rosenberg SelfEsteem Scale (r=.76), while the Empowerment Scale had a much more modest correlation with the Empowerment Decision-Making Scale(r=.46).35,36 Recovery The Recovery Scale from the POP was used to measure respondents’ perceived level of knowledge and engagement in activities known to promote recovery. This scale demonstrated modest test–retest reliability (r=.61), good internal consistency (alpha=.86), and modest concurrent validity when compared to the Recovery Assessment Scale (r=.63).32,37 Program impressions The follow-up PEI included a section that asked Procovery participants about their impressions of the program with ratings from 1 (very negative impression) to 4 (very positive impression). Analysis A 2×2 analysis of variance with repeated measures was used to evaluate whether any observed changes in the subscale scores of the PEI were impacted by the between-participants factor Group Membership (program or comparison), the within-participants factor of time (initial or follow-up), or a complex interaction between these two factors.

Facilitator and administration evaluation Design and data collection This portion of the evaluation used a semi-structured questionnaire, focus groups, and personal interviews to identity perceived consumer outcomes associated with the Procovery Circles, the strengths and weaknesses of the Procovery model, and issues associated with statewide implementation. Only certified Procovery facilitators were eligible for the study. In order to be a Procovery facilitator, one had to be certified through completion of a day-long training covering the Procovery philosophy, structure of the model, practices to be employed within the group, and fidelity components. Questionnaires were mailed to all certified Procovery facilitators at the time of the study (n=92). The questionnaire was created in collaboration between the investigators and state department of mental health personnel. It consisted of the following areas: (1) 25 Likert-type statements about respondents perceptions of the Procovery model with a five-item response set that ranged from strongly disagree to strongly agree; (2) 30 treatment areas in which respondents were asked to identify whether they thought Procovery had either a positive, negative, or no impact; (3) three open-ended questions measuring respondents’ perceptions of the model. Six focus groups were conducted with a total of 60 facilitators regarding their perceptions of the Procovery Circles. Participants were recruited at monthly facilitator meetings at five locations across the state. All facilitators were expected to attend these meetings as part of the demonstration project each month to discuss success and failures of the model, fidelity issues, and other clinical issues and concerns. Focus groups lasted 1 hour and were audio recorded and transcribed. The first

394

The Journal of Behavioral Health Services & Research

40:4

October 2013

and second authors were involved in facilitating the focus groups. Attendance at these focus groups ranged from 7 to 15 participants with an average of 10 participants. Participants were asked to discuss the ways in which they thought that Procovery was helpful or not helpful to their clients, how the approach differed from other approaches they used, and the challenges they experienced implementing the model. Finally, personal interviews were conducted with 13 administrative/team leaders at different agencies across the state representing the full range of geographic regions in which Procovery Circles were implemented. Administrators were asked about their overall impressions of the Procovery model, and the benefits and challenges they have experienced in implementing the model at their agencies. Interviews lasted approximately 45 to 60 minutes and were audio recorded. Eight interviews were face to face, and five were conducted over the telephone. Analysis Questionnaires were analyzed using descriptive statistics, including means, standard deviations, percentages of participant responses, and number of respondents. Focus groups and personal interviews were analyzed for themes regarding perceptions and attitudes about Procovery in general and the specific demonstration project. Commonalties in participant responses were identified and explored using a cross comparative analysis.38,39 Codes were developed for participant responses, which were then collapsed into broader categories and sub-categories using a constant-comparative method across all focus groups.39 Focus group transcripts were coded by two trained researchers. Peer debriefing between researchers was utilized in order to come to a consensus regarding commonalities and differences in codes and categories that emerged. Administrator interviews were analyzed by one researcher.

Protection of human subjects All phases of this study were approved by the Institutional Review Boards of the state department of mental health, the universities from which the evaluators were associated, and the state psychiatric hospital in which the consumer outcome study was completed. Limitations Several methodological limitations should be noted. First, selection bias is possible since both arms of the evaluation used small, nonrandom samples of self-selected individuals, and so it is possible that groups may have differed on some unmeasured characteristics or biases. Second, the small consumer sample size prohibited the use of any type of dosage analyses from being conducted. Third, the fact that the intervention was conducted in a secure and congregate hospital setting also increased the potential of contamination and masking of treatment effects, as well as multiple intervention effects. Fourth, the PEI used to measure consumer outcomes possessed limited psychometric information and demonstrated only modest test–retest reliability and concurrent validity. Likewise, the survey and interview protocols used to assess facilitator and administrator perspectives were not previously tested for reliability or validity and peer debriefing methods were not used with the administrative interviews to enhance trustworthiness relying instead on a single analyst.

Results Consumer perception of outcomes The participants in the program and comparison groups were similar on most background and clinical variables, as they most frequently reported to be male, diagnosed with schizophrenia and/or

Statewide Implementation of Recovery Support Groups

MANCINI et al.

395

affective disorders, to have never been married, to have a heterosexual orientation, and to have mixed ethnicity. The average age of the participants in both groups was 37 years. Differences between the program group and the comparison group in mean scores on the initial questionnaire were small and not statistically significant for five of the six dimensions. The one exception was well-being. A significant group×time interaction was observed for responses on the well-being subscale (F [1, 66]=4.70, pG.05). Examining this complex relationship further revealed that the program group members who attended at least four Procovery Circles showed relatively higher initial well-being subscale scores than the comparison group (F [1, 66]=7.08, p=.01). However, there were no differences in the well-being subscale between the two groups in responses to the follow-up questionnaire. The program group also showed a slight decrease in well-being subscale over time (t 42 =−1.97, p=.05). Thus, the interaction stemmed from a mild decrease in well-being in the program group over time that was not observed in the comparison group, and an initial difference between the groups that was not maintained at follow-up. It should be noted, too, that the program group scores on the well-being subscale were higher than comparison group scores at both time points. Regarding consumers’ perception of the recovery attitudes held by hospital treatment staff, the program group exhibited a significant increase in scores over time (t 41 =3.62, pG.01), while no such change was observed among comparison group scores. Further, the program group had significantly higher scores at follow-up than did comparison group (F [1, 66]=8.11, pG.01). These findings indicate that after attending Procovery Circles regularly, program group members believed that treatment staff members were more supportive of their recovery over time, while the comparison group showed no such change in their perceptions of staff. Overall quality of life ratings for the program group significantly increased over time (t 40 =2.12, pG.05), while comparison group ratings did not change. This indicates that those attending Procovery Circles realized an increase in their overall self-perceived quality of life, while those not attending Procovery did not experience such change. No significant interactions or main effects were observed among the social relationships, consumer recovery attitudes, or quality of life subscales. Table 1 presents findings for the program and comparison groups. Program group members reported higher than average satisfaction (i.e., scores at least above 3 on a four-point scale) with Procovery Circles across a variety of dimensions. These participants indicated that Procovery Circles tended to meet their needs, provided services and supports they were looking for, and helped them manage problems more effectively. They also reported they would refer a friend in need to a Procovery Circle. Comments offered by several Program members indicated that they enjoyed the socialization aspects associated with attending Procovery Circles, as well as the hopeful message of recovery emphasized in the Circles.

Procovery facilitator questionnaire demographics Questionnaires were sent to 92 Procovery facilitators. Seventy-one completed the questionnaire for a response rate of 77 %. Among the respondents, 85 % were female and 92 % were Caucasian, with an average age of 42 years and a range of 24 to 64 years of age. Most respondents had college degrees, including 60 % with a Master’s degree and 17 % with a Bachelor’s degree. The majority of respondents (61 %) were at their respective agencies from 1 to 4 years as peer or non-peer professional staff, while 50 % had 8 or more years of mental health experience. Procovery Circles were conducted in long-term and acute inpatient settings, consumer run drop-in centers, assertive community treatment teams and community mental health centers, clubhouses, and day programs. Specific percentages of these programs were not collected, although approximately 24 % of the subject pool worked in inpatient settings with the remaining in community-based settings. To project confidentiality, respondents were not asked to report their location.

396

The Journal of Behavioral Health Services & Research

40:4

October 2013

Table 1 Means, standard deviations, and analysis of variance (ANOVA) results for experimental and control groups as a function of time Initial Follow-up questionnaire questionnaire Group Well-being Program group Comparison group Staff recovery attitudes Program group Comparison group Overall quality of life Program group Comparison group Quality of life Program group Comparison group Social relationships Program group Comparison group Consumer recovery attitudes Program group Comparison Group

Error df Group (G) Time (T) G×T

M

SD

M

SD

3.51 3.21

0.37 0.55

3.39 3.31

0.49 0.51

3.07 2.95

0.60 0.81

3.46 3.06

0.48 0.63

3.83 4.09

1.95 1.60

4.67 4.08

1.92 1.89

2.78 2.54

0.49 0.39

2.75 2.71

0.52 0.61

2.31 2.11

0.92 0.90

2.16 2.17

0.90 0.64

3.35 3.00

0.90 0.96

3.02 3.09

ANOVA F

66

2.97

0.01

4.70*

65

3.93*

7.66**

2.34

61

0.00

3.85

0.80

66

1.65

1.04

2.16

61

0.19

0.32

1.00

65

0.27

0.65

1.76

1.08 0.81

*p≤.05; **pG.01

Facilitator perceptions of consumer outcomes At least 95 % of respondents reported that Procovery Circles had a positive or very positive impact on consumers’ internal recovery attributes as outcomes such as motivation, problem solving, hope, socialization, communication, self-esteem, and confidence among other outcomes. However, only 58 and 63 % thought that Procovery Circles had a positive or very positive impact on consumers’ homelessness and employment, respectively. One respondent reported that Procovery Circles had a negative or very negative impact for the outcomes of goal development, assertiveness, and substance use. Table 2 includes complete results. Three themes emerged from the focus groups, administrator interviews, and responses to the open-ended question on the survey (N-59). First, Procovery was viewed as positive and helped participants gain a sense of hopefulness for the future as illustrated by the following facilitator quote, “I think it’s that hope, you know, that I can have a better life, and it has really pushed them to look at life differently.” A second finding was that facilitators stated that Procovery Circles assisted in the socialization of participants through mutual support and self-help. Through this process, participants were able to provide help and hope to one another. A third theme was the development of increased expectations and motivation to further participate in treatment as illustrated by the following quote, “I think it inspires hope, self advocacy and realizing that they’re not the only ones who are struggling with a psychiatric illness.”

Statewide Implementation of Recovery Support Groups

MANCINI et al.

397

Table 2 Responses by circle facilitators to treatment areas impacted by Procovery Sign Mean (SD) Treatment participation Personal responsibility Motivation Decision making Problem solving Socialization Hope Articulating needs Goal development Alcohol/substance use Commitment to developing wellness Use of medication Developing insight into problems Crisis/relapse prevention Substance use Making choices Articulating choices Assertiveness Literacy Homelessness Communication skills Illness self-management strategies Ability to make friends and build supports Treatment engagement Expressing feelings Developing meaningful activities Employment Understanding feelings Self-esteem Confidence in abilities

No

Sign

Neg (%)

Neg (%)

Imp (%)

Pos (%)

Pos (%)

N

4.4 4.3 4.5 4.2 4.2 4.6 4.7 4.2 4.2 3.9 4.3 4.1 4.4 4.2 3.8 4.2 4.3 4.2 4.1 3.7 4.3 4.3 4.4

(0.5) (0.5) (0.6) (0.5) (0.5) (0.5) (0.5) (0.5) (0.6) (0.6) (0.5) (0.6) (0.5) (0.6) (0.7) (0.5) (0.4) (0.6) (0.7) (0.7) (0.5) (0.5) (0.5)

0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 2 0 0 0 0 0

2 0 3 6 3 2 0 5 6 24 2 15 2 7 25 3 0 3 18 42 0 3 2

55 68 47 70 69 37 30 73 64 61 68 59 59 66 63 73 73 68 53 46 67 68 60

43 32 50 24 27 62 70 22 29 15 31 25 39 27 10 24 27 28 30 12 33 29 39

60 63 66 63 62 65 67 60 63 46 59 59 64 55 48 67 63 65 57 33 66 62 65

4.3 4.5 4.2 3.7 4.3 4.4 4.3

(0.5) (0.5) (0.6) (0.6) (0.4) (0.5) (0.5)

0 0 0 0 0 0 0

0 0 0 0 0 0 0

2 0 8 37 0 0 2

63 49 64 54 74 63 70

36 51 28 9 27 36 28

64 67 61 43 68 66 67

SD standard deviation, Sign Neg significant negative impact, Neg some negative impact, No Imp no impact, Pos some positive impact, Sign Pos significant positive impact, N total number of respondents to question

Statewide expansion of Procovery Circles Another focus of the evaluation was to obtain an overall impression of the Procovery model to help identify implementation issues. Data from each of the sources provided guidance for this. Ninety-four percent of facilitators at least agreed that they support the model (83 % strongly), while 85 % of facilitators perceived that staff at their agencies supported Procovery, and 96 % reported supervisors and administrators supported the model. Almost all respondents (97 %) agreed the

398

The Journal of Behavioral Health Services & Research

40:4

October 2013

model had a positive impact on consumers. Further, 92 % of facilitators at least agreed that the Procovery model had a positive influence on both their professional practice and personal lives. Ninety-seven percent of facilitators at least agreed (81 % strongly) that Procovery should continue at their sites, and 93 % supported full statewide implementation. Table 3 includes complete results. Responses from the facilitator focus groups and administrator interviews also provided overall support to Procovery. Administrators were unanimous in their opinion that Procovery was a strategy that was commensurate with a recovery-oriented mission as illustrated by the following administrator quote: “[Procovery] is a strength-based model that…mobilizes; it empowers consumers and recipients of services, family members…And I think the term “Procovery” is a very positive branding of the recovery philosophy…. We want everyone that we’re working with not just to have symptoms and their disorders directly treated, but we want to help them to remove any impediments that they have to leading a full, productive life. And Procovery is a very strong tool in being able to do that.” A second and related aspect of the program that was most often cited was that participation in Procovery was a voluntary choice. This was perceived to lead to better engagement of participants in both Procovery and other treatments as illustrated by the following facilitator quote: “I work at a state hospital and of course people are restricted in many ways when they live there, and part of the beauty of this model is that it’s freeing and it gives them choice, whether or not to come, whether or not to participate when you’re in the group…in a world where you have very little choice about many things, that ends up being a big thing.” A majority of administrators talked about Procovery as useful only as part of a broader menu of psychiatric rehabilitation programming that met the employment, housing and other health and mental health needs and goals of consumers as illustrated in the following quote: “[Procovery] is not a substitute for our having jobs programs and working with people on housing and community support and treatment. It needs to be part of a larger toolbox available to us in terms of helping people achieve all of their goals.” Implementation issues Overall, 86 % of facilitators at least agreed that implementation of the Procovery demonstration initiative had gone smoothly. Ninety-six percent at least agreed that the level of training they received was adequate, although about one third of facilitators believed they needed additional training. Ninety-three percent of facilitators believed the training resulted in a good understanding of the philosophy and principles of the Procovery model, and 96 % agreed or strongly agreed that they were capable of facilitating Procovery Circles. See Table 3 for full results. Not all aspects of the implementation of the demonstration project were positive. Facilitators and administrators indicated that they believed that strict fidelity constraints to Procovery made effective tailoring and implementation problematic as illustrated by the following facilitator quote: “We have people who are actively psychotic and actively having problems with thought disorder and tracking….And so sometimes we’ve had to adapt the groups a little bit for them. And I think that was the thing that our staff struggled with the most, was not being able to adapt to what they saw as the immediate need.” A second type of constraint was the training and reporting requirements of the Procovery model. Agencies were required to send all facilitators to a full-day training session, attend ongoing 2 hours monthly regional fidelity meetings, submit weekly attendance logs, and purchase refreshments for the socialization period costing agencies staff time and money leading to frustration as illustrated in this administrator quote: “It’s like this is such an easy thing to do and we have training after training after training to do it.…And sometimes I can’t spare the people that need to go to the fifteenth ‘let's learn how to do this’ meeting.” Another administrator stated the following: “I think that part of the reluctance of some of the staff members is because it’s not just the group; it’s also the requirement to come to the

Statewide Implementation of Recovery Support Groups

MANCINI et al.

399

Table 3 Facilitators’ overall perception of the Procovery model and implementation Str

Overall, I support the Procovery Model. I believe participation in Procovery Circles has had a positive impact on participants. The Procovery Model has had a positive influence on my professional practice. Facilitating Procovery circles has had a positive influence in my personal life. The Procovery Model is supported by all or most other staff at my agency. The Procovery Model is supported by administrators and supervisors at my agency. I believe that the Procovery Model is something I have always done. I believe that Procovery works well for some participants but not for others. There are parts of Procovery that are ineffective. I support the continuation of use of the Procovery model at my site. I think the Procovery model should be implemented across the state. I believe the implementation of the Procovery initiative has gone smoothly. I have received an adequate level of training in Procovery facilitation. I need additional training in Procovery facilitation. I fully understand the philosophy and principles of the Procovery Model. I feel completely capable of facilitating Procovery Circles. I support family members facilitating Procovery Circles. I support consumers facilitating Procovery Circles.

Not

Str

Mean (SD)

Dis (%)

Dis (%)

Sure (%)

Agr (%)

Agr (%)

N

4.7 (0.9) 4.7 (0.7)

4 1

0 0

1 1

11 25

83 72

71 71

4.3 (0.8)

2

2

5

46

46

57

4.3 (0.9)

1

4

6

45

44

71

4.2 (0.8)

0

5

11

48

37

65

4.5 (0.7)

2

0

3

37

59

65

3.0 (1.2)

10

33

16

31

9

67

3.1 (1.0)

7

23

26

40

3

69

2.3 (1.0)

23

35

30

11

1

71

4.8 (0.6)

1

0

1

16

81

70

4.6 (0.7)

1

0

6

23

70

71

4.1 (0.8)

0

7

7

58

28

71

4.4 (0.7)

1

1

1

48

48

71

2.6 (1.1)

14

32

21

30

3

71

4.5 (0.8)

1

1

4

37

56

71

4.5 (0.7)

1

0

3

35

61

71

4.0 (0.9)

2

3

19

43

34

68

4.6 (0.7)

0

1

7

24

68

71

SD standard deviation, Str Dis strongly disagree (1), Dis disagree (2), Not sure (3), Agr agree (4), Str Agr strongly agree (5), N total number of respondents to question

meetings…so not only do you lose the day but it affects personal time and children, arrange childcare….So I mean those kinds of things make it really difficult on top of the time at work.”

400

The Journal of Behavioral Health Services & Research

40:4

October 2013

Furthermore, facilitators and administrators voiced concern that the lack of funding for staff time devoted to Procovery for reimbursement was an implementation barrier as illustrated in the following quote from an administrator: “One of the auditors said, ‘Oh, but you can't bill for Procovery.’ And I said, ‘It’s crazy. You’re telling me I can bill for them playing pool and interacting with people at the pool table but I can’t bill for them going to the group?’ But we didn’t get any additional money to do any of it, and we weren't compensated for staff away or anything like that…That’s why we had to cut back on how many staff could participate in it.”

Discussion Program participants reported an improved overall satisfaction with their lives after participating in Procovery Circles on at least four occasions over 6 months when compared to those who did not attend Procovery Circles. Additionally, Procovery participants reported an increase in their beliefs that staff supported their recovery from baseline to the 6-month follow up. One possible interpretation for these results is that consumers’ positive feelings about the different approach employed by staff facilitating Procovery Circles generalized to their overall perceptions of staff. This may be a significant finding given the importance of a positive therapeutic relationship on consumer outcomes.40 It is uncertain why program participants did not improve in well-being, quality of life, social relationships, and social exclusion. One can speculate Procovery by itself is not enough to overcome the negative effects of the residing in a long-term state psychiatric hospital. Another explanation may have to do with the dose of treatment. Consumers were categorized as Procovery participants if they attended four groups or more. Attending small numbers of groups may not provide these additional benefits. Small sample size prohibited analyses based on increased attendance at Procovery Circles. This evaluation also found that Procovery is highly supported by facilitators due to the positive focus of the model and its emphasis on mutual support. These factors have been found to be important ingredients in the recovery process.1,2,5–9 Facilitators also stated that consumers were highly engaged in the model due to the model’s client and strength-centered focus, nonhierarchical structure, and voluntariness. Consumer-based approaches will continue to play a role in mental health reform efforts due to their specific recovery-oriented values such as consumer choice, voluntary participation, self-determination, and staff respect.25,26,29

Implications for Behavioral Health Recent research indicates that effective implementation of innovative and evidence-based practices in community mental health settings is a challenge.41–43 The evaluation identified some specific issues and challenges related to implementation in the areas of support, fidelity, and training. First, data indicate that administrators struggled with a lack of clear financial support for Procovery from the state department of mental health. In order for innovative practices to be sustained in system transformation efforts, it is important that the mental health authorities governing change provide financial incentives and support for its initiatives in the form of reimbursement for services, transportation, and materials.26,44 Second, facilitators and administrators also struggled with some of the strict fidelity components and structure of the Procovery model. Providing facilitators with flexibility to creatively adapt the model to meet the needs of their clients, while preserving the model’s essential core functions, may have beneficial effects both for consumer outcomes and the evolution of the model itself.41,45,46 Third, administrators and facilitators also identified ongoing continuous quality improvement monitoring in the form of monthly facilitator meetings as a barrier to continued implementation due to time and transportation costs. Technology in the form of web-based learning, e-learning communities of practice, audio and video teleconferencing, DVDs, and DVD ROMS could provide

Statewide Implementation of Recovery Support Groups

MANCINI et al.

401

a more efficient means for continuous quality improvement and training tailored to meet the needs of diverse staff and locations.41,47,48 Combined with financial incentives and support, these approaches could deliver training and continuous quality improvement without the cost barriers to implementation seen in some sites. The Procovery Model may be an effective consumer-based option when combined with other evidence-based practices for mental health systems seeking to be more recovery oriented. However, adequate resources and opportunities in the community must be developed to help consumers access housing, employment, healthcare, and transportation. It is recommended that further research be conducted on the effectiveness of Procovery in both the short and long term, particularly including measures that go beyond self-reported outcomes and perceptions to more behaviorally oriented outcomes, including active participation in treatment, release from long-term psychiatric hospitals, and maintaining community housing, to name a few. Additional formative and summative evaluations of the model that specifically look at differences between communitybased settings and inpatients settings are needed. Studies that examine consumer outcomes and cost-effectiveness using randomized, controlled designs in a variety of community and inpatient settings are especially recommended. Conflict of Interest The authors do not have any conflicts of interest to report.

References 1. Anthony, WA. Recovery from mental illness: The guiding vision of the mental health service system in the 1990’s. Psychosocial Rehabilitation Journal 1993;16:11–23. 2. Author, et al. Making sense of it all: Consumer providers’ theories about factors facilitating and impeding recovery from psychiatric disabilities. Psychiatric Rehabilitation Journal 2005; 29(1): 48–55. 3. Deegan, P. Recovery and empowerment for people with psychiatric disabilities. Social Work and Health Care 1997; 25: 11–24. 4. Substance Abuse and Mental Health Services Administration. Working definition of mental disorders and substance use disorders. Available online at: http://www.samhsa.gov/newsroom/advisories/1112223420.aspx. Accessed June 15, 2012. 5. Andresen, R, Oades, L, & Caputi, P. The experience of recovery from schizophrenia: Towards an empirically validated stage model. Australian and New Zealand Journal of Psychiatry 2003; 37(5): 586–594. 6. Davidson, L, & Strauss, J. Sense of self in recovery from severe mental illness. British Journal of Medical Psychology 1992; 65: 131–145. 7. Estroff, SE. Self, Identity and subjective experiences of schizophrenia: In search of the subject. Schizophrenia Bulletin 1989; 15: 189–196. 8. Author. Empowering people with severe mental illness: A practical guide. New York: Oxford University Press, 2006. 9. Resnick, SG, Fontana, A, Lehman, AF., et al. An empirical conceptualization of the recovery orientation. Schizophrenia Research 2005; 75(1): 119–28. 10. Link, BG, Struening, EL, Neese-Todd, S, et al. Stigma as a barrier to recovery: The consequences of stigma for the self-esteem of people with mental illnesses. Psychiatric Services 2001; 52: 1621–1626. 11. Lally, SJ. “Does being here mean there is something wrong with me?” Schizophrenia Bulletin 1989; 15: 253–265. 12. Onken, SJ, Slaten, E. Disability identity formation and affirmation: The experiences of persons with severe mental illness. Sociological Practice: A Journal of Clinical and Applied Sociology 2000; 2: 99–111. 13. Roe, D. Progressing from “patienthood” to “personhood” across the multidimensional outcomes in schizophrenia and related disorders. Journal of Nervous & Mental Disease 2000; 189(10): 691–699. 14. Carney, CP, Jones, L, Woolson, RF. Medical co-morbidity in women and with schizophrenia: a population-based controlled study. Journal of General Internal Medicine 2006; 21(11): 1133–1137. 15. Caton, CL, Shrout, PE, Eagle, PF, et al. Risk factors for homelessness among schizophrenic men: A case–control study. American Journal of Public Health 1994; 84: 265–270. 16. Miller, BC, Paschall, CB, Svendsen, DP. Mortality and medical comorbidity among patients with serious mental illness. Psychiatric Services 2006; 57(10): 1406–1415. 17. Regier, DA, Farmer, ME, Rae, DS, et al. Comorbidity of mental disorders with alcohol and other drug abuse: Results from the Epidemiologic Catchment Area (ECA) study. Journal of the American Medical Association 1990; 264: 2511–2518. 18. Brunette, MF, Mueser, K, Drake RE. A review of the research on residential programs for people with severe mental illness and cooccurring substance use disorders. Drug and Alcohol Review 2004; 23: 471–481. 19. Carey, K. Substance use reduction in the context of outpatient psychiatric treatment: A collaborative, motivational, harm reduction approach. Community Mental Health Journal 1996; 32(3): 291–306. 20. Author, et al. Impact of staff attention on predicting post-discharge community tenure of psychiatric inpatients. Psychological Services 2007; 4: 306–315.

402

The Journal of Behavioral Health Services & Research

40:4

October 2013

21. Drake, RE, Mueser, KT, Brunette, MF, et al. A review of treatments for people with severe mental illnesses and co-occurring substance use disorders. Psychiatric Rehabilitation Journal 2004; 27(4): 360–374. 22. Minkoff, K. An integrated model for the management of co-occurring psychiatric and substance disorders in managed care systems. Disease Management and Health Outcomes 2000; 8(5): 251–257. 23. President’s New Freedom Commission on Mental Health. Achieving the promise: Transforming mental health care in America. DHHS Publication No. SMA-03-3832, Rockville, MD: Department of Health and Human Services, 2003. 24. Campbell, J. The historical and philosophical development of peer-run support programs. In: S Clay, B Schell, PW Corrigan, RO Ralph (Eds).On our own together: Peer programs for people with mental illness. Nashville: Vanderbilt University Press, 2005, pp. 17-64. 25. Holter, MC, Mowbray, CY, Bellamy, CD, et al. Critical ingredients of consumer-run services: Results of a national survey. Community Mental Health Journal 2004; 40(1): 47–63. 26. Nelson, G, Janzen, R, Trainor, J, et al. Putting values into practice: Public policy and the future of mental health consumer-run organizations. American Journal of Community Psychology 2008; 42: 192–201. 27. Solomon, P. Peer support/peer provided services: Underlying processes, benefits, and critical ingredients. Psychiatric Rehabilitation Journal 2004; 27: 392–401. 28. Solomon, P, Draine, J. The state of knowledge of the effectiveness of consumer provided services. Psychiatric Rehabilitation Journal 2001; 25: 20–27. 29. Hardiman, ER, Theriot, MT, Hodges, JQ. Evidence based practice in mental health: Implications and challenges for consumer-run programs. Best Practices in Mental Health 2005; 1: 105–122. 30. Crowley, K. The power of Procovery in healing mental illness: Just start anywhere. Los Angeles, CA: Kennedy Carlisle Publishing, 2000. 31. Campbell, J, Cook, JA, Jonikas, JA, et al. The Peer Outcomes Protocol Questionnaire. Chicago: University of Illinois at Chicago National research and Training Center on Psychiatric Disability, 2004. 32. Campbell, J, Einspahr, K. Evenson, R, et al. The Peer Outcomes Protocol (POP): Psychometric properties of the POP. Chicago: University of Illinois at Chicago National Research and Training Center on Psychiatric Disability, 2004. 33. Campbell, J, Schraiber, R. The Well-being Project: Mental health clients speak for themselves. Sacramento, CA: California Department of Mental Health, 1989. 34. Lehman, AF. A Quality of Life Interview for the chronically mentally ill. Evaluation and Program Planning 1988; 11: 51–62. 35. Rosenberg, M. Society and the adolescent self-image. Princeton, NJ: Princeton University Press, 1965. 36. Rogers, ES, Chamberlin, J, Ellision, ML, et al. A consumer-constructed scale to measure empowerment among users of mental health services. Psychiatric Services 1997; 48(8): 1042–1047. 37. Giffort, D, Schmook, A, Woody, C, et al. Recovery Assessment Scale. Chicago, IL: Illinois Department of Mental Health, 1995. 38. Krueger, RA, Casey, MA. Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage, 2000. 39. Miles, M, Huberman, A. Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage, 1994. 40. Castonguay, LG, Beutler, LE. Principles of therapeutic change that work. New York: Oxford University Press, 2006. 41. Beidas, RS, Koerner, K, Weingardt, KR, et al. Training research: Practical recommendations for maximum impact. Administration and Policy in Mental Health 2011; 38: 223–237. 42. Rapp, CA, Etzel-Wise, D, Marty, D, et al. Barriers to evidence-based practice implementation: Results of a qualitative study, Community Mental Health Journal 2010; 46: 112–118. 43. Swain, K, Whitley, R, McHugo, GJ, et al. The sustainability of evidence based practices in routine mental health agencies, Community Mental Health Journal 2010; 46: 119–129. 44. Roussin-Isett, K, Burnam, MA, Coleman-Beattie, B, et al. The role of state mental health authorities in managing change for the implementation of evidence-based practices. Community Mental Health Journal 2008; 44: 195–211. 45. Webb, C, DeRubeis, R, Barber, J. Clinician adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology 2010; 78: 200–211. 46. Kendall, PC, Gosch, E, Furr, J. Flexibility within fidelity. Journal of the American Academy of Child and Adolescent Psychiatry 2008; 47: 987–993. 47. Cucciare, M, Weingardt, K, Villafranca, S. Using blended learning to implement evidence-based psychotherapies. Clinical Psychology: Science and Practice 2008; 15: 299–307. 48. Weingardt, K, Cucciare, M, Bellotti, C, et al. A randomized trial comparing two models of web based training in cognitive behavioral therapy for substance abuse counselors. Substance abuse 2009; 27: 19–25.

Statewide Implementation of Recovery Support Groups

MANCINI et al.

403