Article Alcohol
Factors that enable and hinder the implementation of projects in the alcohol and other drug field Abstract
Sarah MacLean Turning Point Alcohol and Drug Centre, Victoria, and Centre for Health and Society, University of Melbourne, Victoria
Objective: Few studies systematically explore elements of successful project implementation across a range of alcohol
Lynda Berends
and other drug (AOD) activities. This
Turning Point Alcohol and Drug Centre, Victoria, and Department of Epidemiology and Preventive Medicine, Monash University, Victoria
paper provides an evidence base to inform project implementation in the AOD field. Approach: We accessed records for 127
Barbara Hunter, Bridget Roberts
completed projects funded by the Alcohol,
Turning Point Alcohol and Drug Centre, Victoria, and Eastern Health Clinical School, Monash University, Victoria
Education and Rehabilitation Foundation from 2002 to 2008. An adapted realist synthesis methodology enabled us to develop categories of enablers and barriers
Janette Mugavin
to successful project implementation, and to
Turning Point Alcohol and Drug Centre, Victoria
identify factors statistically associated with successful project implementation, defined
P
as meeting all funding objectives. Thematic
rojects have limited scope to achieve their intended impact if they are poorly implemented. We report here on a study designed to identify enabling factors critical to successful project implementation and barriers that impeded implementation within a sample of 127 completed projects funded by the Alcohol, Education and Rehabilitation (AER) Foundation between 2002 and 2008. Since its inception in 2001, the AER Foundation (now known as the Foundation for Alcohol Research and Education) has provided time-limited funding for more than 1,000 projects intended to reduce the burden of harm arising from alcohol and/or inhalant misuse in Australia. Records of funded projects held at the AER Foundation provide rich information on successes and difficulties encountered in relation to funded activities in the alcohol and other drug (AOD) field. Projects in the study sample entailed diverse activities. Some were treatment programs; others comprised workforce training or community education and Submitted: August 2010
prevention activities. An additional group of projects involved strategies to enhance the efficiency of AOD agencies through, for instance, policy development. All projects in the study sample were implemented within AOD agencies, or were designed to address AOD misuse and were implemented in health and welfare agencies. While substantial research effort is devoted to identifying the outcomes of AOD interventions, few studies systematically explore elements of successful project implementation across a range of activities. Evaluations of AOD and other health interventions attend to issues of implementation; however they usually concern one intervention type.1-4 Other research provides theoretical rather than empirical analyses, offering useful typologies of factors that contribute to intervention success.5-9 Drawing on this literature and on a heterogeneous sample of 127 AER Foundation funded projects, this study provides empirically-based categories of barriers and enablers to project implementation ranked according to the frequency with which they
Revision requested: April 2011
Accepted: June 2011
Correspondence to: Dr Sarah MacLean, Turning Point Alcohol and Drug Centre, Gertrude Street, Fitzroy, Victoria 3065; e-mail:
[email protected]
analysis of eight case study projects allowed detailed exploration of findings. Results: Nine enabler and 10 barrier categories were identified. Those most frequently reported as both barriers and enablers concerned partnerships with external agencies and communities, staffing and project design. Conclusion: Achieving supportive relationships with partner agencies and communities, employing skilled staff and implementing consumer or participant input mechanisms were statistically associated with successful project implementation. Implications: The framework described here will support development of evidencebased project funding guidelines and project performance indicators. The study provides evidence that investing project hours and resources to develop robust relationships with project partners and communities, implementing mechanisms for consumer or participant input and attracting skilled staff are legitimate and important activities, not just in themselves but because they potentially influence achievement of project funding objectives. Key words: health services research, health planning guidelines, capacity building, community health services Aust NZ J Public Health. 2012; 36:61-8 doi: 10.1111/j.1753-6405.2011.00804.x
2012 vol. 36 no. 1
AUSTRALIAN AND NEW ZEALAND JOURNAL OF PUBLIC HEALTH © 2012 The Authors. ANZJPH © 2012 Public Health Association of Australia
61
MacLean et al.
Article
occurred within the study sample. Additionally, the study identifies a small number of barriers and enablers that were statistically associated with meeting all funding objectives for projects in the study sample.
Methods Realist synthesis methodology10,11 attempts to grapple with the complex factors influencing a social intervention’s impact by reviewing underlying mechanisms or sequences of events that enable interventions to work, rather than assessing specific interventions or programs. For example, Pawson11 explored the effect of ‘incentivisation’ (providing incentives to people to use services or engage in programs) across the contexts of smoking cessation, housing and correctional services. An adapted version of realist synthesis enabled us to explore a range of variables associated with successful and less successful project implementation across projects involving different activities, as discussed in greater detail elsewhere.12 Ethical approval to conduct the research was obtained from the University of Melbourne Human Research Ethics Committee.
The 127 projects included in our eventual study sample were selected in order to maximise similarity of reporting. We excluded projects where final reporting was unavailable, which were funded for less than $20,000, or which funded research or infrastructure. The AER stores project documentation electronically and in paper files. We analysed funding agreements, interim and final reports (and external evaluations for the 41 projects where they existed) to identify and record barriers and enablers to project implementation for each project. Data analysis entailed first and second coding cycles.13 Lower level (more specific) enabling and barrier factors were identified for each project and coded. Searches were conducted to develop a brief literature review concerning implementation of health and welfare interventions. Enablers and barriers to successful project implementation were then developed through an iterative process involving clustering factors into conceptually coherent categories which also reflected themes emerging from our literature review. Enabler and barrier factors for each project were then coded into these categories. We also identified which projects were successfully implemented, defined as having met all project funding objectives.
Table 1: Enabler and barrier categories identified. Enablers External communication and relationships
Barriers Engaging communities and partner agencies
Support from partner agencies or from participating communities, used existing networks Staffing and leadership
Lack of partner agency or community interest in or commitment to project, partner agency withdrawal Identifying and retaining staff
Employed suitable staff or contractors, staff or management provided leadership, staff training activities undertaken
Delay in staff recruitment, staff or management turnover, staff lacked required skills
Project planning and design
Project planning and design
Evidence-based model, good fit to needs, flexible design, appropriate scoping, holistic approach
Poor fit, inadequate scoping, poor timing
Organisational governance and capacity
Governance
Organisation already experienced in project work, good policies & procedures, effective internal reference group Sensitivity to service users and settings
Lack of management involvement, reference group unrepresentative or ineffectual Meeting cultural needs of specific demographic groups
Model culturally appropriate, employed culturally/gender appropriate staff
Resources or approach not culturally appropriate, failed to engage specific demographic groups
Staff team communication and relationships
Clarity of staffing roles
Staff engaged and enthusiastic about project, staff consultation mechanisms Participatory approach to service delivery
Staff roles unclear, staff conflict, staff didn’t prioritise project involvement, placed additional stress on staff Complexities of service users
Target group involved in development , used role models or peer approaches, activity based approaches Funding and resourcing
Participation poor, challenging behaviours
Well funded, used existing resources, gained additional funding, partner agency contributed resources
Submissions for ongoing funding unsuccessful, other funding problems
Research and data collection
Research, evaluation and data collection
Well documented, effective data collection, ongoing research
Poor data collection systems, poor response to evaluation, datasets missing Wider service system challenges
Service system issues not identified as an enabler by projects in the study sample
Funding and resourcing
Lack of other services, inter-professional problems, philosophical differences in addressing AOD
Note: Enablers are listed alongside a corresponding barrier. They are presented in order of enabler to implementation most frequently mentioned in project records to least frequently mentioned.
62
AUSTRALIAN AND NEW ZEALAND JOURNAL OF PUBLIC HEALTH © 2012 The Authors. ANZJPH © 2012 Public Health Association of Australia
2012 vol. 36 no. 1
Alcohol
Factors affecting project implementation
Enablers to project implementation As noted above, evaluations of AOD and other public health interventions identify a range of issues implicated in successful project implementation. For example, in the US, Guydish et al.8 recommended an extensive period of implementation planning for substance abuse treatment programs to account for practical considerations and ensure consistency in the theoretical paradigm underpinning operations. A review of factors that contribute to long-term community AOD harm-prevention projects found that honouring community values, cultivating key leader support, and utilising Indigenous staff were key characteristics of successful projects.3 Reflection on the implementation of centralised intake models for AOD treatment across three jurisdictions (the US; Ontario, Canada; and Victoria, Australia) noted the importance of preparatory activities so treatment sectors were ready and amenable to change.5 Key success factors in community-based lifestyle interventions have been identified as: extensive reach, adoption and engagement; sufficient time for social and societal changes to take place; substantial commitment from all stakeholders; and sufficient funding.15 Strempel and colleagues2 documented five examples of ‘best’ practice in Australian Indigenous AOD
2012 vol. 36 no. 1
100
Percentage of projects
90 80 70 60
73 61
50
61 53
40
42
30
38
36
36 28
20 10
nd de sig n is n ati o an n a dc lg o ap ac vern Se it y an ns ce itiv it y to ser vic eu ser Sta s ff t ea m int era cti Pa on rti cip ato ry ap pro Fu ac nd h in g an dr eso urc in g Re sea rch an dd ata
ga
de rsh ip
n in
Or ga
Pla n
lea
Sta ff
ter n
al
rel ati on sh ip
s
0
Ex
Data were collected through Access and exported to SPSS for analysis. Fisher’s Exact Test was used to assess relationships between successful project implementation and each of the enabling and barrier factors, with a p value of 0.05 or less denoting significance. Values are stated for factors significantly associated with successful implementation. Table 1 shows the enablers and barriers identified through this process, with examples of factors included in each of these categories listed in italics: Eight case studies were selected from the overall study sample to give greater depth to our analysis by exploring implementation across a mix of project types, urban and non-urban settings, and implementation success. Selection criteria included availability of external evaluations, recent funding and organisational agreement to participate. A total of 22 interviews were conducted with a thematic analysis14 of interview transcripts used to identify arguments about factors associated with project implementation. Inevitably, the study entailed limitations, not least of which concerned the heterogeneity of projects in the sample. Barrier and enabling factors were clearly articulated for some projects; in others we were required to identify them from available documentation. Barriers and enablers may be categorised in many ways, and ‘successful implementation’ is also open to various interpretations. These decisions reflect the study team’s wish to identify definitions that both reflect the data and make sense to AOD agencies and funding bodies. Further, some barriers and enablers do not apply equally across the study sample (e.g. the barrier ‘meeting cultural needs of specific demographic groups’) and the importance of these issues as major themes for implementation may thus be underrepresented. Percentages are provided for each barrier and enabler category for consistency of reporting. Findings should be interpreted with these limitations in mind.
Enablers
Figure 1: Enablers to project implementation by frequency noted in project documentation (n=127).
projects, concluding that each entailed “clearly defined and effective management structures and procedures; trained staff and ongoing staff development programs; good multi-strategy and collaborative approaches; strong leadership; adequate and continuing funding; and clearly defined realistic objectives aimed at the provision of appropriate services that address community needs”. These frameworks informed our identification of enabler and barrier categories, as outlined below and illustrated in Figure 1.
External communication and relationships Community-based initiatives are unlikely to be effective without support and participation from the communities concerned or integration with other elements of the service system.16 An evaluation of partnership strategies in Victorian primary care found that partnerships were most successful when there was: a common purpose; effective communication and clarity about roles and relationships; a positive attitude among stakeholders and member agencies towards the partnership and each other; supportive policy, planning and resources; and relevant effective skills and leadership.17 Collaboration requires substantial investments of time and energy.18 Many projects in our study sample relied on partnerships with other agencies or engagement with communities and their representatives to implement activities. Across all projects in our study sample, ‘external communication and relationships’, emerged as the most commonly cited enabler, reported in relation to 73% (n=93) of projects. Within this category, partner agency enthusiasm or having already established relationships with key agencies was the most commonly cited enabling factor, noted in relation to 52% (n=66) of projects. Community enthusiasm for the project and for the organisation conducting the project was cited as an enabling factor in relation to 28% (n=36) of projects. Political or government support was a further factor that facilitated project implementation. As described above, each project in the study sample was classified as successfully implemented or not, according to whether all objectives stipulated in funding agreements had been met. Barrier and enabler factors were then tested for statistical association
AUSTRALIAN AND NEW ZEALAND JOURNAL OF PUBLIC HEALTH © 2012 The Authors. ANZJPH © 2012 Public Health Association of Australia
63
MacLean et al.
Article
with successful implementation. For the majority of successfully implemented projects (78%, n=99), reports or evaluations noted community and/or elder enthusiasm (in the case of projects in Indigenous settings) for the organisation and project, while only 22% of projects not successfully implemented showed evidence of this support. This was a statistically significant difference (p=
