Prev Sci (2015) 16:1123–1127 DOI 10.1007/s11121-015-0606-3
Studying Program Implementation Is Not Easy but It Is Essential Joseph A. Durlak 1
Published online: 23 September 2015 # Society for Prevention Research 2015
Abstract This study offers a commentary on the articles contained in the special issue of Prevention Science, BReadiness to implement Social- Emotional Learning interventions.^ The commentary also puts these articles into current context by summarizing important findings in implementation research and listing some priorities for future work. Keywords Implementation . Social and emotional development . School interventions Readers who have carefully read the articles in this special issue (Wanless and Domtrovich, Readiness to Implement Social-Emotional Learning Interventions) are apt to come away amazed about the complexity of research on implementation. For example, the authors of these seven studies examined a total of 65 variables that might influence implementation, and they employed 22 measures to capture four different components of implementation. Finally, they collected data on implementation up to six times within an academic year and over a 4-year period. Some of the variables that the authors in this special issue have examined as potentially affecting the level of achieved implementation include such things as teachers’ attitudes toward the intervention and their sense of self-efficacy, various aspects of school climate and work relationships, and student behavior and responsiveness to intervention. The components of implementation that were studied included dosage, fidelity, quality of delivery, and participant responsiveness that were assessed using self-reports, direct observation, video recordings, or consultant ratings. * Joseph A. Durlak
[email protected] 1
Loyola University Chicago, Chicago, IL 60660, USA
Readers might also be curious about some of the current results. For example, sometimes authors’ findings were not consistent with previous research and sometimes were not consistent with the findings from other articles in this special issue in terms of which variables were significantly related to implementation. For example, teacher demographic characteristics such as educational background or experience predicted implementation in two reports (Domitrovich et al. 2015; Williford et al. 2015) but not in a third (Wanless et al. 2014). Within the same study, a predictor might be significantly related to one component of implementation such as quality of delivery but not to dosage (Malloy et al. 2014). Moreover, when different components of implementation were assessed, they typically showed modest relationships with each other (Domitrovich et al. 2015; Malloy et al. 2014; Williford et al. 2015), and the data obtained from multiple efforts to assess the same component of implementation such as teacher responsiveness or fidelity were not always highly correlated with each other (Roberts et al. 2014; Williford et al. 2015). What is going on here? Two questions might come to mind. First, does studying implementation need to be this complicated? Second, should one expect more consistency in the results from current and previous studies and among those in the current issue? The simple answer to the first question is BYes,^ studying implementation is very complicated. The answer to the second question is Bnot necessarily.^ Inconsistency in findings is apt to occur for several reasons. To understand the answers to these two questions, it is helpful to step back and place the investigations described in this special issue into the current status of implementation science. To do so, I need to define implementation, provide a brief history of research on implementation, and discuss several major findings that bring the reader up to date on what we know and do not know about implementation.
1124
What Is Implementation? The terminology used to study implementation varies across disciplines. For purposes of the current discussion, implementation can be defined as the ways a program is put into practice and delivered to participants. Implementation thus refers to what a program looks like Bon the ground^ when it is being conducted, as opposed to what a program looks like in theory or on the drawing board. For a variety of reasons, there are often differences between the conceptualization of a program and its subsequent day-to-day operations and delivery. Implementation is not an all-or-none phenomenon but exists along a continuum, so one can think of the level or degree of implementation that is achieved such as low, medium, or high.
Brief History of Implementation Research Although research on implementation goes back more than a hundred years to when extension agents began helping farmers apply scientifically based practices to increase their productivity and crop yields (Rogers 2003), it was not until the 1970s and early 1980s that implementation began receiving specific attention in the applied social sciences and education. Research on implementation has grown substantially over the past three to four decades. Now there is at least one journal devoted exclusively to implementation research ( I m p l e m e n t a t i o n S c i e n c e , h t t p : / / w w w. implementationscience.com); several publication outlets require data on implementation to be included in any trial reports; and verification of implementation has become incorporated into the standards of evidence in some research areas, such as prevention (Gottfredson et al. 2015). Implementation is currently important for all types of interventions (prevention and treatment) for all types of participants (youth and adults) and for all types of services and programming (mental and physical health, education, and social services).
Major Findings in Implementation Research Findings in implementation science have confirmed the fundamental importance of program implementation (Durlak 2015). For example, we now know that it is not evidencebased programs that are effective, but it is well-implemented evidenced-based programs that are effective. Research has consistently shown that the level of implementation that is achieved has an important effect on program outcomes, and this can be manifested in two important ways. First, as a general rule, when it is possible to detect that the same program has varying levels of implementation across settings (e.g., better implemented in school A than in school B), the better
Prev Sci (2015) 16:1123–1127
implemented program usually yields significantly better outcomes than the less well implemented program. Second, poorly implemented programs often have little or no significant positive impact on their participants. The implications of these findings are straightforward. Monitoring implementation and examining how the level of achieved implementation relates to different program outcomes are now essential aspects of all program evaluations. Moreover, research has also confirmed that when they are disseminated into new settings, the implementation of many evidence-based programs is often sub-optimal. For a variety of reasons, high levels of implementation are not achieved. Therefore, knowledge about implementation is crucial to the development, evaluation, and successful dissemination of evidence-based initiatives. If we do not assess implementation, we do not know if a program has been put to an adequate test. It may fail not because the intervention lacks value, but because the intervention was not implemented at a sufficiently high enough level to produce its effects. Assessing implementation might also inform continuous improvement efforts by suggesting how program outcomes could be stronger if implementation was improved. Studying implementation presents several challenges because of additional findings that have appeared (Durlak 2015). Namely, research has identified at least eight different components to implementation (e.g., fidelity, quality, dosage, adaptation, participant engagement, program reach, program differentiation, and monitoring of control or comparison conditions) over 20 contextual factors that influence the level of achieved implementation (e.g., various features of the intervention, front-line providers, and the organization hosting the intervention; Durlak and DuPre 2008) and 14 steps that need to be successfully accomplished to increase the chances of effective implementation (Meyers et al. 2012). A few of these steps involve adequately assessing the need for the program and how well it fits into the local ecology, in addition to delivering effective training and consultation services, and establishing an on-going monitoring system for tracking implementation along the way and making improvements if necessary. Several conceptual frameworks have appeared that attempt to describe the complex relationships that may occur among different variables that can influence the course and final result of implementation (Damschroder and Hagedorn 2011; Domitrovich, et al. 2008; Han and Weiss 2005: Wandersman, et al. 2008). The eight components of implementation, the over 20 contextual factors potentially affecting implementation, and the 14 steps necessary to achieving effective implementation leave a staggering array of possible permutations that could affect any attempt at implementation. Researchers are thus faced with the task of investigating multiple factors that might operate in any situation and influence program outcomes. Further complicating the picture is that it is not yet clear what methods should be used in different circumstances to measure implementation most reliably and validly, and how often such
Prev Sci (2015) 16:1123–1127
measurements should be taken because levels of achieved implementation can vary over time. Finally, there is likely to be a threshold level of implementation at which desired program outcomes are obtained. Going beyond this threshold may not yield any appreciably better benefits for participants. In other words, implementation does not have to be perfect for a program to be effective, but it does have to be Bgoodenough.^ Unfortunately, we do not know the most effective implementation threshold for different evidence-based interventions or whether this threshold varies over time. Ultimately, we need to learn how the level of implementation achieved for different components of implementation affects the types of outcomes demonstrated by participants in different types of programs and what contextual factors facilitate or hinder achieving an effective level of implementation for each intervention.
Articles in This Special Issue Given the current context of research on implementation, the articles in this special issue illustrate useful ways that investigators have approached the daunting but necessary task of understanding implementation. Because there are so many potentially important variables that can affect implementation, researchers have to make decisions about which ones to assess. They cannot study all possibilities at once. The current group of authors has chosen wisely by usually including multiple variables related to characteristics of the school staff, the school system, and the intervention. They have selected these variables based on previous theory and empirical findings, offered a priori hypotheses about what they expected to find, and used sophisticated multivariate statistical techniques in their analyses. Moreover, several of the studies included large data sets; one report evaluated over 1000 teachers from 37 schools who were followed over a 4-year period (Pas et al. 2014). Managing such large longitudinal undertakings is impressive. There are several other notable features of the included studies. One report evaluated which variables were related to the sustainability of an intervention (Lochman et al. 2014). This is an important issue because there is no guarantee that if an intervention is successful initially, that it will be continued. Another study collected data from both intervention and comparison schools (Pas et al. 2014). Monitoring what is occurring in control schools is often overlooked in implementation research but extremely important. It merits careful study because of the fantasy of the untreated control group in school-based studies (Durlak 1995). Schools often offer multiple programs or services to their students, and some or many of these offerings can overlap with some dimensions of the newly introduced program. One survey of American schools found that the typical school offered a median of 14 preventive
1125
activities (Gottfredson and Gottfredson 2001). Activities included in sex education, drug prevention, bullying, or sexual abuse programs can contain elements related to self-management, effective communication, and problem-solving and decision-making skills that are also part of a new evidencebased SEL program. Therefore, it is critical to identify the major elements of new initiatives and compare them to the usual or current practices that are occurring in so-called control schools. If a new program is not distinct enough from what is already occurring in schools, it may not be effective (e.g., Cook et al. 1999). The achieved levels of implementation reported among the current studies are worthy of note. Investigators used different metrics and procedures to assess different implementation components. When these methods could be converted into a percentage, the results are quite revealing. Measures of quality of implementation were fairly high to high in four reports, ranging from 69 to 89 % (Domitrovich, et al. 2015; Malloy et al. 2014; Pas, et al. 2014; Williford et al. 2015). In other cases, implementation was at less than ideal levels. For example, levels of fidelity did not surpass 55 % in two reports (Wanless et al. 2014; Williford et al. 2015). Dosage levels varied from 48 to 53 % in two investigations (Malloy et al. 2014; Williford et al. 2015). In their sustainability study, Lochman and colleagues found that counselors were only using 18 % of the original intervention components 2 years later. The sub-optimal implementation levels achieved in several studies on several components of implementation are not necessarily a negative reflection on the investigators; they indicate how difficult it is to achieve high levels of implementation in field research. In most cases, authors found substantial variability in the different components of implementation that were assessed; some school staff members were much more effective in delivering the program than others. It is important to stress that even slight changes in some situations can make a substantial difference. In one study, a one point difference in implementation scores reflected an 18 % increase in intervention dosage (Domitrovich et al. 2015). Such results underscore the importance of identifying which variables are related to higher levels of implementation so future trials can concentrate on these variables to improve implementation.
Priorities for Future Research In addition to issues that have already been discussed (e.g., the need to develop efficient and reliable measures of different implementation components and determine implementation threshold effects for different interventions), there are at least three more research priorities. These involve the need to (a) assess the influence of adaptations, (b) evaluate the impact of
1126
training, and (c) ascertain which implementation components are associated with which program outcomes. Adaptation refers to changes made by providers that depart from the original program design or operation. Initially, adaptations were seen as detrimental to achieving program success and they were strongly discouraged. However, two findings in implementation research have appeared to change this view. First, adaptations are the rule rather than the exception, particularly in school-based research regardless of the training or instruction program providers receive (Dusenbury, Brannigan, Hansen, Walsh, and Falco 2005; Ringwalt et al. 2003). Teachers invariably make some changes when they implement new programs. Second, adaptations may have positive, negative, or no effect on eventual program outcomes depending on the nature of the adaptations. For example, teachers may decide to change the session length or some features of a new program to fit the regular classroom routine, to heighten the cultural relevance of a program, or in some way to capture more effectively the interest and involvement of their students. Therefore, it is important to document the types of adaptations that occur during implementation and assess their potential influence on different outcomes. This is nicely illustrated by a study of teachers implementing the All Stars drug prevention program in which the researchers used video recordings to categorize the types of adaptations that teachers were making (Hansen et al. 2013). The adaptations that occurred were rated as either potentially positive in nature (those that could enhance program outcomes), negative (those likely to diminish program outcomes), or neutral (those unlikely to have an effect either way). Results indicated that all teachers made some adaptations, and some made many more than others. In fact, teachers averaged nearly six adaptations in each program lesson. Student drug use was the major program outcome. As expected, neutral adaptations had no significant effect on student drug use, whereas positive adaptations had a positive effect, and negative adaptations had a negative effect. Implementation research has verified that pre-program trainings followed by some form of ongoing technical assistance and consultation once the program begins are necessary although not sufficient conditions for achieving high levels of implementation (Durlak and DuPre 2008). However, implementation research has mostly been silent on documenting how training impacts the specific skill levels of potential program providers. Psychosocial interventions are not easy to conduct, and we must not assume that training levels the playing field so that all trainees become equally adept at implementing all aspects of the intervention. Therefore, it is important to document providers’ abilities or skill levels at delivering the program at the beginning and end of training. As Wanless and colleagues note (2014), the more that trainers and coaches understand provider needs, the more they can fine-tune or individualize their training and consultative practices to assist those in more need of attention.
Prev Sci (2015) 16:1123–1127
A third and final priority for future work involves specifying which components of implementation are related to which specific program outcomes. We should not assume that each component is equally important for all possible outcomes. To date, fidelity and dosage have been the most frequently studied; in most cases, higher levels of these components are associated with better results, but not in every situation and for every assessed outcome (Durlak and DuPre 2008). In some circumstances, quality of delivery, participant responsiveness, or certain adaptations may be more strongly related than other implementation components to some program benefits, or, perhaps, be more important for some participant subgroups. For example, some teachers may be able to adapt components of interventions effectively to improve benefits for some ethnic or cultural groups.
Concluding Comments Implementation science has become an essential field of inquiry for understanding what happens when evidence-based programs are brought into new settings. Assessing program implementation is now an important piece of all program evaluations. The studies in this special issue provide models for how to approach the complex world of implementation. The goal is straightforward, but not easy to achieve. How can we maximize the chances that evidence-based programs will be effectively conducted in more settings so that more individuals can potentially benefit? Compliance with Ethical Standards This research does not involve human subjects. Informed consent There is no need for informed consent because this manuscript does not involve research with human subjects. Funding There was no funding for the preparation of this manuscript. Conflict of Interest There author declares that they have no conflict of interest.
References Cook, T. D., Habib, F., Phillips, M., Settersten, R. A., Shagle, S. C., & Degirmencioglu, S. M. (1999). Comer’s school development program in prince George’s county, Maryland: A theory-based evaluation. American Educational Research Journal, 36, 543–597. doi:10. 3102/00028312036003543. Damschroder, L. J., & Hagedorn, H. J. (2011). A guiding framework and approach for implementation research in substance use disorders treatment. Psychology of Addictive Behaviors, 25, 194–205. doi: 10.1037/a0022284. Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K. E., Buckley, J. A., Olin, S., & Ialongo, N. S. (2008). Maximizing the
Prev Sci (2015) 16:1123–1127 implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28. Domitrovich, C. E., Pas, E.T., Bradshaw, C. P., Becker, K., Keperling, J., Embry, D. .Ialongo, N. (2015). Individual and school organizational factors that influence implementation of the PAX Good Behavior Game intervention. Prevention Science. doi: 10.1007/s11121-0150557-8 Durlak, J. A. (1995). School-based prevention programs for children and adolescents. Thousand Oaks: Sage. Durlak, J. A. (2015). What everyone should know about implementation. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 395–405). New York: Guilford. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventivein terventions. Health Education Research, 20, 308–313. Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science. doi:10.1007/ s11121-015-0555-x. Gottfredson, G. D., & Gottfredson, D. C. (2001). What schools do to prevent delinquency and promote safe environments. Journal of Educational and Psychological Consultation, 12, 313–344. Han, S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679. Hansen, W. B., Pankratz, M. M., Dusenbury, L., Giles, S. M., Bishop, D. C., Albritton, J., et al. (2013). Styles of adaptation: The impact of frequency and valence of adaptation on preventing substance use. H e a l t h E d u c a t i o n , 11 3 , 3 4 5 – 3 6 3 . d o i : 1 0 . 1 1 0 8 / 09654281311329268. Lochman, J.E, Powell, N.P., Boxmeyer, C.L., Qu, L., Sallee, M., Wells, K.C., Windle. M. (2014). Counselor-level predictors of sustained
1127 use of an indicated preventive intervention for aggressive children. Prevention Science. doi 10.1007/s11121-014-0511-1. Malloy, M., Acock, A. DuBois, D. L. Vuchinich, S., Silverthorn, N. Ji, P. Flay B. R. (2014). Teachers’ perceptions of school organizational climate as predictors of dosage and quality of implementation of a social-emotional and character development Program. Prevention Science. doi: 10.1007/s11121-0140534-7. Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The quality implementation framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50, 462–480. Pas, E.T., Waasdorp, T.E., Bradshaw, C.P. (2014). Examining contextual Influences on classroom-based implementation of positive behavior support Strategies: Findings from a randomized controlled effectiveness trial. Prevention Science. doi:10.1007/s11121-014-0492-0. Roberts, A.M., LoCasale-Crouch, J., DeCoster, J., Hamre, B.K., Downer, J.T., Williford, A.P., Pianta, R.C. (2014). Individual and contextual factors associated with pre-kindergarten reachers’ Responsiveness to the My Teaching Partner coaching intervention. Prevention Science. doi: 10.1007/s11121-014-0533-8. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press. Ringwalt, C. L., Ennett, S., Johnson, R., Rohrbach, L. A., SimonsRudolph, A., Vincus, A., & Thorne, J. (2003). Factors associated with fidelity to substance use prevention curriculum guides in the nation’s middle schools. Health Education & Behavior, 30, 375–391. Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., & Saul, J. (2008). Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41, 171–181. Wanless, S.B., Rimm-Kaufman, S.E., Abry, T., Larsen, R.A., Patton, C.L. (2014). Engagement in training as a mechanism to understanding fidelity of Implementation of the Responsive Classroom approach. Prevention Science. doi: 10.1007/s11121-014-0519-6. Williford, A.P., Sanger Wolcott, C., Vick Whittaker. J., Locasale-Crouch, J. (2015). Program and teacher characteristics predicting the implementation of Banking Time with preschoolers who display disruptive behaviors. Prevention Science. doi: 10.1007/s11121-015-0544-0.
Copyright of Prevention Science is the property of Springer Science & Business Media B.V. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
