Violence Against Women

6 downloads 0 Views 394KB Size Report
Mar 31, 2011 - Shabnam Javdani, Nicole E. Allen, Nathan R. Todd and Carolyn J. .... councils examined in this study (Allen, Todd, Anderson, Davis, & Javdani,.
Violence Against Women http://vaw.sagepub.com/

Examining Systems Change in the Response to Domestic Violence: Innovative Applications of Multilevel Modeling Shabnam Javdani, Nicole E. Allen, Nathan R. Todd and Carolyn J. Anderson Violence Against Women 2011 17: 359 DOI: 10.1177/1077801211398621 The online version of this article can be found at: http://vaw.sagepub.com/content/17/3/359

Published by: http://www.sagepublications.com

Additional services and information for Violence Against Women can be found at: Email Alerts: http://vaw.sagepub.com/cgi/alerts Subscriptions: http://vaw.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations: http://vaw.sagepub.com/content/17/3/359.refs.html

>> Version of Record - Mar 31, 2011 What is This?

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

VAW3986 21 VAW17310.1177/1077801211398621

Article

Examining Systems Change in the Response to Domestic Violence: Innovative Applications of Multilevel Modeling

Violence Against Women  17(3) 359­–375 © The Author(s) 2011 Reprints and permission: http://www. sagepub.com/journalsPermissions.nav DOI: 10.1177/1077801211398621 http://vaw.sagepub.com

Shabnam Javdani1, Nicole E. Allen1, Nathan R. Todd2, and Carolyn J. Anderson1

Abstract Facilitating systems change in the response to domestic violence has been touted as a central goal in the effort to hold systems accountable and create a coordinated response for survivors. However, examination of systems change and whether particular social change efforts (e.g., coordinating councils) contribute to such change is a notoriously difficult research endeavor due in large part to methodological barriers, including those that stem from nonexperimental designs and complex data that are characterized as nested and measured in proportions. This article describes important methodological challenges and proposes innovative techniques to address these challenges. Specifically, multilevel modeling is applied to examine two key systems markers, including protection order and domestic violence program referral rates over time in one state. For each marker, the methodological approach is highlighted and innovations in employing multilevel modeling are discussed. Keywords collaboration, coordinated community response, coordinating council, domestic violence shelter/program, multilevel modeling, systems change

Systems change has been broadly defined as “efforts that strive to shift the underlying infrastructure within a community or targeted context to support a desired outcome, including shifting existing policies and practices, resource allocations, [and] relationship structures” 1

University of Illinois at Urbana-Champaign, Champaign DePaul University, Chicago, IL

2

Corresponding Author: Shabnam Javdani, University of Illinois at Urbana-Champaign, Department of Psychology, 603 E. Daniel Street, Champaign, IL 61820 Email: [email protected]

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

360

Violence Against Women 17(3)

(Foster-Fishman & Behrens, 2007, p. 192). Examination of systems change is an important goal of research on domestic violence, particularly because it presents an opportunity to maintain a systems focus and to understand the effects of social change interventions that aim to affect change at this level. In these investigations, the spotlight is often placed on understanding the function and patterns of the systems that comprise the response to the occurrence of domestic violence and on leveraging changes in those systems, both to improve the current response and to decrease violence in the future (e.g., Shepard & Pence, 1999). Coordinating councils are one common vehicle through which communities attempt to affect change in the response to domestic violence by bringing relevant stakeholders together to encourage interagency coordination and systems change. The current illustration focuses on two important systems change markers and details the associated methodological challenges of examining change and whether such changes can be attributed to the formation and development of coordinating councils.

Protection Orders Prominent systems that respond to domestic violence include criminal justice and the courts, domestic violence shelters, and human services. Because systems are complex and are comprised of multiple interrelated parts, specific markers are often examined as indices for systems change. In research on domestic violence, the “usual suspects” indexing the system’s response to survivors and batterers have often been markers of the criminal justice and court systems. Indeed, many scholars agree that there are important gaps in the criminal justice system’s response (e.g., Buzawa, 2002), including ambiguities in the extent to which state and federal policy and guidelines are implemented locally and translated into shifts in practice (Diviney, Parekh, & Olson, 2009). One meaningful hallmark of the criminal justice response to examine with respect to implementation is that of orders of protection, as they can meaningfully translate to increasing survivor safety and batterer accountability (see Buzawa, 2002; Buzawa & Buzawa, 1996). Notably, evidence for the effectiveness of protection orders is equivocal, with some studies linking permanent protection orders with increased survivor safety (Holt, Kernic, Lumley, Wolf, & Rivara, 2002), especially when they are adequately enforced (Harrell & Smith, 1996). Other studies, however, find that protection orders may not adequately protect survivors from reabuse (Klein, 1996). Despite such equivocal findings, access to protection orders remains a common reform effort at state and local levels. Thus the accessibility of protection orders is an important index of systems change and innovative methods will allow for an examination of such access over time. Specifically, shifts in the proportion of emergency protection orders that become plenary (or longer term) orders is an important outcome because such shifts may reflect increased system coordination, facilitating survivors’ continued engagement with the criminal justice system and access to legal advocacy resources. Throughout the article, we will refer to the “protection order rate” as this proportion of emergency orders that become plenary orders.

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

361

Javdani et al.

Domestic Violence Agency Referrals A second marker important to understanding systems change in the response to domestic violence is the rate of referral from key agencies to domestic violence programs. This marker captures potential cross-system collaboration because it documents the extent to which one system (e.g., law enforcement) communicates with and refers to another system (e.g., shelters). Such cross-agency referrals are one form of interagency coordination, a common component of change efforts. Attention to the extent to which other systems are coordinating with domestic violence programs, including shelters, is centrally important, as such programs have historically been at the core of domestic violence movements and continue to provide a multitude of services (e.g., health care, legal services) to battered women and their children. Furthermore, criminal justice agencies are an important source of referrals because survivors may come into contact with the criminal justice system before they are linked to any other sources of assistance.

Coordinating Councils In addition to understanding whether and to what extent systems change has occurred over time, another central goal of this article is to describe and employ innovative methodology, multilevel modeling (MLM), to examine whether systems changes can be attributed to particular social change efforts. Coordinating councils are one prominent social change vehicle charged with facilitating systems change in local communities or regions (e.g., Allen, 2005, 2006). Ideally, in these councils interagency coordination is promoted by bringing together multiple stakeholder groups (e.g., law enforcement, the judiciary, domestic violence advocates, faith-based and human service providers) and engaging them in shared reform efforts. The coordinating councils examined in this study were established through an initiative of the state court system with the primary goal of improving the institutional response to domestic violence. Each of 21 coordinating councils corresponds to a judicial circuit and was formed at different points in time. Thus a second goal of the current article is to highlight how this innovative methodology helps us understand whether potential observed patterns in criminal justice and domestic violence program referral markers can be attributed to the work of a social intervention stemming from state policy—coordinating councils. Notably, institutionalized changes including increasing the accessibility of protection orders and criminal justice system referrals to domestic violence agencies were important goals of councils examined in this study (Allen, Todd, Anderson, Davis, & Javdani, 2011). Thus any observed changes in protection order and domestic violence referral rates would closely reflect the focus of council efforts.

Overview In this study, our primary substantive goals were to investigate shifts in key systems change indicators—namely, protection order and shelter referral rates—over time. Furthermore,

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

362

Violence Against Women 17(3)

we examine whether such systems changes could be attributed to the work of coordinating councils, a social change effort implemented largely to leverage systems changes. The data used to inform these questions were archival data from the state that documented annual protection order rates (i.e., the proportion of issuing plenary versus emergency protection orders) and annual shelter referral rates (i.e., the proportion of referrals to shelters that came from criminal justice agencies). Our primary methodological goals, therefore, were to (a) model rate (i.e., proportion) data over time and to (b) compare different communities (i.e., as indexed by our 21 councils) to discover whether coordinating councils influenced the systems changes observed. In the next sections, we detail particular methodological challenges inherent in our approach and, later, discuss methodological innovations that helped to address each challenge.

Methodological Challenges Despite its importance, examination of systems change is notoriously difficult due to the inherent complexity and dynamic nature of systems, with the methods used to examine systems and their markers notably less developed (Hirsch, Levin, & Miller, 2007). Two key methodological challenges in the investigation of systems change arise because (a) data need to be examined via nonexperimental designs because experimental designs are likely implausible (if not impossible), and (b) the nature of data are complex given that (i) observations are “nested” or clustered within other broad categories (e.g., individuals live in communities) and (ii) outcome measures are rates or proportions.

Nonexperimental Designs Are Common Establishing whether systems change has occurred and whether it can be attributed to a particular effort (e.g., coordinating councils) often begs for an experimental design from a methodological standpoint. Yet classic experimental designs with “control” group comparisons are often not viable because communities cannot usually be randomly assigned to intervention/nonintervention groups. Thus comparison communities are often chosen to approximate “controls.” However, employing comparison control communities is also problematic because it introduces multiple sources of “error” that may actually be an important part of the systems change “story.” For example, in a recent study, Visher, Harrell, Newmark, and Yahner (2008) compared three communities to examine the effectiveness of the Judicial Oversight Demonstration (JOD) Project. This was a laudable effort but also resulted in many complexities regarding interpretations of differences across communities. Absent true experimental controls, it becomes exceedingly difficult to attribute observed differences or similarities to particular community efforts because “preexisting differences between JOD and comparison sites—not the JOD intervention—might account for differences in outcomes” and statistical controls employed in modeling “cannot control for unobserved [italics added] differences” in outcomes (p. 518). Furthermore, efforts to reform the systems response to domestic violence have been so widespread, there may be no true comparison communities because a vast majority of comparable communities have some

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

363

Javdani et al.

coordinated activity and/or have a local change effort (e.g., in response to new federal and state mandates). An alternative quasi-experimental approach is to use longitudinal designs to examine within-community change over time. In this approach, the control group is the group itself but at an earlier point in time. Typically, this approach has important limitations, especially with regard to the introduction of historical effects that occur outside of the intervention being examined, and which hinder the capacity to clearly attribute change to a particular effort. Longitudinal designs are also challenging given that researchers have often not had the opportunity to establish a baseline prior to systems change work being underway. For this reason, using existing archives gathered as part of routine practice within systems can be valuable because this establishes how the system functioned (e.g., with regard to protection order rates) prior to a given community initiative being implemented.

Complexity of Data Clustered or nested data. Systems change markers are often multilevel or “nested” within other meaningful categories. Multilevel or nested data are characterized by multiple sources of meaningful (i.e., nonerror) variation. Multilevel data arise in a number of different contexts. For instance, individuals may be embedded in larger settings and the measurements on individuals within a setting are a cluster of data. This type of clustering or grouping is perhaps the most commonly employed and has demonstrated utility across disciplines (e.g., in educational research, students nested within classrooms; also see Todd, Allen, & Javdani, 2010). Figure 1(a) illustrates this type of clustering where the top row of boxes represents (in this case) councils and the bottom row represents individuals (or members) within each council. The top row is referred to as Level 2 units and the bottom row as Level 1 units. We may have multiple measurements on each person (i.e., Level 1 measures) and on the councils themselves (i.e. Level 2 measures). Last, the number of individuals in each council need not be the same, and in Figure 1(a), the number of people in each council j equals nj. Another type of cluster can occur in the context of longitudinal investigations, when researchers are interested in understanding phenomena over time (Singer & Willett, 2003; Snijders & Bosker, 1999). This instance is illustrated in Figure 1(b) where time points are nested within a person or individual (also see Raudebush & Bryk, 2002). In Figure 1(b) time points are labeled generically as “time”, because there is much flexibility in how time is measured. The variable to represent “time” (e.g., age, grade in school, chronological) and the metric for time (e.g., minutes, months, years) can be chosen to best meet the needs of the investigation. The number of time points and their spacing need not be the same for all individuals. Another instance of longitudinal data is given in Figure 1(c) where “person” in Figure 1(b) has been replaced by “circuit” (for a given judicial circuit within the state), such that the investigation focuses on change over time for a given circuit. In all of these scenarios, data are “multilevel” and the variation observed on measures at Level 1 (e.g., time points in Figures 1b and 1c) is dependent on the variation observed at Level 2 (e.g., circuit coordinating councils). In other words, observed changes over time may be dependent on differences between coordinating councils. When data are nested,

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

364

Violence Against Women 17(3)

Figure 1. Examples of nested or clustered data

the practices within one setting tend to be more similar to each other than to such practices in other settings because of the shared features of a given setting (e.g., the judges of a particular judicial circuit may vary in their judicial practices even subject to the same laws). In the current example, the protection order and referral rate data for each of 21 judicial circuits were examined over a period of 15 years with attention to annual rates where each annual measurement observation was nested within judicial circuits (i.e., Figure 1c). We were particularly interested in whether there were such meaningful and systematic differences between circuits. However, nested data violate critical statistical assumptions of many mainstream analyses because data within clusters are not independent. Failing to account for dependencies in data invalidates statistical significance test results and coverage rates of confidence intervals of estimates. Violations of independence generally lead to inflated Type I error rates because estimates of standard errors for para­ meters are too small (Raudenbush & Bryk, 2002; Snijders & Boskers, 1999). This presents a challenge for research using markers of systems change. Yet the structure of these data also provides an opportunity. In the next section, we describe how multilevel modeling (MLM) provides a flexible and elegant innovation to address the challenges of nested data structures.

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

365

Javdani et al.

Outcomes are rates or proportions. Another challenge resulting from the complex nature of data is that systems change outcomes in domestic violence research are often proportions, rates, or other types of nonnormally distributed variables (Anderson, Verkuilen, & Johnson, 2010). Proportions and rates arise when the outcome measure is dichotomous. The natural distribution of dichotomous variables is the binomial distribution and the logical modeling approach is logistic regression. A binomial random variable is the sum of independent occurrences of one of the categories of the dichotomous variable (i.e., the number of orders of protection). In our case, the number of extensions of emergency protection orders divided by the total number of emergency orders equals the proportion of plenary/emergency orders in a circuit at a specific time point. This proportion estimates the probability of an emergency order becoming a plenary order in that circuit at that time point. These protection order proportions cannot be accurately modeled using Ordinary Least Squares Regression, because proportions are not normally distributed as assumed in ordinary regression. Furthermore, ordinary regression can result in inadmissible estimates of probabilities (i.e., probability estimates less than 0 or greater than 1; Agresti, 2002; Anderson et al., 2010; Kutner, Nachtsheim, Neter, & Li, 2005). The natural choice of a modeling approach is logistic regression; however, we must also take into account that the number (or proportion) of protection orders is measured over time (i.e., each rate is “nested” in a give year). Advancing important questions about the systems response to domestic violence requires addressing significant methodological challenges as described below. Generalized linear mixed models, of which multilevel logistic regression models are special cases, meet these demands.

Methodological Innovations in Multilevel Modeling Because of the challenges involving nonexperimental designs and data complexity, the use of more complex statistical models is often required. Innovative use of MLM can address each of the challenges reviewed above in meaningful ways. The most well-known MLM is Hierarchical Linear Modeling (HLM). HLMs have been used to assess patterns in data that have a hierarchical structure including multiple levels of analysis and time (Raudenbush & Byrk, 2002). However, because HLMs only apply to normally distributed outcome measures, we require another version of an MLM, in particular multilevel logistic regression. A logistic regression model where protective order rate is the response variable and time is an explanatory variable is assumed to hold for each council. This logistic regression model is the “Level 1 model.” We are also interested in differences between councils, which are captured by allowing the parameters of these logistic regression models to differ. The parameters are assumed to follow a normal distribution and normal linear regression may be proposed for these regression parameters. These are the “Level 2 models.” In this section, we elaborate on how MLM can be applied to address challenges associated with nonexperimental designs examining nested, longitudinal data.

Employing Powerful Quasi-Experimental Designs One of the primary challenges with examining systems change lies in our lack of ability to employ “true” experimental designs. One way to address this challenge is to construct a

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

366

Violence Against Women 17(3)

creative quasi-experimental design within an MLM framework. While the use of quasiexperimental designs is not without limitations, coupling this approach with MLM allows for a focused look within given communities (i.e., the change in rates observed in a judicial circuit) and does not assume complete “equivalence” or similarity across communities. At the same time, this design allows for examination of multiple communities simultaneously to look for trends in change over time due to a particular intervention (in this case councils). This strengthens our ability to draw conclusions about the associations between a given community effort and systems change. Specifically, MLM provides a conceptual framework to examine change over time, by specific nested group (e.g., judicial circuit). Two main types of MLM analyses are conducted toward this end. First, our data are grouped according to each cluster (e.g., judicial circuit) to examine the descriptive change over (chronological) time for our outcome of interest (e.g., protection order rates). Doing so allows us to observe if and how change occurs (i.e., does the rate increase overall?). Second, MLM allows us to treat time creatively by examining not only historical time (annual rates) but also council age (i.e., how long a given council has been in place).

Allowing for Complex Data Accounting for variance at multiple levels. MLM is well suited to analyzing nested data because it accounts for the dependence attributed to different levels of analysis in the data (Raudenbush & Byrk, 2002; Snijders & Bosker, 1999). In fact, MLM directly models and disaggregates the effects of different levels (e.g., within circuit change via a logistic regression model and between circuit differences in protection order rates via a normal linear regression model). This approach captures within circuit dependency and yields better estimates of standard errors for parameters. Better standard errors in turn lead to valid statistical inferences. In our data and illustration for this article, we focus on longitudinal data, where we have measurement occasions of yearly protection order rates (Level 1) nested within 21 different judicial circuits (Level 2). This case is illustrated in Figure 1(c). Although these circuits are geographically bound within the same state, we only have a two-level model and are examining how time is nested within these circuits. We can then use MLM to examine changes in protection order rates within each circuit across time and patterns of between-circuit variation (i.e., whether all circuits have the same average rate). Finally, we can use MLM to see how the presence of coordinating councils may explain shifts in our systems change markers. That is, MLM directly assesses within- and between-circuit variation in the rate of change over time for both protection orders and referrals to domestic violence programs. This longitudinal application of MLM allows us to examine research questions regarding shifts in systems change markers over time and more specifically shifts in systems change markers by circuit. Examining rate data using generalized linear mixed models. MLM can also be used to examine rates, or proportions of emergency/plenary orders of protection, across time by employing generalized linear mixed models (GLMMs; Anderson et al., 2010; Molenberghs & Verbeke,

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

367

Javdani et al.

2005; Singer & Willett, 2003). In this model, the probability of occurrence of an outcome (e.g., that a protection order is granted or a referral is made) can be predicted from a number of independent variables using maximum likelihood estimation. For our purposes, this type of modeling is a helpful tool to understand how rates change across time (e.g., the annual protection order rate for a given circuit) and how rates may be different in different circuits. GLMMs then allow us to model random intercept and slopes, where a random intercept is necessary when circuits have different rates at a certain point in time. A random slope is necessary when the trajectory of change for circuits is different (e.g., protection orders in one circuit increase over time while they decrease in another circuit). We use this flexible modeling strategy to creatively examine the influence of a systems change initiative on relevant outcomes.

Applications of Multilvel Modeling Protection Order Rates Approach. We were interested in examining the extent to which the ratio of issuing plenary versus emergency protection orders (herein referred to as protection order rates) has shifted over time, and whether such shifts could be attributed to the work of coordinating councils. Given the nature of our methodological design (i.e., no true comparison communities) and the complexity of the data (i.e., nested, rate data), we used MLM to examine these questions using archives across 15 years (e.g., see Figure 2a). Specifically, we used a fourfold strategy to disentangle historic trends from the possible contribution of coordinating councils on the protection order ratio. In these models, we were looking for possible historical effects. This series of four analyses is illustrated in Figure 2 for three hypothetical circuits A, B, and C. These three councils illustrate the analyses conducted and reflect the pattern of our actual findings. Circuits A, B, and C differ in terms of when a council was created and the overall level of proportions of orders of protection. The circles represent time points prior to the existence of a council and the dots represent time points after a council was in place. First, we tested a series of models to assess the change of protection order rates across historic time. Note that there appears to be an overall upward trend in Figure 2(a). In the next two analyses, subsets of data were analyzed, including data from before council formation and data from after council formation. These steps were taken to allow us to disentangle whether this overall upward trend occurs before versus after council formation. Second, we inspected a series of models to assess the change of protection order rates over time for the period prior to council formation. The before council formation model is illustrated in Figure 2(b). Third, we examined a series of models to assess the change of protection order rates over time for the period following council formation. In this step, we still use chronological year to examine whether council formation has an effect on the protective order rate pattern. This is illustrated in Figure 2(c). In combination, these analyses allow us to draw conclusions about whether changes in protection order rates are due to the presence or absence of councils, while accounting for historical effects. Thus we can begin to untangle how our phenomenon of interest changes across time and also consider

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

368

Violence Against Women 17(3)

Figure 2. Illustration of four-stage multilevel modeling strategy for three hypothetical councils (A, B, and C).

the potential influence of council formation. However, these models do not take into account the age of councils that might also affect the extent to which protective order rates experience a shift over time. To address this, our fourth step tests a series of integrated models to assess protection order ratio change including both council age and the presence/absence of a council. In this last model, illustrated in Figure 2(d), age of council is used as our measure of time irrespective of chronological year of council formation (e.g., councils formed in different years may both be 2 “years old”). In addition, we add a second Level 1 variable to the model: a dichotomous variable to account for the variation prior to council formation (no council; coded “0” if a council exists, and “1” if no council exists).1

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

369

Javdani et al.

Taken together, this four-stage modeling strategy allowed us to examine if and how protection order rates changed over historic time as well as to investigate the impact of council formation and age.2 Throughout this process, we also examined descriptive graphs to aid in interpretation (e.g., those like Figure 2), and found that much of the “story of the data” could be told by examining these graphs. Specifically, separate graphs were generated for each judicial circuit, marking protection order patterns over time. In each graph, a mark was made indicating the year when the council in that circuit was formed. This approach allowed us to examine each circuit’s individual systems change pattern (i.e., did protection order rates increase over time?), and visually compare the pattern of change before and after council formation in that circuit (i.e., does the pattern shift after the formation of the council?). Findings and interpretations. As described in Allen et al. (2011), a consistent story

emerged from the MLM process suggesting that councils had a modest positive influence on the rate of emergency orders that moved to plenary orders of protection. First, we found that there was an increase in the protection order rate over time. However, it was unclear whether this pattern was due to general historical effects or if it was affected by the presence of councils. For a more exact understanding, we examined change across historic time for locations during periods of time when councils were not present versus when councils were present. Through examination of graphs of the data and the use of MLM, we found no change in the rates across time in geographic regions that did not have a council present; however, there was an increase across time in geographic regions that did have a council present. This design, and comparison of change over time for locations with and without councils, shows that the presence of a council augmented the transition to plenary orders. Second, we examined an integrated model that conceptualized time in terms of council age, or how many years the coordinating council had been in existence. As described in Allen et al. (2011), this model allowed for the simultaneous estimation of the linear effect for the age of the council (e.g., how the rate changed as the council grew older) as well as the effect for the presence/absence of a council within a circuit. Using multilevel logistic regression to examine this pattern of effects, we found a linear effect (on the logit scale) for council age, indicating that as the council grew older, the protection order rate increased. Furthermore, we found that the odds of an order moving from emergency to plenary were significantly greater in circuits that had councils, compared to circuits without councils. In summary, this model indicates that the presence of a council is important (e.g., greater odds of an emergency order becoming plenary) and that the longer a council is present in a circuit predicts a higher protection order rate. Thus MLM allowed for a more exact understanding of if and how the presence of community coordinating councils, a systems level intervention, influenced a system change outcome (i.e., the accessibility of orders of protection) in the response to domestic violence. Challenges. Although multilevel logistic regression addressed important systems change questions, we faced challenges and recognize potential for future work. First, although we demonstrated patterns of protection order rate change, we did not have enough power to use the other variables included in our design to explain this change because, as is often the

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

370

Violence Against Women 17(3)

case, we had a fixed and relatively small number of Level 2 units (i.e., 21 judicial circuits). With larger numbers of Level 2 units (e.g., circuits), MLM could incorporate other variables to explain what accounts for observed changes (e.g., modeling to explain why the slopes are different across judicial circuits). In addition, although the current design improves on the study of change over time in a single circuit for a single council, it is still difficult to make firm causal statements regarding the influence of councils given the possible presence of historic or other factors not accounted for methodologically. This highlights that a statistical technique is limited by methodological factors such as design and sampling, and interpretations need to be carefully constructed to reflect the totality of the strengths and limitations of the study.

Domestic Violence Program Referral Rates Approach. Using the same four-stage strategy as that described for protection order rates, we also examined the extent to which changes in the proportion of domestic violence program referrals from criminal justice agencies (i.e., law enforcement, state’s attorneys, circuit clerks, and the legal system) occurred from 1998 to 2008, and whether potential changes could be attributed to the work of coordinating councils. Archival data were acquired from the Information Network database,3 which is informed by victim service providers in the State of Illinois (InfoNet Manual; also see Grossman, Lundy, & Beniston, 2007 and Allen et al., 2011). Using these data, we examined the proportion of referrals from criminal justice agencies to domestic violence programs (i.e., criminal justice referrals/total number of referrals across all sources). Findings and interpretations. Results provided support for a random intercept, meaning that different judicial circuits were characterized by different referral rates over time.4 However, in the random intercept model, we did not find evidence for time as a potential linear effect (in the logit scale), indicating that, on average, domestic violence referral rates did not evidence appreciable changes over time during the period for which we had data.5 Next, we examined the possibility that time demonstrated nonlinear trends (e.g., quadratic, cubic), but also found no supporting evidence. Finally, we examined graphs for each of our judicial circuits, as well as graphs averaging effects across circuits (i.e., for the state as a whole), and found that this visual evidence corroborated what we were finding statistically: While criminal justice referrals significantly vary between circuits (i.e., in their intercepts), they do not change over time. Because of this null finding, we did not pursue questions about the effects of councils on the trajectory of referrals. Challenges. Given that increasing cross-agency collaboration is a major goal of councils, these findings lead to some speculation around the reasons why no observable changes in domestic violence program referrals were found, especially given evidence for such changes in protection order rates. Indeed, our null findings may underscore a much broader challenge important for systems change in general: that of “missing the baseline” because the starting year in these analyses is 1998. That is, it is possible that criminal justice referral ratios indeed see a significant increase over time (and councils may have affected this increase). However, these data would not index this pattern if criminal justice referrals began to increase prior

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

371

Javdani et al.

to 1998. In fact, there may be limited capacity to empirically document such shifts in this state, given unreliability of these referral data prior to 1998 (e.g., Grossman et al., 2007). Furthermore, most councils in our study were formed after this “baseline,” suggesting that many local council efforts were not yet fully underway prior to the increase in collaboration that may have occurred between criminal justice and domestic violence programs prior to 1998.

The Big Picture: Bringing Methods into Alignment With Conceptual Aims Though systems patterns are notoriously difficult to study, using MLM invites creative methodological approaches to examine important questions. In this article, we have highlighted critical methodological challenges posed by data informing systems change, including those that arise because of the implausibility of experimental designs, and the complexity inherent in systems change data, particularly when data are nested and when rates are the outcome of interest. Importantly, an examination of protection order and domestic violence program referral rates was made possible through MLM techniques that helped account for the multilevel structure, quasi-experimental longitudinal design (time), and complexity (nested and rate nature) of these data. Due to use of this modeling, our results inform the extent to which a social change effort—the formation of local coordinating councils—had an influence on the systems response to domestic violence. Substantively, we find partial evidence for such an influence, where meaningful changes in protection order rates were associated with the formation of coordinating councils, while no evidence of such shifts in domestic violence program referral rates emerged. The MLM approach presented here provides a way to conceptualize systems-level interventions within the framework and confines of statistical designs, while simultaneously honoring the complexity of our data in the context of the natural environment in which they occur. This approach may be in greater alignment with the conceptual aims of research on systems change in the response to domestic violence than comparing small numbers of communities that are unlikely to vary on only the dimension of interest (e.g., the presence or absence of a domestic violence council). Focusing on multiple data points, over time, across multiple communities allows us to better understand change in critical systems markers without assuming or imposing the equivalence required when we focus on singe comparison communities. Furthermore, with sufficient power (a larger number of communities) we can also explore meaningful variation in the trajectory and quality of the systems response to domestic violence (e.g., the fidelity of implementation of a given intervention, the length of time coordinated efforts have been in place, the intensity of resources available to survivors). This approach retains the complexity of understanding the systems response to domestic violence, which never involves just a single change, but simultaneous changes across multiple organizations and, indeed, systems. In our examination of councils, we advance an interesting metaframework for understanding systems change across time. Despite the methodological challenges inherent in all of the data we presented, our modeling approach created a powerful quasi-experimental

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

372

Violence Against Women 17(3)

design within which nested and rate data were modeled over time. Specifically, due to the overlap between council age and historic time (i.e., as historic time increases, so does council age), we needed a strategy that would allow us to examine council influence without confounding this association with effects attributed to historic time (i.e., history or maturation confounds). To address this, our metaframework employed a strategy that first examined patterns of systems change markers over time separately for instances when councils were present and compared this pattern with instances when councils were not present. Though we did not have explicit “comparison communities,” each of our 21 councils varied in age, meaning that they were formed at separate historical moments. Thus, for every circuit, we had systems marker data both “before” and “after” the formation of that circuit’s coordinating councils, providing the basis for multiple within and across community comparisons. In this way, our approach addresses this classic historical confound to some extent by “mixing” historic time in a “pre” and “post” council formation framework (councils were created in different years, so historic time is varied from one council to the next). Our integrated model also directly assessed the effect of council age; we collapsed council age across historic time and compared systems change patterns before and after council formation with a single integrated dataset. It is important to note that this quasi-experimental design is not without limitations. Indeed, use of this creative design does not avoid all threats to internal validity because it remains susceptible to selection-related threats that arise when groups are not randomly assigned to (in this case) communities. For instance, we cannot fully rule out the possibility that there were preexisting differences between communities that formed councils earlier versus communities that formed councils later. Moreover, it could be the presence of such preexisting differences, rather than the councils themselves, that produced observed changes in protection order rates. Despite these threats, our use of multiple communities compared before and after council formation provides a way to distill and replicate the observed changes in protection order rates. Furthermore, our study underscores the use of multiple methodological tools to help “tell the story” portrayed by the data. Specifically, using graphs helped to visually piece together a coherent story about systems change patterns across the state. In combination, this graphical evidence largely supported our modeling conclusions showing that protection order rates evidenced an increase over time, whereas domestic violence program referral rates did not. These graphs also lend credit to our conclusion that council formation was attributable to the linear increase in protection order rates, as multiple circuits demonstrated no linear pattern in protection order rate prior to council formation while they showed a positive linear pattern after council formation (see Allen et al., 2011); such trends were not observed for domestic violence program referral rates. These visual data coupled with other sources (e.g., qualitative data) would further augment our understanding of the systems change “story” being told by each community coordinating council. Beyond the methodological innovations, understanding creative ways to examine systems changes represents an important agenda for research on domestic violence precisely because it centralizes the systems’ response to violence. Empirical examinations of systems markers allow for a better understanding of the community response to domestic violence and place

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

373

Javdani et al.

the focus of the problem at an ecological level as a complement to studies of individual women, men, and families that are affected. As we advance methodological approaches that examine the systems level, we retain a focus on the critical shifts that illustrate whether and the degree to which communities advance a cutting edge response to domestic violence.

Acknowledgments The authors would like to thank Jennifer Hiselman for her invaluable assistance with data analysis and access. They would also like to thank Vernie Beorkrem, Heather Dorsey, Sally Foster, and council coordinators and members for helping make this study possible.

Authors’ Note The content of this article reflects the views of the authors and not necessarily those of the funding agency.

Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the authorship and/or publication of this article.

Funding The authors disclosed that they received the following support for their research and/or authorship of this article: This project was supported by funding from the National Institute of Justice (NIJ) Grant Award No. 2005-WG-BX-0005.

Notes 1. Thus, when a council is present, “nocouncil” drops out of the model but the “age” variable remains in the model. In addition, a random effect is included for the “nocouncil” variable and, in effect, this random component for precouncil is only part of the model when no council is present (because when a council is present, this variable is 0 and drops out of the model along with the associated random effect). 2. This modeling can be implemented in statistical software packages such as SAS 9.2, using the glimmix procedure. 3. This is a system maintained by the Illinois Criminal Justice Information Authority, the Illinois Coalition Against Sexual Assault (ICASA), and the Illinois Coalition Against Domestic Violence (ICADV). 4. Criminal justice referrals comprise the number one source of domestic violence program referrals in our sample across judicial circuits (M = 47.0% of domestic violence program referrals come from criminal justice, SD = 370 referrals). Though the proportion of referrals from criminal justice agencies remains relatively stable across years (range = 45.2%-51.3%), this rate is variable across the 21 judicial circuits, with a range of proportions between 20% and 60%. 5. To draw these conclusions, we used a host of fit indices, including P-AIC, P-AICC, P-BIC, P-CAIC, and P-HQIC. Furthermore, we calculated a chi-square statistic, which did not drop in magnitude on introduction of time as a linear effect, providing further support for our conclusion.

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

374

Violence Against Women 17(3)

References Agresti A. (2002). Categorical Data Analysis (2nd ed.). New York: Wiley. Allen, N.E. (2005). A multi-level analysis of community coordinating council. American Journal of Community Psychology, 35, 49-63. Allen, N.E. (2006). An examination of the effectiveness of domestic violence coordinating councils. Violence Against Women, 12, 46-67. Allen, N. E., Todd, N. R., Anderson, C. J., Davis, S., & Javdani, S. (2011). Council-based approaches to intimate partner violence: Evidence of distal change in the systems response. Unpublished manuscript. Anderson, C. J., Verkuilen, J., & Johnson, T. (2010). Applied generalized linear mixed models: Continuous and discrete data. New York: Springer. Buzawa, C. G. (2002). Domestic violence: The criminal justice response. Thousand Oaks, CA: SAGE. Buzawa, E. S., & Buzawa, C. G. (Eds.). (1996). Do arrest and restraining orders work? Thousand Oaks, CA: SAGE. Diviney, C. L., Parekh, A., & Olson, L. M. (2009). Outcomes of civil protective orders. Journal of Interpersonal Violence, 24, 1209-1221. Foster-Fishman, P. G., & Behrens, T. R. (2007). Systems change reborn: Rethinking our theories, methods, and efforts in human services reform and community-based change. American Journal of Community Psychology, 39, 191-196. Grossman, S. F., Lundy, M., & Beniston, M. (2007). Analysis of InfoNet data from domestic violence agencies, January 1998 through December 11, 2006. Chicago: Illinois Criminal Justice Information Authority. Harrell, A., & Smith, B. E. (1996). Effects of restraining orders on domestic violence victims. In E. S. Buzawa & C. G. Buzawa (Eds.), Do arrest and restraining orders work? (pp. 214-242). Thousand Oaks, CA: SAGE. Hirsch, G. B., Levine, G., & Miller, R. L. (2007). Using systems dynamics modeling to understand the impact of social changes initiatives. American Journal of Community Psychology, 39, 239-253. Holt, V. L., Kernic, M. A., Lumley, T., Wolf, M. E., & Rivara, F. P. (2002). Civil protection orders and risk of subsequent police-reported violence. Journal of the American Medical Association, 288, 589-594. Information Network Manual. Retrieved from http://www.icjia.state.il.us/infonets/public/PDF/ DVInfonetManual.pdf Klein, A. R. (1996). Re-abuse in a population of court-restrained male batterers: Why restraining orders don’t work. In E. S. Buzawa & C. G. Buzawa (Eds.), Do arrest and restraining orders work? (pp. 192-213). Thousand Oaks, CA: SAGE. Kutner, M. H., Nachtsheim, C. J., Neter, J., & Li, W. (2005). Applied linear statistical models (5th ed.). Boston: McGraw-Hill. Molenberghs, G. and Verbeke, G. (2005). Models for Discrete Longitudinal Data. New York: Springer. Raudenbush, S. W., & Byrk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: SAGE.

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014

375

Javdani et al.

Shepard, M. F., & Pence, E. L. (1999). Coordinated community response to domestic violence: Lessons from Duluth and beyond. Thousand Oaks, CA: SAGE. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change andevent occurrence. New York: Oxford University Press. Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advancedmultilevel modeling. Thousand Oaks, CA: SAGE. Todd, N. R., Allen, N. E., & Javdani, S. (2010). Multilevel modeling: Method and application in community psychology. Unpublished manuscript. Visher, C. A., Harrell, A., Newmark, L., & Yahner, J. (2008). Reducing intimate partner violence: An evaluation of a comprehensive justice system-community collaboration. Criminology & Public Policy, 7, 495-523.

Bios Shabnam Javdani, MA is a doctoral candidate in clinical/community psychology at the University of Illinois Urbana-Champaign. Her research examines the development of and social response to violence across levels of analysis (e.g., individual, social) and in gender-specific ways. She is particularly interested in adolescent girls’ involvement in the juvenile justice system and the development and consequences of women’s use of violence in the context of intimate relationships. She has helped develop and coordinate an advocacy program for juvenile justice-involved girls and hopes to continue to integrate research and community-based action with a focus on women and girls. Nicole E. Allen, PhD, is associate professor of community psychology at the University of Illinois Urbana-Champaign. Her research examines community collaboration and systems change processes with a focus on the community response to intimate partner violence and Systems of Care development and implementation for youth and families. She has contributed to the scholarly literature via numerous papers and presentations and is committed to bridging scholarship and action by working closely with community partners in both research and action. She also directs an advocacy program for women with abusive partners and girls at risk for entry into or residing in juvenile detention. Nathan R. Todd, PhD, is assistant professor of community psychology at DePaul University in Chicago. His research examines contextual factors that influence individual and group engagement with social justice. He focuses on how religious settings (i.e., congregations and interfaith groups) and Whiteness influence engagement in social justice with specific attention to multiple levels of analysis. Carolyn J. Anderson, PhD, is a professor in the Department of Educational Psychology at the University of Illinois Urbana–Champaign. Her research interests lie at the intersection of statistical models for categorical data analysis and psychometrics. Her major line of research deals with the development of multivariate categorical data that have latent variable interpretations. She teaches advanced statistical methods courses mostly to students in the social and behavioral sciences. She has served in leadership positions in the Psycholometic Society, Classification Society, and Division 5 of APA.

Downloaded from vaw.sagepub.com at NEW YORK UNIV LIBRARY on June 3, 2014