Sample Chapter

6 downloads 299 Views 123KB Size Report
Second, I discuss adaptation of methods—literature reviews, quantitative studies, ... problem with U theory is the conditions in new settings vary from the ...
From: Research, Actionable Knowledge, and Social Change: Reclaiming Social Responsibility Through Research Partnerships By: Edward P. St. John Chapter 3 Critical-Empirical Approach The critical-empirical approach is a meta-methodology that can be used in literature reviews, quantitative research, and qualitative research to discern and address critical challenges in policy and practice. The critical-empirical approach provides an alternative for testing how social theory can inform intervention processes in education and social agencies. As noted earlier (Table 1.2), both assessment and inquiry-based intervention processes appropriately include all of the steps in the research process. In this chapter, I first compare the criticalempirical approach to the neo-scientific approaches to educational and social research. While the critical-empirical approach can be used in basic research that reconstructs theory (e.g. St. John, Hu, & Fisher, 2010), I focus on how it can be used in support of locally-situated educational and social reform. Second, I discuss adaptation of methods—literature reviews, quantitative studies, and qualitative inquiries—in support of assessment and action-inquiry research processes within organizations. As part of this discussion, I develop sets of propositions about research design and methods to guide action research projects that seek to identify and address critical issues within research partnerships.

St. John   

Chapter 3 



Comparison of Research Approaches The critical-empirical approach to social and educational research involves critical analysis of theories related to educational and social problems. As an introduction to the criticalempirical approach, I compare it to the traditional approach to social research. Theory in Problem Solving The critical-empirical approach involves deconstructing theories about solutions to a social problem, generating and using appropriate evidence to test whether the claims about the reform hold up, and reconstructing situated understanding of solutions to local problems. The process can involve review of universal theories (U theories) and related evidence.1 Most commonly used social, economic, psychological, and educational theories make claims about universal patterns of problems and solutions (U theories). Typically, U theories provide explanations about the ways both prior conditions and interventions influence outcomes. The problem with U theory is the conditions in new settings vary from the conditions present when the theory was developed. In contrast, to solve problems in practice we need situated theories (S theories) that provide testable explanations about how a recurrent problem might be solved in context. Practitioners typically adapt U theories to make claims about action strategies as rationales for reform, but they seldom test those theories in actionable situations. The goal of S theory is to solve problems, while the goal of U theory is to provide universal explanations. Both types of action theories are needed and should work in tandem in the social sciences. While theory building is not the primary intent of research conducted in partnerships, it is an integral part of the process.

St. John   

Chapter 3 



Many of the common U theories were developed during an earlier period of social and economic progress, from the Enlightenment through the emergence of the global period, a problem discussed in the prior chapter. Since the education excellence movement was predicated on U theories of change, the reforms systematically overlooked the role of S theories about local action within educational reform. This devalued the professional discretion and expertise of educators at all levels. Looking at the theories related to college access (discussed in Chapter 2), an early theory of social attainment was used to explain the ways family background related to goals, high school courses, and educational attainment, making it possible for policy researchers to hypothesize how interventions in education, especially changes in course requirements, would influence educational attainment. Much of the federally funded research used to inform this rationalization about new course requirements had examined the correlations between courses completed and eventual success for student cohorts (e.g. longitudinal studies of students in graduating classes). The broader theories about how social contexts influence attainment, including social capital and reproduction theories, were overlooked in this correlation analysis. The policy recommendations that followed, that all students be required to take higher level courses to complete high school, actually changed the conditions facing students, teachers, and school administrators, causing instability in the social context of high schools and the lives of families. A broader framing that tested assumptions from social capital, human capital, and social reproduction probably would have yielded a different strategy for reform. The contexts of education and the lived lives of families were more difficult to change than anticipated by those who mandated the new requirements because these factors were overlooked. It is necessary to

St. John   

Chapter 3 



consider situated factors and pay attention to educational practices—methods of instruction and how they fit learning needs—along with the goal of raising content standards. This process of local adaption to new policies involves addressing challenges locally, developing and using S theories. For illustrative purposes, I discuss this problem in relation to U and S theories below. U Theory and Policy Change: Consider the theory problem related to the implementation of new mandates for high school graduation which largely overlooked the organizational change and adaptation processes needed within schools. The curriculum and instruction methods that had worked for a minority of students—those who had achieved on math tests—were used for the broader population that had not experienced prior success. Policymakers who changed requirements overlooked the contexts of school and community change, as had the analysts who conducted the research on educational attainment. Both public high schools and charter high schools were slow to change curriculum because of these factors, and educational outcomes did not improve (Ravitch, 2010). Case studies indicate that successful charters actually tried out multiple approaches to curriculum to find methods of teaching advanced subjects to inner city students, while public schools had to acquire external support from nonprofit organizations and colleges to change their culture when faced with the requirements and constrained curriculum, including district-chosen texts (St. John, Masse, Lijana, & Bigelow, in review). The new quasi-private charter schools had more freedom to adapt, while the public schools had constrained curriculum, but neither type of school was able to handle the challenge. There were at least two underlying problems. One problem related to social contexts. General status attainment theory had previously been reconstructed to include social capital focusing on the roles of networks, information, and trust (e.g., Coleman, 1988). The singular focus on educational attainment in the statistical

St. John   

Chapter 3 



modeling in the NCES studies largely overlooked the social support families would need. The sad irony is there had been research on student choice and capital formation that could have better informed these policy decisions. For example, Indiana developed statewide programs to provide social and community support, building trust that led students and parents to use the information provided about college and career pathways (e.g. Hossler & Schmidt, 1995; Hossler, Schmidt, & Vesper, 1999; St.John & Musoba, 2011). But this type of socially adaptive change was not widely adopted; most states implemented new graduation requirement without a social support (St. John, Daun-Barnett, & Moronski-Chapman, 2013). Second, there were changes in conditions external to schools that undermined efforts to encourage academic preparation: college tuition rose and student grants declined. The older theories of human capital (e.g., Becker, 1964) had been used by advocates of student financial aid in the 1960s and 1970s in rationales for federal need-based student grant programs, developed as part of the Higher Education Act (HEA) and its reauthorization. Had the social and political context remained stable, with low college tuition and relatively high need-based grants, then costs would not have been as much of a discouraging force for low- and middle-income students at it turned out to be. The family concerns about costs influenced student aspirations (e.g. St. John, 2003), and more recent economic theory recognizes the important role of these types of perceptions (e.g. Stiglitz, 2012) on aspirations and, consequently, college preparation. Clearly, different U theories were applicable and could have been used to inform debates about changing high school graduation requirements, but the narrow choice of educational attainment theory led to a constrained framing of the educational problem. Education attainment theory was used to argue for changing requirements and expanding the availability of college preparatory courses within comprehensive high schools (e.g. Adelman, 1995; Berkner & Chavez,

St. John   

Chapter 3 



1997); newer concepts of social capital were overlooked; and economic theories of human capital that had an established role in access policy (i.e. meeting financial need for low-income students) were ignored. The Role of S Theory: The implementation of centralized policies put in place by the Global Transition and corporatization of education complicates governance, change and improvement in schools. As schools, colleges, and other organizations adjust to new requirements, they face new challenges that must be solved locally. The chapters that follow provide many examples of the ways research has been and can be used to inform local change and adaptation. Looking at the transformation of comprehensive high schools into college preparatory schools, the challenges facing most urban and rural schools differ from those in the majority of suburban high schools. Most suburban schools already had college preparatory tracks that could be expanded, but many rural schools had not developed advanced courses in math and other subjects, and while most cities had some preparatory high schools, the majority of urban high schools did not have advanced courses. Further, most low-income parents in urban and rural communities did not have prior personal experience with the advanced courses confronting their children. Making choices about what school to attend, how to help their children in advanced courses, and whether their children will be able to pay for college were critical issues that had to be solved locally. Even in Indiana, a state that took a comprehensive approach to policy, the shift in public finance, graduation requirements, and education markets caused myriad problems in schools and colleges. Indiana policymakers found it necessary to develop research informed strategies for reform in early reading, college support for high school students, and tactics for admission,

St. John   

Chapter 3 



orientation, and student support within public and private colleges. Solving these problems starts with a focus on situated challenges and solutions (e.g. S theory). I use the term actionable research to refer to research that is situated locally and considers a range explanations related to emergent problems. Examples of actionable research informing local adaptations are provided in subsequent chapters. Design Proposition 1: While basic research seeks to prove universal propositions proposed by theory and applied research seeks to use universal claims to inform practice, actionable research seeks to inform interventions in situated contexts that focus on improving outcomes and reducing inequalities. Problem solving that cuts across the boundaries of U and S theories is needed. This chapter focuses on the research aspect of the problem: how to frame research that might inform practitioners about problem solving, a process of resituating (or reconstructing) U theories. The next chapter digs into the application of the critical-empirical approach in applied problem solving, the action inquiry model (AIM), and the following chapters examine professional, organizational, and policy contexts for using action inquiry. Comparison of Approaches The scientific and critical-empirical approaches to social science research are compared in Table 3.1 as they relate to theory, the application of research methods, the role of research, and their implications and limitations. Both approaches are important, but there is a need for more social research that helps solve problems in situated contexts (i.e., builds and uses S theory) and that reconstructs social theory (U theory) to fit the new global context. The scientific approach tends to be paradigmatic, which involves making implicit assumptions related to the theory and testing explicit in elements to advance understanding, as

St. John   

Chapter 3 



illustrated above by the use of attainment theory in federal research on college preparation and access discussed above. Using this approach, prior research within a theoretical tradition is used to frame new research problems. Typically, problems are framed as educational, social or economic, but the combination of frames is seldom used. In education, research is typically framed with theories from sociology, economics, or education (e.g., theories related to practice of teaching), but generally multiple theories are not used to develop hypotheses. Over time, some areas of theory, like college choice,2 have developed that are rooted in economic, social, and educational theory. In the scientific method, using quantitative data makes it easier to confirm or disconfirm hypotheses generated using theory-based assumptions. Experiments with random assignment are generally thought to be the best way to evaluate an intervention, although quasi-experimental methods with multivariate statistics have more frequently been used in education3 deriving a choice of sets of independent variables from theory or logical models and using the model to predict an outcome (e.g. regressions of different types).4 Using the significance or nonsignificance of independent variables in quasi-experimental studies and in outcomes in experimental studies provide the means of hypothesis testing.

Table 3.1 Comparison of the Neo Scientific and Critical-Empirical Approaches to Educational and Social Research Dimension Relation to Theory and Prior Research

St. John   

Scientific Approach Review prior research and theory to develop a “hypothesis” that can be tested in a well-defined research study.

Critical-Empirical Approach Review competing theories and diverse research pertaining to the policy problem. Identify different, possibly competing, claims.

Chapter 3 



Accepted Methods

Quantitative studies allow for accepting or rejecting hypotheses. Experimental designs, including natural experiments, are emphasized.

Methods appropriate for “testing” specific claims are used; appropriateness of alternative methods depends on the nature of claims to be examined.

Depending on the context and topic, Large-scale data collections and researchers can use quantitative secondary data analyses are also research, qualitative research, and/or frequently used. critical reviews of research. Functionalist qualitative research used to asses impact of interventions and encourage bringing innovation to scale

Research is used to confirm and verify claims. Often offered as “proof” of the theory, claim, or model. Implications Research often used for building rationales for reform. and Research tends to be “selfLimitations sealing” and to overlook competing views. Adapted from: St. John, 2007, p. 69 Role of Research

This approach is highly compatible with action inquiry focusing on problem solving. It can be used to pilot test strategies before full scale adoption and adaptation in large organizations Research is used to build understanding, develop theory, and inform action. Emphasis is on actionable knowledge. Research examines competing views and can be used to open conversation. Research tends to be overlooked in policy forums because of complexity.

In actionable research focusing on situation challenges, it is crucial to consider multiple explanations for problems when designing and interpreting research and determining how it applies to problem solving. Design Proposition 2: In contrast to the notion of control implicit in attempts to replicate best practices, actionable research seeks to promote professional learning, innovation, and adaptation of best practices within human systems. New statistical methods from sociological and economic research have been increasingly used in education research: regression discontinuity has been used to examine the effects of an St. John   

Chapter 3 



intervention at the cut point of award, where recipients and non-recipients are essentially the same; instrumental variables have been used to control for selection for the intervention (Heckman adjustment); and difference in differences analyses have been used to compare conditions prior to and after an intervention for comparable groups. To the extent these advanced methods can be used in the study of education reforms, they should be used. The quantitative studies used in this volume were completed before these newer standards were generally accepted in educational research; the examples of quantitative methods may not be the most current, but they still have value for illustrative purposes. Qualitative methods have also been used as a form of exploratory or confirmatory evidence within the research science approach. This type of qualitative research typically uses case studies or interviews to test concepts or frames of theory, contributing to the development of functionalist theories and informing interventions by trying out or testing different concepts or theories (e.g. Bastedo, 2009; Bensimon, 1989). However, such work does not go as far as causal research in confirming links between practices and outcomes. To generate actionable knowledge, it is crucial to not limit inquiry to preconceived hypotheses, but instead to analyze evidence that emerges which can inform knowledge reconstruction. The critical-empirical approach is rooted in critical theory as a process of theory deconstruction, claims testing, and reconstruction (Habermas, 1984, 1987) and action research when designing experiments in actionable situations to test strategies for change (Argyris, Putnam & Smith. 1985; Argyris & Schön, 1975; Lewin, 1952). The difference in the framing of problems using the two approaches relates to identifying and testing different, possibly competing, claims about the causes and solutions to social problems in action situations. This approach to problem solving was generated by Habermas in The Theory of Communicative

St. John   

Chapter 3 

10 

Action (1987), where he demonstrated the analytic process of critically examining social theories, using evidence from various sources to critique and test the logic of claims central to the theory, and reconstructing ideas from the theory. The core elements of the critical-empirical approach are:  Deconstructing claims from theories, frameworks, and/or models that can be tested;  Generating evidence from reviews, quantitative studies, qualitative studies, and/or action projects to test the claims; and  Reconstructing new understandings and developing them into strategies that can be tested in action. Using this critical stance allows researchers and practitioners to look across theories to find a range of feasible explanations that can be used when examining problems in practice. Existing theories from education, psychology, sociology and economics can inform reasoning about potential underlying causes. For each problem area, the different theoretical forms of reasoning provide lenses for thinking about causes related to prior learning, predispositions and personal strengths, family background and culture, and financial barriers and expectations. At the assessment stage, in addition to establishing a baseline against which to compare future success, a goal is to speculate about possible underlying problems that could be remedied, the local challenges that should be the focus of action inquiry. Integrated theories like academic capital formation may help set the frame, but each major challenge has a local context and constraints on action, so engaging in the process of constructing hypotheses about possible remedies is a crucial part of the process.

St. John   

Chapter 3 

11 

Design Proposition 3: Actionable research seeks to address critical challenges in their situated contexts through informed human action, a process that necessarily engages practitioners in the design, implementation and evaluation of interventions. The action component emphasizes designs for interventions that can test hypotheses in action situations, but it is important to go through an analysis process to get to that point. The action dimension involves researchers and practitioners in discerning why problems recur in the local setting, looking internally and externally for possible solutions, assessing the feasibility of the solutions in relation to the understanding of the problem, designing and implementing actions that test (or pilot test) the plan, and using evaluation to continue to refine strategy. While the remainder of this chapter focuses on using traditional methods of research to construct actionable knowledge in situated contexts, the remainder of the book focuses on using the critical-empirical approach in thought partnerships with practitioners. Research Methods in Critical-Empirical Research Literature reviews, quantitative methods, and qualitative methods are professional practices researchers use to address social and educational problems. Graduate courses in education and the social sciences introduce students to basic and advanced methods researchers use as part of research design. Below I discuss strategies for using these same methods to design research and use generally accepted methods to support and inform professionals in educational and social service agencies engaged in research partnerships that support and inform reform. Critical Reviews Informing Reform Literature reviews play a central role in actionable research, as they do in basic research testing theoretical concepts. As part of putting together a baseline assessment in collaboration with an organization, it is important to develop potential explanations for evident patterns that

St. John   

Chapter 3 

12 

might be changed through intervention. In most areas of reform, there are usually extensive literatures on intervention designs, research methods, and results of interventions, much of which encourages replication, an emphasis reinforced by neo science in education and social science. When considering a problem related to policy implementation at the local level, there are multiple applicable areas of research that can inform action. Rather the selecting a single theory to guide policy implementation, it is important to consider a range of concepts of the problem at the outset of planning for implementation. Using an alternative, critical-empirical approach involves critically reviewing the literature on reforms to discern evidence that can inform practitioners about strategies they might try out. In this case the review process should emphasize:  The program features of reform models and how they differ from the practices currently used in the partner organization: How would adoption or adaptation of the reform alter practice?  The types of professional development needed to learn about the new program in practice: What would practitioners in the partner organization need to learn to try out the new approach, and how would they acquire this knowledge?  The underlying values of the reform: Is the reform model compatible with the mission and goals of the partner organization?  How the reform might engage citizens (e.g. students and parents in schools) in the core functions of the organization: How would adopting the reform meet the actual needs of citizens the organization serves?

St. John   

Chapter 3 

13 

 The intended outcome of the reform and how the program features actually link to these outcomes: Does the reform address challenges evident in the organization?  The findings from the research: Does the research show that the intervention, perhaps adopted and adapted from a reform implemented elsewhere, address critical issues in the partner organization? Methods Proposition 1: To inform practitioners about potential interventions, reviews should provide practitioners with information on prospective interventions, how they have worked in the past, and how they might fit into or be improved upon with partner organizations. Literature reviews can also be used to aid state agencies, schools and practitioners in making informed choices about practices to consider; reviews can then be developed into guidebooks or frameworks. For example, the Indiana Education Policy Center (IEPC) collaborated with the Indiana Department of Education on the development of a critical review informing teachers about strategies for improving early reading (St. John, Loescher, & Bardzell, 2002). As part of the new grant programs for schools, the center reviewed a wide range of reading reforms that had a research base and developed a guidebook for schools to encourage them to assess current practices and outcomes, review the reform models to discern which might improve their reading outcomes—both through improved test scores and enabling more students to make yearly progress (i.e. reduce retention in grade level and special education referrals)— and develop an approach for reform. The IEPC also conducted workshops for proposal writers from schools and school systems, encouraging them to use interventions as pilot tests of new methods and to use insights gained to develop reform strategies that cut across grades K-5. Schools were encouraged to consider how their reading programs helped students develop a set St. John   

Chapter 3 

14 

of skills—from reading readiness through phonemic awareness, comprehension, writing, and reading across the curriculum—that would provide scaffolding for reading. The guidebook provided a foundation for proposals to the state for improvements in early reading, and there was evidence of improvement in both achievement and equity outcomes in the period that followed (Spradlin, Kirk, Walcott, Kloosterman, Zaman, McNabb, Zapf, & Associates.2005). In another important example of review informing practice, Estela Bensimon and colleagues (Bensimon & Malcolm, 2012) developed a theory of change and a guiding framework for colleges and state systems to use as they assess gaps in minority enrollment and retention. This work has yielded a series of publications that inform intervention theory and methods with respect to reducing inequality. . The guides are written as sequences of questions teams from campuses can use to make informed judgments about possible causes of problems, refine their listening skills in discerning unintended discriminatory practices, and identify possible interventions that merit testing. This approach informs practitioners as they ponder interventions that merit testing on their campuses. Indeed, this work illustrates how critical reviews can be used to build a culture of reform among practitioners who share a commitment to improving diversity. Bensimon and colleagues also recently developed a guidebook focusing on building equity-focused college-going cultures in high schools (Jones, Bensimon, McNarir, & Dowd, 2011). This guidebook develops frameworks and questions for reflection by educators on schoolbased cultures and intervention strategies and by system administrators on data systems for assessing (tracking and monitoring diversity) and evaluating interventions. Developed as part of a “tool kit” project for Kresge Foundation, this new guide builds on the underlying logic of earlier reviews.

St. John   

Chapter 3 

15 

Methods Proposition 2: Through critical reviews, researchers can construct guidebooks that help practitioners design interventions they can test by adapting current strategies. Not only are literature reviews an integral part of the research process, they can be used to inform practice. As noted, Estela Bensimon and her colleagues have demonstrated that reviews can be used as the basis for guidebooks for schools and college as they address inequalities. The Action Inquiry Model (Chapter 4) also provides a basis for engaging practitioners. The AIM model is appropriate when there I s not a wealth of prior research and the problem is not a simple matter of choosing a pre-tested remedy. If used as part of organizational management, it can also help practitioners build their capabilities as change agents. Quantitative Research Supporting Action research Quantitative methods have changed substantially in the past decade as neo science approaches have been adopted within the government-corporate-nonprofit complex. There is new awareness that quantitative methods can be used in research that addresses critical issues in education (e.g., Stage, 2007). My argument is that more researchers should use appropriate statistical methods in support of educational partnerships to reform practices to improve the quality of services and reduce inequality in their access. When discussing statistical methods, it is appropriate to distinguish three types: descriptive analyses that establish trends and basic relationships among variables and groups; correlation methods that illustrate relationships between independent variables and outcomes (using theory to select variables thought to be related to outcomes); and causal methods that attempt to prove a relationship between interventions and outcomes. Basic Statistics and Intervention Design: The bottom line for professional organizations with accountability systems are the rates on key indicators of quality and equity (e.g. test scores

St. John   

Chapter 3 

16 

as a quality measure and gaps across groups in scores as an equity measure), what we can refer to as critical success indicators (CSIs). When focusing on improvement in outcomes used as CSIs in educational and social services, it is crucial to consider trends and differences in outcome across groups. The CSIs illustrate patterns of change, be it progress or regress. For example, educational accountability systems tend to use trends of outcomes related to attainment (e.g. percent completing high school, percent of graduates going to college, and percent of undergraduates completing college) and achievement (i.e. trends in average test scores) as indicators of the quality of the system. Disparities in the measures of success over groups are typically used as simple equity measures (e.g. differences in scores between minority and majority students)5. Methods Proposition 3: Trends in critical outcomes related to quality and equity provide baseline success indicators for practitioners to consider when choosing or designing intervention methods that seek to improve quality and reduce inequalities in outcomes across groups. Simple measures of differences (i.e. the comparison of means for groups) are frequently used to measure the efficacy of interventions in improving quality. In a designed experiment, the treatment group is compared to the group that does not experience the intervention being tested. To control for differences in a population before treatment, individuals must be randomly assigned to treatment and comparison groups (Campbell, & Stanley, 1963). There are two serious problem with the application of this experimental method in education and social services: 1) since people frequently make choices about schools, colleges, courses in college and so forth, it is difficult to enact experiments using random selection; and 2) even when an experiment proves successful as measured by significant differences for critical success

St. John   

Chapter 3 

17 

indictors, repeating the experiments and achieving similar results proves difficult in human systems with different people, resources and contexts (the replication problem). The problems with randomization and replication do not mean practitioners cannot learn from experiments. In the action inquiry model (Chapter 4), I argue there is substantial value to maintaining an experimental attitude in organizational and social change even when randomization is not possible. It is still possible to use trends as they compare to historical outcomes in the organization and to compare treatment groups whether or not randomization is used in the assignment to a group. Other types of quantitative and qualitative research can also inform learning about how and why outcomes differ, if they do. Such an inquiry-based approach improves learning by practitioners. The inquiry-based method also recognizes the role of human agency: citizens—clients of organizations, students in schools, and the professionals who serve them—are not subjects to be studied, but rather human agents whose actions are not controlled by policies or mandates. By engaging practitioners in research and students in the choice of educational programs, it is possible to promote learning at client/student, professional, and organizational levels. Professional development with open experimentation and client/student engagement in choosing educational opportunities that align with interests can become strategies for promoting learning, innovation, and adaptation within systems. Methods Proposition 4: When feasible and morally appropriate, action experiments should use random assignment to treatment group and control group as part of the pilot test of intervention methods, an approach that meets the highest standards of statistical proof.

St. John   

Chapter 3 

18 

Methods Proposition 5: Practitioners and researchers should maintain an experimental attitude about and follow through on measuring whether interventions result in changes in quality and equity measures compared to historical trends and/or control groups. Correlations, multivariate statistics, and quasi-experimental methods: Most quantitative research in education and the social sciences uses U theory to design research on interventions. A large array of multivariate statistical methods can be used to examine how interventions relate to outcomes. Typically, the second statistics courses in graduate programs introduces students to these methods, including various regression models that can be used to assess whether, controlling for background variables, an intervention alters an outcome. Most education and social science researchers should be competent in generally accepted multivariate methods. Studies using these methods have provided a great deal of information on how social and economic factors relate to educational outcomes, but educational research has not made sufficient use of these methods in evaluations of interventions. However, further testing of sound education theories with well defined linkages, like Tinto’s theory of college persistence (Tinto, 1982, 1987), have progressed (e.g. Braxton & Lien, 2000). This type of research is not without problems. For example, research on college retention has been widely used to inform the design of interventions, like learning communities (i.e. linked courses offered in college dormitories). Research that showed student interaction with faculty is associated with persistence was used to rationalize new methods for involving faculty in learning communities that also build strong social ties among students. Most of these studies, however, did not meet generally accepted standards on learning communities or other interventions rationalized based on Tinto’s theory (Braxton, McKinney, & Reynolds, 2006; Patton, Morelon, Whitehead, & Hossler, 2006).

St. John   

Chapter 3 

19 

Therefore it became important to reframe persistence research to focus on evaluating the impact of actual interventions (St. John & Wilkerson, 2006). There has been a paucity of actionable research addressing local challenges in implementing best practices. For example, adapting theory and using multivariate models to document whether interventions have the intended effects and how intervention methods can be adapted and refined to evaluate the effects of interventions like learning communities is necessary (Musoba, 2006; St. John, 2006; St. John, McKinney, & Tuttle, 2006). However, there continue to be limitations in the application and use of multivariate methods in the evaluation of interventions in practice: 1) U theories are not always easy to adjust to S circumstances and extant databases or surveys; 2) initial interventions that include pilot tests of new strategies may not include a sufficient number of subjects to allow the use of multivariate methods; and 3) more substantive information about practitioner and client/student experiences may be needed (i.e. qualitative research). Methods Proposition 6: It is desirable to use theory to inform the design of multivariate statistical studies that seek to measure the impact of interventions when it is feasible to do so. Causal Methods: A new generation of causal methods has been developed and tested for large scale research on interventions, including regression discontinuity (RD), a method that can be used to judge whether the treatment had an impact at the “cut point” of selection for an intervention (e.g. the text score differential for inclusion in remedial programs), and differences in differences (DID), a method for comparing entities (e.g. groups organizations, states) implementing the intervention to those who do not (e.g. Dynarski, 2002; Daun-Barnett & St. John, 2012). These methods also have limitations. For example, generalization in RD is limited

St. John   

Chapter 3 

20 

to groups near the cut point studied and DID is generally used within a regular regression model that compares across groups. Nevertheless, causal models represent the most recent advance in statistical methods and should be used in research on interventions whenever appropriate and possible. These methods are especially appropriate after there has been sufficient evidence from pilot tests on an intervention method that can be replicated, and there is an effort to bring the intervention to scale. Use of these methods raises further questions, especially with respect to the role of replication. Nevertheless, when appropriate they should be used as part of research within partnerships between universities and educational and social systems engaged in reform. Methods Proposition 7: When feasible and appropriate, causal statistical methods should be used to evaluate interventions, especially when there are efforts to bring an innovation to scale within large educational systems. Qualitative Research to Inform Action Inquiry Given the limitations of statistical methods, qualitative research has an especially important role to play in action research informing improvement in schools, service organizations, and CBOs. When critical success indicators of quality and equity are used as the bottom line in administration and accountability, practitioners should seize opportunities to engage in action research that seeks to improve outcomes through innovations in practice because of the opportunities for professional development and impact. Qualitative research— collecting and interpreting information from students, parents, and other citizens—can be used to provide insight into the limitations of current practices (assessment of challenges) and in pilot testing interventions.

St. John   

Chapter 3 

21 

Before engaging in qualitative research to support action inquiry, researchers should have a background in qualitative methods, both courses with content on methods and experience using the methods; practitioners engaged in conducting focus groups and interviews should also have some training, possibly through working with a research partner. Using Qualitative Information in Assessment: While the CSIs may provide a bottom line for leaders in schools and organizers in CBOs, they generally do not provide the types and quality of information practitioners can use to identify critical challenges and judge the impact of interventions. Practitioners can gain substantial insight from observations and talking with citizens about practice. Asking about personal challenges and observing how different groups engage in learning and service opportunities provide insights into how people interact and react. Practitioners have perspectives on what is happening and should happen that shape their interpretations of what they hear and observe. Based on their experiences they frequently have theories of the problems in practice and rationales as to how they can contribute to solutions. In communications about problems, it is important to get past defensive stances, a process facilitated by open discourse (see Chapter 5). Open conversations, interviews and focus groups provide opportunities to gain insights into the voices of potential clients and the perspectives of practitioners. Interviews, focus groups, and surveys conducted by researchers who are not directly involved can provide fresh insights into the ways programs or services actually work. One option, especially in large organizations, is for teams of practitioners to conduct qualitative research designed to gain insights into the voices of other practitioners within the organization. Qualitative research used in the assessment process should focus on patterns that can be influenced by professional action, taking into account the cultural contexts of the lived lives of

St. John   

Chapter 3 

22 

community members and practitioners as resources. Learning about cultural differences helps build understanding of the nature of challenges and resources for addressing them. Methods Proposition 8: As part of the assessment process, interviews and observations of practice provide insights into the ways educational and service programs actually function within the constraints of systems, including information to inform hypotheses about the ways practices might be changed to improve outcomes and reduce inequalities. The insights gained from these qualitative sources provide information about the reasons for troubling trends. Specifically, there are three sources of information needed to start identifying and addressing challenges: 

Trends in critical indicators including achievement, declines in program completion rates, and so forth, along with quantitative information of possible causes;



Research reviews that provide sound theoretical frameworks and an array of explanations for the challenges; and



Qualitative information derived from insights of practitioners and citizens served by the organizations.

Practitioners gain new insights by triangulating on problems using these multiple sources of information. At the level of practice within organizations, policy is an external matter, a constraint. It is crucial to build an understanding of the problems that can be addressed within these constraints: the forms of action that improve outcomes and reduce gaps within those constraints. It also helps facilitate discourse that focuses on identifying critical challenges that can be addressed by changes in practice—adaptation of known policies and practices and

St. John   

Chapter 3 

23 

innovations that create new forms of practice—when both practitioners and researchers engage in the process of triangulation on these three types of data. Qualitative Research in Pilot Tests of New Strategies: As part of action inquiry, teams of professionals investigate possible solutions to the challenges before deciding on strategies to pilot test. Further research and reviews can inform that process as discussed in Chapter 5. It is often crucial to design an evaluation of the intervention as part of its implementation plan. Both internal and external researchers have critical roles in the design and completion of pilot tests. Of course, it is important to collect information on use of the services, but it is also important to have some basis for comparing the ways the intervention actually works for the treatment group compared to the control group. As part of the design, there should be routine collection of interviews, observations, and reflections for both groups. The comparison of qualitative information—the comparison groups or insights from historical practice gained as part of the assessment—is crucial information for the organizational improvement process. Methods Proposition 9: Interviews, observations, and reflections on interventions should be designed into pilot tests and compared to similar information from prior practice (i.e. assessment) or from control groups. Collaboration between researchers and practitioners is crucial for successful completion of pilot tests that inform educational improvement. Using generally accepted research standards in action research also helps build high-quality actionable knowledge within areas of professional practice. Without meeting quality standards, research on practice will languish as second rate compared to conventional research which treats practitioners and their clients as subjects rather than agents of reform.

St. John   

Chapter 3 

24 

Qualitative Research on Intervention Theory and Methods: This book provides theory to guide research within partnerships. The core argument is that we can integrate state-of-the-art methods into reform processes as means of creating actionable knowledge. The research presented in subsequent chapters illustrates prior attempts to achieve this aim through development and testing of the action inquiry model (AIM) (Chapter 4); pedagogical interventions that encourage development of the analytic skills needed for action research (Chapter 5); using action inquiry within ongoing organizational change strategies (Chapter 6); and the use of multi-level approaches to research informing policy development and system change (Chapter 7). But these chapters provide only illustrative examples of the process of using actionable research in support of reform. The propositions in Chapters 2 and 3 provide intermediate understandings of the uses of state-of-the-art research methods to build actionable knowledge. The propositions set forth can be further tested, refined, and revised as part of the research process, including the development of doctoral dissertations and research partnerships that address critical issues. Research that tests these propositions, including qualitative research that examines the experiences of professors, graduate students, practitioners, and others serving in organizations undergoing change, can inform further testing of these propositions. Methods Proposition 10: Qualitative research on intervention theory and methods, including research testing the propositions stated above, can help inform and build a practice of actionable research. Critical Inquiry: Critical qualitative inquiry that focuses on equity and informs advocacy (Pasque, Carducci, Kuntz, & Gildersleeve, 2012) has an important role to play in reframing policy. Critical inquiry not only helps illuminate challenges and provides information about the

St. John   

Chapter 3 

25 

ways policies and interventions influence people, it can inform advocacy and resistance to problematic policy. Qualitative researchers who take critical positions typically recognize the agency of citizens and encourage conscious action. In my view, critical inquiry is central to the use of action inquiry in organizations. Researchers who collaborate with educational agencies and CBOs should encourage thoughtful action based on evidence. However, I also caution researchers to use “easing in” methods when testing out new ideas about action (see Chapter 6). When citizens and professionals voice critical attitudes in situations of unequal power, they need to test the extent of their managers’ openness when critiqued and questioned about their positions on issues. A core principle of professional responsibility is to “do no harm.” Encouraging activism and resistance can put citizens and professionals in jeopardy when there are unequal power relationships. Testing the extent of openness through public social interaction is, in my view, an important part of the art of using critiques based on evidence in the change process. Using the steps in the action inquiry process, especially figuring out why the problem exists in the first place and testing alternative forms of action (pilot testing), provides space for innovations using critical evidence to guide reform (Chapter 4) and to integrate critical information into learning-oriented governance within educational and other social systems. Qualitative research methods and critical interpretation with an eye on equity are crucial to addressing equity issues within educational and service organizations. Methods Proposition 11: Critical qualitative research is vital and necessary to uncover inequities and unfairness in practice because it illuminates the voices of those disenfranchised by centrally mandated policies and informs possible organizational adaptations.

St. John   

Chapter 3 

26 

In the current era, bringing reforms to scale has become the bottom line for intervention theory and methods, a problem discussed in Chapter 4. Quantitative evidence of going to scale and having an impact, however, provides only part of the information needed. Qualitative research which observes, critiques, and reflects on the process of action inquiry in research partnerships is perhaps even more crucial in building knowledge and skills for a new generation of actionable research focusing on social justice in education and social services in the global period. Guidance Advanced methods in social, economic and educational research can easily be adapted to support improvement and reform in schools, colleges, CBOs and social service organizations. In contrast to neo scientific research, which usually focuses on proving a single solution to a complex problem, the critical-empirical approach involves drawing from research across theories about critical challenges as part of assessment that defines challenges and frames strategies to test interventions to reduce inequality and improve outcomes. This approach to research uses stateof-the-art methods, but focuses on testing reforms in collaboration with practitioners working in partnerships. My argument is not that researchers should consider every possible explanation for social problems; rather, I recognize that reviews of the ways specific problems have been treated by researchers from different fields will yield competing explanations for the problem, consistent with the propositions guiding design of research of this type (Text Box 3.1.) By testing competing explanations, it is possible to come up with alternative approaches to problem solving in educational and other organizations that promote educational uplift. Text Box 3.1 about here

St. John   

Chapter 3 

27 

It is also possible to use state-of-the-art methods for review of literature, quantitative research, and qualitative research in support of partnerships that seek to improve equality and equity in education and services that support uplift of underserved and underrepresented groups (Text Box 3.2). Action research is highly aligned with the use of mixed and multiple methods, providing flexibility to individual researchers and their partner organizations. Text Box 3.2 here Guidance for Researchers Research that supports practitioners in their efforts to find solutions to recurrent problems can be conducted as part of an individual faculty member’s research agenda or as part of partnerships involving teams of practitioners and researchers. It is also possible to design doctoral programs that make it possible for students to focus on actionable knowledge and social/educational problem solving, although it is necessary to have a structure to support this approach (see Addendum). The choice is really one that involves recognizing personal research interests. Faculty researchers who have an interest in using research to inform social change should consider the real problems confronting practitioners in their field. Talking with practitioners will uncover myriad recurrent problems in schools, colleges and social organizations that promote and encourage educational uplift. The best way to get started with a new research agenda focusing on specific problems is to settle on a few related issues and review diverse studies. The review process can reveal different explanations for critical social problems, especially when studies from different research traditions are reviewed. The search capacities now available yield large numbers of studies and reports. Be sure to include studies that reach differing conclusions about causes and cures. When reviewing prior research, it is important to think critically about the assumptions made by researchers and the limited number of topics St. John   

Chapter 3 

28 

actually tested in the research. The review should consider the factors that must be considered to assess whether a remedy works. Guidance for Practitioners Increasingly, educators, policymakers, and other professionals need to use research to justify and rationalize their reform strategies. A great deal of research has been funded and conducted on reform models that can be used to construct rationales for funding proposals. University professors and other researchers can assist with the instrumental process of using research to rationalize reform. Such work has value, but I recommend a different approach to research partnerships. Compiling research to rationalize a specific reform strategy misses the step of assessing why the problem exists in the first place, especially locally-situated challenges that can occur as a result of policy implementation. Professionals who want to engage in solving recurrent problems that contribute to inequalities they observe in practice need to take a step back from the marketplace of solutions and engage in thinking through reasons why problems exist in their situated contexts, including consideration of the parameters of practice they can change. Collaboration with thought partners is crucial to this process.

                                                             1

The use of universal (U) and situated (S) theory is adapted from Habermas’s analyses in the two volume A Theory of Communicative Action (1984, 1987). 2 Both economists and higher education researchers considered social, economic, and academic factors in early studies of college enrollment (e.g. Jackson, 1978; Manski & Wise, 1983). Eventually, higher education scholars integrated the theories into stages of college choice that included searching for a college, applying, and choosing among colleges to which students were admitted (e.g., Hossler, Schimt & Vesper, 1997) 3 Students studying research methods should read about experimental and quasi-experimental design. Campbell and Stanley (1963) remains the classic reference, although there are updates to the original. 4 At different times over the past forty years, different multivariate methods have been in or out of favor including path analysis, LISREL, logistic regression, and so forth. Using more advanced or newer methods is generally preferred by academic journals and dissertation committees. 5 Given that test scores are used as measures of the quality of educational systems, it is important to realize that differences in achievement are indicators of differences in how well the education systems serves different groups, rather than an indicator of differences in the basic ability of groups.

St. John   

Chapter 3 

29