Teaching for social justice: Using vignettes to assess ...

Barnatt, J., Shakman, K., Enterline, S., Cochran-Smith, M., Ludlow, L.

Teaching for social justice: Using vignettes to assess attitudes and beliefs

Joan Barnatt, Karen Shakman, Sarah Enterline, Marilyn Cochran-Smith, & Larry Ludlow Boston College Paper presented at the American Educational Research Association Annual Meeting April 2007

Page 1 of 7

Barnatt, J., Shakman, K., Enterline, S., Cochran-Smith, M., Ludlow, L.

Teaching for social justice: using vignettes to assess attitudes and beliefs Abstract Despite rich conceptual work about teacher education and social justice, there are currently very few ways to measure differences among teacher candidates or to document change over time. This study reports on the development of a social justice vignettes (SJV) assessment tool for use in pre-service teacher education. The tool contains five scenarios that relate to social justice issues in schools and classrooms, which were constructed and piloted with different groups of teachers/teacher candidates and in different settings. Data from interviews, field notes, and questionnaires, and quantitative analysis of response range, as well as factor analysis of response items, offer support for the use of vignettes as a measurement tool. Social justice vignettes also offer promise as a pre-service or professional development tool for addressing issues of social justice in the classroom. Objectives & Research Question This study is part of a larger university-based initiative in teacher education reform, Teachers for a New Era (TNE), funded primarily by the Carnegie Corporation, which focuses on the outcomes of teacher education, including teaching for social justice. At this university, researchers have taken a ‘portfolio’ perspective to the problem of assessing the impact of teacher education, wherein a collection of complementary studies gather evidence, from multiple perspectives, to explore the impact of teacher education on teachers and their pupils, and the larger purposes of education. This study addresses the question: Can vignettes be developed as a reliable and valid measure of social justice attitudes, beliefs, and dispositions of teacher candidates/ teachers? This question is posed recognizing that, despite rich conceptual work on social justice, there are few measurement tools for capturing aspects of teaching for social justice, identifying differences in issues of social justice, and documenting change in teacher candidates’ beliefs about social justice through the teacher education experience.

Theoretical Frameworks A number of teacher education programs in the US and abroad include social justice as one of their goals. (See programs and frameworks described in Cochran-Smith, 1991, 1999; Ladson-Billings, 1999, 2001; Quartz & TEP Research Group, 2003; Zeichner, 1993; Zeichner & Liston, 1987.) Multicultural education, culturally relevant pedagogy, and high expectations for all learners in the constructivist classroom are central tenets of teaching for social justice (Banks, 1995; Ladson-Billings, 1995). Another aspect, often overlooked in programs, is the role of activism and civic participation (Greene, 1998 in Ayres; Kahne & Westheimer, 2003). Implicit, and sometimes explicit, is acknowledgement of the political position of the teacher (Cochran-

Page 2 of 7

Barnatt, J., Shakman, K., Enterline, S., Cochran-Smith, M., Ludlow, L. Smith, 1998; El-Haj, 2003; Kohl, 1998). Experts also acknowledge the challenge of working for equity in inequitable school contexts (El-Haj, 2003). The vignettes, developed for this assessment, draw on this body of work, especially Cochran-Smith’s (1998) “theory of teacher education for social change,” and “principles of practice” in learning to teach for social justice (1999). Gauging social justice beliefs poses particular difficulties in instrument development. Social justice issues appear in extraordinarily complex situations that are compounded by interpersonal relationships and varying school contexts. Teacher’s beliefs about social justice are highly sensitive, reflecting moral values and professional images. In addition, differences may exist between the beliefs expressed and actions taken. Finally, social justice has different meanings to different individuals, creating a serious challenge of measurement error when utilizing conventional survey methods. Two formats, case studies and surveys, represent the current research formats currently in use in the field to gauge understanding of teacher candidates’ beliefs regarding social justice issues (Cochran-Smith, Davis, and Fries, 2004). The TNE portfolio does approach social justice through these methods, but recognizes their limitations. Case studies offer rich description and insight into particular contexts, but limit generalizability due to small sample size, and cannot provide program-wide data or comparison over time. Standard surveys offer potential for efficiently targeting greater numbers of respondents, but are troubled by limitations of addressing the complexity of real events and detailed insight into patterns among responses. Recent research utilizing vignettes to measure instructional practices in mathematics classrooms, conducted by the RAND Corporation (Stecher, et al., 2006) offers cautious support for this format as a valid means of measuring behaviors that are consistent with what is found in classroom observations, and on surveys and logs. Additional evidence in favor of vignettes as a format for surveying complex and sensitive social issues comes from the fields of psychology, sociology, medicine, and mental health. There is evidence that vignettes offer a way of determining the cognitive processes that are utilized in making decisions, and suggest which elements of a situation are important in solving a dilemma (Morrison, Stettler, and Anderson, 2004). Furthermore, vignettes have been shown to address complex issues effectively and economically with a large number of respondents, thus offering the efficiency of quantitative data with a richness that is closer to qualitative research (Finch, 1987). The concrete and realistic nature of the vignette contextualizes the scenario so that it makes the situation familiar and interesting to the respondent, prompting reflective responses (Morrison, Stettler, and Anderson, 2004; Schoenberg and Ravdal, 2000). In ascribing the situations to another person, vignettes are found to be less threatening to respondents as they make judgments (Barter and Renold, 2000; Schoenberg and Ravdal, 1999). While differences in understanding and defining social justice may remain, the vignette grounds the respondent in a framework so that responses reflect the participant’s personal beliefs and attitudes in response to complex, difficult, and realistic situations. Methods, techniques and modes of inquiry in Vignette Development

Page 3 of 7

Barnatt, J., Shakman, K., Enterline, S., Cochran-Smith, M., Ludlow, L. The vignettes for the assessment tool described here were developed drawing on formats found in the literature, including classic moral dilemmas by Kohlberg (1976), and the subsequent development of the Defining Issues Test by Rest, et al. (1986), which streamlined Kohlberg’s open-ended questions with multiple choice responses. The following formats were also considered: the Social Work Values Inventory (Pike, 1996), which provides brief scenarios, requiring respondents to indicate the action they would take based on anchoring statements; and the Moral Justification Scale (Gump, et al., 2000), with extended vignettes and embedded statements that reflect caring and/or justice responses. Respondents are asked to rate the statements, pulled directly from the vignette, on a scale of 1 to 10, basing judgments on the importance of the statement in making a decision about the dilemma. Drawing from these resources, five social justice vignettes were developed, based on the social justice theoretical frameworks above. Respondents were asked to read each dilemma, respond to a series of statements by rating them on a five-point Likert scale, and indicate actions they would be likely to take in response to the scenario. Finally, respondents were asked to explain, in an open-ended response, why they would take these actions. Four vignettes were short written dilemmas reminiscent of Kohlberg’s work (1976), and a fifth followed the longer rendition of Gump, et al. (2000). Each vignette is drawn from real-world examples in the classroom, either from the researchers own background or from qualitative case studies of learning to teach, which are part of the TNE portfolio mentioned above. For example, one of the vignettes, “Speak English, Please!” takes place in a high school classroom in which students are reading a novel by a Mexican American author. During the discussion of the text, two Latina girls begin speaking excitedly to one another in Spanish, and an African-American girl responds, “You’re in America now. Speak English, please!” The questions that follow this scenario ask respondents to consider implications of Spanish language use in the English classroom, the role of linguistic and cultural diversity in the classroom, and the responsibility of the teacher in this situation. Other vignettes focus on topics such as differentiated instruction and evaluation, racial diversity, and mandated curriculum. Data sources and Analysis In piloting and refining the vignettes, four different analyses were conducted. Analysis 1 involved the use of a think-aloud protocol with two veteran teachers to explore the authenticity of scenarios, the effectiveness of the response formats, and clarity of information and directions. This initial analysis provided feedback for how vignettes and questions should be edited, what additions should be made to action statements, and which of the two different formats was more viable. Additional scale adjustment was also indicated. In general, the think aloud protocol endorsed the vignettes’ realism and indicated that they captured the tensions between personal beliefs and realistic possibilities for action in the school setting. Analysis 2 took place after the draft vignettes instrument was revised and used as part of a professional development program for a middle school staff of 25. Participants individually completed all vignettes, and then took part in self-selected small group

Page 4 of 7

Barnatt, J., Shakman, K., Enterline, S., Cochran-Smith, M., Ludlow, L. assessments of a single vignette. A protocol for review and discussion of the individual vignettes in small groups was provided, and participants completed written comments. Analysis of responses, field notes of discussion, and small group reports indicated further revisions to the vignettes, and rating and action statements. Based on participant feedback, the vignette with embedded statements was rewritten to conform to the format used in the other vignettes. The Likert scale items were analyzed for range of responses and for consistency on responses highlighting an individual principle. Again, the response was affirming: the vignettes are realistic, reflect the complexity of teaching for social justice, and initiate rich discussion and reflection. Analysis 3 occurred after the vignettes were piloted in three university classrooms, with a total of 59 respondents. Here, the vignettes were presented in a variety of ways to determine whether the format impacted the responses. In one set of vignette questionnaires, the action choices followed directly from the scenarios and preceded the Likert scale questions, whereas in the other sets, the Likert scale items followed the scenario. In another set of questionnaires, respondents were asked to describe actions they would take in response to the scenarios, rather than offered suggestions. This allowed us to check the actions we had generated, and make additions or modifications, based on their responses. Analysis of these data indicated that fatigue may have had an impact on responses, suggesting that the vignettes should be offered in a staggered sequence across the sample. Open-ended action responses did not provide new actions, and a range of responses was noted, as was consistency among items addressing the same principle across vignettes. Analysis 4 was similar to the second round in which the tool was used for professional development. The revised vignettes were presented to university clinical faculty who supervise student teachers as part of a two-day workshop. The vignettes were used as a tool for generating discussion. Based on the four analyses described above, a revised Social Justice Vignettes instrument has been developed. This will be administered to approximately 200 entering teacher candidates, at undergraduate and graduate levels, and will provide substantial quantitative data. Analysis 5, which will be completed in December, will provide data to determine reliability of the Likert scale items and construct validity through factor analysis. Additional analysis will include exploring differences between and among curriculum areas and grade levels represented in the population. The intention is to use the SJV as an instrument to assess teacher candidates at the beginning and end of the teacher education program. This instrument has the potential to measure changes in the beliefs, dispositions and attitudes of teacher candidates over time. These findings will also be analyzed in comparison to other elements of the portfolio of TNE research, including twelve items on entering, exit, and one year out surveys being conducted on all teacher candidates; qualitative responses on social justice from a study following 22 teacher candidates through their master’s program and into the first year of teaching; and a comparison study of our teacher candidates with novice teachers from other programs.

Page 5 of 7

Barnatt, J., Shakman, K., Enterline, S., Cochran-Smith, M., Ludlow, L.

Results and Educational Significance Preliminary results indicate that the instrument is useful. That is, we have found that a range of responses exists across participants, suggesting that differences in attitudes and beliefs about social justice issues are reflected in the ratings. In addition, consistency across individual participant ratings for responses intended to measure the same principles was also demonstrated in the pilot study. These findings suggest that vignettes may provide a useful tool that responds to the limitations presented by standard survey methods, while maintaining efficiency in handling large participant numbers. Reports by participants suggest that the vignettes genuinely reflect the complex and tension-filled events that define social justice issues in the classroom. Additionally, the vignette format is reported to serve as a potential means of gaining access to respondents’ beliefs, attitudes, and dispositions in a non-threatening manner, encouraging reflection on practice. As such, the vignettes offer promise both as a measurement tool and as a pre-service or professional development tool for addressing issues of social justice in the classroom. This instrument may be relevant to other institutions as there is little empirical evidence of the impact or effectiveness of preparing teachers to teach for social justice, despite the fact that more and more teacher education programs highlight a commitment to diversity and equity, and even explicitly to preparing teachers for social justice. In an era of accountability for K-12 schools and teacher preparation institutions, teacher educators have a responsibility to challenge what counts as success in a K-12 classroom, and be able to measure that success. To remain true to the goal of promoting social justice in and through teacher preparation, educational research must establish valid and reliable ways to measure success in this area. This tool is a step in capturing this educational outcome. References Banks, J.A. (1995). Multicultural education: Historical development, dimensions, and practice. In J.A. Banks (Ed.), Handbook of research on multicultural education (p. 3-19). New York: Macmillan. Barter, C., & Renold, E. (1999). The use of vignettes in qualitative research. Social Justice Research, 25, Department of Sociology, University of Surrey: UK. Cochran-Smith, M. (1991). Learning to teach against the grain. Harvard Educational Review, 61(3), 279-310. Cochran-Smith, M. (1998). Teaching for social change: Toward a grounded theory of teacher education. In A. Hargreaves, A. Lieberman, M. Fullan & D. Hopkins (Eds.), The international handbook of educational change (pp. 916-951). The Netherlands: Kluwer Academic Publications. Cochran-Smith, M. (1999). Learning to teach for social justice. In G. Griffin (Ed.), The education of teachers: Ninety-eighth yearbook of the National Society for the Study of Education (pp. 114-144). Chicago: University of Chicago Press. Cochran-Smith, M. (2004). Walking the Road: Race, Diversity and Social Justice in Teacher Education. New York: Teacher College Press, pp. 102-119.

Page 6 of 7

Barnatt, J., Shakman, K., Enterline, S., Cochran-Smith, M., Ludlow, L. Cochran-Smith, M., Davis, D., & Fries, K. (2004). Multicultural teacher education: Research, practice and policy. In J. Banks (Ed.), Handbook of research on multicultural education (3rd ed., pp. 931-975). San Francisco: Jossey Bass El-Haj, T.R.A.(2003). Practicing for equity from the standpoint of the particular: Exploring the work of one urban teacher network. Teachers College Record. 105 (5), 817-845. Finch, J. (1987). The vignette technique in survey research. Sociology. 21, 105-114. Green, M. (1998). Teaching for Social Justice (Introduction), W. Ayers, J.A. Hunt, and T. Quinn (Eds.), Teaching for Social Justice. New York: Teacher College Press, pp. xxvii-xlvi. Gump, L.S., Baker, R.C., Roll, S. (2000). The moral justification scale: Reliability and validity of a new measure of care and justice orientations. Adolescence. 2000 Spring; 35 (137): 67-76. Kahne, J. & Westheimer, J. (2003). Democracy and civic engagement - what schools need to do. Phi Delta Kappan, 85(1) 34-46. Kohl, H. (1998). The Discipline of Hope: Learning from a Lifetime of Teaching. Simon & Schuster: New York. Kohlberg, L. (1976). Moral stages and moralization: The cognitive-developmental approach. In T. Lickona (ed.), Moral Development and Behavior: Theory, Research and Social Issues. New York: Holt, Rinehart and Winston. Ladson-Billings, G. (1995) Toward a Theory of Culturally Relevant Pedagogy, American Educational Research Journal, Vol. 32 (3), pp. p. 465-491. Ladson-Billings, G. (1999). Preparing teachers for diverse student populations: A critical race theory perspective. In A. IIan-Nejad & D. Pearson (Eds.), Review of Research in Education (24, 211-248). Washington DC: American Educational Research Association. Morrison, R.L., Stettler, K., & Anderson, A.E. (2004). Using vignettes in cognitive research on establishment surveys. Journal of Official Statistics. 20 (2), 319-340. Pike, C. K. (1996). Development and initial validation of the Social Work Values Inventory. Research on Social Work Practice, 6, 337-352. Quartz, K.H. & TEP Research Group (2003). “Too angry to leave:” Supporting new teachers’ commitment to transform urban schools. Journal of Teacher Education. 54(2), 99-111. Rest, J. R. (1986). Moral Development: Advances In Research and Theory. New York: Praeger. Schoenberg, N.E., Raveal, H. (2000). Using vignettes in awareness and attitudinal research. Social Research Methodology, 3 (1), 63-74. Stecher, B., Le, V., Hamilton, L., Ryan, G., Robyn, A., & Lockwood, J.R. (2006). Using structured classroom vignettes to measure instructional practices in mathematics. Educational Evaluation and Policy Analysis, 28(2), 101-130. Zeichner, K.M.(1993). Educating teachers for cultural diversity. E. Lansing, MI: Michigan State University, National Center for Research on Teacher Learning. Zeichner, K., & Liston, D. (1987). Teaching student teachers to reflect. Harvard Educational Review, 57, p. 1-22.

Page 7 of 7