American Journal of Evaluation

American Journal of Evaluation http://aje.sagepub.com/

The State of the Empirical Research Literature on Stakeholder Involvement in Program Evaluation Paul R. Brandon and Landry L. Fukunaga American Journal of Evaluation 2014 35: 26 originally published online 3 October 2013 DOI: 10.1177/1098214013503699 The online version of this article can be found at: http://aje.sagepub.com/content/35/1/26

Published by: http://www.sagepublications.com

On behalf of:

American Evaluation Association

Additional services and information for American Journal of Evaluation can be found at: Email Alerts: http://aje.sagepub.com/cgi/alerts Subscriptions: http://aje.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav

>> Version of Record - Feb 6, 2014 OnlineFirst Version of Record - Oct 3, 2013 What is This?

Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

Article

The State of the Empirical Research Literature on Stakeholder Involvement in Program Evaluation

American Journal of Evaluation 2014, Vol. 35(1) 26-44 ª The Author(s) 2013 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/1098214013503699 aje.sagepub.com

Paul R. Brandon1 and Landry L. Fukunaga2

Abstract Evaluators widely agree that stakeholder involvement is a central aspect of effective program evaluation. With the exception of articles on collaborative evaluation approaches, however, a systematic review of the breadth and depth of the literature on stakeholder involvement has not been published. In this study, we examine peer-reviewed empirical studies as the first step in establishing a firm foundation for a discussion of stakeholder involvement. We excluded all studies that did not provide descriptions of their methods, helping ensure that we only reviewed studies with strong methodological warrants for their findings. We address three key questions about the state of the empirical literature on stakeholder involvement, focusing on the studies’ methodological and contextual characteristics, and the involvement themes and substantive nature (positive, negative/ mixed, or explanatory/normative) of the findings. We provide detailed findings in an appendix at http://aje.sagepub.com/supplemental or http://goo.gl/W8oMnl.

Keywords stakeholder involvement, literature review, program evaluation, research on evaluation

Research on evaluation (RoE) is an expanding endeavor. In this article, we report the results of a systematic literature review of the empirical RoE literature on stakeholder involvement in program evaluation, a venerable topic in the program evaluation literature. We discuss the methods and contexts of the empirical studies of the topic; identify the major stakeholder-involvement topics that have been addressed in the studies; and provide an overview of the studies’ positive and negative findings about stakeholder involvement as well as explanations and suggestions that the study authors have provided for evaluators’ consideration. Unlike some RoE literature reviews that

1 2

Curriculum Research & Development Group, College of Education, University of Hawaii at M¯anoa, Honolulu, Hawaii, USA Department of Educational Psychology, University of Hawaii at Manoa, Honolulu, Hawaii, USA

Corresponding Author: Paul R. Brandon, Curriculum Research & Development Group, University of Hawaii, 1776 University Avenue, Castle Memorial Hall, Room 118, Honolulu, HI 96822, USA. Email: [email protected]


Brandon and Fukunaga

27

include reflective narratives in their studies, we review only articles that have reported the methods of their endeavors.

Background RoE is defined as systematic inquiry into the methods, practices, and profession of program evaluation, with potential implications of its findings for evaluation theory. Evaluators collect and report findings about programs; researchers on evaluation collect and report findings about the many aspects of evaluation itself. The breadth of RoE is reflected in part in the growing number of reviews of the literature presented in recent years. Since 2004, authors in the American Journal of Evaluation alone have published 10 systematic literature reviews (Brandon & Singh, 2009; Chouinard & Cousins, 2009; Christie & Fleischer, 2010; Coryn, Naokes, Westine, & Schro¨ter, 2011; Johnson et al., 2009; Labin, Duffy, Meyers, Wandersman, & Lesesne, 2012; Miller & Campbell, 2006; Peck, Kim, & Lucio, 2012; Ross, Barkaoui, & Scott, 2007; Trevisan, 2004). Reviews such as these are valuable and appropriate when multiple studies of a topic appear over many years, because they can contribute considerably to our understanding of the findings on the topics in their entirety. They provide insights into the methodological soundness of the research, show the degree of attention paid to various aspects of evaluation, provide nuanced findings about the strengths and weaknesses of prevailing evaluation guidelines and practices, and present suggestions and recommendations. Stakeholder involvement in program evaluation is a topic that has received considerable attention in the RoE literature. We define stakeholder involvement as the participation of program stakeholders— that is, those who have a stake in the evaluation or its results, typically ranging from the beneficiaries, or recipients, of program services to administrators and funding agencies—in any phase of an evaluation. It is one of the most discussed aspects of evaluation practice, beginning with some of the first formal modern program evaluations. Ralph Tyler, the first major theorist of the 20th century to use the term evaluation to describe the systematic appraisal of educational or social programs, involved teachers in the landmark Eight-Year Study of Progressive Education during the 1930s (Tyler, 1991). Tyler stated that college faculty ‘‘played a principal part’’ in the evaluation of program objectives and materials in the Cooperative Study in General Education in the 1940s (Tyler, 1987, p. 125). Stakeholder involvement began appearing as an integral part of evaluations in the mid-1970s, perhaps characterized best by the work of the National Institute of Education’s evaluations of the Cities in Schools and Push/Excel programs (Farrar & House, 1983; Gold, 1981, 1983; Stake, 1983). Evaluators began to perceive stakeholder involvement as a manifestation of the political nature of evaluation (Datta, 2011; House, 1991); its purposes included enhancing the use of evaluation findings (e.g., Patton, 1978), promoting social justice (e.g., Greene, 1987), and helping to ensure the validity of evaluations (e.g., Brandon, 1999; Brandon, Lindberg, & Wang, 1993). By the early 1990s, House (1991, p. 239), among many others, concluded that ‘‘the stakeholder concept seems firmly entrenched.’’ Stakeholder involvement is central in several evaluation approaches and models, including utilization-focused evaluation (Patton, 2008), responsive evaluation (Stake, 1976), stakeholder-based evaluation (Bryk, 1983; Mark & Shotland, 1985), and collaborative evaluation in its various forms such as empowerment evaluation (Fetterman, 2001), deliberative democratic evaluation (House & Howe, 1999), transformative evaluation (Mertens, 2009), and practical participatory evaluation (Cousins & Chouinard, 2012). The major sets of evaluation standards and evaluator competencies developed in North America address stakeholder considerations, although to varying degrees (American Evaluation Association, 1995; Canadian Evaluation Society, 2011; Stevahn, King, Ghere, & Minnema, 2005; Yarbrough, Shulha, Hopson, & Caruthers, 2011). By the onset of the 21st century, ‘‘an indicator of how widely accepted and infused’’ stakeholder issues had become in evaluation was that ‘‘texts on evaluation theory and Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

28

American Journal of Evaluation 35(1)

method written from a quantitative perspective now include discussions of stakeholder involvement’’ (Datta, 2006, p. 423). Stakeholder involvement in program evaluation may be widely accepted as a component of evaluation, but evaluators have not based their conclusions about it on the results of systematic and comprehensive reviews of the empirical research. Published summaries focusing explicitly and broadly on the stakeholder literature (e.g., Taut, 2008) have not claimed to be thorough reviews or have presented findings based in part on the results of reflective narratives, which are a questionable form of empirical research (Cousins, 2003). Reflective narratives are essay-like articles describing aspects of stakeholder involvement. These studies present problems: They tend to include many findings without weighting their importance, the unit of analysis—a segment of text, a journal entry, an account of an evaluator recollection, or some combination thereof—is unclear, and they are open to cherry-picking by the author and susceptible to bias such as memory and forgetting effects. More important, they omit information about their data collection or analysis methods. Without knowledge of methodological warrant, it is difficult to know the degree to which we can be confident in the conclusions of the studies covered in a literature review. The body of RoE literature is not necessarily known for its methodological sophistication, but weeding out studies that do not present information about their methods can elevate our level of confidence in its findings. With these considerations in mind, in our review we report only the results of peer-reviewed studies of empirical research that include discussions of data collection and analysis methods. This is key to our review: The oversight that peer reviewers had of the methods of the studies that we reviewed helps ensure that the studies were sufficiently sound.

Purpose, Research Questions, and Intended Audiences Purpose The purpose of this article is to help close a gap in the RoE literature by providing an overview of the state of the empirical research on stakeholder involvement in program evaluation. To achieve this end, we conducted a broad-based systematic review of the empirical literature. We consider a systematic review to be an effort that ‘‘(1) emphasizes systematic decision rules, (2) generally uses qualitative and quantitative means to summarize findings, and (3) usually includes or integrates qualitative and quantitative information from various data sources and designs’’ (Labin, 2008, p. 92). Our conclusions are in the form of a categorized list of themes about stakeholder involvement that reflect the foci of the reviewed research, with further categorization of the findings into three types showing positive results, negative or mixed results, or explanatory or normative results. In an appendix (available only as a supplement to the online version of this article; go to http:// aje.sagepub.com/supplemental or http://goo.gl/W8oMnl), we provide detailed summaries of the three types of findings for evaluators to examine in light of their needs, circumstances, or settings. As we show, the number of studies that we reviewed is too small, and the diversity of their results is too great to allow widely generalizable conclusions about the participants, methods, or outcomes of stakeholder involvement. The nuances of the findings are quite varied; to attempt to summarize them as prescriptive guidelines for evaluators to follow would go beyond the epistemological and methodological heft of the body of reviewed literature.

Research Questions We address the purpose of our study by answering three questions. The questions are about (a) the methods and contexts of the reviewed studies, helping to situate their results; (b) descriptive findings showing the topics that researchers have addressed; and (c) findings of a more prescriptive bent that provide insights into the strengths and weaknesses of stakeholder involvement as well as explanations and advice that might be useful for researchers and evaluators. Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014


29

Research Question 1: What were the methodological and contextual characteristics of the studies? We report the number and percentage of the reviewed studies that were original research and the number and percentage that were reports of stakeholder involvement in actual evaluations. We also describe the study designs, the data collection methods used in the studies, the professions in which the studies took place, and the stakeholder groups participating in the studies. These findings (a) show the breadth, depth, and warrant of the conclusions of the reviewed literature; (b) situate the studies within real-world practice; (c) help show the extent to which we can have confidence in the findings; and (d) help serve as the foundation for making recommendations about how research on stakeholder involvement might be improved. Research Question 2: What categories and subcategories of themes about stakeholder involvement emerged in the review of the studies’ findings? Themes were the topics, variables, dimensions, aspects, components, or features of stakeholder involvement that we found in discrete sentences, paragraphs, or sections of the literature (Ryan & Bernard, 2003). We refer to the categories of themes throughout the remainder of this article as components of stakeholder involvement and to the subcategories as component features. We provide the components and component features as a means of summarizing the findings and as fodder for developing conclusions about the breadth and depth of the study of stakeholder involvement. Our response to Research Question 2 describes what practicing evaluators and researchers have considered worthy of study and shows the extent to which authors have addressed the components and component features across the body of reviewed studies. Furthermore, by its omissions, the list of components and component features shows what remains for researchers to study. The findings addressing this question are primarily descriptive, with some prescriptive implications about the degree to which authors have covered certain components in the literature and about other potential components that they have not addressed at all. Research Question 3: Of the results reported in the studies, what were the major positive findings, the negative or mixed findings, and the explanatory or normative findings about stakeholder involvement? Our discussion of the types of findings contributes to an understanding of the strengths and weaknesses of stakeholder involvement and how it might be improved. This discussion forms the detailed portion of our summary of the literature, showing how stakeholders have been involved in evaluations and presenting issues that evaluators might want to consider when involving stakeholders.

Following a description of the methods of our review, we address each research question in turn, followed by a concluding section.

Intended Audiences We intend this review primarily for researchers, with applicability for practitioners who are willing to explore the findings in depth. Possible uses include using the results to flesh out understanding of the literature; preparing hypotheses for studies of variations in stakeholder involvement, such as in causal analyses; finding gaps in the literature by identifying partially addressed stakeholder-involvement topics or those not considered at all; preparing survey items about stakeholder involvement; identifying features of stakeholder involvement to consider when conducting evaluations; and specifying potential barriers to successful involvement. The appendix available at http://aje.sagepub.com/supplemental or http://goo.gl/W8oMnl gives a detailed compilation of conclusions of the literature. Evaluators can use it to find issues to consider when involving stakeholders, learn about potential problems with stakeholder involvement, or prepare research studies on stakeholder involvement.

Methods The steps of broad-based systematic reviews include (a) specifying research questions, (b) identifying and downloading documents, (c) developing inclusion criteria and selecting documents that address the criteria, (d) identifying and coding the appropriate information from the documents, Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

30


Table 1. Steps of the Literature Review. Step

Description

Identified and downloaded documents

Developed inclusion criteria and selected documents that address the criteria

Identified and coded the appropriate information from the documents; analyzed the information

Searched the literature for the period January of 1985 through May of 2010 in Ebscohost (Academic Search premier, ERIC, MasterFILE Premier, Professional Development Collection, Behavioral Sciences Collection, and Public Administration), PsycINFO, Sage Journals Online, Science Direct, Scopus, and the Web of Science. Looked for keywords, subject terms, and titles that included the words stakeholder, collaborative, empowerment, and participatory, each in combination with the word evaluation. Found 7,580 entries; one author reviewed all abstracts and identified 466 for further consideration. Both the authors reviewed the abstracts for these and found 322 about stakeholder involvement in evaluation Developed and applied inclusion criteria specifying that the studies had to (a) involve stakeholders in the conduct or study of program evaluation; (b) be reported in peer-reviewed journals; and (c) report empirical data on stakeholder involvement, with descriptions of the study methods. Identified 41 (12%) that were peer-reviewed articles with discussions of the methods for examining stakeholder involvement and were reports of one of the two types: (a) reports about stakeholders in actual evaluations or (b) reports of research on stakeholder involvement Each author independently coded the articles for (a) article type, (b) research designs and data collection methods, (c) the professions or discipline within which the studies were conducted, (d) type of participating stakeholders, and (e) themes about stakeholder involvement. Frequency and percentage distributions were prepared. Each author independently reviewed and prepared notes summarizing each article. One author reviewed each summary, coded the appropriate segments (with themes emerging during the coding), and grouped the themes. The second author validated the codings and themes. The authors reviewed discrepancies between their two sets of results. Collaboratively, the authors organized the themes into 76 subcategories and clustered them into 10 main categories. For each subcategory, the authors summarized the results according to whether they were positive findings about stakeholder involvement, negative or mixed-result findings, and/or explanatory or normative findings



31

(e) analyzing the information, and (f) preparing and presenting the results of the review (Labin, 2008). The methods of our literature review addressed all of these steps, although our research questions emerged as the study progressed. We provide a summary of the activities addressing Steps (b) through (e) in Table 1, including the bibliographic databases that we searched, the search terms we used, the number of articles we identified with the search terms, the inclusion criteria we adopted, and how we winnowed and coded the identified articles. We selected only studies with descriptions of their methods, resulting in the exclusion of reflective narratives and articles focusing solely on theory or methods. We excluded literature reviews as well. We included one reflective narrative-like piece that provided descriptions of the methods that the researcher used. Overall, we identified 41 articles for review. The first step in the analysis of the 41 studies was to assign codes for the methodological and contextual characteristics of the studies with MS Excel. Next, we analyzed the findings of the studies in a multistage process, summarizing and coding them using an iterative, grounded-theory approach with MAXQDA software (a qualitative analysis package). We began by independently and carefully summarizing the 41 articles in sentences and paragraphs. This resulted in a list of 659 ‘‘comments’’ (i.e., discrete findings of varying length) within the MAXQDA files. Second, we reviewed the comments, developed a set of themes that emerged during the review, and categorized the comments using the list of the themes. We accomplished this by independently preparing draft lists of themes, merging similar themes, and ensuring that themes addressed all parts of our summaries of the articles. The themes were informed in part by our knowledge of the theory and methods of stakeholder involvement, thereby making them familiar to evaluators who know the literature or have experience in involving stakeholders and in part by identifying new themes that we found in the articles. Third, we grouped related themes into categories (components) and subcategories (component features). We organized the 659 summary comments into three types, including (a) positive findings (phrases or statements that suggested the benefits of stakeholder involvement or, in some instances, statements about beneficial aspects of evaluators’ involvement with stakeholders); (b) negative or mixed-result findings (phrases or statements about findings that suggested detrimental aspects of stakeholder involvement or were mixed positive and negative); and (c) explanatory or normative findings or topics (phrases or statements about findings that provided elucidation about how the study addressed stakeholder involvement, further explained positive or negative/mixed findings, or provided guidance about how stakeholder involvement should be conducted). The positive and negative/mixed-result findings show what researchers found in particular instances, and the explanatory/normative findings provide elaboration. Some comments were about either positive or negative/mixed findings and provided explanatory or normative suggestions. One author conducted the first categorization and wrote summary statements addressing each type of finding, and the second author validated the categorization and revised the summary statements as needed. Both authors agreed on the final categorization.

Results and Discussion Methodological and Contextual Characteristics of the Studies We show our summary of the studies’ designs, their data collection methods, the professions addressed in the studies, and the types of stakeholders participating in the studies in Table 2. These results suggest at least seven conclusions or implications: 1.

The small number of empirical reports that we reviewed is surprising. Given such a widely discussed topic, we expected to find more, particularly since it is minimally difficult to collect data in evaluations that involve stakeholders. On examining the number of studies included in the 10 literature reviews published in the American Journal of Evaluation since 2004, however, the number is less surprising: With the exclusion of two reviews that examine Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

32


2.

3.

4.

5.

6.

methodological issues only, the mean number of articles reviewed was 44.6 (standard deviation ¼ 11.4). We conclude that the research on evaluation on many topics is thin. The limited number of studies that we reviewed should not be taken to imply that stakeholder involvement has received little attention in the broader literature, however. Such an implication is contradicted by the large number of reflective narratives that we did not examine. We hope that our findings, underpinned by evidence of methodological adequacy, will serve as the foundation for discussions of the results of such narratives. The 41 studies were either reports of original research on stakeholder involvement, independent of actual evaluations (n ¼ 11; 27%), or reports of actual evaluations or meta-evaluations (n ¼ 30; 73%). The small percentage of studies reporting original research—despite widespread discussion of stakeholder involvement among evaluators—reflects a concern expressed by leaders in the field (e.g., Mark, 2008; Smith, 1993) about the lack of empirical RoE in general. It probably also reflects the lack of support for studies of evaluation. Agencies have made few substantial efforts to fund research on evaluation. With federal funding, researchers at the Northwest Regional Educational Laboratory and at the University of California at Los Angeles in the late 1970s and early 1980s conducted several years of work on the topic. The National Science Foundation (NSF) funded the Evaluative Research and Evaluation Capacity Building program in the early years of the first decade of this century and currently funds the Promoting Research on Innovative Methodologies for Evaluation program. In addition to a lack of funding support, King and Stevahn (2012) suggested that little research has been conducted because of the relative newness of the discipline, the lack of agreement on what evaluation means and how it should be conducted, and the practical focus of evaluation that inhibits time and funds for theory building. Stakeholder involvement, however, is a topic that can be addressed in studies more readily than others. The first author, for example, conducted a dozen studies on various aspects of stakeholder involvement, including reports of actual evaluations and original research done in conjunction with evaluations, over about a decade (Brandon, 2012). The 30 studies of stakeholder involvement in actual evaluations included 26 single or multiple case studies. Case studies are likely to yield nuanced, contextualized findings about stakeholder involvement. Thus, the use of case study methods in the majority of the 41 studies suggests that we can expect rich results providing a variety of insights into stakeholder involvement that might be difficult to summarize succinctly. Nearly half of the studies that we reviewed took place in health or education settings. The predominance of these professions might suggest that stakeholder involvement is emphasized more in them than in other disciplines and might also suggest the disciplines of academics who publish on the topic. Health, education, and social programs are the focus of attention in much of the evaluation literature. Indeed, two journals—Educational Evaluation and Policy Analysis and Studies in Educational Evaluation—focus solely on education evaluations and another—Evaluation and the Health Professions—focuses solely on health. To the extent that stakeholder involvement is part of evaluations in other professions, broadening the research on stakeholder involvement to include these professions would benefit the literature on the topic. Questionnaires and interviews were the predominant method of data collection reported in the studies; these are the most feasible methods to use, particularly for studies of actual evaluations that involved stakeholders (the majority of the 41 studies). The results produced are likely to be descriptions and self-reports that are vulnerable to the biases and limitations inherent in these methods. The distribution of stakeholders serving as respondents, as shown in Table 2, reflects a difference between original research on evaluation and reports of stakeholder involvement in actual evaluations. The authors of the former type of study reported results for a large number Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014


33

Table 2. Methodological and Contextual Characteristics of the 41 Reviewed Articles. N ¼ 41(%)

Study characteristic Design Post only Single case study Multiple data collection rounds Multiple case study Meta-evaluation Pre-post Data collection method Survey questionnaires Interviews Fields notes or observations Document review Personal reflections Focus groups Informal discussions Profession or discipline Evaluationa Health Education Social services Type of participating stakeholder group Program implementers or staff Administrators or board members Program beneficiaries Evaluators Organization or advisory board representatives Community members Not identified Funders or clients University or research faculty Parents or family members Participants serving as experimental group Number of stakeholders involvedb 1–25 26–50 51–100 >100

22 21 17 5 4 2

(54%) (52%) (41%) (12%) (10%) (5%)

30 23 14 13 6 4 2

(73%) (56%) (34%) (31%) (15%) (10%) (5%)

13 10 9 8

(32%) (24%) (22%) (20%)

19 16 12 11 10 6 4 3 3 2 1

(46%) (39%) (29%) (27%) (24%) (15%) (10%) (7%) (7%) (5%) (2%)

16 5 3 12

(44%) (14%) (8%) (33%)

a

Studies of evaluation outside of any particular discipline. bPercentages are of the 36 studies reporting the number of involved stakeholders. Percentages do not add to 100 because of rounding.

7.

of respondents (e.g., from surveys and simulations). In contrast, the authors of the latter type of study collected data from the relatively few stakeholders who typically participate in actual evaluations. The small number of participating stakeholders is, in itself, a telling finding about stakeholder involvement. The stakeholders participating in the studies were mostly program personnel, suggesting that the results cannot be expected to apply to other stakeholder groups. Although they reflect the third highest group shown in Table 2, the small number of studies involving program beneficiaries (12) is disappointing, because beneficiaries often have insights into programs that program personnel might lack (Brandon, Lindberg, & Wang, 1993; Brandon, Newton, & Harman, 1993). The relative lack of discussion of this component may reflect the infrequent Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

34

American Journal of Evaluation 35(1) participation of beneficiaries in evaluation. Practitioners interested in involving project beneficiaries might refer to the appendix at http://aje.sagepub.com/supplemental or http://goo.gl/ W8oMnl for guidance. Knowing more about the planning, implementation, and effects of beneficiary involvement would be a significant contribution to the literature.

Stakeholder Involvement Components and Component Features We show the 10 components and 76 component features (with the number of articles addressing each) in Table 3. We offer several reflections and conclusions about the list of components and component features: 1.

2.

3.

4.

5.

We found considerable overlap in the components and component features that the studies addressed. The percentage of articles addressing the 10 components ranged from 48% to 78%, showing the commonality among researchers’ conceptions of stakeholder involvement. Certainly this is due in part to the theoretical conceptualizations underlying the many approaches to evaluation that have been developed over the years, but it is also due in part to the features and exigencies of stakeholder involvement in practice. Stakeholder involvement as a means of promoting evaluation use, as reflected in the Evaluation Use and Influence Component, is one of the rare topics in the evaluation literature that has been studied repeatedly. However, it might be suggested that the general topic deserves more nuanced examination in the empirical literature. Researchers might be well advised to examine some of the less studied aspects of use and influence that are shown in the list of component features (e.g., stakeholder learning about other than the evaluand, the level of responsiveness of program personnel, stakeholders’ future willingness to participate in evaluations, and awareness of diversity issues). Such topics are sometimes addressed in reflective narratives about collaborative evaluations (e.g., see Cousins & Chouinard, 2012); they deserve to be addressed in other forms of research including those using interviews, questionnaires about evaluation practice, and simulations. That more comments were made about instrumental use than symbolic, conceptual, or process use (three of the component features in the Evaluation Use and Influence Component) is not surprising, given its early emphasis on the literature (e.g., Mark & Shotland, 1985). The relatively large number of articles addressing process use is somewhat surprising, because researchers have studied this form of use only in the past few years. We believe that the literature produced after the period of our review is continuing this trend. The two components, Affective Aspects of Involvement and Collaboration, Communication, and Interaction, reflect much of the methodological center of stakeholder involvement, in which parties on both sides must enter into collaboration with the appropriate degree of willingness to participate as fully and forthrightly as necessary, drawing on the strengths of each while respecting the positions and expertise of each other. As reflected in King and Stevahn’s (2012) recent volume on interaction in evaluation, stakeholder involvement in evaluation requires psychological sophistication in ways that are unnecessary in evaluations that do not involve stakeholders substantially. Because of the complexity of stakeholder–evaluator interaction and collaboration among stakeholders, researchers have room to do considerably more research on the topic. Issues of affect have not surfaced as the primary focus of many commentaries about stakeholder involvement over the years. The component features listed in the Evaluator Characteristics Component clearly indicate the importance of not only evaluator technical skill but interpersonal skills and leadership as well. Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014


35

Table 3. Number and Percentage of Articles Addressing the Stakeholder Involvement Components and Component Features.

Components and component features Evaluation use and influence 1. Instrumental use of findings 2. Process use 3. General comments about effects of stakeholder involvement on use or on the level of use 4. Enhanced knowledge of evaluand or organization 5. Effects of stakeholder feedback on evaluation findings, recommendations or decisions 6. Conceptual use of findings 7. General empowerment issues 8. Symbolic use of findings 9. Effects on advocating and getting funding for programs 10. Stakeholders’ expectations that involvement would help solve problems 11. Stakeholder learning about other than the evaluand 12. Level of responsiveness of program personnel 13. Stakeholders’ future willingness to participate 14. Awareness of diversity issues Affective aspects of involvement 1. Motivation or commitment issues 2. Stakeholder satisfaction with involvement 3. Stakeholders felt input was valued 4. Degree of appreciation of evaluation 5. Insufficient or wavering levels of stakeholder involvement 6. Stakeholder turnover 7. Trust issues 8. Stakeholder involvement was perceived to be imposed 9. Participation limited because of stresses in stakeholders’ lives Resource issues 1. Time or stakeholder duties 2. Degree of stakeholders’ evaluation experience, knowledge, skills, or training 3. General resource topics 4. Lack of incentives for stakeholder participation 5. Reduced level of technical quality of evaluation 6. Problems with technology

Total articlesa

Positive findingsb

32 (78%) 19 (46%) 13 (68%) 16 (39%) 12 (75%) 14 (34%) 6 (43%) 8 (20%) 8 (20%)

8 (100%) 4 (50%)

7 (17%) 5 (12%) 4 (10%) 3 (7%)

7 (100%) 5 (100%)

5 (12%) 5 (12%) 3 (7%)

5 (26%) 4 (25%) 6 (43%)

1 (5%)

1 (13%)

3 (38%)

3 (75%)

1 (25%)

2 (67%)

1 (33%)

3 2 2 1

(100%) (100%) (100%) (100%)

5 7 9 7

(38%) (64%) (100%) (88%)

6 (46%) 4 (36%)

7 (54%)

8 (20%) 3 (7%) 3 (7%) 2 (5%)

Collaboration, communication, and interaction 26 (63%) 1. Quality of communication and interaction 11 (27%) 2. Collaboration throughout the process 8 (20) 3. Sharing of stakeholder knowledge and expertise 6 (15%) 4. Evaluators’ and stakeholders’ differing perceptions 6 (15%) of program or evaluation purpose, philosophy, goals, or expectations

7 (50%)

1 (13%) 6 (100%) 5 (100%)

2 (40%) 2 (67%)

2 (5%) 26 (63%) 22 (54%) 9 (21%)

Explanatory or normative findingsb

3 (100%)

3 (7%) 3 (7%) 2 (5%) 2 (5%) 1 (2%) 27 (66%) 13 (32%) 11 (27%) 9 (22%) 8 (20%) 6 (15%)

Negative/ mixed findingsb

4 (80%) 1 (33%) 2 (100%)

3 (14%)

15 (68%) 5 (56%)

8 (36%) 8 (89%)

1 (13%)

5 (63%)

5 (63%) 3 (100%) 1 (33%)

2 (67%) 2 (100%) 7 1 6 1

(64%) (13%) (100%) (17%)

4 (36%) 7 (88%) 4 (67%)

2 (33%)

(continued) Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

36


Table 3. (continued)

Components and component features 5. Enhancements in communication, sharing, or networking 6. Communication issues 7. Instructions about how to be involved Equity and bias 1. The democratic and equitable nature of stakeholders’ involvement in the evaluation 2. General potential for bias 3. Inequitable involvement; domineering stakeholders; power issues 4. Representativeness of stakeholders 5. Lack of stakeholders’ forthrightness or honesty

Total articlesa

Positive findingsb

5 (12%)

4 (80%)

4 (10%) 1 (2%) 26 (63%) 16 (39%) 11 (69%)

Explanatory or normative findingsb 1 (20%)

3 (75%)

1 (25%) 1 (100%)

3 (19%)

4 (25%)

6 (86%) 4 (58%)

3 (43%) 4 (58%)

3 (50%)

2 (33%) 4 (67%)

1 (17%) 3 (50%)

2 (20%) 2 (25%) 5 (100%)

6 (60%) 4 (50%)

2 (20%) 2 (25%)

1 (25%) 4 (100%) 3 (100%)

3 (75%)

7 (17%) 7 (17%) 6 (15%) 6 (15%)

Negative/ mixed findingsb

Degree and kind of stakeholder involvement 1. General comments about degree of involvement 2. Involvement of stakeholders in various roles 3. Diverse stakeholder input 4. Issues about organizational or political context 5. Lack of buy-in or support 6. Bureaucratic or career impediments to involvement 7. Overreliance on the evaluator

23 (56%) 10 (24%) 8 (20%) 5 (12%) 4 (10%) 4 (10%) 3 (7%)

Evaluation capacity building 1. Level of or changes in evaluation capacity 2. Stakeholder training topics 3. Learning opportunity 4. Level of satisfaction with evaluation capacity building activities Evaluator characteristics 1. Level or importance of evaluator methodological expertise 2. Evaluator communication, negotiation, or facilitation skills 3. Evaluator role 4. Evaluator leadership 5. Evaluator level of involvement 6. Evaluator background Evaluation quality 1. Credible evaluation findings 2. Quality of evaluation design or methods 3. Reporting findings 4. General comments about quality Stakeholder group characteristics 1. Primary users’ involvement 2. Powerful groups’ or decision makers’ involvement 3. Program beneficiaries’ involvement 4. Program size

20 (49%) 17 (42%) 14 (82%) 7 (17%) 2 (29%) 4 (10%) 3 (75%) 2 (5%) 2 (100%)

2 (5%)

1 (33%) 2 (100%)

3 (18%)

1 (6%) 5 (71%) 1 (25%)

20 (49%) 11 (27%)

5 (45%)

3 (27%)

6 (55%)

7 (17%)

3 (43%)

1 (14%)

4 (57%)

6 (15%) 6 (15%) 3 (7%) 3 (7%) 18 (44%) 9 (22%) 7 (17%) 6 (15%) 5 (12%) 18 (44%) 7 (17%) 7 (17%) 7 (17%) 5 (12%)

3 (50%) 5 (83%)

2 (33%)

4 (67%) 2 (33%) 3 (100%) 2 (67%)

2 (67%) 1 (33%) 8 3 3 4

(89%) (43%) (50%) (80%)

1 (14%) 3 (43%) 3 (43%)

1 (11%) 5 (71%) 4 (67%) 1 (20%)

2 (29%) 1 (14%)

6 (86%) 2 (29%) 3 (43%) 5 (100%)

a Percentage of all 41 articles that addressed the component or component feature. bPercentage of the articles addressing the component feature. Percentages across the three types of findings add to more than 100 when articles had more than one type of findings about the component feature. Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

Brandon and Fukunaga 6.

7.

8.

37

Given that the quality of an evaluation is crucial to producing valid or useful results, the relative lack of attention to it compared with the other nine components, as shown in the relatively small number of articles addressing the Evaluation Quality Component, is surprising. The door is open for researchers to examine the degree to which stakeholder involvement affects quality, perhaps by considering other aspects such as bias, affective issues, and communication or collaboration issues. It is surprising and disappointing that we did not find more studies systematically examining the degree of stakeholder involvement. (See the results for the Degree and Kind of Stakeholder Involvement Component in Table 3.) Descriptions of how stakeholders were involved were provided in all the studies, but details about the extent to which they were involved were often not reported. Brandon and Singh (2009) similarly found that the authors of only 10% of the empirical studies of evaluation use discussed the level or the extent of use. The degree of involvement can be inferred from the descriptions provided about involvement, but more specific quantitative findings would lend support to claims about the importance and effects of involvement. By its omissions, the list of components and component features shows aspects of stakeholder involvement that remain to be examined in systematic empirical studies. It does not include, for example, anything about the effects of stakeholder involvement on social justice, even though that is a primary purpose of stakeholder involvement mentioned in the literature. Perhaps social justice is a difficult and diffuse construct to measure as an effect of involving stakeholders. Likewise, the list does not include cultural competence—a notable omission, probably due to the relatively recent emphasis on the topic in the literature and by professional associations. It simply had not been discussed in the empirical stakeholder involvement literature published from 1985 to 2010. These and other features of stakeholder involvement deserve attention in empirical studies.

Types of Findings (Positive Results, Negative or Mixed Results, and Explanatory/Normative Results) We show the distribution of the types of findings for each component feature in Table 3. In the appendix available at http://aje.sagepub.com/supplemental or http://goo.gl/W8oMnl, we provide detailed summaries of the components and component features. The appendix results show accolades for some aspects of stakeholder involvement and present cautions about others, and they allow evaluation practitioners, theorists, and researchers to examine these features in considerable depth. The explanatory and normative findings are likely to be the most elucidating for practitioners planning to involve stakeholders, as they provide suggestions about contextual issues and descriptions of the nuances of involving stakeholders. The positive and negative or mixed-result findings are descriptive, whereas the explanatory and normative findings address the issue of how evaluators might involve stakeholders for maximum benefit. The positive findings are most often accounts that laud stakeholder involvement and are probably those that receive the most attention in the literature. The negative or mixed-result findings are likely to be helpful by pointing out some of the difficulties that evaluators committed to involvement might encounter. We note several key observations about these findings: 1.

The substantial proportion of results in Table 3 and in the appendix that provide explanatory or normative conclusions for the Evaluation Use and Influence Component (compared to the proportion of these findings for most of the other components) reflects, we believe, the importance that authors have placed on understanding the use of evaluation findings and on promoting ways to enhance their use. Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

38

American Journal of Evaluation 35(1) 2.

3.

4.

5.

The results for the Affective Aspects of Involvement Component were fairly evenly distributed between positive and negative/mixed findings. In several studies, the authors commented on challenges in maintaining consistent motivation and involvement over time due to involving large numbers of diverse stakeholder groups, staff turnover, limited willingness to participate, or the amount of time necessary to commit when participating in evaluations. The affective aspects of stakeholder involvement are a potential strength, yet the empirical research that we examined shows risks as well. The findings about resources show more negative/mixed-result findings than positive ones. The comments about resource issues sometimes stood out when we reviewed articles because of their harshly negative slant. The extra demands on evaluators and program personnel when involving stakeholders in program evaluations require a considerable time commitment and, in some cases, material resources. These demands tend to be a source of contention, especially when they exceed the expectations of the participants. Thus, not surprisingly, evaluators need to be aware of the potential problems about resource issues when they involve stakeholders. Future studies might address ways to remedy these issues. For example, stakeholders’ experience, knowledge, skills, and training can affect the success of evaluations in which stakeholders are involved. Other ways to address the issues that researchers might study and that practitioners might attend to can be found in the appendix. The negative or mixed-result findings in the Equity and Bias Component far outnumbered the positive findings, perhaps reflecting the potential for inadequate degrees of equity when stakeholders are involved. The explanatory or normative findings suggest that taking steps to assure equity and lack of bias has required evaluators to pay specific and considerable attention to power imbalances, representative participation, and organizational climate. Equity issues are similar to resource issues, in the sense that both are in the background of involvement and can be overlooked, but without equitable participation among stakeholder groups, involvement can be a sham. In our view, the potential for problems of this nature is not reflected in the exuberance for stakeholder involvement in the professional literature about evaluation approaches that promote collaboration among stakeholders. (To the authors’ credit, Cousins & Chouinard’s 2012 volume is an exception.) Much more could be done to examine these issues, including not only the extent to which they exist but also the methods for addressing them. Qualitative studies in particular could identify ways to help ensure that the potential for bias is minimized. Unlike all the other components (with the exception of Evaluation Use and Influence) that we identified in the review, most of the comments in the Evaluator Characteristics Component are of an explanatory or normative nature and include guidance that might assist evaluators when involving stakeholders.

Limitations We note two limitations of our study. First, although we are confident that we identified all the articles with the terms stakeholder and evaluation in the title, abstract, or keywords, we also know from our own research that other articles about involvement were not identified (e.g, Brandon, Lindberg, & Wang, 1993; Brandon, Wang, & Heck, 1994; Brandon & Heck, 1998).We purposefully focused on the reports whose authors intended to explicitly address stakeholder involvement, since they were most likely to contribute fully to the literature and because others would be much more difficult to identify. Some valid and useful findings might have been shown in reflective narratives. However, as we have described, without knowledge of data collection and analysis procedures, we know little or nothing about the extent to which the results of such narratives are subject to various biases and Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014


39

errors such as evaluator recall or social desirability. Furthermore, reflective narratives are very difficult to summarize across studies, raising serious questions about the unit of analysis. Lacking sound answers to these questions is one reason why we chose to exclude reflective narratives from our study. Second, some of the component features we identified are not completely discrete (e.g., Affective Aspects of Involvement overlaps with Collaboration, Communication, and Interaction). Taxonomies with nonoverlapping schema are notoriously difficult to construct and a challenge inherent to qualitative summaries of research findings. In addition, the distribution of themes reported in the findings does not necessarily reflect their distribution in actual practice; instead, it might reflect the purposes of the articles. Articles about the use of evaluation findings, for example, could reflect a publication bias toward positive reports about evaluation use. Reviewers of the literature sometimes summarize their results in the form of guidelines and recommendations for evaluators. Some readers of the present article (such as one anonymous reviewer) might consider the lack of these here to be a limitation of our study. We did not go beyond the categorization of findings into components and component features because we believe that the number of reviewed studies is too small and the details of the findings are too nuanced and varied. Our purpose is simply to provide an empirical foundation for others researching or practicing stakeholder involvement, with the hope that more generalizable findings based on sound research ultimately will constitute much of the empirical research. In the meantime, the appendix at http://aje.sagepub.com/ supplemental or http://goo.gl/W8oMnl provides considerable detail for drawing conclusions appropriate for researchers’ and practitioners’ purposes and relevant to varied research and practice contexts.

Conclusions Our review of the state of the empirical research on stakeholder involvement in evaluation suggests a body of work that is far more limited than desirable, uses fairly simple designs and data collection methods, and occurs mostly in a few professions or disciplines but nevertheless has contributed considerably to our knowledge of the topic. It is the first grounded-theory review of the literature on the topic that has addressed strict inclusion criteria, includes a detailed description of major topics (i.e., components and component features), and provides useful information about the strengths and weaknesses of involving stakeholders. The small number of studies reporting original research on stakeholder involvement, outside of actual evaluations, reflects the lack of empirical RoE in general—more accurately, perhaps, the lack of funding for such research. History suggests that such funding is often not generous or even available, thus leaving RoE to academic evaluators with the time to conduct the research. Fortunately, this work has grown in recent years. For example, a review of stakeholder involvement among people with disabilities was recently published (Jacobson, Azzam, & Baez, 2013), as was one reporting survey findings on RoE itself (Szanyi, Azzam, & Galen, 2013). Indeed, of the articles published in the American Journal of Evaluation from 1998 (when the journal took its current name) through 2011, 219 (35%) were classified as RoE (Brandon, Vallin, & Philippoff, 2012). Furthermore, the preponderance of case studies stemming from the actual evaluations that we reviewed, with their rich descriptive findings, suggests that practicing evaluators can add considerably to the ongoing discussion about stakeholder involvement. We hope that studies using additional designs—perhaps even causal methods—will be added to the compendium of studies in the future. The findings on stakeholder involvement beyond those reviewed here include the gray literature and published reflective narratives, which report results of evaluators’ ruminations on the topic or insights that they believe can contribute novel, confirmatory, or disconfirmatory results to the literature. We chose not to include reflective narratives primarily because they provide no information allowing reviewers to judge the quality of their analysis and reporting methods. We believe, Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

40


however, that findings about stakeholder involvement in reflective narratives might help elucidate the components and component features that we present here or serve to triangulate and elaborate on the positive, negative/mixed, and explanatory/normative findings. As Cousins and Chouinard (2012) noted, the participant observers who report reflective narratives typically provide insights from actual practice, present the narratives in an accessible narrative form, and build their reports upon well-founded constructivist learning principles. We do not claim that the themes we identified in this review were uninfluenced by previous reviews, theorists’ frameworks, or practitioners’ insights published elsewhere. For example, Resource Issues, Evaluator Characteristics, and Stakeholder Group Characteristics are universally considered by evaluators of all theoretical stripes (e.g., see Rog, Fitzpatrick, & Conner, 2012), and Use and Influence and Evaluation Capacity Building are the primary features of utilization-focused evaluation and participatory evaluation. Some of our components reflect the dimensions of stakeholder participation that Vo (2013) described in her review of three evaluation approaches. Nevertheless, our conclusions are more firmly grounded in empirical results than other reviews of the literature on stakeholder involvement. The results show that most of the components were addressed in half or nearly half of the studies, suggesting that evaluators might consider them when involving stakeholders in their studies or conducting research on the topic. The relative lack of discussion of some topics, such as evaluation quality, equity and bias, and the degree to which stakeholders are involved, suggests deficits that researchers might consider addressing. Other topics have not been addressed at all—for example, cultural competence comes to mind. No doubt we have overlooked others. The categorization of the results into positive findings, negative/mixed-result findings, and explanatory or normative findings provides insights into some component features that we believe others have not fully discussed elsewhere. For example, most of the findings about resources were negative, reflecting a potential significant difficulty in involving stakeholders. Similarly, we found more negative and mixed-result findings than positive findings about equity and bias. The results for evaluation quality tended to be more positive, showing that the inclusion of stakeholders can serve to enhance an evaluation and its findings. Unlike other discussions of these issues, our comparison of the three types of findings readily lends itself to enumerating the relative frequency of each. The explanatory and normative findings provide elaborations on the meaning of the component features and help interpret the positive and negative or mixed-result findings. In the chronicles of program evaluation, stakeholder involvement is a venerable topic. The empirical research on the topic provides numerous insights but deserves to be conducted more systematically, thoroughly, and frequently. We hope that our study provides a foundation for researchers selecting involvement-related topics on which to focus and encourages studies of the topic in new settings.

Appendix Detailed Findings About Stakeholder Involvement (http://aje.sagepub.com/supplemental or http://goo.gl/W8oMnl). Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding The author(s) received no financial support for the research, authorship, and/or publication of this article. Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014


41

References American Evaluation Association, Task Force on Guiding Principles for Evaluators. (1995). Guiding principles for evaluators. In W. R. Shadish, D. L. Newman, M. A. Scheirer, & C. Wye (Eds.), Guiding principles for evaluators. New Directions for Program Evaluation, 66, 19–16. Brandon, P. R. (1999). Involving program stakeholders in reviewing evaluators’ recommendations for program revisions. Evaluation and Program Planning, 22, 363–372. Brandon, P. R. (2012, April). Ruminations on research on evaluation. Paper presented at the meeting of the American Educational Research Association, Vancouver, Canada. Brandon, P. R., & Heck, R. H. (1998). The use of teacher expertise in decision making during school-conducted needs assessments: A multilevel perspective. Evaluation and Program Planning, 21, 323–331. Brandon, P. R., Lindberg, M. A., & Wang, Z. (1993). Involving program beneficiaries in the early stages of evaluation: Issues of consequential validity and influence. Educational Evaluation and Policy Analysis, 15, 420–428. Brandon, P. R., Newton, B. J., & Harman, J. W. (1993). Enhancing validity through beneficiaries’ equitable involvement in identifying and prioritizing homeless children’s educational problems. Evaluation and Program Planning, 16, 287–293. Brandon, P. R., & Singh, J. M. (2009). The strength of the methodological warrants for the findings of research on program evaluation. American Journal of Evaluation, 30, 123–157. Brandon, P. R., Vallin, L. M., & Philippoff, J. (2012, October). A quantitative summary of research on evaluation published in the American Journal of Evaluation from 1998 through 2011. Paper presented at the meeting of the American Evaluation Association, Minneapolis, MN. Brandon, P. R., Wang, Z., & Heck, R. H. (1994). Teacher involvement in school-conducted needs assessments: Issues of decision-making process and validity. Evaluation Review, 18, 458–471. Bryk, A. (Ed.). (1983). Stakeholder-based evaluation. New Directions for Program Evaluation, 17. San Francisco, CA: Jossey-Bass. Canadian Evaluation Society. (2010). Competencies of Canadian evaluation practice. Retrieved from http:// www.evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf Chouinard, J. A., & Cousins, J. B. (2009). A review and synthesis of current research on cross-cultural evaluation. American Journal of Evaluation, 30, 457–494. Christie, C. A., & Fleischer, D. N. (2010). Insight into evaluation practice: A content analysis of designs and methods used in evaluation studies published in North American evaluation-focused journals. American Journal of Evaluation, 31, 326–346. Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schroeter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32, 199–226. Cousins, J. B. (2003). Utilization effects of participatory evaluation. In T. Kellaghan, D. L. Stufflebeam, & L. A. Wingate (Eds.), International Handbook of Educational Evaluation (pp. 245–266). Boston: Kluwer. Cousins, J. B., & Chouinard, J. A. (2012). Participatory evaluation up close: An integration of research-based knowledge. Charlotte, NC: Information Age. Datta, L. (2006). The practice of evaluation: In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), The SAGE handbook of evaluation (pp. 419–428). Thousand Oaks, CA: Sage. Datta, L. (2011). Politics and evaluation: More than methodology. American Journal of Evaluation, 32, 273–294. Farrar, E., & House, E. R. (1983). The evaluation of Push/Excel: A case study. In A. S. Bryk (Ed.), Stakeholderbased evaluation. New Directions for Program Evaluation, 17, 31–58. Fetterman, D. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage. Gold, N. (1981). The stakeholder process in educational program evaluation. Washington, DC: National Institute of Education. Gold, N. (1983). Stakeholders and program evaluation: characterizations and reflections. In A. S. Bryk (Ed.), Stakeholder-based evaluation. New Directions for Program Evaluation, 17, 63–72.


42


Greene, J. C. (1987). Stakeholder participation in evaluation design: Is it worth the effort? Evaluation and Program Planning, 10, 375–394. House, E. R. (1991). Evaluation and social justice: Where are we? In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: At quarter century (pp. 233–247). Chicago, IL: National Society for the Study of Education. House, E. R., & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks, CA: Sage. Jacobson, M. R., Azzam, T., & Baez, J. E. (2013). The nature and frequency of inclusion of people with disabilities in program evaluation. American Journal of Evaluation, 34, 23–44. Johnson, K., Greenseid, L., Toal, S. A., King, J. A., Lawrenz, F. & Volkov, B. (2009). Research on evaluation use: A review of the empirical literature from 1986–2005. American Journal of Evaluation, 30, 377–410. King, J. A., & Stevahn, L. (2012). Interactive evaluation practice: Managing the interpersonal dynamics of program evaluation. Newbury Park, CA: Sage. Labin, S. N. (2008). Research synthesis: Toward broad-based evidence. In N. Smith & P. R. Brandon (Eds.), Fundamental issues in evaluation (pp. 89–110). New York, NY: Guilford. Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338. Mark, M. M. (2008). Building a better evidence for evaluation theory: Beyond general calls to a framework of types of research on evaluation. In N. L. Smith & P. R. Brandon (Eds.), Fundamental issues in evaluation (pp. 111–134). New York, NY: Guilford. Mark, M. M., & Shotland, L. (1985). Stakeholder-based evaluation and value judgments. Evaluation Review, 9, 605–626. Mertens, D. M. (2009). Transformative research and evaluation. New York, NY: Guilford. Miller, R. L., & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296–319. Patton, M. Q. (1978). Utilization-focused evaluation. Beverly Hills, CA: Sage. Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Los Angeles, CA: Sage. Peck, L., Kim, Y., & Lucio, J. (2012). An empirical examination of validity in evaluation. American Journal of Evaluation, 33, 350–365. Rog, D. J., Fitzpatrick, J. L., & Conner, R. (Eds.). (2012). Context and evaluation. New Directions for Evaluation, 135. San Francisco: Jossey-Bass. Ross, J. A., Barkaoui, K., & Scott, G. (2007). Evaluations that consider the cost of educational programs: The contribution of high quality studies. American Journal of Evaluation, 28, 477–492. Ryan, G. W., & Bernard, H. R. (2003). Techniques to identify themes. Field Methods, 15, 85–109. Smith, N. L. (1993). Improving evaluation through the empirical study of evaluation practice. Evaluation Practice, 14, 237–242. Stake, R. E. (1976). A theoretical statement of responsive evaluation. Studies in Educational Evaluation, 2, 19–22. Stake, R. E. (1983). Stakeholder influence in the evaluation of Cities-in-Schools. New Directions for Program Evaluation, 17, 15–30. Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43–59. Szanyi, M., Azzam, T., & Galen, M. (2013). Research on evaluation: A needs assessment. Canadian Journal of Program Evaluation, 27, 39–64. Taut, , S. (2008). What have we learned about stakeholder involvement in program evaluation? Studies in Education Evaluation, 34, 224–230. Trevisan, M. S. (2004). Practical training in evaluation: A literature review. American Journal of Evaluation, 25, 255–272. Tyler, R. W. (1987). Education: Curriculum development and evaluation, an oral history conducted 1985– 1987 by Malca Chall. Berkeley, CA: University of California, Bancroft Library, Regional Oral History Office. Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014


43

Tyler, R. W. (1991). General statement on program evaluation. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and Education: At quarter century (pp. 3–17). Chicago, IL: National Society for the Study of Education. Vo, A. (2013). Visualizing context through theory deconstruction: A content analysis of three bodies of evaluation theory literature. Evaluation and Program Planning, 38, 44–52. Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.

Articles Examined in the Literature Review Alpert, B., & Bechar, S. (2007). Collaborative evaluation research: A case study of teachers’ and academic researchers’ teamwork in a secondary school. Studies in Educational Evaluation, 33, 229–257. Arden, C., McLachlan, K., & Cooper, T. (2009). Building capacity through sustainable engagement: Lessons for the learning community from the GraniteNet Project. Australian Journal of Adult Learning, 49, 75–101. Ayers, T. D. (1987). Stakeholders as partners in evaluation: A stakeholder-collaborative approach. Evaluation and Program Planning, 10, 263–271. Azzam, T. (2010). Evaluator responsiveness to stakeholders. American Journal of Evaluation, 31, 45–65. Brandon, P. R. (1999). Involving program stakeholders in reviews of evaluators’ recommendations for program revisions. Evaluation and Program Planning, 22, 363–372. Campbell, B., & Mark, M. M. (2006). Toward more effective stakeholder dialogue: Applying theories of negotiation to policy and program evaluation. Journal of Applied Social Psychology, 36, 2834–2863. Campbell, R., Dorey, H., Naegeli, M., Grubstein, K. L., Bennett, K. K., Bonter, F., . . . Davidson, W.S. (2004). An empowerment evaluation model for sexual assault programs: Empirical evidence of effectiveness. American Journal of Community Psychology, 34, 251–262. Cartland, J., Ruch-Ross, H. S., Mason, M., & Donohou, W. (2008). Role sharing between evaluators and stakeholders in practice. American Journal of Evaluation, 29, 460–477. Chen, S., Poland, B., & Skinner, H. A. (2007). Youth voices: Evaluation of participatory action research. Canadian Journal of Program Evaluation, 22, 125–150. Cousins, J. B. (1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational Evaluation, 22, 3–27. Cousins, J. B. (2001). Do evaluator and program practitioner perspectives converge in collaborative evaluation? Canadian Journal of Program Evaluation, 16, 113–133. Cousins, J. B., Donohue, J. J., & Bloom, G. A. (1996). Collaborative evaluation in North America: Evaluators’ self-reported opinions, practices, and consequences. Evaluation Practice, 17, 207–226. Curran, V., Christopher, J., Lemire, F., Collins, A., & Barret, B. (2003). Application of a responsive evaluation approach in medical education. Medical Education, 37, 256–266. Davidsdottir, S., & Lisi, P. (2007). Effects of deliberative democracy on school self-evaluation. Evaluation, 13, 371–386. Fleischer, D. N., & Christie, C. A. (2009). Evaluation use: Results from a survey of U.S. American evaluation association members. American Journal of Evaluation, 30, 158–175. Forss, K., Kruse, S. E., Taut, S., & Tenden, E. (2006). Chasing a ghost? An essay on participatory evaluation and capacity development. Evaluation, 12, 128–144. Gibbs, D. A., Hawkins, S. R., Clinton-Sherrod, M., & Noonan, R. K. (2009). Empowering programs with evaluation technical assistance: Outcomes and lessons learned. Health Promotion Practice, 10, 38–44. Greene, J. C. (1987). Stakeholder participation in evaluation design: Is it worth the effort? Evaluation and Program Planning, 10, 379–394. Greene, J. C. (1988). Communication of results and utilization in participatory evaluation. Evaluation and Program Planning, 11, 341–351. Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014

44


Greene, J. G. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12, 91–116. Henry, G. T., Dickey, K. C., & Areson, J. C. (1991). Stakeholder participation in educational performance monitoring systems. Educational Evaluation and Policy Analysis, 13, 177–188. Iutcovich, J. M. (1993). Assessing the needs of rural elderly: An empowerment model. Evaluation and Program Planning, 16, 95–107. Jivanjee, P., & Robinson, A. (2007). Studying family participation in system-of-care evaluations: Using qualitative methods to examine a national mandate in local contexts. Journal of Behavioral Health Services and Research, 34, 369–381. Johnson, R. L., Willeke, M. J., & Steiner, D. J. (1998). Stakeholder collaboration in the design and implementation of a family literacy portfolio assessment. American Journal of Evaluation, 19, 339–353. Lennie, J. (2005). An evaluation capacity-building process for sustainable community IT initiatives: Empowering and disempowering impacts. Evaluation, 11, 390–414. MacLellan-Wright, M. F., Patten, S., Dela Cruz, A. M., & Flaherty, A. (2007). A participatory approach to the development of an evaluation framework: Process, pitfalls, and payoffs. Canadian Journal of Program Evaluation, 22, 99–124. Mayberry, R. M., Daniels, P., Yancey, E. M., Akintobi, T. H., Berry, J., Clark, N., & Dawaghreh, A. (2009). Enhancing community-based organizations’ capacity for HIV/AIDS education and prevention. Evaluation and Program Planning, 32, 213–220. McDuff, M. D. (2001). Building the capacity of grassroots conservation organizations to conduct participatory evaluation. Environmental Management, 27, 715–727. Miller, R. W. (1987). Using evaluation to support the program advisory function: A case study of evaluatorprogram advisory-committee collaboration. Evaluation and Program Planning, 10, 281–288. Northcote, J., Lee, D., Chok, S., & Wagner, A. (2008). An email-based Delphi approach to tourism program evaluation: Involving stakeholders in research design. Current Issues in Tourism, 11, 269–279. Papineau, D., & Kiely, M. C. (1996). Participatory evaluation in a community organization: Fostering stakeholder empowerment and utilization. Evaluation and Program Planning, 19, 19–93. Parkinson, S. (2009). Power and perceptions in participatory monitoring and evaluation. Evaluation and Program Planning, 32, 229–237. Poth, C. A., & Shulha, L. (2008). Encouraging stakeholder engagement: A case study of evaluator behavior. Studies in Educational Evaluation, 34, 218–223. Preskill, H., & Caracelli, V. (1997). Current and developing conceptions of use: Evaluation use TIG survey results. Evaluation Practice, 18, 209–225. Quintanilla, G., & Packard, T. (2002). A participatory evaluation of an inner-city science enrichment program. Evaluation and Program Planning, 25, 15–22. Ribisl, K. M., Steckler, A., Linnan, L., Patterson, C. C., Pevzner, E. S., Markatos, E., . . . Peterson, A. B. (2004). The North Carolina Youth Empowerment Study (NCYES): A participatory research study examining the impact of youth empowerment for tobacco use prevention. Health Education & Behavior, 31, 597–614. Rice, M., & Franceschini, M. C. (2007). Lessons learned from the application of a participatory evaluation methodology to healthy municipalities, cities and communities initiatives in selected countries of the Americas. Promotion & Education, 14, 68–73. Ross, J. A., & Jaafar, S. B. (2006). Participatory needs assessment. Canadian Journal of Program Evaluation, 21, 131–154. Thayer, C. E., & Fine, A. H. (2001). Evaluation and outcome measurement in the non-profit sector: stakeholder participation. Evaluation and Program Planning, 24, 103–108. Toal, S. A., King, J. A., Johnson, K., & Lawrenz, F. (2009). The unique character of involvement in multi-site evaluation settings. Evaluation and Program Planning, 32, 91–98. Turnbull, B. (1999). The mediating effect of participation efficacy on evaluation use. Evaluation and Program Planning, 22, 131–140. Downloaded from aje.sagepub.com at American Evaluation Association on April 3, 2014