“You Get the Great Feeling That You’re Being Heard But in the End You Realize That Things Will Be Done Differently and in Others’ Favor”: An Experimental Investigation of Negative Effects of Participation in Evaluation

Abstract

Practical-participatory evaluation is often considered an effective approach to enhancing evaluation use. Despite this generally positive connotation, potential negative effects of participatory evaluation have also been discussed in the literature but never empirically investigated before. Drawing upon the social–psychological procedural justice literature, we assumed that stakeholders’ perceptions about the justice of an evaluation might explain possible negative effects of participatory evaluation on evaluation use. Negative effects of participatory evaluation on instrumental and conceptual use were explored in two experiments (Study 1: N = 85; Study 2: N = 78). The results show that negative effects of participation on evaluation use are mediated by stakeholders’ justice perceptions. Implications of the present research include considering stakeholders’ socioemotional needs, negotiation, and transparency when conducting participatory evaluation.

Keywords

participatory evaluation negative effects procedural justice experimental approach

Programs, interventions, and initiatives are increasingly being required to address accountability and development in our complex, fast-changing world. Scholars and evaluation practitioners have shifted from seeing stakeholders as the mere recipients of evaluation to seeing them as active contributors, since they are the ones who will eventually utilize the evaluation processes or results in their programs. This latter form of evaluation is called participatory evaluation. Participatory evaluation is generally valued for its positive effects on numerous outcomes, including satisfaction with the evaluation (Turnbull, 1999), feelings of commitment and ownership toward the evaluation (Rodríguez-Campos, 2005; Taut, 2007), trust and respect (Taut, 2008), fairness (Weiss, 1986), improved validity and credibility (Cullen, Coryn, & Rugh, 2011), and most of all for its positive effect on evaluation use (Brandon, 1998; Cousins & Chouinard, 2012; Patton, 2008). The majority of the scholarly literature on participatory evaluation promises that participatory evaluation should indeed positively contribute to evaluation use. But what if participatory evaluation cannot keep this promise?

Consider a client who asks an evaluator for a participatory evaluation but does not allow stakeholders to actually influence decisions important to them. For example, program staff are invited to participate in framing the evaluation questions, collecting the data, and interpreting the raw data in cooperation with the evaluator. But when it comes to determining how the evaluation will affect the program, the client makes a unilateral decision—without considering or after only marginally considering the evaluation results. How would participatory evaluation be associated with evaluation use in such an instance? Despite the considerable scholarly efforts made describing the positive relationship between participatory evaluation and evaluation use, two important issues remain: (1) Research on the possible negative effects of participatory evaluation is scarce and (2) the underlying psychological mechanisms behind the effects of participatory evaluation have not yet been empirically investigated.

We wanted to address these gaps in the present research. Accordingly, the aims of the current article are to assess the possible negative effects of participatory evaluation on evaluation use and to better understand the psychological mechanisms that might potentially underlie participatory evaluation effects. This will hopefully sensitize scholars and evaluation practitioners to the importance of taking an informed and critical approach to participatory evaluation and provoke discussion on critical issues in participatory evaluation.

Participatory Evaluation and Evaluation Use

Participatory evaluation is defined as “evaluation in which trained evaluators work in partnership with [program] stakeholders to produce evaluative knowledge” (Cousins & Chouinard, 2012, p. 10). Within the broader framework of participatory evaluation, two important approaches can be identified: While transformative participatory evaluation focuses on empowerment and giving equitable voice to marginalized stakeholder groups (Cousins & Whitmore, 1998; Mertens, 2009), practical participatory evaluation aims at enhancing evaluation use and decision-making (Cousins & Chouinard, 2012; Cousins & Whitmore, 1998). The latter approach in particular has received considerable attention in research on participatory evaluation.

We conceptualize evaluation use in accordance with Alkin and Taut (2003). They integrated (1) two components of evaluation use, that is, instrumental use, the utility of the evaluation for subsequent behavior, and conceptual use, the more indirect utility of the evaluation for learning and attitude changes (Leviton & Hughes, 1981; Shadish, Cook, & Leviton, 1991) with (2) evaluation influence (Henry & Mark, 2003; Kirkhart, 2000; Mark & Henry, 2004), the evaluation’s effect on attitudes, motivations, and behaviors on an individual, interpersonal, or collective level. It is argued that both instrumental use and conceptual use might arise as a consequence of the evaluation process and findings, either immediately after the evaluation or with a delayed effect.

Possible Negative Effects of Participatory Evaluation on Evaluation Use

Numerous scholars have empirically investigated the relationship between participatory evaluation and evaluation use. For example, positive relationships have been found between participatory evaluation and the use of both evaluation processes and results (Fleischer & Christie, 2009; Preskill & Caracelli, 1997), instrumental and conceptual use (Cousins, 1996; Cousins, Donohue, & Bloom, 1996; Jacob, Ouvrard, & Bélanger, 2011; Roseland, Lawrenz, & Thao, 2015), as well as symbolic use, where the evaluation results are used to inform political debate (Greene, 1987, 1988; Turnbull, 1999). Moreover, several literature reviews capture the range of scholarly investigations of the proposed relationship and confirm the predominant perception among professional evaluators and other stakeholders that participatory evaluation is positively associated with evaluation use (Brandon & Fukunaga, 2014; Cousins & Chouinard, 2012; Daigneault, 2014; Johnson et al., 2009).

Compared to such extensive research on the positive effects of participatory evaluation for evaluation use, research on possible mixed or negative effects of participatory evaluation is marginal (but see Daigneault, 2014, for exceptions). Nevertheless, more than 30 years ago, Mark and Shotland (1985) discussed negative stakeholder reactions or attitudes with regard to participatory evaluation. For example, if power over programmatic issues that are important to stakeholders is not shared to the expected level, participatory evaluation can be perceived as a sham, leading to feelings of resentment and dissatisfaction toward the evaluation. This reaction might even be destructive for evaluation use, as stakeholders who feel resentment toward the evaluation might also refuse to take action. This problem is different from misuse or nonuse as described by Alkin and King (2017): Misuse occurs when stakeholders intentionally use an evaluation in unethical ways, while nonuse occurs when stakeholders do not use the evaluation findings at all, either intentionally or unintentionally. The reaction described in this article is indeed intended by the acting stakeholder, but not because he or she planned to behave or feel that way from the outset; rather, it is failed participatory processes that create reactions such as refusing to take action or other, not yet described consequences.

Justice Perceptions as Fundamental Psychological Processes Affecting the Outcomes of Participation

Mark and Shotland (1985) briefly refer to the procedural justice literature when explaining possible negative effects of participatory evaluation. Procedural justice describes the way in which people judge outcomes by means of the perceived fairness of the process that led to these outcomes (Greenberg & Folger, 1983). When a procedure is perceived as fair, there are positive consequences in terms of both attitudes, including satisfaction or organizational commitment (McFarlin & Sweeney, 1992), and behaviors, including job performance or organizational citizenship behavior (Cropanzano, Rupp, Thornton, & Shao, 2016; Moorman & Byrne, 2005). One frequently used experimental manipulation in procedural justice research is to give people voice to generate fair procedures and no voice to generate unfair procedures (e.g., Lind, Kanfer, & Earley, 1990; van den Bos, Wilke, Vermunt, & Lind, 1998). When procedures are perceived as fair, people tend to react positively to outcomes even when those outcomes are unfavorable to them. This is known as the fair process effect (Greenberg & Folger, 1983; van den Bos, Lind, Vermunt, & Wilke, 1997). However, there are situations in which even giving people voice can lead to negative reactions. For example, when people suspect that their input will not be taken into account in the final decisions, when they see that their input did not affect the outcome in the way they had hoped (i.e., the outcome turned out to be somewhat different from what was expected), or when they believe that voice was only induced in order to generate stakeholder acceptance (Potter, 2006a, 2006b), the procedure is seen as pseudo-participation (Goodwin & Ross, 1992) and negative reactions occur. This is known as the frustration effect (Cohen, 1985; Lind & Tyler, 1988). Observed negative reactions include aversion, resentment, and dissatisfaction (Greenberg & Folger, 1983; Potter, 2006a, 2006b) as well as counterproductive working behaviors (Cohen-Charash & Mueller, 2007). These negative reactions may be worse than reactions in situations where no voice was offered at all. They are more destructive because they involve the active and intentional rejection of certain outcomes, processes, or even people initiating the process (Brockner et al., 2007).

In participatory evaluation, stakeholders are given voice, which may be perceived as an indication of procedural justice and its associated positive effects. In cases where the participatory procedure fails (i.e., is viewed as a sham because stakeholders feel that their input did not affect the outcome as much as they expected it to), the stakeholders’ perception of the justice of an evaluation might elicit the frustration effect, negatively affecting outcomes such as evaluation use. Following procedural justice reasoning, it can be hypothesized that if participatory evaluation causes stakeholders to perceive the procedure as unfair (e.g., as pseudo-participation), this perceived injustice will lead to a significant reduction in evaluation use: Desirable actions will not be taken (instrumental use) and attitudes and cognitions toward the evaluation will become negative (conceptual use).

Purpose of the Current Research

Participatory evaluation and especially practical participatory evaluation are generally considered to have positive effects on evaluation use. Possible negative effects of participatory evaluation have also been discussed but never empirically investigated before. Drawing upon the procedural justice literature, it can be hypothesized that pseudo-participation might have destructive consequences for attitudes and behavior—even more so than in the case of participation and even nonparticipation. Assuming that these negative voice effects can also be applied to participatory evaluation, our first hypothesis is:

Hypothesis 1: Pseudo-participatory evaluation leads to less evaluation use (instrumental, conceptual) than participatory evaluation and nonparticipatory evaluation.

Drawing upon the procedural justice literature, we argue that stakeholders’ perceptions about the justice of an evaluation might account for the negative effect of pseudo-participatory evaluation. We thus further hypothesize that:

Hypothesis 2: The effect of pseudo-participatory evaluation on evaluation use (instrumental, conceptual) is mediated by stakeholders’ justice perceptions, such that pseudo-participation leads to lower justice perceptions and this in turn is associated with lower evaluation use (instrumental, conceptual).

These hypotheses will be tested using two different samples: university students (Study1) and a sample with more practical experience (Study 2) in order to ensure participants’ practice understanding of the study context.

Method for Study 1 and Study 2

The results of two online studies with different samples will be reported: one sample consisting of university students, the other of practicing social workers. Both studies use an experimental vignette design in order to maintain as much internal validity and external validity as possible (Aguinis & Bradley, 2014) while acknowledging that such designs cannot capture real-life dynamics essential to participatory evaluation approaches such as personal relationships, shifting ideas during the process, or changing questions and methods.

Procedure and Materials

A single-factor experiment with three levels (participation, pseudo-participation, nonparticipation) was conducted to test the hypotheses. Participants were first introduced to an online scenario: They were asked to put themselves in the position of a school social worker whose activities were being evaluated.¹ The scenario briefly described the aim of the evaluation (e.g., monitoring school social work activities and exploring participants’ satisfaction with the services as well as potential areas for improvement), data collection methods (e.g., questionnaires, interviews, observations), and an evaluation steering committee (academics, representatives of different schools, school social workers, and the municipality).

The participants were then randomly assigned to one of the three conditions (participatory, pseudo-participatory, and nonparticipatory evaluation): In the first and second conditions, participants were told that they were taking part in a workshop where evaluation data would be discussed and recommendations for future school social work would be developed collectively. A set of five recommendations for school social work was then presented to participants as the result of the workshop (e.g., parents should be better engaged by inviting them to open gatherings at the school and better facilities for group pedagogical workshops should be provided). The participants were told that this set of recommendations was based on collective agreement among the workshop participants and would also be documented in a final evaluation report. In the third condition, no workshop was mentioned. Finally, participants in all three conditions received an excerpt from the final evaluation report. In the participatory evaluation condition, the recommendations listed in the evaluation report were identical to the aforementioned recommendations from the workshop. In the second condition, the recommendations listed in the evaluation report were not identical to those from the workshop. For example, instead of saying that parents would be invited to attend open gatherings at the school, the recommendation said that social workers should visit parents in their homes.² This divergence was introduced to induce the perception that the outcome (the evaluation report) differed from what was expected as a consequence of the participatory process (the workshop), making this group the pseudo-participatory evaluation condition. Finally, in the nonparticipatory evaluation condition, the participants received the set of recommendations from the final evaluation report immediately after the initial brief description of the evaluation.

After reading the final evaluation report, participants in all three conditions were asked to respond to a set of items before being informed about the purpose of the study and being carefully debriefed.

Measurements

Manipulation check

Three items were used to test for the pseudo-participation manipulation, for example, “The recommendations in the evaluation report match the recommendations from the workshop” or “The recommendations in the evaluation report are the ones I expected after the workshop” (Study 1: α = .95, Study 2: α = .90).

Dependent variables

Instrumental use was measured using 3 items. Due to the vignette design, the items reflected behavioral intentions instead of actual behavior: “I can easily picture myself working with the recommendations,” “I can easily picture myself utilizing the recommendations,” and “I can easily picture myself changing my practice as a school social worker based on the evaluation” (Study 1: α = .86, Study 2: α = .91). Conceptual use was measured using 4 items: “I learned something new about my school social work through the evaluation,” “I expanded my understanding of my school social work through the evaluation,” “I got new ideas and stimuli through the evaluation,” and “My attitude toward my school social work improved through the evaluation” (Study 1: α = .88, Study 2: α = .94). Items on conceptual use were partly taken, translated, and adapted from Greene (1988).

Mediation variable

Justice perceptions were measured using 2 items, “I think the evaluation process was fair” and “I think the evaluation process was just” (Study 1: r _SB = .97, Study 2: r _SB = .98). All items were answered using 7-point Likert-type scale ranging from 1 = not true at all to 7 = very true. ³

Study 1

Participants

Participants were students at the Department of Psychology, FernUniverstitaet in Hagen, Germany. Participants were invited to take part in the study via the department’s online laboratory website. Of an initial sample of 153, 68 participants were excluded due to response errors such as randomly clicking through the items or incomplete data sets. This led to a sample of N = 85 participants, of which n = 62 were female and n = 23 were male, with an average age of 34 (min = 21, max = 57, SD = 10.5). Participants received course credit for their participation in the study.

Results

For information on intercorrelation of measured variables, see Supplemental Appendix 1, available in the online version of this article.

Manipulation check

The manipulation check tested whether or not the recommendations in the evaluation report were in accordance with the workshop recommendations. Since only participants in the participatory and the pseudo-participatory conditions received information about the workshop, the manipulation check items were presented only to these groups. Thus, the following analyses concern only these two groups. Independent t tests revealed significant differences: Participants in the pseudo-participatory condition perceived the two sets of recommendations as less similar (M = 3.95, SD = 1.71) than participants in the participatory condition (M = 5.59, SD = 1.12): t(55) = 4.38, p = .007, d = 1.16, representing a large effect.⁴

Experimental effects and mediation analysis

The first hypothesis predicted that pseudo-participatory evaluation would lead to less evaluation use (instrumental, conceptual) than participatory and nonparticipatory evaluation. To test this hypothesis, a multivariate analysis of variance (MANOVA) was conducted with the evaluation procedure (pseudo-participatory, participatory, and nonparticipatory evaluation) serving as the independent variable and instrumental use and conceptual use as the dependent variables. The results revealed a multivariate effect for these variables, V = .28, F(4, 164) = 6.60, p < .001. Univariate effects were found for both instrumental use, F(2, 82) = 10.15, p < .001, with an achieved power of .98,⁵ and conceptual use, F(2, 82) = 14.78, p < .001, with an achieved power of .99.

Participants in the pseudo-participation condition reported lower instrumental use (M = 4.19, SD = 1.38) and lower conceptual use (M = 3.53, SD = 1.24) than participants in the participation condition (instrumental use: M = 5.00, SD = 1.07; conceptual use: M = 4.41, SD = 1.20) and in the nonparticipation condition (instrumental use: M = 5.56, SD = 0.84; conceptual use: M = 5.26, SD = 1.02). Planned contrasts for pseudo-participatory evaluation compared to participatory and nonparticipatory evaluation show that these differences in instrumental use and conceptual use are significant, with medium to high effect sizes, instrumental use: t(33.93) = −3.59, p = .001, d = 0.86;⁶ conceptual use: t(82) = −4.75, p < .001, d = 1.13.

To test the second hypothesis, which posits that the effect of pseudo-participatory evaluation on evaluation use (instrumental, conceptual) is mediated by justice perceptions, a mediation analysis was conducted using a bootstrap-based procedure with the SPSS (Version 24) macro PROCESS (Hayes, 2013). Effect coding was used for the multicategorical predictor (Hayes & Preacher, 2014). The results of the mediation analysis are shown in Figure 1.

Figure 1.

Effect of pseudo-participatory evaluation on (a) instrumental use and (b) conceptual use mediated by justice perception, Study 1.

Justice perceptions mediated the effect of pseudo-participatory evaluation on instrumental use, with a negative indirect effect of pseudo-participation (b = −.67, SE = .15). Justice perceptions also mediated the effect of pseudo-participatory evaluation on conceptual use, with a negative indirect effect of pseudo-participation (b = −.45, SE = .14).

Discussion

Confirming our first hypothesis, pseudo-participatory evaluation clearly reduced reported instrumental and conceptual use compared to participatory and nonparticipatory evaluation. The results for the mediation hypothesis show that the effect of pseudo-participatory evaluation on instrumental and conceptual evaluation use was mediated by participants’ justice perceptions. Justice perceptions had a negative indirect effect on evaluation use, as expected due to the frustration effect (Cohen, 1985; Lind & Tyler, 1988).

Study 1 was conducted in an online laboratory with student participants, who might lack familiarity with the content of the practice-based scenario (i.e., a real understanding of school social work). Since participants were randomly assigned to the experimental conditions, we can assume a comparatively high internal validity for Study 1 (Shadish, Cook, & Campbell, 2002). However, in order to improve external validity, a second study was conducted with a different group of participants who were presumably more familiar with scenarios such as the one presented in Study 1. This was the aim of Study 2.

Study 2

Participants

Participants in Study 2 were either school social workers or social workers working with youth who were also part-time students in an online bachelor of arts program in social work in Germany. Recruitment was carried out via e-mail to national school social work associations in Germany for school social workers and via the Bachelor of Arts program administration for the social workers working with youth. From the initial sample of 269 participants, 191 participants were excluded due to response errors such as randomly clicking through the items or incomplete data. This lead to a final sample of N = 78, of which n = 49 were school social workers and n = 29 were social workers working with youth. Fifty-five participants were female, 21 were male (1 nonresponse and 1 neither male nor female). Participants were 41 years of age on average (min = 21, max = 65, SD = 10.6). Participants were offered a subsequent summary of the study results as well as the chance to win one of the three book vouchers (each worth 15 Euros).

Results

For information on intercorrelation, see Supplemental Appendix 2, available in the online version of this article.

Manipulation check

An independent t test confirmed that the manipulation had a significant effect. Participants in the pseudo-participatory evaluation condition perceived the recommendations as less similar (M = 3.89, SD = 1.56) than participants in the participatory evaluation condition (M = 5.35, SD = 1.07), with a strong effect: t(48) = 3.89, p < .001, d = 1.08.

Experimental effects and mediation analysis

A MANOVA was conducted to test the first hypothesis that evaluation use is lower in the pseudo-participatory evaluation condition than in the participatory and nonparticipatory evaluation conditions. Again, the evaluation procedure (pseudo-participatory, participatory, and nonparticipatory evaluation) served as the independent variable and instrumental use and conceptual use served as the dependent variables. The results revealed an overall multivariate effect: V = .19, F(4, 150) = 3.99, p = .004. In terms of univariate effects, there were differences between the groups on instrumental use, F(2, 75) = 7.48, p = .001, with an achieved power of .93, but not for conceptual use, F(2, 75) = 1.56, p = .218, with an achieved power of .32.

Participants in the pseudo-participatory evaluation condition reported lower instrumental use (M = 3.46, SD = 1.63) and lower conceptual use (M = 3.01, SD = 1.82) compared to participants in the participatory evaluation (instrumental use: M = 4.96, SD = 1.24; conceptual use: M = 3.65, SD = 1.67) and in the nonparticipatory evaluation condition (instrumental use: M = 4.56, SD = 1.40; conceptual use: M = 3.79, SD = 1.63). Planned contrasts for pseudo-participatory evaluation compared to participatory and nonparticipatory evaluation show meaningful differences for instrumental use, t(75) = −3.78, p < .001, d = 0.91, and marginal differences for conceptual use, with almost moderate effect sizes, t(75) = −1.72, p = .09, d = 0.41.

PROCESS (Hayes, 2013) was used to test for mediation. Again, effect coding was used for the multicategorical predictor (Hayes & Preacher, 2014). The results of the mediation analysis are shown in Figure 2.

Figure 2.

Effect of pseudo-participatory evaluation on (a) instrumental use and (b) conceptual use mediated by justice perception, Study 2.

As in Study 1, justice perceptions mediated the effect of pseudo-participatory evaluation on instrumental use (b = −.39, SE = .16) and on conceptual use (b = −.42, SE = .19).

Discussion

The results of Study 2 replicate the findings of Study 1 for the first hypothesis in that pseudo-participatory evaluation led to lower instrumental use and (at least marginally) lower conceptual use relative to participatory and nonparticipatory evaluation. These effects were again mediated by justice perceptions. Since there was only a marginal total effect of pseudo-participatory evaluation on conceptual use to be mediated, the relative indirect effect of justice perceptions can be considered to be meaningful (Rucker, Preacher, Tormala, & Petty, 2011; Zhao, Lynch, & Chen, 2010).

General Discussion

The results for the pseudo-participatory evaluation conditions showed that a participatory evaluation process that resulted in an unexpected set of recommendations was less likely to be used. Of course, other aspects in participatory evaluation can also go wrong. For example, participatory evaluation can raise unrealistic expectations among participants (Cousins, 1996), it can be viewed as a heavy burden by the community (Sullins, 2003), committees or even a single person might not cooperate as planned (Gastic, Irby, & Zdanis, 2008; Whitmore, 1998), or the diversity of a steering group might become too complex to handle (Mathie & Greene, 1997). However, the findings in the current studies add weight to Mark and Shotland’s (1985) argument that when participatory evaluation is viewed as a sham, the result might be resentment and dissatisfaction or, as was shown in the current studies, significantly more negative attitudes and cognitions regarding the evaluation as well as a significant decrease in behavioral intentions.

The finding of different reactions as a function of the perceived degree of influence in evaluation is also in line with other research. For example, Lennie (2005) showed that a perceived lack of ownership and control over project activities may lead participants to leave the participatory evaluation process. Also, Rebolloso, Fernández-Ramírez, and Cantón (2005) found that using a participatory approach to evaluation leads to better dissemination of evaluation findings compared to a directive approach. As discussed by Gregory (2000) and Rebien (1996), participatory evaluation that is conducted poorly or even faultily highlights power issues. It is impossible for participatory evaluation to leave power issues between evaluators and stakeholders unaddressed. Several participant comments from the current study illustrate this issue, with one participant bringing it to a head: “What was described is something that happens over and over again. On political and official levels. You get the great feeling that you’re being heard but in the end you realize that things will be done differently and in others’ favor.” While participatory evaluation might have suggested stakeholder control over substantive issues, it instead only confirmed existing power differences, resulting in lowered intentions for instrumental use and lowered conceptual use responses.

Of course, power is not the only factor that influences behavior: Strong intentions, environmental constraints, or personal skills, to name a few, will also influence whether a certain behavior is performed (Fishbein et al., 2001). Nevertheless, the current research highlights that more needs to be known about pseudo-participatory evaluation—its antecedents, perceptions, and its consequences as well as what consequences might be expected when other power issues arise, for example, due to information disparities or when stakeholders perceive that they have lost expert status (Taut & Brauns, 2003).

For the first time, the theory of procedural justice was used to explore and understand possible underlying psychological mechanisms of participatory evaluation. For participants in the current studies—imaging themselves in one of the three evaluation conditions described—the effects of pseudo-participatory evaluation were mediated by their perceptions of the justice of the evaluation. These new insights into the role of justice perceptions in participatory evaluation might also influence the way scholars conceptualize participatory evaluation mechanisms, as will be discussed in the next section.

Implications for Participatory Evaluation Theory

We suggest that it might be wise for evaluators to become sensitive to stakeholders’ perceptions of fairness in participatory evaluation processes, just as evaluators need to be concerned about relationship quality, trust, or power and authority. Our study showed that justice perceptions mediate the effect of pseudo-participatory evaluation on participants’ responses concerning evaluation use. But why are justice perceptions so important? The procedural justice literature suggests that people care about justice due to relational concerns: People have socioemotional interests in their social relationships (Blader & Tyler, 2015). They want to be seen as valued members of a group and feel respected by its members and representatives because that gives them crucial information about their belonging and status within the group (de Cremer & Tyler, 2005). One way to provide this information is through a procedure that is perceived as fair. This kind of positive justice information influences cooperative behaviors in positive ways (Tyler & Blader, 2003) and has several other positive effects, such as the aforementioned fair process effect. Most influential models of procedural justice are based on this relational and socioemotional assumption (Blader & Tyler, 2015).

However, program evaluation textbooks that attempt to explain psychological concepts within interactive evaluation practices (e.g., Johnson, Johnson, & Stevahn, 2011; King & Stevahn, 2013) refer to social interdependence theory instead (Deutsch, 1949; Johnson & Johnson, 1989). Social interdependence theory suggests that people strive to reach personal goals and act in either cooperative (goals can only be achieved cooperatively with others; positive interdependence) or competitive (goals can only be achieved in competition with others; negative interdependence) ways in order to do so. Evaluators are advised to facilitate positive interdependence and cooperative goals in order to develop promotive interaction (Johnson et al., 2011; King & Stevahn, 2013).

Blader and Tyler (2015) argue that according to social exchange theories, people view the group which they cooperate or compete with primarily as a vehicle to achieve desired personal outcomes—their interest in cooperation is thus instrumental in nature. In contrast, socioemotional interests as suggested by the procedural justice literature and as conceptualized in the current research view the relationship to the group as most important for people. Thus, by only considering instrumental interests in participatory evaluation, one could easily overlook the potential and—as was shown in the current research—particularly the pitfalls of socioemotional interests: A failure to promote procedures that are perceived as just can lead to negative consequences for evaluation use. Thus, scholars are well advised to consider stakeholders’ socioemotional interests in addition to their instrumental interests when conceptualizing and conducting participatory evaluation and when trying to understand and account for the associated processes.

Implications for Participatory Evaluation Practice

In the current research, we remarked that participatory evaluation addresses power issues (Gregory, 2000; Rebien, 1996), which can have negative consequences if not handled appropriately (Mark & Shotland, 1985). Groves and Guijt (2015) outline practical recommendations for practicing participatory evaluators that address exactly these power issues: (1) Evaluators should think about and decide on the extent to which they want and are able to share power so that participation is more than just an aspiration, (2) evaluators should be conscious about their reasons for and values regarding participation in order to make clearer choices within the participatory process, and (3) evaluators should take actions to avoid participatory evaluation becoming a sham. Moreover, Shulha, Whitmore, Cousins, Gilbert, and Al Hudib (2015) describe principles to guide collaborative approaches to evaluation: fostering meaningful relationships, developing a shared understanding of the program, promoting appropriate participatory processes, monitoring and responding to resource availability, monitoring evaluation progress and quality, promoting evaluative thinking, following through to realize use, and clarifying motivations to collaborate. Since these principles were empirically derived and validated on the basis of evaluators’ experience and expertise, they can help to guide collaborative approaches to evaluation and avoid pitfalls and backlashes.

Limitations

The current study applied an experimental vignette design, which clearly cannot capture the dynamics of real-life participatory evaluation, for example, personal factors, organizational factors, or factors stemming from the program under evaluation (see Cousins & Chouinard, 2012), or the long-term effects of participatory evaluation. In addition, participants did not have a real personal interest in the evaluation process or its outcomes since the scenario was fictitious. However, since the scenario was developed with utmost care, we assume that it at least roughly corresponds to real-life evaluation situations.

Wording the instrumental use items in the first person in Study 1 might have been confusing for student participants. It might have been hard for them to make informed judgments regarding their intentions for behavior they are presumably not familiar with.

The experimental vignette design forced us to conceptualize instrumental use as behavioral intentions, not actual behavior. However, even though actual behavior could not be measured in the current studies, the results still provide valuable insights because behavioral intentions are considered strong antecedents of actual behavior (Ajzen, 1991).

Lastly, the order in which the manipulation check items and the mediation items were presented might have led to expectancy effects in participants’ responses. Better designs in which possible expectancy effects can be ruled out are needed.

Conclusion and Future Directions

Participatory evaluation has had a huge impact on the evaluation community due to its association with many positive outcomes, most prominently evaluation use. Of course, there are also other valuable reasons to conduct participatory evaluations apart from fostering evaluation use, including emancipatory or transformative goals. However, in the current research, we found differential experimental effects of participatory evaluation on aspects of evaluation use. We further explored the role of justice perceptions on the negative effects of participatory evaluation. Our results point to the advent of a new field within systematic research, discussion, and practice on participatory evaluation: Participatory evaluation can have a negative effect on evaluation use, with critical consequences for the positive connotation of participatory evaluation. Participatory evaluation that is viewed as pseudo-participatory evaluation not only reduces evaluation use but also leads stakeholders to actively reject evaluation use as a function of their justice perceptions. If the evaluation community is to further enhance and develop the field of participatory evaluation in order to tap into its potential, it should also be conscious of and think critically about negative effects when conceptualizing, investigating, and conducting participatory evaluation.

Supplemental Material

Supplemental Material, AJE813447_Supplemental_Appendix_1 - “You Get the Great Feeling That You’re Being Heard But in the End You Realize That Things Will Be Done Differently and in Others’ Favor”: An Experimental Investigation of Negative Effects of Participation in Evaluation

Supplemental Material, AJE813447_Supplemental_Appendix_1 for “You Get the Great Feeling That You’re Being Heard But in the End You Realize That Things Will Be Done Differently and in Others’ Favor”: An Experimental Investigation of Negative Effects of Participation in Evaluation by Benjamin Froncek, and Anette Rohmann in American Journal of Evaluation

Supplemental Material

Supplemental Material, AJE813447_Supplemental_Appendix_2 - “You Get the Great Feeling That You’re Being Heard But in the End You Realize That Things Will Be Done Differently and in Others’ Favor”: An Experimental Investigation of Negative Effects of Participation in Evaluation

Supplemental Material, AJE813447_Supplemental_Appendix_2 for “You Get the Great Feeling That You’re Being Heard But in the End You Realize That Things Will Be Done Differently and in Others’ Favor”: An Experimental Investigation of Negative Effects of Participation in Evaluation by Benjamin Froncek, and Anette Rohmann in American Journal of Evaluation

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Benjamin Froncek

Supplemental Material

Supplemental Appendices 1 and 2 are available in the online version of this article at .

Notes

References

Aguinis

Bradley

K. J.

(2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17, 351–371. doi:10.1177/1094428114547952

Ajzen

(1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179–211. doi:10.1016/0749-5978(91)90020-T

Alkin

M. C.

King

J. A.

(2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 76, 1–17. doi:10.1177/1098214017717015

Alkin

M. C.

Taut

S. M.

(2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1–12. doi:10.1016/S0191-491X(03)90001-0

Blader

S. L.

Tyler

T. R.

(2015). Relational models of procedural justice. In Cropanzano

R. S.

Ambrose

M. L.

(Eds.), The Oxford handbook of justice in the workplace (pp. 351–370). New York, NY: Oxford University Press.

Borenstein

(2009). Effect sizes for continuous data. In Cooper

H. M.

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–237). New York, NY: Russell Sage Foundation.

Brandon

P. R.

(1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325–337. doi:10.1177/109821409801900305

Brandon

P. R.

Fukunaga

L. L.

(2014). The state of the empirical research literature on stakeholder involvement in program evaluation. American Journal of Evaluation, 35, 26–44. doi:10.1177/1098214013503699

Brockner

Fishman

A. Y.

Reb

Goldman

Spiegel

Garden

(2007). Procedural fairness, outcome favorability, and judgments of an authority’s responsibility. Journal of Applied Psychology, 92, 1657–1671. doi:10.1037/0021-9010.92.6.1657

10.

Cohen

R. L.

(1985). Procedural justice and participation. Human Relations, 38, 643–663. doi:10.1177/001872678503800703

11.

Cohen-Charash

Mueller

J. S.

(2007). Does perceived unfairness exacerbate or mitigate interpersonal counterproductive work behaviors related to envy? Journal of Applied Psychology, 92, 666–680. doi:10.1037/0021-9010.92.3.666

12.

Cousins

J. B.

(1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational Evaluation, 22, 3–27. doi:10.1016/0191-491x(96)00001-6

13.

Cousins

J. B.

Chouinard

J. A.

(2012). Participatory evaluation up close: An integration of research-based knowledge. Greenwich, England: Information Age.

14.

Cousins

J. B.

Donohue

J. J.

Bloom

G. A.

(1996). Collaborative evaluation in North America: Evaluators’ self-reported opinions, practices and consequences. American Journal of Evaluation, 17, 207–226. doi:10.1177/109821409601700302

15.

Cousins

J. B.

Whitmore

(1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5–23. doi:10.1002/ev.1114

16.

Cropanzano

R. S.

Rupp

D. E.

Thornton

M. A.

Shao

(2016). Organizational justice and organizational citizenship behavior. In Podsakoff

P. M.

Mackenzie

S. B.

Podsakoff

N. P.

(Eds.), The Oxford handbook of organizational citizenship behavior. Oxford, England: Oxford University Press.

17.

Cullen

A. E.

Coryn

C. L. S.

Rugh

(2011). The politics and consequences of including stakeholders in international development evaluation. American Journal of Evaluation, 32, 345–361. doi:10.1177/1098214010396076

18.

Daigneault

P. M.

(2014). Taking stock of four decades of quantitative research on stakeholder participation and evaluation use: A systematic map. Evaluation and Program Planning, 45, 171–181. doi:10.1016/j.evalprogplan.2014.04.003

19.

de Cremer

Tyler

T. R.

(2005). Am I respected or not? Inclusion and reputation as issues in group membership. Social Justice Research, 18, 121–153. doi:10.1007/s11211-005-7366-3

20.

Deutsch

(1949). A theory of cooperation and competition. Human Relations, 2, 129–152. doi:10.1177/001872674900200204

21.

Fishbein

Triandis

H. C.

Kanfer

F. H.

Becker

Middlestadt

S. E.

Eichler

(2001). Factors influencing behavior and behavior change. In Baum

A. S.

Revenson

T. A.

Singer

J. E.

(Eds.), Handbook of health psychology (pp. 3–17). Mahwah, NJ: Lawrence Erlbaum.

22.

Fleischer

D. N.

Christie

C. A.

(2009). Evaluation use: Results from a survey of U.S. American evaluation association members. American Journal of Evaluation, 30, 158–175. doi:10.1177/1098214008331009

23.

Gastic

Irby

D. J.

Zdanis

(2008). When stakeholders rebel: Lessons from a safe schools program. Studies in Educational Evaluation, 34, 208–211. doi:10.1016/j.stueduc.2008.10.004

24.

Goodwin

Ross

(1992). Consumer responses to service failures: Influence of procedural and interactional fairness perceptions. Journal of Business Research, 25, 149–163. doi:10.1016/0148-2963(92)90014-3

25.

Greenberg

Folger

R. G.

(1983). Procedural justice, participation, and the fair process effect in groups and organizations. In Paulus

P. B.

(Ed.), Springer series in social psychology: Basic group processes (pp. 235–256). New York, NY: Springer.

26.

Greene

J. C.

(1987). Stakeholder participation in evaluation design: Is it worth the effort? Evaluation and Program Planning, 10, 379–394. doi:10.1016/0149-7189(87)90010-3

27.

Greene

J. C.

(1988). Communication of results and utilization on participatory program evaluation. Evaluation and Program Planning, 11, 341–351. doi:10.1016/0149-7189(88)90047-x

28.

Gregory

(2000). Problematizing participation: A critical review of approaches to participation in evaluation theory. Evaluation, 6, 179–199. doi:10.1177/13563890022209208

29.

Groves

Guijt

(2015). Positioning Participation on the Power Spectrum. Retrieved from http://www.betterevaluation.org/en/blog/positioning_participation_on_the_power_spectrum

30.

Hayes

A. F.

(2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. New York, NY: Guilford Press.

31.

Hayes

A. F.

Preacher

K. J.

(2014). Statistical mediation analysis with a multicategorical independent variable. The British Journal of Mathematical and Statistical Psychology, 67, 451–470. doi:10.1111/bmsp.12028

32.

Henry

G. T.

Mark

M. M.

(2003). Beyond Use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293–314. doi:10.1177/109821400302400302

33.

Jacob

Ouvrard

Bélanger

J. F.

(2011). Participatory evaluation and process use within a social aid organization for at-risk families and youth. Evaluation and Program Planning, 34, 113–123. doi:10.1016/j.evalprogplan.2010.08.002

34.

Johnson

D. W.

Greenseid

L. O.

Toal

S. A.

King

J. A.

Lawrenz

Volkov

(2009). Research on evaluation use. A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30, 377–410. doi:10.1177/1098214009341660

35.

Johnson

D. W.

Johnson

R. T.

(1989). Cooperation and competition: Theory and research. Edina, MN: Interaction Book.

36.

Johnson

D. W.

Johnson

R. T.

Stevahn

(2011). Social interdependence and program evaluation. In Mark

M. M.

Donaldson

S. I.

Campbell

(Eds.), Social psychology and evaluation (pp. 288–317). New York, NY: Guilford Press.

37.

King

J. A.

Stevahn

(2013). Interactive evaluation practice: Mastering the interpersonal dynamics of program evaluation. Los Angeles, CA: Sage.

38.

Kirkhart

K. E.

(2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 2000, 5–23. doi:10.1002/ev.1188

39.

Lakens

(2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 1–12. doi:10.3389/fpsyg.2013.00863

40.

Lennie

(2005). An evaluation capacity building process for sustainable community IT initiatives. Empowering and disempowering impacts. Evaluation, 11, 390–414. doi:10.1177/1356389005059382

41.

Leviton

L. C.

Hughes

E. F.

(1981). Research on the utilization of evaluations: A review and synthesis. Evaluation Review, 5, 525–548. doi:10.1177/0193841X8100500405

42.

Lind

E. A.

Kanfer

Earley

P. C.

(1990). Voice, control, and procedural justice: Instrumental and noninstrumental concerns in fairness judgments. Journal of Personality and Social Psychology, 59, 952–959. doi:10.1037/0022-3514.59.5.952

43.

Lind

E. A.

Tyler

T. R.

(1988). The social psychology of procedural justice. New York, NY: Plenum Press.

44.

Mathie

Greene

J. C.

(1997). Stakeholder participation in evaluation: How important is diversity? Evaluation and Program Planning, 20, 279–285. doi:10.1016/S0149-7189(97)00006-2

45.

Mark

M. M.

Henry

G. T.

(2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10, 35–57. doi:10.1177/1356389004042326

46.

Mark

M. M.

Shotland

R. L.

(1985). Stakeholder-based evaluation and value judgments. Evaluation Review, 9, 605–626. doi:10.1177/0193841X8500900504

47.

McFarlin

D. B.

Sweeney

P. D.

(1992). Distributive and procedural justice as predictors of satisfaction with personal and organizational outcomes. Academy of Management Journal, 35, 626–637. doi:10.2307/256489

48.

Mertens

D. M.

(2009). Transformative research and evaluation. New York, NY: Guilford Press.

49.

Moorman

R. H.

Byrne

Z. S.

(2005). How does justice affect organizational citizenship behavior? In Greenberg

Colquitt

J. A.

(Eds.), Handbook of organizational justice (pp. 355–380). Mahwah, NJ: Lawrence Erlbaum Associates.

50.

Oelerich

(2013). Evaluation Schulsozialarbeit im Rahmen des Bildungs- und Teilhabepakets (BuT) in Wuppertal [School social work evaluation in the context of the education and participation package in Wuppertal]. Retrieved from http://www.sozpaed.uni-wuppertal.de/fileadmin/sozpaed/Abschlussbericht_Evaluation_Schulsozialarbeit_BuT_Wuppertal-Oelerich-9.2013.pdf

51.

Patton

M. Q.

(2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

52.

Poth

C. A.

Shulha

(2008). Encouraging stakeholder engagement: A case study of evaluator behavior. Studies in Educational Evaluation, 34, 218–223. doi:10.1016/j.stueduc.2008.10.006

53.

Potter

P. W.

(2006a). Procedural justice and voice effects. Journal of Organizational Culture, Communications and Conflict, 10, 33–62.

54.

Potter

P. W.

(2006b). Procedural justice, voice effects and sham: Examining the decision maker from a research context perspective. Journal of Organizational Culture, Communications and Conflict, 10, 61–76.

55.

Preskill

Caracelli

(1997). Current and developing conceptions of use: Evaluation use TIG survey results. American Journal of Evaluation, 18, 209–225. doi:10.1016/s0886-1633(97)90028-3

56.

Rebien

C. C.

(1996). Participatory evaluation of development assistance: Dealing with power and facilitative learning. Evaluation, 2, 151–171. doi:10.1177/135638909600200203

57.

Rebolloso

Fernández-Ramírez

Cantón

(2005). The influence of evaluation on changing management systems in educational institutions. Evaluation, 11, 463–479. doi:10.1177/1356389005060263

58.

Rodríguez-Campos

(2005). Collaborative evaluations: A step-by-step model for the evaluator. Tamarac, FL: Llumina Press.

59.

Roseland

Lawrenz

Thao

(2015). The relationship between involvement in and use of evaluation in multi-site evaluations. Evaluation and Program Planning, 48, 75–82. doi:10.1016/j.evalprogplan.2014.10.003

60.

Rucker

D. D.

Preacher

K. J.

Tormala

Z. L.

Petty

R. E.

(2011). Mediation analysis in social psychology: Current practices and new recommendations. Social and Personality Psychology Compass, 5, 359–371. doi:10.1111/j.1751-9004.2011.00355.x

61.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

62.

Shadish

W. R.

Cook

T. D.

Leviton

L. C.

(1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.

63.

Shulha

Whitmore

Cousins

J. B.

Gilbert

Al Hudib

(2015). Introducing evidence-based principles to guide collaborative approaches to evaluation: Results of an empirical process. American Journal of Evaluation, 37, 193–217. doi:10.1177/1098214015615230

64.

Speck

(2006). Qualität und Evaluation in der Schulsozialarbeit: Konzepte, Rahmenbedingungen und Wirkungen [Quality and evaluation in school social work: Concepts, general conditions and impacts] . Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.

65.

Sullins

C. D.

(2003). Adapting the empowerment evaluation model: A mental health drop-in centre case example. American Journal of Evaluation, 24, 387–398. doi:10.1177/109821400302400307

66.

Taut

(2007). Methodological and conceptional challenges in studying evaluation process use. Canadian Journal of Program Evaluation, 22, 1–19.

67.

Taut

(2008). What have we learned about stakeholder involvement in program evaluation? Studies in Educational Evaluation, 34, 224–230. doi:10.1016/j.stueduc.2008.10.007

68.

Taut

Brauns

(2003). Resistance to evaluation: A psychological perspective. Evaluation, 9, 247–264. doi:10.1177/13563890030093002

69.

Turnbull

(1999). The mediating effect of participation efficacy on evaluation use. Evaluation and Program Planning, 22, 131–140. doi:10.1016/s0149-7189(99)00012-9

70.

Tyler

T. R.

Blader

S. L.

(2003). The group engagement model: Procedural justice, social identity, and cooperative behavior. Personality and Social Psychology Review, 7, 349–361. doi:10.1207/S15327957PSPR0704_07

71.

van den Bos

Lind

E. A.

Vermunt

Wilke

H. A. M.

(1997). How do I judge my outcome when I do not know the outcome of others? The psychology of the fair process effect. Journal of Personality and Social Psychology, 72, 1034–1046. doi:10.1037/0022-3514.72.5.1034

72.

van den Bos

Wilke

H. A. M.

Vermunt

Lind

E. A.

(1998). Evaluating outcomes by means of the fair process effect: Evidence for different processes in fairness and satisfaction judgements. Journal of Personality and Social Psychology, 74, 1493–1503. doi:10.1037//0022-3514.74.6.1493

73.

Weiss

C. H.

(1986). The stakeholder approach to evaluation: Origins and promise. In House

E. R.

(Ed.), New directions for educational evaluation (pp. 145–157). Abingdon, England: Routledge Falmer.

74.

Whitmore

(1998). “We need to rebuild this house”: The role of empowerment in evaluation of a Mexican Farmers’ cooperative. In Jackson

E. T.

Kassam

(Eds.), Knowledge shared: Participatory evaluation in development cooperation (pp. 217–230). West Hartford, CT: Kumarian Press.

75.

Zhao

Lynch

J. G.

Jr Chen

(2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37, 197–206. doi:10.1086/651257

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB