Principles for Stakeholder Involvement in Evaluation in Latin America

Abstract

In the last three decades, the promotion of stakeholder involvement in evaluation has been gaining relevance in the Latin American and internationally, across varied agencies, institutions, and civic organizations. The 2030 Agenda and the Global Evaluation Agenda have also recognized the centrality of participation in evaluation. This article explores stakeholder involvement in evaluation based on collaborative work with stakeholders from 15 evaluative experiences. It shows what characterizes participatory evaluation in the region today and the principles of this practice.

Keywords

evaluation practice principles stakeholder involvement in evaluation Latin America

Over the last three decades, stakeholder involvement in evaluation has increased globally, as reflected in the number of theories and methodologies that prioritize participation in the evaluation process (Cardozo Brum & Rosas Huerta, 2021; Fetterman & Wandersman, 2018; Guijt, 2014; Hay & Kumar-Range, 2014; Patton, 2010; Rodríguez-Campos, 2018; Zukoski & Bosserman, 2018). This interest in participation in evaluation is also evident in the Global Evaluation Agenda (2016–2020)¹ and the 2030 Agenda, in which the UN makes it clear that combating social inequalities means moving toward policies that “leave no one behind” (EvalPartners, 2015; United Nations Development Group, 2017). Although there is evidence of an increase in these participant engagement practices in Latin America (Sanz et al., 2019; Tapella & Sanz, 2019), research on this practice specific to Latin America is limited.

This article explores this evaluative practice in Latin America by examining a sample of 15 initiatives selected from within the framework of the EvalParticipativa project.² We also propose a set of principles that reflect current practice and offer a guide to stakeholder involvement in evaluation in Latin America. In order to link our regional discussion regarding evaluation approaches to stakeholder involvement to that of the Global North, this article addresses the following questions: What characterizes evaluation practices that integrate stakeholder participation in Latin America? What principles can be drawn from this practice to define stakeholder involvement approaches in the region?

The article begins by summarizing existing literature on participation in evaluation. We then outline our methodology before presenting the results alongside theoretical reflections on the practice of stakeholder involvement in evaluation. The final section discusses our findings and links them to stakeholder involvement theory and practice in Latin America.

Existing Research on Participation in Evaluation

The relationship between evaluation and participation arose in the 1960’s along with the emergence of qualitative evaluation and Participatory Action Research as social science researchers began to question positivist approaches (Plottu & Plottu, 2009). The classical evaluation models focused on measuring results from an external “independent” perspective, generally with applied quantitative techniques. Stakeholder involvement in evaluation, on the other hand, highlighted reflexivity as a part of the evaluation process, proposed to include stakeholders’ voices to assess programs, and began to develop new qualitative techniques (Cousins et al., 2013; Everitt & Hardiker, 1996; Guba & Lincoln, 1989; House, 1980; Scriven, 1973).

However, it was not until the mid-1990’s—and within an international framework aimed at promoting human development—that the field began to witness a flourishing of reflection and debate on participation and evaluation (Cartland et al., 2008; Chouinard & Milley, 2018; Cornwall, 2008; Cullen et al., 2011; Daigneault & Jacob, 2009; King et al., 2007; Ryan et al., 1998). During these three decades, various versions of stakeholder involvement in evaluation arose and evolved. These included empowerment evaluation, transformative evaluation, democratic deliberative evaluation, and collaborative evaluation. All of these approaches focused on both the dynamic and continuous involvement of stakeholders in evaluation processes and the recognition of stakeholders’ distinct expertise (Cartland et al., 2008; Chouinard & Milley, 2018; Cousins & Chouinard, 2012; Cousins & Whitmore, 1998; Fetterman et al., 2018; Mbava, 2017; Monnier & Conan, 1995; Nugroho et al., 2018). In Latin America, “participatory evaluation” has been used as an umbrella term. More broadly, the field has identified three main evaluation approaches to stakeholder involvement: collaborative evaluation, participatory evaluation, and empowerment evaluation.³ In order to distinguish among these evaluation approaches, developed in the Global North, Fetterman et al. (2018, pp.2–3) consider the role of the evaluator or the evaluation team in each type.

Collaborative evaluators are clearly in charge of the evaluation and are interested in creating ongoing engagement between the stakeholders and themselves. We contend that this contributes to stronger evaluation designs, enhanced data collection and analysis, and results that stakeholders can easily understand and use (Rodríguez-Campos, 2018). Participatory evaluators share control of the evaluation in a position of equality with the stakeholders, encouraging them to become involved in defining the evaluation, developing the instruments, collecting and analyzing the data, and reporting and disseminating the results (Zukoski & Bosserman, 2018). Empowerment evaluators understand the various stakeholders (program staff members, program participants, and community members) to be those in control of the evaluation, while the evaluator serves as a “critical friend” or coach to ensure a high-quality process (Fetterman & Wandersman, 2018). Figure 1 illustrates how these three main approaches vary.

Figure 1.

Stakeholder involvement approaches. Source: Prepared by the authors.

This article presents an integrated definition of the principles of stakeholder involvement in evaluation in Latin America, making connections across these three evaluation types. Taking a “stakeholder involvement in evaluation approach” requires involving varied stakeholders—including program managers, technical staff, end-users, and other relevant social actors—throughout the evaluation process. Specifically, it gives them a voice and establishes moments for consultation regarding issues that include what is evaluated; who participates; when it happens; what data collection and analysis methods are used; what the results, conclusions, and recommendations are; and how the results are communicated (Coupal, 2000; Shulha, 2010; cited in Fetterman et al., 2014). Stakeholder involvement in evaluation, therefore, has a normative component, focused on democratizing the inquiry process, co-creating knowledge, and responding better to the needs and particularities of each specific context (Chouinard & Milley, 2018; Plottu & Plottu, 2009).

Today, stakeholder participation constitutes a dynamic process that cannot be detached from the policy and program type, the community and its participation capacity and history, the cultural context, and the purpose and conditions under which the evaluation is conducted (Trickett & Ryerson-Espino, 2004). In stakeholder involvement approaches, trained evaluators work in partnership with the stakeholders of a social intervention to produce evaluative knowledge concerning the intervention, generate learning, and make decisions in a concerted manner (Cousins & Whitmore, 1998; Cousins et al., 2013).

Within the broader framework of stakeholder involvement in evaluation, the following two specific developments focus on promoting greater collaboration in evaluative processes (Cousins & Whitmore, 1998; Cousins et al., 2013). Practical evaluation—developed mainly in the developed Anglo-Saxon world and most applicable to the collaborative and participatory approaches to evaluation—emphasizes the involvement of stakeholders from a pragmatic problem-solving approach that seeks to improve the use and results of evaluation (Cousins et al., 2013). Transformative evaluation, attributed to the empowerment evaluation approach, on the other hand, has emerged in Latin America, India, and Africa and seeks emancipation, empowerment, and social change.

With regard to these two developments, there is an interesting debate around whether stakeholder participation should be instrumentalist or empowering (Arnstein, 1969; Hart, 1992; Mark & Shotland, 1985). From an instrumentalist point of view, participation is a top-down endeavor that implies involving stakeholders as informants. From this perspective, evaluation may lose its potential to democratize or transform (Cornwall, 2008). In contrast, an empowerment-style evaluation places stakeholders at the center of the evaluation. They co-design the methodology, carry out the field work, analyze the data, and present the results and recommendations, all with the support of a facilitating evaluation team (IDS, 1998). In short, these two conceptualizations of participation in evaluation are in tension with each other as one emphasizes the methodological demands of evidence-based practices, while the other focuses on social justice (Chouinard, 2013; cited in Chouinard & Milley, 2018). These two perspectives are not only marked by evaluative culture and politics, but also by the power relations that exist between those who fund the policy and its evaluation, and those to whom the policy is addressed. These two perspectives have implications for who participates, how they participate, and to what degree they participate.

In the Latin American context, the transformative approach to stakeholder involvement in evaluation has been driven and promoted by a rich tradition in the field of social action that has placed an explicit and notable emphasis on the participatory dimension inscribed in a liberating perspective of reality. Specifically, stakeholder involvement in evaluation has been nourished by three traditions of inquiry-action, widely developed in the region: Popular Education, Participatory Action Research, and the Systematization of Experiences. Despite their distinct nuances, these traditions all recognize the relevance of dialogue and participation, as well as the facilitating role of those who promote inquiry-action processes. Likewise, they all focus on social change and empowerment⁴ (Sanz et al., 2019; Tapella & Sanz, 2019).

Since the 1970’s, knowledge of local stakeholders’ particular conditions has become integrated into the notion and practice of participation in evaluation. In particular, stakeholders’ well-being and vulnerability have been central to Latin American considerations of participation in evaluation. This includes native peoples (Kushner & Rotondo, 2012), peasants (Romero Sarduy et al., 2017), older adults (Huenchuan & Paredes, 2007), at-risk communities (Letichevsky & Penna Firme, 2012), and citizens and organized social groups in social policy management and evaluation (Cardozo Brum, 2008).

In brief, this paper explores stakeholder involvement in evaluation in Latin America, where the literature on this subject is still scarce. We next turn to the methodology we have used in the research. Then, we link stakeholder involvement in evaluation in Latin America to current debates in the wider field of evaluation before offering a set of principles.

Methodology

This study aims to understand the complexity of stakeholder involvement in evaluation practice within the specific context of Latin America. To this end, we opted for “moderate generalization”⁵ (Payne & Williams, 2005), which is based on rich data collection, does not postulate grand theories, and expresses its findings hypothetically. In this vein, the research was carried out using a qualitative and collaborative approach in two connected phases (see Figure 2), based on concrete initiatives and joint work with stakeholders.

Figure 2.

Phases, activities, and dimensions of analysis. Source: Prepared by the authors.

In a first phase, the EvalParticipativa team⁶ set up a Virtual Forum⁷ as a collective learning space, aimed at generating an extended conversation regarding the field of stakeholder involvement in evaluation. In order to generate new lessons from real and concrete initiatives, the team issued an open invitation (through its webpage, social networks, and evaluators’ networks) to identify examples of initiatives that integrate stakeholder involvement approaches to evaluation in Latin America. As a result of this process, and following a non-probability sampling procedure oriented by six criteria⁸ (Patton, 2002), the team selected 15 initiatives that reflected the diversity of existing initiatives in the region. Some of these initiatives (see Table 1) worked at local level while others addressed regional issues. Most focused on social and community development and were often interested in transforming social conditions. They were multi-method evaluations that generated tools and instruments for the evaluation and generally made use of qualitative approaches.

Table 1.

Sample of Evaluations for This Project.

Evaluation	Location(s)	Stakeholders Involved
Monitoring and evaluation of the Experience Capitalization Project	Argentina, Brazil, and Colombia in Latin America. Also, East Africa and Asia	Technical Centre for Agricultural and Rural Cooperation (CTA) and International Fund for Agricultural Development (IFAD)
Strengthening the participatory monitoring of effects and impacts of non-governmental organization projects	11 countries of Central America and the Andean Region	Consortium of 7 German NGOs and 39 Latin American partners, Federal Ministry for Economic Cooperation and Development
Evaluation of DW Akademie's media development cooperation project in Latin America	Guatemala, Colombia, Ecuador, Bolivia	DW Akademie and Federal Ministry for Economic Cooperation and Development
Update to the appraisal of the human rights situation and Human Rights Program of Mexico City	Mexico	Human Rights Program of Mexico City, 4 NGOs, 3 academic institutions and UN High Commissioner for Human Rights
Participatory evaluation of the Mesa de Trabajo intervention tool in the community of San Mateo, Mendizábal	Mexico	TECHO International
Intermediate evaluation of the Regional Agricultural Research Consortiums Program (CRIA)	Guatemala	United States Department of Agriculture (USDA), Ministry of Agriculture, Livestock and Food (MAGA), and the Inter-American Institute for Cooperation in (IICA)
Participatory evaluation of cancer prevention and care in Valle de la Estrella	Costa Rica	Costa Rican Social Security Fund (CCSS)
Participatory evaluation of the Mesa de Trabajo intervention tool in the rural community of San Enrique, Guayaquil	Ecuador	TECHO International
Systematizing the experience of the project “Strengthening Generational Handover in Junin Cooperatives”	Peru	RETAR
Participatory evaluation of the Mesa de Trabajo intervention tool in the Santa Teresa settlement, San Bernardo Commune, Metropolitan Region	Chile	TECHO International
Pilot participatory evaluations of six interventions of the National Service Program	Chile	Fundación para la Superación de la Pobreza and Programa Servicio País
A participatory self-evaluation of educational quality (IACE)	Argentina	Center for Local Development Support (CEADEL) and UNICEF Argentina
Second self-evaluation of the National University of Lanús (UNLa)	Argentina	National University of Lanús (UNLa)
Participatory evaluation of the Mesa de Trabajo intervention tool in the communities of Nuevo Horizonte and San Martín	Paraguay	TECHO International
Participatory evaluation of the implementation of TECHO Uruguay's community work model in Barrio 7 de Diciembre, Montevideo	Uruguay	TECHO International

Note. TECHO is also known as Un Techo Para Mi País (UTPMP; Spanish: A Roof For My Country).

Source: Prepared by the authors.

After the sample was selected, the EvalParticipativa team invited those who facilitated the evaluations to the First Gathering of Participatory Evaluation Experiences in Latin America,⁹ which took place in Quito (Ecuador) from November 18–22, 2019. The purpose of the event was to share illustrative experiences from the region, identify common success factors and recurring problems, and document these processes, which often go unnoticed to the group of evaluators in the region. To this end, the event was structured around five modules: (1) The Exchange of Experiences; (2) The “What” of Stakeholder Involvement in Evaluation; (3) The “How” of Stakeholder Involvement in Evaluation; (4) How to Facilitate Stakeholder Involvement Processes in Evaluation; and (5) The Stakeholder Involvement in Evaluation Tools Fair.¹⁰ The collaborative work carried out in this event, and specifically in modules 1 and 2, fed the results presented below regarding the practice and principles of stakeholder involvement in evaluation in the region.

The focus of module 1 was the exchange of experiences as a starting point for building shared knowledge on stakeholder involvement in evaluation approaches. This module was based on the main results that emerged from the Virtual Forum, and these same results were used throughout the event to stimulate reflection and analysis. Prior to the event, the participants received a set of instructions to facilitate the preparation and communication of each experience. Organizers also provided the participants with the following categories of analysis: (1) the main difficulties or limitations presented by the evaluation; (2) the main advantages and positive aspects of the evaluation experience; (3) key lessons (new or reinforced); and (4) challenges from the development of stakeholder involvement approaches to evaluation.

At the meeting, the participants presented a brief description of the intervention that was the object of the evaluation and described the relevant actors in the evaluation process, indicating their roles at each stage.

The EvalParticipativa team promoted collaboration in Module 1 using the activity “Pausing to Take Stock.” This activity, which split the group into four at random, promoted joint reflection on each of the indicated dimensions of analysis based on the presentations of the evaluative initiatives. Participants shared a synthesis of the written material in a plenary session and this provided another opportunity for future comments. Thus, the categories analyzed reached their “theoretical saturation” point,¹¹ and together, the participants defined the current state of stakeholder involvement in evaluation in practice in Latin America.

Module 2 focused on the collective construction of a set of principles or assumptions on stakeholder involvement in evaluation, based on existing practice in Latin America. For this purpose, the EvalParticipativa team designed the “Participatory Evaluation Playing Card,” a tool in the form of a board game for evaluators, program and project managers, and evaluation facilitators. It uses 40 playing cards that feature statements related to stakeholder involvement in evaluation¹² and the profile of the facilitator in this type of evaluation.

The groups played six rounds of the game. In each round, each participant had to choose and justify one of the cards that they felt best reflected what they understood to be the most important characteristic of this type of evaluation. At the end of each round, each group chose one of the cards by consensus and this became part of the group's agreed statements on the topic. At the end of the game, each group came up with a list of the most important principles or assumptions for stakeholder involvement in evaluation. In the plenary session, each group presented their “results” and then, everyone present came together to choose the most representative ones by consensus. Repeated or similar statements were discarded or integrated into another that they considered to be superior or more profound. The principles presented in the following section are the ones highlighted through this process by the region's facilitators of this type of evaluation.

The EvalParticipativa team organized the data by categories of analysis, considering previous and new categories present in the debates on stakeholder involvement in evaluation. With an inductive approach, the analysis remained open to other categories of analysis that emerged during the fieldwork. In this sense, the analysis combined the perceptions of key stakeholders with our own perceptions, which are steeped in general evaluation theory and more specifically, stakeholder involvement approaches to evaluation.

From Practice to Principles: Conceptual Considerations of Stakeholder Involvement in Evaluation in Latin America

Based on our empirical work and the existing literature, this section starts by exploring the characteristics of stakeholder involvement in evaluation in Latin America. We provide evidence of this practice in the region and identify its strengths and challenges. We then propose a set of principles that currently define this approach in the region and which remain open to future discussion in Latin America and beyond.

What Characterizes Evaluation Practice in the Region?

Stakeholder involvement approaches in evaluation in the Latin American region are heirs of the rich tradition of Popular Education, Participatory Action Research, and the Systematization of Experiences, which focus on the desire to transform human realities. This transformative perspective is key to the Latin American practice of stakeholder involvement in evaluation, and yet, documented research is scarce in the region.

Below, we define characteristics—strengths and challenges—of participatory evaluation in practice in Latin America.

An Evaluation Culture and Policy Marked by Classical Evaluation Approaches

According to our empirical research, the way evaluation culture and policy are developing in the region is strongly influenced by the commitment of governmental bodies and international development agencies to classical evaluation models and approaches. These are sometimes linked to the Logical Framework Approach and usually assume a managerial approach. Organizations that have a clear focus on social change and that can self-fund their evaluations face less challenges when they seek to apply stakeholder involvement in evaluation approaches. Nevertheless, these Latin American organizations have a limited capacity to influence general evaluation culture and policy. Furthermore, many funders and commissioners have doubts about the rigor of participatory evaluation and believe that stakeholder involvement in evaluation is not suitable for revealing long-term results or impacts.

Furthermore, as in other regions (Chouinard, 2013), our research shows that policymakers and funders only consider evaluation to be useful for the purposes of accountability, institutional transparency, and control. They give reduced importance to “formative” evaluation as an instrument for continuous learning and capacity building.

Diversity of Interests in Evaluation: Making Progress in the Creation of Spaces for Negotiation

Traditionally, in Latin America, governmental bodies and international development agencies have requested evaluations that follow a top-down approach (Cullen et al., 2011; Guba & Lincoln, 1989; House, 1980; Scriven, 1973). That has, in turn, generated attitudes of detachment and disinterest, if not resistance, among local stakeholders. On the contrary, and according to the initiatives studied, stakeholder involvement in evaluation implies and promotes negotiation between the various actors—including funding agents and political authorities—for each specific evaluation exercise and that evaluation's possible scope. Our study reveals that a commitment to stakeholder involvement in evaluation also brings the task of reconciling conflicts of interest given the diversity of actors, and balancing power dynamics in evaluation decision-making. According to the interventions that we have examined, stakeholder involvement in evaluation practice in Latin America recognizes the diversity of stakeholders engaged and generates social spaces for meeting, dialogue, and joint knowledge creation (Tapella et al., 2021).¹³

Stakeholder Involvement: A Tool for Improving Evaluations

According to the experiences we have examined from the region, the active involvement of affected individuals and entities allows the evaluation to view the object more closely than in conventional external evaluations. This favors a better understanding of the local context. In accordance with the Latin American literature, stakeholder involvement enhances methodological designs that are flexible and adaptable to varied contexts and evaluative purposes (Cavanna, 2019). Likewise, it favors a greater understanding of the processes and, therefore, of the results, both expected and unexpected.

In the initiatives reviewed, the varied actors involved in the evaluation felt that participatory evaluation approaches would be beneficial in future work. One way of encouraging institutional spaces to take on participatory approaches could be to share with them the advantages experienced in similar contexts (Sanz et al., 2019).

The Challenge of Effectively Involving Excluded Voices

From a social transformation perspective, stakeholder involvement initiatives emphasize the inclusion of all relevant voices, especially groups that are often excluded. They also insist on treating stakeholders as subjects of rights rather than mere informants. However, in the evaluations analyzed, the degree of effective involvement varies among the evaluations.

Firstly, most stakeholder involvement in evaluation processes is encouraged by the actors closest to the management of the intervention: technical teams, mid-level program officials, donors interested in enhancing participatory initiatives, and etc. Only in some long-term initiatives with strong stakeholder involvement have these actors pushed and advocated for a greater influence in the evaluation dynamics.

Secondly, on many occasions—and especially in governmental projects—the identification of relevant stakeholders is based on an a priori and often standardized list, adopting the aforementioned top-down approach. In the initiatives we have examined, a broad and complete definition of what constitutes a relevant stakeholder in each specific evaluation is considered essential. This ensures that historically excluded stakeholders take part in the evaluation process.

Thirdly, not all stakeholders want to get involved and those who do get involved may not be clear about their roles or may not perceive themselves as evaluators.¹⁴ This was the case in half of the initiatives analyzed. We also detected a challenge in maintaining consistency in participation over long periods of time.

What Principles Emerge as Defining?

During the First Gathering of Participatory Evaluation Experiences in Latin America, participants engaged in a collaborative process of defining the principles of stakeholder involvement in evaluation (see Table 2). These principles were drawn from the practical experiences of the participants and are presented below. These principles—at times, in line with previous theoretical reflections—serve as postulates to guide this type of evaluation as it takes root in the region. They do not, however, constitute a closed set of principles. They remain open to dialogue with other present and future initiatives, as well as conceptual frameworks.

Table 2.

Principles of Stakeholder Involvement in Evaluation in Latin America.

1	Actors are actively and consciously incorporated into the evaluation process as subjects of rights.
2	Local knowledge is recognized as valid and essential knowledge for evaluation.
3	Institutional or organizational representatives work in partnership with local stakeholders.
4	External evaluation teams act as facilitators of the evaluation process.
5	The use of didactic techniques and materials facilitates dialogue and exchange.
6	Stakeholders take ownership of both the processes and the results of the evaluation.

Source: Prepared by the authors based on the discussions of the First Gathering of Participatory Evaluation Experiences in Latin America.

Actors are Actively and Consciously Incorporated Into the Evaluation Process as Subjects of Rights

In the case of Latin America, the participants in the First Gathering of Participatory Evaluation Experiences in Latin America consider stakeholders to be subjects of rights. They do not view them as objects of the intervention, individuals in need of protection, or mere beneficiaries, as often happens in conventional evaluations. Coming from a transformative perspective, they recognize stakeholders as beings that are empowered for speech and critical thinking, decision-making, and autonomy. They also recognize these stakeholders to be bearers of their own interests, expectations, and priorities. As pointed out by various authors, these proponents of stakeholder involvement in evaluation also recognize that participation relates to the democratic function of public policy evaluation (Cornwall, 2008; Monnier & Conan, 1995; Plottu & Plottu, 2009).

A commitment to the active involvement of the various actors has implications at every stage of the evaluation process, from its design through to the dissemination of results (Sanz et al., 2019). Such active involvement entails that those who make the decision to carry out an evaluation with stakeholder involvement must assume that this implies sharing decision-making at each stage of the evaluative process. This is nothing short of a ceding of power (Chouinard & Milley, 2018; Fetterman & Wandersman, 2018). Likewise, the understanding that participants are subjects of rights implies also recognizing the responsibilities determined by this condition. These characteristics align this first principle with the empowerment evaluation approach.

Local Knowledge is Recognized as Valid and Essential Knowledge for Evaluation

A fundamental facet of the incorporation of stakeholders in evaluation requires that governmental bodies and international development agencies recognize and value the knowledge and know-how of the local actors. This is also asserted in the literature (Chouinard & Milley, 2018; Gadotti, 2017; Kushner & Rotondo, 2012; Mbava, 2017; Nugroho et al., 2018). The legitimization of their knowledge (at times, the fruit of the actors’ reading of their own reality) forms the basis for advancing in transformative processes (Zukoski & Bosserman, 2018).

In the evaluation experiences that were shared, the way this knowledge unfolds is significant and is linked both directly and indirectly to the evaluation's specific focus. In fact, emerging elements of local knowledge, which may seem insignificant to the evaluative purpose at first glance, can become productively integrated into the evaluation when facilitators are sensitive to it. These characteristics align this principle more closely with stakeholder involvement approaches to evaluation that acknowledge the role of people in producing evaluative knowledge.

Institutional or Organizational Representatives Work in Partnership with Local Stakeholders

Stakeholder involvement in evaluation is a process, facilitated by professional evaluators, in which various stakeholders work in partnership to generate evaluative knowledge. These stakeholders include program managers as well as local stakeholders. In accordance with the framework of stakeholder involvement approaches to evaluation, the stakeholders are the ones who should define what is evaluated, who participates, when it happens, what data collection and analysis methods are used, and how the results are communicated.

Collaboration is required at every stage of the evaluation process. However, the delimitation of the focus of the evaluation and the negotiations that take place among the various actors can set the tone for the rest of the activities. Aubel (2000) notes that this first stage aims to generate moments of participation that are effective both for clearly delimiting the objective sought by participation and for structuring and channeling it during the evaluation process. This principle emphasizes the key role of the stakeholders during the whole evaluation process; thus, it reflects an empowerment evaluation approach.

External Evaluation Teams Act as Facilitators of the Evaluation Process

External evaluation teams can use participation in evaluation to contribute to empowerment and learning as well as improve the democratic culture of organizations and social groups. However, evaluation teams can also use it in a purely instrumentalist manner, reproducing existing power asymmetries and diminishing its transformative capacity (Cornwall, 2008; Hart, 1992; Mark & Shotland, 1985).

In this regard, the role of the evaluation facilitator gains relevance, as they are key to the success of any stakeholder involvement in evaluation approach. According to the Latin American experiences examined, and in line with other initiatives (Espinosa-Fajardo, 2019), it is important that the evaluation team is respectful and capable of managing groups as they collectively define what is important in the evaluation, decide how it should be carried out, and determine the results, conclusions, and recommendations of the evaluation.

The Use of Didactic Techniques and Materials Facilitates Dialogue and Exchange

In diverse and culturally complex contexts, inequalities in terms of power, voice, and the capacities of the various actors present a challenge to the effective development of participation. Stakeholder involvement in evaluation has confronted this challenge by creating innovative didactic materials and participatory tools that help communicate and recover the perspectives of all involved in the evaluation process (Chouinard & Milley, 2018).

In line with the experiences we have analyzed from Latin American practice, we have noted that the techniques activate both the mind and emotions, contributing to a greater and more real ownership of the reflections and findings. In addition, the tools enhance the emergence and development of critical thinking in the people involved in the process. The techniques, workshops and games seek to generate exchanges that deepen individual and collective knowledge while placing the stakeholders firmly at the center of the process. While all evaluation approaches that value and make space for stakeholders pay attention to the use of didactic techniques and materials, this principle most closely aligns with the framework and spirit of the collaborative evaluation approach.

Stakeholders Take Ownership of Both the Processes and the Results of the Evaluation

Stakeholder involvement in evaluation requires local actors to capture contextual specificities and ensure that their perspectives are included. This translates, from a transformative perspective, into a horizontal distribution of power in decision-making and deeper democratization. Active participation enhances the understanding of other points of view and facilitates access to a common interpretation of the program under evaluation. Furthermore, according to the experiences examined from the region, stakeholder involvement in evaluation generates a process through which local actors increase their opportunities to take ownership of the evaluation processes and, thus, assume more ownership of the results. This principle aligns perfectly with empowerment evaluation.

Discussion

Research on participation in evaluation is constantly evolving and is now a key issue in the 2030 Agenda and the “Global Evaluation Agenda” (2016–2020). In this regard, the launch of the Sustainable Development Goals constitutes a turning point in the theory and practice of evaluation given the prominence awarded to civil society (EvalPartners, 2015). The Latin American experiences shared here reveal the characteristics of the practice in this region. Some of these characteristics are common to other territories and reveal some shared advances and challenges related to stakeholder involvement in evaluation practice at global level. Our research has led us to propose a set of principles which, based on experiences taken from Latin America, we think are relevant to stakeholder involvement approaches in evaluation in other regions. We will now reflect on the key conditions required to advance stakeholder participation in evaluation in Latin America. These conditions are open to future discussion in this and other regions.

The Latin American experience underscores that stakeholder involvement in evaluation is based on the active and conscious participation of all individuals as subjects of rights and on the recognition of their local knowledge. It also relies on the collaborative and horizontal work of all stakeholders, promoted by the facilitation of evaluation teams and the use of didactic techniques and materials to ensure that stakeholders take on a sense of ownership of the evaluation. Facilitation can act like a lubricant for the gears that make up stakeholder involvement in the evaluation process. However, facilitation is sometimes a complex task that requires specific training.¹⁵ Techniques, games, or workshops can contribute to the expression of multiple perspectives from several social actors. This implies facilitating debates and trust, putting one's whole self into the dynamics of learning (not only thinking) as well as generating exchanges that allow one's own and mutual knowledge to deepen. In this sense, it requires detaching stakeholder involvement in evaluation from merely technocratic approaches, in which techniques and instruments are used for purely entertainment purposes and do not really help integrate the perspectives of local actors.

Progress in stakeholder engagement also requires changes to the evaluation culture and policy, evaluation approaches, and purposes, as well as the degree and mode of fostering horizontal relationships among the varied stakeholders. At present, the influence of governmental bodies and international development agencies (who fund the vast majority of the evaluations) define how evaluation is understood and implemented. While civil society organizations that have a focus on social justice tend to promote participation in evaluation, governmental bodies and agencies usually prefer more classical approaches and adopt a style of participation that is closer to an instrumentalist perspective than an empowering one. Therefore, proponents of stakeholder involvement in evaluation should prioritize working with public administrations and agencies to show them the benefits of integrating participation into their evaluations.

In the same vein, to ensure that evaluations do not reduce participation to mere consultation under the symbolic pretense of participation, the experiences we examined reveal the need to highlight social and power inequalities, manage conflicts of interest that emanate from them, and promote dialogue and exchange throughout the evaluation process. This would open up evaluative culture and policy to non-classical approaches that are committed not only to accountability, but also to learning and improvement. Likewise, evaluators need to define concrete mechanisms and apply tools to promote the effective involvement and participation of a diverse set of stakeholders that evaluations have historically left out in the region.

Furthermore, after examining the experiences we have mentioned, we believe that evaluation in the region has two pending tasks if varied stakeholders are to have an effective voice in initiatives. Firstly, it needs to build shared language on stakeholder involvement in evaluation which breaks away from the flexible and multiple uses of the concept of “participation.” Secondly, it needs to find ways to encourage active and empowering involvement among all stakeholders. This implies looking at two interconnected sides of the same coin: reflecting on the opportunity to participate and the capacity to participate. The first, linked to the evaluative culture and policy of the context, is determined by the institutional and political will of those who design and conduct an evaluation to create real spaces for participation. The second, linked to the evaluation process itself, is mainly determined by the attitudes and skills that the various stakeholders have developed through previous participatory experiences.

Regarding the opportunity to participate, the analyzed initiatives show that not all institutions or organizations that request an evaluation are really willing to facilitate and accompany processes of stakeholder involvement in evaluation. Challenges include whether or not the commissioning body is actually willing to go through with this kind of evaluation or question its legitimacy, and the availability of spaces and mechanisms required for dialogue among the varied social actors. Furthermore, Latin American institutions often reject or question the very democratic values that are required to bring about these changes. In line with the lessons drawn from the initiatives analyzed, it is particularly important to involve those who are in decision-making positions and who promote the institutionalization of participation in the entities that commission the evaluations. Likewise, time and economic resources are required to develop these types of processes which, given the above, are usually very limited and insufficient.

In relation to the capacity to participate, although in the Latin American context there is an increasing number of stakeholder involvement in evaluation initiatives, there are important challenges to be tackled. The capacity for participation is enhanced when institutions offer opportunities for participation and encourage stakeholders to become decisively involved in projects. However, as we have indicated, in Latin America there is a predominance of institutions where citizen participation is only promoted to a limited extent. These participation capacities can also be promoted during the evaluation process itself through active facilitation.¹⁶

In sum, we contend that evaluation practice needs to broaden its perspectives and recognize participation as a right. The initiatives analyzed here illustrate the value of involving stakeholders in all stages of the evaluation. In addition, these initiatives indicate a need to establish dialogue in evaluation so that facilitators can manage differences, train evaluation teams as facilitators of these types of practices, and know and make available techniques and materials that favor collaborative work.

In general terms, the experiences we have studied show that stakeholder involvement in evaluation in Latin America is emerging and developing. As in other parts of the world, stakeholder involvement in Latin American evaluation is advancing slowly. The evaluation community is becoming aware of the challenges involved in moving toward participatory activity that have the potential to transform institutions, organizations, and social life.

There are many types of participation in evaluation. The Latin American initiatives we examined indicate that not all types are of equal footing. A participatory approach should serve to contribute new and distinct evidence on policies and programs. Likewise, participatory evaluation should serve to build capacity so that organizations can have greater control over their own development. Stakeholder involvement in evaluation can also function as a tool to improve the capacity of various actors to reflect, analyze, and propose solutions from their multiple perspectives.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was partially funded by the National University of San Juan (UNSJ) and the National Council for Scientific and Technical Research (CONICET) from Argentina. The fieldwork was supported technically and financially by the German Institute for Development Evaluation (DEval).

ORCID iD

Julia Espinosa-Fajardo

Notes

References

Arnstein

S. R.

(1969). The ladder of citizen inquiry. Journal of the American Institute of Planners, 35(4), 216–224. https://doi.org/10.1080/01944366908977225

Aubel

(2000). Manual de evaluación participativa del programa. Involucrando a los participantes del programa en el proceso de evaluación. https://evalparticipativa.net/wp-content/uploads/2019/05/11.-manual-de-evaluacic3b3n-participativa-del-programa.pdf

Cardozo Brum

(2008). Gestión y evaluación participativas en políticas sociales. Política y Cultura, (30), 137–163.

Cardozo Brum

M. I.

Rosas Huerta

(2021). Avances recientes en la evaluación de políticas y programas públicos. Universidad Autónoma Metropolitana.

Cartland

Ruch-Ross

H. S.

Mason

Donohue

(2008). Role sharing between evaluators and stakeholders in practice. American Journal of Evaluation, 29(4), 460–477. https://doi.org/10.1177/1098214008326201

Cavanna

J. M.

(2019). El semáforo de eliminación de la pobreza. CAF.

Chouinard

J. A.

(2013). The case for participatory evaluation in an era of accountability. American Journal of Evaluation, 34(2), 237–253. https://doi.org/10.1177/1098214013478142

Chouinard

J. A.

Milley

(2018). Uncovering the mysteries of inclusion: Empirical and methodological possibilities in participatory evaluation in an international context. Evaluation and Program Planning, 67, 70–78. https://doi.org/10.1016/j.evalprogplan.2017.12.001

Cornwall

(2008). Unpacking “participation”: Models, meanings and practices. Community Development Journal, 43(3), 269–283. https://doi.org/10.1093/cdj/bsn010

10.

Coupal

(2000). Seguimiento y evaluación participativos basados en los resultados. Ottawa, Canada. https://evalparticipativa.net/wp-content/uploads/2019/05/03.-seguimieno-y-evaluacic3b3n-participativo-basado-en-resultado.pdf.

11.

Cousins

J. B.

Chouinard

J. A.

(2012). Participatory evaluation up close: An integration of research-based knowledge. Information Age Publishing.

12.

Cousins

J. B.

Whitmore

(1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5–23. https://doi.org/10.1002/ev.1114

13.

Cousins

J. B.

Whitmore

Shulha

(2013). Arguments for a common set of principles for collaborative inquiry in evaluation. American Journal of Evaluation, 34(1), 7–22. https://doi.org/10.1177/1098214012464037

14.

Cullen

A. E.

Coryn

C. L. S.

Rugh

(2011). The politics and consequences of including stakeholders in international development evaluation. American Journal of Evaluation, 32(3), 345–361. https://doi.org/10.1177/1098214010396076

15.

Daigneault

P. M.

Jacob

(2009). Toward accurate measurement of participation: Rethinking the conceptualization and operationalization of participatory evaluation. American Journal of Evaluation, 30(3), 330–348. https://doi.org/10.1177/1098214009340580

16.

Espinosa-Fajardo

(2019). La evaluación de programas y políticas de infancia: Aproximaciones clásicas y emergentes. In Del Moral

Gálvez

(Eds.), Infancia y bienestar. Una apuesta política por las capacidades y los cuidados (pp. 353–398). Editoral Deculturas.

17.

EvalPartners. (2015). EvalAgenda 2020. Global evaluation agenda (2016–2020). https://www.evalpartners.org/global-evaluation-agenda.

18.

Everitt

Hardiker

(1996). Evaluating for good practice. MacMillan Press.

19.

Fetterman

Rodríguez-Campos

Wandersman

O’sullivan

R. G.

(2014). Collaborative, participatory, and empowerment evaluation: Building a strong conceptual foundation for stakeholder involvement approaches to evaluation (A response to cousins, Whitmore, and Shulha, 2013). American Journal of Evaluation, 35(1), 144–148. https://doi.org/10.1177/1098214013509875

20.

Fetterman

Rodríguez-Campos

Zukoski

A. P.

(2018). Collaborative, participatory, and empowerment evaluation: Stakeholder involvement approaches. Guilford Press.

21.

Fetterman

D. M.

(2015). Empowerment evaluation and action research: A convergence of values, principles, and purpose. In Bradbury

(Ed.), The SAGE handbook of action research (pp. 83–89). Sage.

22.

Fetterman

D. M.

(2017). Transformative empowerment evaluation and Freirean pedagogy: Alignment with an emancipatory tradition. New Directions for Evaluation, 2017(155), 111–126. https://doi.org/10.1002/ev.20257

23.

Fetterman

D. M.

Wandersman

(2018). Essentials of empowerment evaluation. In Fetterman

D. M.

Rodríguez-Campos

Zukoski

A. P.

(Eds.), Collaborative, participatory and empowerment evaluation. Stakeholder involvement approaches (pp. 74–89). Guilford Press.

24.

Gadotti

(2017). The global impact of Freire’s pedagogy. New Directions for Evaluation, 155, 17–30. https://doi.org/10.1002/ev.20253

25.

Glaser

B. G.

Strauss

A. L.

(1967). The discovery of grounded theory. Aldine.

26.

Guba

E. G.

Lincoln

Y. S.

(1989). Fourth generation evaluation. Sage.

27.

Guijt

(2014). Participatory approaches: Methodological briefs—Impact evaluation. https://www.unicef-irc.org/publications/750-participatory-approaches-methodological-briefs-impact-evaluation-no-5.html. 21 pages.

28.

Hart

R. A.

(1992). Children‘s participation: From tokenism to citizenship. UNICEF.

29.

Hay

K. E.

Kumar-Range

(2014). Making evaluation matter: Writings from South Asia. Sage Publications.

30.

House

E. R.

(1980). Evaluating with validity. Sage.

31.

Huenchuan

Paredes

(2007). Guía metodológica para la evaluación participativa de políticas y programas, en el marco de la estrategia regional sobre el envejecimiento. CEPAL, CELADE, UN.

32.

IDS. (1998). Participatory monitoring & evaluation: Learning from the change. IDS Policy Briefing, 12. https://www.ids.ac.uk/download.php?file=files/dmfile/PB12.pdf

33.

King

J. A.

Cousins

J. B.

Whitmore

(2007). Making sense of participatory evaluation: Framing participatory evaluation. New Directions for Evaluation, 2007(114), 83–105. https://doi.org/10.1002/ev.226

34.

Kushner

Rotondo

(2012). Editors’ notes. New Directions for Evaluation, 27, 1–5.

35.

Letichevsky

A. C.

Penna Firme

(2012). Evaluating with at-risk communities: Learning from a social program in a Brazilian slum. New Directions for Evaluation, 2012(134), 61–76. https://doi.org/10.1002/ev.20019

36.

Mark

M. M.

Shotland

R. L.

(1985). Stakeholder-based evaluation and value judgments. Evaluation Review, 9(5), 605–626. https://doi.org/10.1177/0193841X8500900504

37.

Mbava

N. P.

(2017). Shifting the status quo: Africa influencing global evaluation practice. Evaluation Matters, 1(3), 12–21. http://idev.afdb.org/sites/default/files/Evaluations/2020-03/article%202%20%20Africa%20influencing%20global%20evaluation.pdf

38.

Monnier

Conan

(1995). Evaluación de la acción de los poderes públicos. Instituto de Estudios Fiscales.

39.

Nugroho

Carden

Antlov

(2018). Local knowledge matters: Power, context and policymaking in Indonesia. Policy Press.

40.

Patton

M. Q.

(2002). Qualitative research & evaluation methods. Sage.

41.

Patton

M. Q.

(2010). Future trends in evaluation. In Segone

(Ed.), From policies to results. Developing capacities for country monitoring and evaluation systems (pp. 44–57). UNICEF.

42.

Payne

Williams

(2005). Generalization in qualitative research. Sociology, 39(2), 295–314. https://doi.org/10.1177/0038038505050540

43.

Plottu

(2009). Contraintes et vertus de l’évaluation participative. Revue Française de Gestion, 2(192), 31–58. https://doi.org/10.3166/rfg.192.31-44

44.

Rodríguez-Campos

(2018). Essentials of collaborative evaluation. In Fetterman

Rodríguez-Campos

Zukoski

A. P.

(Eds.), Collaborative, participatory and empowerment evaluation. Stakeholder involvement approaches (pp. 10–20). Guilford Press.

45.

Romero Sarduy

M. I.

Moreno Moreno

Hernández Chávez

C. N.

(2017). Evaluación participativa por protagonistas: Una experiencia de innovación social. Revista Estudios del Desarrollo Social: Cuba y América Latina, 5(2), 81–91. http://www.revflacso.uh.cu/index.php/EDS/article/view/194

46.

Ryan

Greene

Lincoln

Mathison

Mertens

D. M.

(1998). Advantages and challenges of using inclusive evaluation approaches in evaluation practice. American Journal of Evaluation, 19(1), 101–122. https://doi.org/10.1177/109821409801900111

47.

Sanz

J. C.

Tapella

Meneses

(2019). Participatory evaluation of cancer prevention and care services. A case study from Valle de la Estrella, Costa Rica. In Cousins

J. B.

(Ed.), Collaborative approaches to evaluation: Principles in use (pp. 47–53). Sage.

48.

Scriven

(1973). Goal free evaluation. In House

E. R.

(Ed.), School evaluation: The politics and process (pp. 319–328). McCutchan.

49.

Shulha

L. M.

(2010, November). Participatory evaluation essentials: Highlighting the essential features of participatory evaluation [Paper presentation]. American Evaluation Association Conference, San Antonio, Texas, USA.

50.

Shulha

L. M.

Whitmore

Cousins

J. B.

Gilbert

Al Hudib

(2016). Introducing evidence-based principles to guide collaborative approaches to evaluation: Results of an empirical process. American Journal of Evaluation, 37(2), 193–215. https://doi.org/10.1177/1098214015615230

51.

Tapella

Rodríguez-Bilella

(2014). Shared learning and participatory evaluation: The sistematización approach to assess development interventions. Evaluation, 20(1), 115–133. https://doi.org/10.1177/1356389013516055

52.

Tapella

Rodríguez Bilella

Sanz

J. C.

Chavez-Tafur

Espinosa-Fajardo

(2021). Sowing & harvesting. Participatory evaluation handbook. German Institute for Development Evaluation.

53.

Tapella

Sanz

J. C.

(2019). La otra mirada. Evaluación participativa y mejora de los servicios de prevención y atención del cáncer en Valle de la Estrella, Costa Rica. Revista Gestión y Análisis de Políticas Públicas(22), 102–119.

54.

Trickett

E. J.

Ryerson-Espino

S. L. R.

(2004). Collaboration and social inquiry: Multiple meanings of a construct and its role in creating useful and valid knowledge. American Journal of Community Psychology, 34(1–2), 1–69. https://doi.org/10.1023/B:AJCP.0000040146.32749.7d

55.

United Nations Development Group (2017). Fulfilling promises: A practical guide for UN advocacy to promote implementation of the 2030 Agenda. United Nations.

56.

Zukoski

A. P.

Bosserman

(2018). Essentials of participatory evaluation. In Fetterman

D. M.

Rodríguez-Campos

Zukoski

A. P.

(Eds.), Collaborative, participatory and empowerment evaluation. Stakeholder involvement approaches (pp. 48–56). Guilford Press.