Multiple Paths to Effective National Evaluation Societies

Literature Review

We examined the existing case studies on NES (see among others Mihalache, 2009; Love & Russon, 2000; Segone, 2006, 2008b, 2009, 2010b; Segone & Ocampo, 2006; Russon & Love, 2003; The Presidents Panel, 1999) and visited their websites, guided by resource mobilization theory as well as the theoretical literature on political opportunity structure and performance of nongovernmental organizations (NGOs). This review reveals that NES aim to achieve a diverse set of goals, situated on both the demand and the supply sides of evaluation. NES generate domestic demand that goes well beyond aid-generated demand for evaluation, thus broadening the evaluation object (Ba Tall, 2009). This can help redirect government accountability from an overwhelming focus on donors to an increase in downward accountability to national citizens, clients, and the public in general. NES also aim to strengthen the supply side of evaluation through evaluation capacity development and networking (Ba Tall, 2009; Quesnel, 2006). Evaluation associations have been particularly instrumental in developing national evaluation standards, norms, and ethical codes for evaluation, and in “localizing” international evaluation guidelines and principles (Segone & Ocampo, 2006).

Drawing upon secondary data, the five main goals of evaluation societies can be summarized as (1) stimulate networks between M&E actors; (2) improve the evaluation capacities of their members; (3) increase evaluation practice; (4) strengthen the use/influence of policy and program evaluation; and (5) increase government accountability toward its citizens. For analytical purposes, we regrouped these five goals into two broad categories, that is, “member/organization-oriented” goals and “policy-oriented” goals. More specifically, the first two goals (stimulate networks between M&E actors and improve the evaluation capacities of their members) focus on the strengthening of NES and their members and have therefore been classified as member/organization-oriented goals, whereas the latter three (increase evaluation practice, strengthen the use/influence of policy/program evaluation, and increase government accountability toward its citizens) are better seen as policy-oriented goals. This regrouping is in line with the review study of Kriel (2006) who drew a distinction between associations that focus on the organization and structuring of an existing community of evaluation stakeholders—mainly practitioners, academics, and researchers—and organizations that seek to raise awareness and build a community of evaluation stakeholders. Kriel also points out that contemporary evaluation organizations tend to exhibit characteristics of both the groups.

With respect to identifying factors that may influence society effectiveness, we singled out determinants of organizational performance from the literature on resource mobilization theory and political opportunity structure. Our decision to combine these two theories was to a large extent inspired by a 1999 study on the performance of NGOs in India and Bangladesh, which showed “NGO performance to be the result of a dynamic interaction between external influences (context) and internal influences (organizational choices)” (Edwards, 1999, p. 363). A similar emphasis on both context and organizational determinants is adopted in Joachim’s (2003) analysis of the conditions under which NGOs are able to influence a state’s interests. Joachim found that NGOs’ success in mobilizing support was contingent on the dynamic interaction between the political opportunity structures in which they were embedded and the mobilizing structures they had at their disposal, including organizational entrepreneurs, a heterogeneous international constituency, and experts. Drawing upon these insights, our analytical framework incorporates contextual conditions that reflect the influence of the external environment on society performance alongside organizational characteristics necessary for NES to attain their organization and policy-oriented goals successfully. Although a full account of the two frameworks is beyond the scope of this article, we briefly summarize the main ideas that guided our research.

Research on social movements has primarily focused on the relation between organizational resources and their effect on the movement organizations’ political and mobilization activities. Resource mobilization theory centers on the premise that the amount of action an organization can take depends largely upon the available resources that it combines and uses (Andrews & Edwards, 2004; Cress & Snow, 1996; Edwards & McCarthy, 2004). Over time, a number of authors (Cress & Snow, 1996; Edwards & McCarthy, 2004) have developed typologies of resources that are important in explaining the organizations’ success, with distinctions drawn between material, human, social, cultural, and moral resources. The most straightforward categories are material and human resources, the first of which refers to financial resources, paid jobs, equipment, office facilities, and means of transportation, and the second to the skills, knowledge, and expertise of staff members and volunteers. Human resources are often distinguished from cultural resources, which refer to a kind of tacit knowledge of “how to behave,” and from social resources, which derive from the relations that organizations and their members have with other actors such as civil society organizations, government, and the media (Edwards & McCarthy, 2004). The last category, moral resources such as solidarity with other organizations, may be the most difficult to control, as their acquisition and loss are largely dependent upon external actors (Edwards & McCarthy, 2004). The importance of these resources for collective action by social movement organizations has already been confirmed for various settings, including homeless organizations, trade unions, and civil rights movements (Andrews, 2001; Cress & Snow, 1996; 2000; Ganz, 2000; Garcia & Parker, 2011). Similarly, the closely related concept of organizational capacity, defined by Eisinger as “a set of attributes that help or enable an organisation to fulfil its mission” (2002: 116), was found to be crucial in explaining the effectiveness of street-level charitable organizations. The concept of organizational capacity was operationalized as “resources, effective leadership, skilled and sufficient staff, a certain level of institutionalisation and links to the larger community from which an organisation might draw help” (Eisinger, 2002, pp. 117–118).

As organizations do not function in a social vacuum, and as evaluation is by nature a highly political undertaking (Weiss, 1987), we also assume that more contextual factors, particularly political context, can influence the effectiveness of NES. We borrow insights from the literature on political opportunity structure (POS; see McAdam, 1999; Tarrow, 1994), where POS is considered “the consistent—but not necessarily formal or permanent—dimensions of the political environment that provide incentives for people to undertake collective action by affecting their expectations for success or failure” (Tarrow, 1994, p. 85). The importance of institutional arrangements and changes in political opportunity structures for organizations’ effectiveness was recently demonstrated by Holzacker (2012), among others, who demonstrated that a favorable political climate and governmental interest are crucial in explaining the numerous strategies and effectiveness of lesbian gay bisexual transgender (LGBT) organizations in their struggle against discrimination in European countries. It is unlikely that closed opportunity structures, where checks and balances are curtailed, will stimulate evaluation. This is particularly true in cases where evaluation activities stimulate accountability, as they may jeopardize the positions of those in power. As Gordillo & Andersson (2004, p. 309) observe, “the more dramatic the power asymmetries amongst actors—that is the more power the political elites possess relative to the power of citizens and their political opposition and other actors—the weaker the incentive is for such an elite to transform established M&E programmes into an instrument for more public sector accountability.” In the context of low-income, aid-dependent countries, investment in evaluative exercises and knowledge can also be counterproductive when substantial amounts of aid can be generated without strong poverty-reducing credentials. As Pritchett (2002, p. 268) notes, “if a program can already generate sufficient support to be adequately funded then knowledge is a danger.”

Fine-Tuning the Analytical Framework: Building on Survey Data From NES

Effectiveness in goal achievement

In order to assess society effectiveness in achieving goals, we relied on organization leaders’ self-assessment. In doing so, we were guided by research on organizational performance of labor (Kalleberg & Moody, 1996) and local environmental organizations (Andrews & Edwards, 2005). We asked society leaders to indicate on a 5-point scale (not at all/slightly/somewhat/much/very much) the extent to which they felt their association had been able to contribute to the five goals identified in the literature review. The results (Figure 1) demonstrate that NES believe they have made a more substantial contribution to the two organization-oriented goals than to the three policy-oriented goals, where more factors are beyond the societies’ control. ³

OPEN IN VIEWER

Figure 1.

Perceived performance on five national evaluation societies (NES) goals.

Stimulating networks is considered the “easiest” goal to achieve, with almost 60% of the NES indicating that they had contributed meaningfully (much/very much), while just over 20% gave the same response for increasing government accountability, which is perceived to be the most “difficult” goal to reach. In the third and fourth sections, these goals were selected for more in-depth analysis with respect to factors that influence society performance. The first goal, stimulate networks between M&E actors, exemplifies the organization-oriented goals, whereas increase government accountability toward its citizens is representative of the policy-oriented goals. Although perceived performance on different goals within each broad goal set was found to be positively correlated, ⁴ no positive correlation was found between stimulate networks and increase government accountability.

Actor and context conditions

Survey respondents rated each of the five goals on the extent to which the organizational capacity factors (human resources, financial resources, social resources, etc.) and the context-related factors (political openness, government interest, donor interest) had been beneficial in realizing the society’s goals, using a 7-point scale (1 = completely insufficient and 7 = completely sufficient) to indicate their beliefs. An overview of the findings is presented in Figure 2. Given that one should keep the number of explanatory conditions at a moderate level in a QCA analysis (Schneider & Wagemann, 2010, p. 3), we included only those contextual and actor factors identified by respondents as being most relevant to goal achievement by NES (i.e., society performance on selected goals). These factors are summarized in Figure 3.

OPEN IN VIEWER

Figure 2.

Availability of resources for national evaluation societies (NES) goal achievement. Scale: 1 = completely insufficient; 2 = largely insufficient; 3 = somewhat insufficient; 4 = neutral; 5 = somewhat sufficient; 6 = largely sufficient; 7 = completely sufficient.

OPEN IN VIEWER

Figure 3.

Analytical framework for studying national evaluation societies (NES) performance.

Lack of financial resources was identified as the most important organizational capacity stumbling block in the realization of all five goals. This is not surprising, given that 40% of the NES in our survey had no financial resources, while another 30% had a budget of less than US$10,000 per year. These results are similar to what was found regarding available staff; 75% of all evaluation societies in our sample had no paid staff. The societies that did have paid employees generally had fewer than five staff members. Evaluation societies therefore relied mainly on the voluntary input of highly committed professionals working in M&E functions within the government, Civil Society Organizations (CSOs), universities, or consultancy firms. Approximately 70% of the evaluation societies surveyed relied on the support of 1–10 volunteers, while approximately 20% had more extensive volunteer networks.

The most positively rated asset of NES was the members’ interest in NES activities and training (average score = 5.62). This is also not surprising. Members of national evaluation associations come from diverse backgrounds, including government agencies, civil society organizations (CSOs), academia, the private sector, donor agencies, parliament, and media. These actors often specialize in different types of evaluation methodologies, with staff from statistical agencies specializing in quantitative methods, for example, and CSOs better equipped for participatory and qualitative evaluation methodologies. This diversity in methodological backgrounds offers opportunities for exchange, discussion, and cross-fertilization, while the wide and diverse membership also allows for rapid dissemination of innovative experiences and best practices. Members of NES also tend to have access to different types of information and resources. Parliamentarians, for instance, generally have easier access to national level policy data, academics often have an advantage in terms of analytical and evaluation tools, and CSOs frequently have more information about implementation realities on the ground and citizens’ perceptions of policy effects at the local level. This allows for useful triangulation of data, while facilitating discussion and counteranalysis. In addition to the worst hindrance (financial resources) and the biggest benefit (NES membership), the degree of activity in each society was included in our analytical framework in order to control for the fact that some of the organizations were still in the initial stages of development while others were well established (See Figure 3).

Of the contextual factors drawn from the political opportunity structure literature, respondents indicated that lack of support from parliament (average score = 2.78) and lack of media coverage (average score = 2.97) were the most important hurdles (after lack of financial resources) to overcome when attempting to induce government accountability. With regard to the goal of stimulating networks among M&E actors, lack of participation by government actors (average score = 3.47) ⁵ was listed as a crucial factor. In general, NES indicate the lack of media coverage, support from parliament, and participation by government actors as performance impediments. These factors are included in our analytical framework under the headings general political openness and government interest in NES (Figure 3).

In the remaining sections, we analyze in greater depth how these actor- and context-oriented factors influence societies’ effectiveness in reaching their member/organization-oriented goals (stimulate networks between M&E actors) and policy-oriented goals (increase government accountability toward citizens). As we are interested in the interplay of various factors, we have employed a crisp-set Qualitative Comparative Analysis (csQCA) for this analysis.

Operationalization of the Outcomes, Actor, and Context Conditions

The context and actor conditions are presented and operationalized in Table 1. In order to obtain an overview of the change in a country’s political openness, we selected one of the World Bank Governance subindicators’ voice and accountability (Kaufmann, Kraay, & Mastruzzi, 2012). The analytical framework also incorporated potential incentives for NES generated by an increase in government interest in society activities, operationalized by asking respondents to indicate on a 5-point scale the extent to which they felt that the interest government officials had shown toward evaluation societies had increased/decreased since the introduction of the Poverty Reduction Strategy Paper (PRSP).

OPEN IN VIEWER

Table 1.

Operationalization of the Analytical Framework.

	Condition	Operationalization	Calibration point for dichotomization
Context
	Increased Political Openness (o)	Change in political openness 2006–2010	Improvement in voice and accountability indicator (=1)
	Increased Government interest (g)	Change in Government interest since PRSP	Improvement of government interest in NES (=1)
Actor
	Availability of Financial Resources (e)	Paid employees	1 paid employee or more (=1)
	High Degree of activity (a)	Very high frequency of meetings	Meeting twice a year or more (=1)
	Extensive Membership (m)	Number of individual members	More than 50 members (=1)
Outcome
	High perceived performance stimulating networks (netw)	To what extent has the evaluation society contributed to stimulating networks among various actors involved in M&E (Government, Civil Society, Donors, Academia)?	Having contributed “Much” or more to stimulating networks (=1)
	High perceived performance stimulating government accountability (acc)	To what extent has the evaluation society contributed to increasing government accountability towards its citizens?	Having contributed “Somewhat” or more to stimulating government accountability (=1)

Note. NES = National evaluation societies.

With respect to actor conditions, financial resources were operationalized as the availability of paid employees. The NES budget would have been another possibility, but we preferred the availability of paid staff because the variable needed to be dichotomized. Moreover, size of the society’s budget and the number of its paid employees are positively correlated (Pearson coefficient = 0.504, p = .001). In addition to financial resources, NES indicated that extensive member interest in their activities was one of their most readily available assets. The survey collected data on the absolute number of individual and organizational members as well as on the composition of society membership. The variable retained for the final analysis was the number of individual members. The degree of activity of NES was operationalized as the frequency of organized meetings.

The two goals selected for our analysis, stimulate networks between M&E actors and increase government accountability toward its citizens, are measured by the survey questions that gauge NES’ perceived contribution to reaching different goals (see Appendix A, Questions 3 and 4).

The next step in csQCA is calibration to obtain dichotomized conditions. Using hypothesis-driven cutoff points is the best option (Schneider & Wagemann, 2010). Theory-based calibration was straightforward for most conditions. In the case of “increased political openness,” the calibration was set at zero (higher than zero is considered “increased political openness,” whereas lower or equal to zero is considered the absence of “increased political openness”). For “increased government interest,” NES were asked whether they felt that the interest of the government in society activities had changed since the introduction of the PRSP on a 5-item scale: decreased considerably/decreased slightly/status quo/ increased slightly/increased considerably (see Appendix A, Question 2). The cutoff was set at status quo, with higher than status quo considered to be increased government interest. For “availability of full time staff,” the cutoff was no paid staff versus at least one paid staff member. For “high degree of activity,” the cutoff point was “twice a year,” as we considered that the most standard practice would be to organize a general meeting of the society once a year. For the two outcomes—performance on stimulating networks among M&E actors and stimulating government accountability—the desirable cutoff points were not as self-evident. Both were measured on a 5-item scale (not at all/slightly/somewhat/much/very much). “High performance” needed to be differentiated from “not high performance” on the two goals. We opted not to use the same calibration point for the two goals, because stimulating networking among M&E actors is less challenging (even in ideal circumstances) than stimulating government accountability. Therefore, we placed the cutoff point for “high perceived performance on stimulating networks” at “Much” (i.e., much or very much), while the cutoff point for “high perceived performance on stimulating accountability” was set at “Somewhat.” For “extensive membership,” we used the median number of members as the cutoff.

These conditions were subsequently employed in the csQCA analysis to explore how the interplay between context and actor conditions leads to high performance on policy- or organization-oriented goals. The equifinality principle central to the QCA approach allows us to identify various paths of multiple causal conditions for high performance of NES. For example, an understaffed, moderately active evaluation society might perform well in an increasingly politically open environment without any increase in government interest, while another evaluation society might perform equally well in a politically deteriorating society as long as it had many members and a very high activity rate.

Paths to Increasing Government Accountability Toward Citizens

A raw data table was constructed in which the original data from the survey for the three actor conditions (paid employees, extensive membership, and high degree of activity) and the two context conditions (increased political openness and increased government interest) were dichotomized into 1 (present/high) and 0 (absent/not high).

Ten cases ⁷ were excluded from our population of 40 NES because of missing data on one of the five conditions or on one of the two outcomes. Additionally, removing two contradictory cases reduced the final number to 28 cases. When identical causal paths lead to two different outcomes, csQCA identifies them as contradictory cases (C). The analyses of societies’ effectiveness in stimulating government accountability and stimulating networks generated two contradictory cases (Indonesia–Brazil for accountability and North Uganda–Cameroon for networks). csQCA uses various strategies to handle contradictory cases: reconsider case selection, add conditions, or reconceptualize the outcome (Schneider & Wagemann, 2010). Ragin (1987, p.113) suggests the best way to approach the problem of contradictions is “to follow the lead of case-oriented researchers (…) by examining the troublesome cases in greater detail and attempt to identify omitted causal variables.” Additionally, one can reconsider the calibration of both the conditions and the outcome, check for measurement error, reconceptualize the outcome, or adjust the case selection (Schneider & Wagemann, 2010). However, when this is not possible the only remaining option is the imposition of a consistency threshold—often used in fuzzy set analysis—leading to the exclusion of contradictory cases (see e.g., Sager & Andereggen, 2012). Therefore, after checking for measurement errors and incorrectly specified concepts, our strategy was to explore the contradictory cases. In the case of Brazil, we contacted the society for a more in-depth discussion of the results and potential explanatory factors. The exchange with the Brazilian society made clear that the organization had a clear vision and an extensive knowledge of M&E and the various goals the society wanted to accomplish, and that they were highly self-critical in their assessment. Given the fact that crisp-set QCA requires dichotomization of data and that the actual scores of Indonesia and Brazil differed by only two points, it is likely that a more self-critical attitude, combined with the dichotomization procedure, led to diverging outcomes for identical causal paths. For the North Ugandan and Cameroonian cases, the fact that the North Ugandan society is a regional organization operating in a political opportunity structure that is characterized by conflict might be key. However, given that those two conditions only fit North Uganda, including this case would not strengthen the overall understanding of society performance. Rather, it would reduce the causal paths to idiosyncratic explanations. Therefore, both the Brazilian case and the North Ugandan cases have been omitted. In line with QCA’s objective of learning from contradictory cases, we see the degree of self-critical attitude, the regional scope of NES, and/or having to operate in conflict situations as potentially important factors in explaining the performance of NES.

Five dichotomized conditions generated 2⁵ logically possible combinations. Of those 32 combinations, 20 were observed, 10 of which were classified as having a positive outcome, while 10 were linked to nonoccurrence of the outcome (i.e., no/modest contribution to increasing government accountability). Twelve logical remainders—logical combinations for which no (not enough) cases were found—are listed in the truth table. ⁸

The data were analyzed using Tosmana (Cronqvist, 2011) and fsqca (Ragin, Kriss, & Sean, 2006). In total, five conditions were introduced for 28 cases that yielded a proportion of conditions to cases of 0.18, almost half of what Marx suggests as the upper limit (ratio = 0.33) for conducting meaningful QCAs (Marx, 2010). Our 28-case analysis of 5 conditions thus also surpasses the minimum of 18 cases put forward in the benchmark table developed by Marx and Dusa as a threshold for generating non-random meaningful conjunctural paths (Marx & Dusa, 2011).

Necessary Conditions for High/not High NES Performance in Stimulating Government Accountability

Before proceeding with analysis of sufficient causal conditions, we performed a necessity analysis, as the minimization procedure used for obtaining sufficient causal conditions could mask necessary causes or produce false necessary conditions (Schneider & Wagemann, 2010). Table 2 shows the results of the analysis of necessary conditions for the outcome of high performance (1) and not high performance (0) for both goals (stimulating networks and accountability). Our analysis follows Schneider and Wagemann’s recommendation to use high thresholds for necessity (consistency = 0.90; Schneider & Wagemann, 2010). No single condition was found to be necessary, even though “high degree of activity” comes close to being necessary for high performance on accountability, while a deteriorating or stable political openness comes close to being necessary for low performance on accountability.

OPEN IN VIEWER

Table 2.

Necessity Test of the Presence and Absence of High Performance on Both Goals (Stimulating Networks and Government Accountability).

Outcome = high performance on networking (1)				Outcome = not high performance on networking (0)
	Condition tested	Consistency	Coverage	Condition tested	Consistency	Coverage
Actor	Employee (1)	0.353	0.667	Employee (0)	0.727	0.412
	Member (1)	0.588	0.714	Member (0)	0.636	0.500
	Activity (1)	0.765	0.722	Activity (0)	0.545	0.600
Context	Openness (1)	0.412	0.636	Openness (0)	0.636	0.412
	Gov (1)	0.706	0.800	Gov (0)	0.727	0.615
Outcome = high performance on accountability (1)				Outcome = not high performance on accountability (0)
	Condition tested	Consistency	Coverage	Condition tested	Consistency	Coverage
Actor	Employee (1)	0.429	0.667	Employee (0)	0.786	0.579
	Member (1)	0.571	0.571	Member (0)	0.571	0.571
	Activity (1)	0.857	0.667	Activity (0)	0.571	0.800
Context	Openness (1)	0.643	0.818	Openness (0)	0.857	0.706
	Gov (1)	0.571	0.533	Gov (0)	0.500	0.538

Note. (1) = the presence of a condition; (0) = absence of a condition.

Interestingly, the latter finding does not hold for networking that highlights how political openness plays a different role for stimulating networking than for accountability. The absence of increased government interest is the context condition that comes closest to being a necessary condition for low performance on networking. High degree of activity is, for high performance on networking, an almost necessary condition even though the consistency is considerably lower than for high performance on accountability. Thus, high degree of activity and decreasing/stable political openness are, respectively, almost necessary conditions for high performance and not high performance on stimulating government accountability. However, strict application of the thresholds indicates that there are no single conditions necessary for high/not high performance on either of the goals.

Sufficient Conditions for the Outcome “High Performance on Accountability”

In a second step, we identify which configurations of conditions are sufficient for the outcome of high performance on stimulating government accountability toward citizens.

QCA analysis generates a “truth table” (in Appendix B) in which the cases are grouped according to the identical causal paths displayed. In other words, NES that had the same (dichotomized) score for the five conditions were grouped together. The truth table contains the dichotomized data of the five conditions and the outcome (increasing government accountability) for the various causal paths.

In the next step, QCA applies Boolean algebra to minimize the combinations for the occurrence of the outcome. Boolean algebra employs “+” as OR and “*” as AND. Capital letters indicate the presence of a condition, while lower case letters indicate the absence of a condition. Two context conditions, “O” (increased political openness) and “G” (increased government interest in the evaluation society), and three actor conditions, “A” (high level of society activity), “M” (extensive society membership) and “E” (paid employees) were included in the analysis.

The csQCA output is based on a process of logical minimization ⁹ (Grofman & Schneider, 2009; Ragin, 1987). Many csQCAs suffer from limited diversity, which means that the number of logical combinations (equalling 2 ^k where k is the number of conditions) is much higher than the combinations that are detected empirically. As csQCA examines the combination of conditions, limited diversity can pose problems. Therefore, hypothetical cases (called logical remainders) based on substantive and theoretical knowledge are often introduced to further minimize the outcome, often differentiating between “easy” and “difficult” counterfactuals (Ragin & Sonnett, 2004; Rihoux & Ragin, 2009; Sager &Andereggen, 2012). However, as this study is to a large extent an inductive exploratory study, we could not build upon in-depth theoretical knowledge of the issue and therefore no hypothetical cases are included. The outcome shown in the solution formula below and in Tables 3 and 4 is the complex solution ¹⁰ of the csQCA, which implies that no logical remainders have been included. The “→” symbol indicates that the causal configurations to the left are sufficient for the outcome to the right.

O g m A + O G e M + O G e a + o g M A + g E A + o E m A → A C C O U N T A B I L I T Y . 11

Six different configurations were identified, which indicates that all six combinations of context and actor conditions are sufficient configurations for a positive outcome (i.e., high perceived society performance for increasing government accountability). The first causal path leading to high society effectiveness in terms of government accountability (OgmA) reads as follows: an evaluation society operating in an environment of increasing political Openness combined with no increased governmental interest in the society, with a relatively low number of members but displaying a high level of society Activity, is sufficient for high perceived performance on increasing the government’s accountability toward its citizens. The evaluation society in both Ghana1 (one of the two NES in Ghana) and Zanzibar follows this causal path to stimulating government accountability toward its citizens (Table 3). The second path (OGeM) and the third path (OGea) feature an evaluation society that is functioning in an increasingly open political climate and that can count on an increasing interest in society activities from the government, without having paid employees at its disposal. Although only the second path is also blessed with a wide membership network, the third path is sufficient for the outcome even without being very active. The former (OGeM) seems to be the most prominent path toward an effective evaluation society in terms of generating accountability, since five NES (Pakistan, Ivory Coast, Guinea, Ghana2, and Indonesia) fit this configuration. The combination of a stimulating environment (government interest and political openness) combined with an extensive membership thus seems to be a viable path to success in stimulating accountability for many NES even without having full-time staff to rely upon. Additionally, other NES (Tanzania and Indonesia) can accomplish a positive result in a stimulating environment without necessarily being very active (meeting more than once a year; OGea).

OPEN IN VIEWER

Table 3.

Results QCA Analysis for “Stimulating Government Accountability.”

Outcome = High performance on stimulating accountability			Raw coveragea	Unique coverage
NES	Context	Actor
Pakistan, Ivory Coast, Ghana2, Guinea, Indonesia	Openness (1) Gov(1)	Employee (0) Members(1)	0.357143	0.285714
Tanzania, Indonesia	Openness (1) Gov (1)	Employee (0) Activity (0)	0.142857	0.071429
Ghana1, Zanzibar	Openness (1) Gov(0)	Activity (1) Members (0)	0.142857	0.071429
Georgia, Malaysia, DRC	Openness(0)	Employee(1) Activity (1) Member (0)	0.214286	0.142857
Senegal, Niger	Openness(0) Gov(0)	Activity (1) Members (1)	0.142857	0.071429
DRC, Philippines, Ghana1, Niger	Gov(0)	Employee (1) Activity (1)	0.285714	0.071429
			Solution coverage 1.000
			Solution consistency 1.000

Note. (1) = the presence of a condition; (0) = absence of a condition; QCA = Qualitative Comparative Analysis.

^a The table lists coverage scores that indicate the percentage of cases that follow a particular causal path to the outcome. It allows us to evaluate the importance of different causal paths (Fiss, 2011). “Raw coverage refers to the proportions of memberships of the outcome explained by each term of the solution, while unique coverage refers to the share of the outcome by one of the solution terms that is not covered by any other solution terms. A high (raw or unique) coverage is an indication that a certain path is relevant, i.e., has empirical weight. The consistency score expresses the degree to which a given condition in each solution term is a subset of the outcome. Solution coverage indicates shares of memberships of the outcome that are explained by the complete solution, and solution consistency indicates the degree to which the whole set of solution terms is a subset of the outcome” (Hellstrom, 2011, p. 13).

Next are three causal paths that regroup NES with strong organizational features (two actor conditions present). Building on their extensive membership network and high degree of activity, the NES in Senegal and Niger have been able to contribute meaningfully to stimulating government accountability toward the citizens in their country, even in a nonconductive context (ogMA). Both the fifth (gEA) and sixth (OEmA) causal paths build on the combination of having paid staff and a very high degree of society activity, the former combined with the absence of increased government interest, while the latter has no increased political openness or extensive membership network. The former cases include NES in DRC, the Philippines, Ghana1, and Niger (ogAM) and the latter Georgia, Malaysia and again DRC (oEAm).

Table 3 summarizes the results. As was mentioned, two conditions appear to be very important in explaining evaluation societies’ performance in stimulating accountability, an increasing political openness (O) and a very high degree of activity (A). The results and the solution formula show a first group of paths in which NES operating in an increasingly politically open environment combined with various organizational strengths (number of members or very high degree of activity) are able to stimulate government accountability. A second group of paths consists of very active NES that, building on the organizational advantage of paid employees or extensive membership network, are able to contribute to accountability.

O g A m + G M e + G e a + A o g M + o E m + g E → A C C O U N T A B I L I T Y

The results of the second group suggest that even in not fully stimulating environments, national evaluation societies can contribute to increasing government accountability through a combination of organizational strengths (paid staff, members, and activity).

As not all causal paths are equally frequent in reality, Table 3 shows the various paths as well as the relative importance of the different paths. The most prominent causal path (unique coverage 0.28) combines increasing political openness and government interest with an extensive membership network without being able to rely upon paid staff (OGMe), thus pointing to the importance of having a stimulating context and an extensive membership network. This dominant path to performance is largely in line with our analytical framework emphasizing the importance of the two context conditions and the membership condition, the latter being the most important positive asset identified by the survey. This path suggests that paid staff are not crucial for NES to meaningfully contribute to accountability. The second most prominent causal path to performance occurs when NES do not function in a fully stimulating environment, but where the combination of paid staff members with a very active society enables NES to contribute to accountability (unique coverage 0.14). This latter result confirms the role of financial/human resources and activity for the functioning of evaluation societies. In another causal configuration (unique coverage = 0.07), NES were able to contribute to government accountability through a combination of intense activity and large numbers of members in spite of the absence of a stimulating context. This suggests the possibility that in those cases a lack of resources was to a certain extent compensated by extensive membership.

Sufficient Conditions for the Outcome “High Performance on Stimulating Networks Between M&E Actors”

The following solution formula reflects the findings of the csQCA focused on high performance in stimulating networks between M&E actors.

O G e M + O G e a + O g E A M + o e A + o A m + o E a M → N E T W O R K I N G 12

which can be rewritten as

O G e M + G e a + g E A M + o e A + A m + E a M → N E T W O R K I N G

A comparison of the results from the two analyses reveals similar findings for the membership-oriented and policy-oriented goals. Specifically, two paths toward stimulating networks can be discerned. A first group of NES can function in a stimulating environment of increasing political openness and build on organizational strengths or, in the absence of organizational strengths, combine with government interest to stimulate networking. The second group, confronted with a deteriorating political climate, is able to strengthen networks by building on a combination of organizational strengths, mostly by being very active (A) or having an extensive network in combination with having paid staff (ME).

Table 4 shows the relative importance of the different paths. The combination that is most prominent, as expressed by a raw coverage score of 0.29 (and a unique coverage of 0.24), represents an evaluation society with many members, operating in an environment characterized by increasing political openness and government interest in evaluation societies (such as Pakistan, Ivory Coast, Ghana2, Guinea, and Indonesia). Interestingly, this path is identical to the earlier “textbook” path for stimulating accountability. The second most prominent causal combination (unique coverage = 0.17) identifies NES that stimulate networking by being very active even without having a large membership base in an environment that is not characterized by increased political openness. Moreover, the share of paths (i.e., sum of the unique coverage), where political openness is present is equal to that of the paths leading to positive outcomes in politically deteriorating climates.

OPEN IN VIEWER

Table 4.

Results QCA Analysis for “Stimulating Networks Among M&E Actors.”

Outcome = High performance on stimulating networking			Raw coverage	Unique coverage
	Context
Pakistan, Ivory Coast, Guinea, Ghana2, Indonesia	Openness (1) Gov(1)	Employee (0) Members(1)	0.294118	0.235294
Tanzania, Indonesia	Openness (1) Gov(1)	Employee (0) Activity(0)	0.117647	0.117647
Philippines	Openness (1) Gov(0)	Employee (1) Activity (1) Members(1)	0.058824	0.058827
Mali, Morocco, Cameroon, DRC, Georgia, Malaysia	Openness(0)	Activity(1) Members(0)	0.352941	0.176471
Mali, Morocco, Cameroon, Kenya, Senegal	Openness(0)	Employee(0) Activity (1)	0.294118	0.117647
Zambia, Rwanda	Openness(0)	Employee (1) Activity(0) Members (1)	0.117647	0.117647
			Solution coverage 1.000
			Solution consistency 1.000

Note. (1) = the presence of a condition; (0) = absence of a condition; QCA = Qualitative Comparative Analysis.

Paths to NES Effectiveness: Linking the Two Goals

In order to provide a comprehensive picture of evaluation societies’ effectiveness with respect to both policy- and membership-oriented goals, we combined the two sets of analyses. The most prominent path to high performance on both goals turned out to be a combination of increased political openness and government interest combined with an extensive membership base, though lacking any paid staff. Both analyses clearly indicate that other paths to effective NES exist. Through particular combinations of organizational strengths, other evaluation societies were able to contribute to both policy- and organization/member-oriented goals despite the more difficult contextual circumstances in which they had to operate.

These findings are presented in Figure 4. The x-axis and y-axis represent, respectively, a evaluation society’s contribution to increasing government accountability and to stimulating networks between M&E actors. ¹³ The cutoff score on each question is represented by a horizontal or vertical line, dividing the graph into four quadrants. The colors highlight the environment in which NES operate: black indicates an entirely nonconducive context characterized by lack of improvement in both political openness and government interest, while white signals environments characterized by two positive context conditions.

OPEN IN VIEWER

Figure 4.

National evaluation societies (NES) contribution to stimulating networking and increasing accountability. Black = no increased political openness + no increased government interest; dark Gray = no increased political openness + increased government interest; light gray = increased political openness + no increased government interest; white = increased political openness + increased government interest.

These results show that most NES working in a nonreceptive environment find it difficult to contribute to both the membership/organization and policy-oriented M&E goals, as almost all of them are located in the third quadrant (relatively low contribution to networking and to domestic accountability). In the first quadrant, on the other hand, where a society’s perceived performance on both the goals (networking and accountability) is high, most evaluation societies function in a stimulating environment (highlighted in white), with the exception of Senegal, DRC, Georgia, and Malaysia. Senegal is an exception in this quadrant, as it manages to perform well despite a nonstimulating environment and without having paid staff but building on its extensive membership network and high activity rate. The latter three countries were able to perform well on both goals despite the fact that political openness had not improved during the previous 5 years (though in two countries government interest in NES had increased - dark grey). In all three countries, however, NES employ at least one full-time employee and display a high degree of activity, which seemingly enabled them to perform well on both the goals despite the fact that they operated in a less-than-optimal environment. The same dark gray is found in the second quadrant, where some NES were performing well on the membership/organization-oriented goal but not high on the accountability goal, in environments that were not becoming more politically open. This is not counterintuitive, although inducing accountability may not be straightforward in countries where political openness is not improving (such as Morocco or Zambia), it does not necessarily forego the promotion of networks between M&E actors. However, in countries where the government’s interest in evaluation societies had not increased, but where the political space was widening, NES (such as Zanzibar and Ghana1) seem to perform well on inducing government accountability toward the general public (fourth quadrant).

Applying a more dynamic perspective to Figure 4 may also generate suggestions for strategies toward more effective evaluation society. For example, increasing government interest in NES in nonconducive environments (mostly in the third quadrant) could enable them to perform better in terms of stimulating networks between M&E actors (a shift toward the second quadrant), while the opening up of political space could allow them to contribute more to inducing government accountability (a shift toward the fourth quadrant).