Abstract
In developing countries, an increasing number of civil society organizations (CSOs) engage in independent monitoring and evaluation (M&E) of government programs and policies. Most CSOs rely on a range of M&E tools in combination with advocacy strategies to hold government accountable and improve the implementation of programs and policies. Despite the popularity of such initiatives, their effectiveness and impact remain unconfirmed and are not well understood. In addition, little is known about the influence of this type of CSO-led M&E at the district level. Using a case study design, the current research provides a map of the different influence mechanisms that occurred following the M&E of the National Health Insurance by Ghanaian CSOs. The research further suggests that the built-in dialogue space is acting as a catalyst for certain influence mechanisms.
In the past decade, an increasing number of civil society organizations 1 (CSOs) have engaged in independent monitoring and evaluation (M&E) of government programs and policies. In developing countries, this type of independent M&E initiatives mainly emerged within the context of the Poverty Reduction Strategy Papers (PRSPs) 2 and the related aid reform agenda. Donors actively supported these independent M&E initiatives because they believed that they would increase domestic accountability (Hickey & Mohan, 2008; Lucas, Evans, & Pasteur, 2004) and that CSOs were in a better position to monitor government programs and policies at the decentralized level because of their proximity to the grassroots (Goetz & Jenkins, 2001).
In a more recent evolution, many of these earlier independent M&E initiatives have been “rebranded” as social accountability initiatives. There is a conceptual fuzziness surrounding social accountability initiatives, but in general, they can be defined as “citizen-led action for demanding accountability from providers” and/or public officials (Joshi & Houtzager, 2012, p. 146). Within the spectrum of social accountability initiatives, some focus on M&E through the use of public expenditure tracking, citizen report cards, and community score cards; some concentrate on transparency, policy advocacy, budget literacy, civic education, lobbying, or coalition building; and some combine several of these elements (McNeil & Malena, 2010, p. 8). The type of independent M&E the current article is studying would correspond to a particular subgroup of social accountability initiatives that is carried out by CSOs who primarily rely on a range of monitoring tools to assess the implementation of a particular government program or policy. CSOs use the gathered evidence in combination with a targeted advocacy strategy to hold government officials and/or service providers accountable. Social accountability initiatives such as lobbying efforts of CSOs to pass the Right to Information Bill or pressuring government to open up policy-making spaces are excluded from this particular subgroup as they lack the M&E component. For the purpose of this article, I have used the term CSO-led M&E to refer to this type of M&E that mainly occurs in developing countries.
Social accountability initiatives, and thus also CSO-led M&E, are burgeoning across the globe due to their increasing popularity with donors and the availability of funds such as the World Bank’s Global Partnership for Social Accountability (http://www.worldbank.org/gpsa). However, little is known about the outcomes of such initiatives in terms of their effectiveness and impact (Joshi & Houtzager, 2012; McGee & Gaventa, 2011). This paradox can be partially explained by the conceptual fuzziness, the scarcity of large-scale, comparative research on the topic, and a lack of understanding of how these initiatives work. An exception is the large-scale study by McGee and Gaventa (2011) that reviews the available evidence relating to the effectiveness and impacts of such initiatives and attempts to identify contributing factors. McGee and Gaventa point out that the available evidence is uneven, not conclusive, and at times contradictory, especially regarding impact as only a handful of rigorous impact studies have been carried out (see e.g., Björkman & Svensson, 2009).
The purpose of this article is to draw attention to a particular type of M&E carried out by CSOs and to understand the consequences of this type of CSO-led M&E by focusing on the underlying influence mechanisms (Henry & Mark, 2003; Mark & Henry, 2004, 2012). By doing so, and because of the intertwining of CSO-led M&E and social accountability initiatives, the article will also contribute indirectly to the growing body of research on social accountability initiatives. To study the influence of CSO-led M&E at the district level, the article focuses on the work of the Ghanaian nongovernmental organization, Social Enterprise Development (SEND-Ghana) Foundation. Almost a decade ago, at the height of the PRSP frenzy, SEND-Ghana started monitoring the use of Heavily Indebted Poor Countries (HIPC) funds in combination with advocacy strategies. Since then, they have embarked on the monitoring of several other pro-poor government programs and policies such as the Ghana School Feeding Program, the District Assembly Common Fund, and the National Health Insurance Scheme (NHIS) which is taken as an example in the current study.
The article will proceed as follows. The theoretical framework that underpins the current study is explained in the next section and mainly relies on Henry and Mark’s (2003) and Mark and Henry’s (2004) Theory of Evaluation Influence. The framework will be applied to SEND-Ghana and the way in which SEND-Ghana is carrying out its M&E will be briefly explained. The subsequent sections will then explain the methodology used in the current research, outline and explain the main results, and discuss some implications in the concluding section.
Theoretical Framework
Because social accountability initiatives, including CSO-led M&E, are geared toward demanding accountability from public officials and/or service providers, the first step is to deconstruct 3 the concept of accountability into its different dimensions, which are answerability and enforceability (Schedler, 1999). Answerability refers to the obligation of power holders to provide information about their actions (transparency) and the reasons behind these actions (justification); enforceability refers to the creation of appropriate incentives for power holders to respond and/or take corrective action (Schedler, 1999, p. 15). This second dimension of accountability is particularly challenging for CSOs because of the unequal power relation between CSOs and service providers and/or public officials (Rubenstein, 2007). CSOs mostly resort to “soft” sanctions, such as media exposure, “naming and shaming,” or dialogue (Newell & Bellour, 2002) to pressure public officials and/or service providers to take action. Another documented strategy, especially in the Latin American context, is to trigger the reaction of formal oversight institutions who have more power and autonomy to apply heavier sanctions (Peruzzotti & Smulovitz, 2006; Rubenstein, 2007). Because of these challenges, most social accountability initiatives that rely on CSOs, whether or not they have a strong M&E component, are skewed toward strengthening transparency and answerability at the expense of enforceability.
The other main function of M&E is to improve programs/policies and to provide valuable information for use in decision making (Preskill & Caracelli, 1997). Theories on evaluation use 4 are trying to understand how and when M&E findings are used by program managers and policy makers and to identify contributing factors. The basis of evaluation theory was established early on with the definition of the main uses of evaluation: instrumental use, conceptual use, and symbolic or political use (Leviton & Hughes, 1981; Weiss, 1977, 1979). Later on, Patton (1997) pointed out that not only evaluation findings but also the process of participating in an evaluation could have consequences; this was referred to as process use. While there have been excellent reviews of those earlier theories elsewhere (Fleischer & Christie, 2009; Preskill & Caracelli, 1997; Shula & Cousins, 1997), the main focus here is on the concept of evaluation influence. Kirkhart (2000) proposed the term influence to overcome the instrumental and one-dimensional connotation of “evaluation use.” Henry and Mark (2003) and Mark and Henry (2004) went one step further and developed the Comprehensive Theory of Evaluation Influence. In their Theory, 5 evaluation is considered as an intervention, which opens up the possibility to formulate a theory of change with defined inputs, activities, outputs, and outcomes (see different column boxes in Figure 1) that can result in social betterment (box far right) in the long term. In a nutshell, the “M&E inputs” or context in which the evaluation takes place will influence the “M&E activities” in terms of research design, data collection, information dissemination, and others (see Figure 1). At the output level, general influence mechanisms are triggered as a consequence of the M&E activities, findings or both. In practice, it is often difficult to disentangle both sources of influence (see also Kirkhart 2000). These general influence mechanisms in turn will trigger other outcomes at the cognitive/affective, motivational, behavioral, and relational level. A menu of possible influence mechanisms at each level of analysis—individual, interpersonal, and collective—is provided subsequently (for details, see columns of Table 1). The table illustrates the broad range of ways in which M&E can have an effect at different levels beyond direct, instrumental use.
The influence of SEND-Ghana’s M&E: theoretical model. Adapted from Mark and Henry (2004, p. 46) and complemented with the “Relational Mechanisms” category as described in Mark (2006) and Mark and Henry (2012). For detailed description of the different categories of influence mechanisms, see Table 1. SEND = Social Enterprise Development Foundation; M&E = monitoring and evaluation; LT = Long Term.
Overview of Different Influence Mechanisms.
Source. Mark and Henry (2004, p. 41) complemented with the “Relational Mechanisms” category as described in Mark (2006) and (Mark & Henry, 2012).
Each entry of the table can be both “an outcome of evaluation and mechanisms that stimulate other outcomes” (Mark & Henry, 2004, p. 43). The terms outcome and influence mechanisms are thus used interchangeably in this article. Further, influence pathways can be reconstructed, or simulated, by specifying the particular sequence of influence mechanisms that has occurred, or needs to occur, to reach a specific type of outcome, for example, a change in the way NHIS ID cards are distributed (collective change in behavior, see Table 1; Henry & Mark, 2003; Mark, 2011; Mark & Henry, 2004).
One of the strengths of the Theory of Evaluation Influence is its flexibility and adaptability in building a context-specific, local theory of evaluation influence (Henry & Mark, 2003). Nonetheless, empirical applications of the Theory (see, e.g., Christie, 2007; Weiss, Murphy-Graham, & Birkeland, 2005) have been scarce so far, especially in a developing context. The Theory as depicted in Figure 1 is already adapted to the specific context in which SEND-Ghana operates (M&E inputs) and the way in which SEND-Ghana is carrying out its M&E and engaging stakeholders in discussing the findings during the interface/policy dialogue (M&E activities). The overall aim is to understand the type of influence mechanisms associated with this specific type of CSO-led M&E at the district level in the context of Ghana.
SEND-Ghana’s Participatory M&E (PM&E) Framework
SEND-Ghana monitors the implementation and performance of several pro-poor government programs and policies, as mentioned in the introduction. To carry out its M&E activities, SEND-Ghana operates through a decentralized network structure of District Citizen Monitoring Committees (DCMCs) and focal nongovernmental organizations (FNGOs), which “host” these DCMCs. To ensure a broad constituency, the DCMCs are made up of two District Assembly members and nine representatives from various community-based organizations with special attention paid to the inclusion of People with Disability (PWD) and women. These DCMCs are present in all 50 districts from the four following regions: Greater Accra Region, Upper East, Upper West, and Northern Region (see www.sendwestafrica.org). SEND-Ghana is thus active at district, regional, and national level. The type of M&E SEND-Ghana is carrying out could be described as performance monitoring, with the main purpose of oversight and compliance (Mark, Henry, & Julnes, 2000, p. 13). Because of their nonconfrontational approach, SEND-Ghana starts from the premise that government policies and programs are good. Thus, the emphasis is not on assessing the “merit or worth” of a certain program or policy, rather the focus is on monitoring and evaluating their implementation and compliance with the guidelines and objectives that are established by government. Further, the indicators used during monitoring are directly derived from the government program or policy in question.
This community type of M&E is at the heart of SEND-Ghana’s “Participatory M&E (PM&E)” framework that is composed of several phases. 6 During the research/M&E phase, data are collected, validated, and analyzed. SEND-Ghana goes to great lengths to validate the gathered data with different concerned stakeholders to ensure the validity, credibility, and acceptance of its M&E findings. Once the analysis is completed, SEND-Ghana presents aggregated findings at the national level in the form of an M&E report, policy briefs, and their newsletter, called the HIPC/Citizen Watch.
The other core part of SEND-Ghana’s PM&E framework is the policy advocacy phase, which has two main components: the policy dialogue or interface meetings and engagement with media. Regarding the latter, SEND-Ghana engages with national media at the launch of their M&E reports, and during the discussion of the results at the district, regional, and national level. Because the public officials concerned are usually informed about the findings before they are communicated in the media, one could talk about a “consented” exposure. While media appeared to have played a key role in pressuring public officials to respond (answerability) and honor their promises (soft enforceability) in most of the M&E rounds SEND-Ghana carried out, there was not much media involvement at the district level during the M&E of the NHIS. Therefore, this second dimension of the advocacy phase is not discussed further.
Following the launch of the report at the national level, SEND-Ghana, together with the FNGOs, organize interface or policy dialogue meetings at the district, regional, and national level. During such meetings, general and district-specific results are presented and discussed, usually by staff of SEND-Ghana or the FNGO. The aim of the interface is to foster exchanges between government and citizens and to offer a dialogue space. The concerned stakeholders are able to respond to questions and issues raised, and explain why certain things did or did not occur (answerability). In addition, during the interface, various stakeholders promise to take concrete action. Those issues that cannot be tackled and handled by public officials at the district level, are moved up to the regional level, and if necessary to the national one. The interface meeting is chosen as a variable of interest in this research for different reasons. First of all, SEND-Ghana’s approach is one of dialogue and active engagement with government in a constructive rather than a conflict-related way and it is a crucial part of SEND-Ghana’s PM&E framework. Second, it is the main place where M&E findings are discussed at the district level as opposed to just disseminating the results in a report. Finally, in line with previous research on the research–policy interface in developing countries, these linking moments or dialogue moments have been found to improve research uptake (Carden, 2009; Court & Young, 2003).
SEND-Ghana’s M&E of the NHIS
The NHIS, which was introduced in 2003, is a social protection policy to replace user fees in the health sector. The goal of the NHIS is “ensuring equitable access to quality basic health care for all Ghanaians (including vulnerable and excluded)” (Government of Ghana, 2009, p. 3). Between 2009 and 2010, SEND-Ghana monitored the implementation of the NHIS, focusing on the two main dimensions of its goal, which are people’s access to the NHIS (including the poor) and providing quality basic health care for all (SEND-Ghana, 2010). SEND-Ghana used semistructured questionnaires to collect primary data from four main groups of stakeholders at the district level: NHIS management, the district health management team (who is responsible for health service delivery at district level), the District Assembly, and NHIS clients (community members) within the district. This article focuses on the influence of CSO-led M&E on the NHIS management, the health service providers, and the District Assembly, which, together, will be referred to as the concerned stakeholders. The effects at the level of citizens within the wider community or on the organizations themselves are beyond the scope of this article. To examine the consequences of SEND-Ghana’s M&E of the NHIS at the district level, the Theory of Evaluation Influence (Henry & Mark, 2003; Mark & Henry, 2004) was used as a starting point. The actual data gathering or M&E, together with the interface/policy dialogue and the other surrounding activities, would fall under “M&E activities” (see Figure 1, second column). A first step to understand the influence of SEND-Ghana’s M&E of the NHIS is to map the different influence mechanisms that occurred at the district level (see dark gray boxes in Figure 1 and the menu of influence mechanisms in Table 1). Second, given the crucial role the interface or policy dialogue meeting is playing within the entire SEND-Ghana framework, it was assumed that the interface could act as a catalyst for certain influence mechanisms (see question mark in Figure 1). To sum up, the following research questions can be formulated regarding SEND-Ghana’s M&E in terms of its influence at the district level:
Method
For the current research, a case study approach was adopted, using a comparative case study design. Case studies offer an advantage when studying complex social realities and causal mechanisms in an in-depth way (George & Bennett, 2005; Yin, 2009, p. 4). The current research combines within-case analysis to trace retrospectively the sequence of influence mechanisms in each district, and comparative case analysis to study the role of interface meetings. Mark (2011, pp. 166–167) also suggested choosing open methods that allowed the tracing back of change sequences when studying evaluation influence. To limit the research, it was decided to focus on the study of influence mechanisms that had occurred at the district level, and to choose one of the more recent M&E “rounds,” which, as mentioned before, was the M&E of the NHIS.
Research Design
To investigate which influence mechanisms occurred following SEND-Ghana’s M&E of the NHIS, four districts from the northern regions in Ghana were selected (see Table 2). Because the Theory of Evaluation Influence (Henry & Mark, 2003; Mark & Henry, 2004) has not yet been used in an empirical way to study the influence of CSO-led M&E in a developing context, it was not possible to investigate a specific, well-established influence pathway and select cases accordingly. Instead, the idea was to map, as completely as possible, all the influence mechanisms that had occurred following the M&E of the NHIS, and therefore four districts were chosen. Because SEND-Ghana has a well-established M&E approach, the following variables can be considered similar across all chosen districts: M&E capacity of DCMCs, years of M&E experience of the DCMCs, type of capacity building, M&E survey/instrument used, and sequence of M&E activities. The most important variable that differed across the four districts, and which is of interest for the second research question, is the presence or absence of a formal interface meeting. In two districts, A and B, an interface meeting was organized by SEND-Ghana together with the FNGO, while in the two remaining districts, C and D, no such meeting was organized. These interface meetings had not occurred because of lack of resources and internal challenges faced by the organization; however, in previous M&E rounds, interface meetings had been organized in those two districts. In addition, some informal discussions around the NHIS M&E findings did take place in District D under the lead of the FNGO. The setup is summarized in Table 2.
Overview of Selected Districts.
Note. In District C, the focal nongovernmental organization (FNGO) was particularly weakened in the period following the national launch of the National Health Insurance Scheme (NHIS) report.
Within-case analysis was used to study the different outcomes of SEND-Ghana’s M&E of the NHIS in each of the districts, while comparative analysis was used to examine the potential role of dialogue spaces, such as the interface meetings, in triggering and enhancing certain influence mechanisms.
Research Process and Analysis
Data were gathered over a period of 2 months at the end of 2011. Data collection was done primarily through key informant interviews using a topic guide that was partially adapted to the respondent. For example, the interview guide for the DCMC and FNGO contained additional questions on the data collection process and associated challenges, while the one for the NHIS and health management contained questions on their internal M&E/feedback system. Open questions were used to leave enough room for the respondent to recount the events. Some examples of questions are as follows: Which issues came up during the discussion of SEND-Ghana’s M&E report? Have any of the raised issues been resolved and how? Further probing was also done to improve recall and get more in-depth information. A total of 4 group discussions and 16 key interviews from the four selected districts and SEND-Ghana staff were recorded, transcribed, and analyzed using NVivo 9. The interviews lasted between 45 min and 90 min. In addition, the following primary and secondary data sources were coded: internal documents, donor reports, participant observation notes, field research journal, and media journal articles from and about a range of stakeholders (DCMCs, FNGOS, NHIS staff, District Assembly staff, Ghana Health Service staff, and SEND-Ghana staff). Qualitative software, in this case NVivo 9, was used to facilitate a more systematic, explicit, and rigorous analysis process without losing flexibility (Lu & Shulman, 2008).
For the identification of themes, theoretical coding was used in combination with emergent coding, especially for the identification of the different influence mechanisms (Ryan & Bernard, 2003). Queries were run to refine the coding and further explore the data both within and across the four cases. In addition, contextual factors that would constitute alternative explanations for the occurrence of the different influence mechanisms were identified and coded. References to other M&E rounds carried out by SEND-Ghana were also coded but excluded from the analysis for this article.
For the generation of the overview table (see Table 3), a matrix query was run to compare the various districts in terms of the different influence mechanisms present in those districts. To take into account multiple referencing of the same example of a particular influence mechanism, the number of times a particular influence mechanism occurred was taken as a measure instead of the number of references to it. Triangulation of data sources was done as much as possible to reduce possible biases. For example, one of the common pitfalls of research about evaluation use is the overstatement of use by the respondents (Brandon & Singh, 2009). To reduce this bias as much as possible, the idea was to include only those instances of influence that could be confirmed by other interviewees in the table. However, because some type of influence mechanisms appeared challenging to confirm, for example, individual influence mechanisms, it was decided to retain them in the table and mark them instead with an asterisk (*).
Overview of Influence Mechanisms Following the National Health Insurance Scheme (NHIS) Monitoring.
Note. The numbers in the table refer to the number of times a particular influence mechanism occurred. The letters between parentheses refer to the level at which the mechanisms can be “located”(I = individual, IP = interpersonal, and C = collective). The influence mechanisms marked with an * indicate that those were challenging to confirm by other interviewees and/or documents. Numbers in parentheses indicate a potential for those mechanisms to occur in the future.
Results
Although a broad range of outcomes were mentioned during the different interviews, this section focuses on those that occurred in relation to SEND-Ghana’s M&E of the NHIS and thus excludes references to other M&E rounds carried out by SEND-Ghana.
Mapping of Influence Mechanisms
As a first step to understanding the influence of SEND-Ghana’s M&E at the district level, a map of the different influence mechanisms that took place after the NHIS monitoring was “reconstructed” in each of the districts (see Table 3). The first columns contain the five overall categories of influence mechanisms followed by their specific types. These are taken from the “menu table” of possible influence mechanisms Mark and Henry (2004, 2012) proposed (see Table 1) combined with some influence mechanisms newly created by the author (see gray shaded cells in the second column). Table 3 gives a summary of the general overview of the results per district, grouped according to the presence (columns District A and B) or absence (columns District C and D) of a formal interface meeting. In what follows, the most important influence mechanisms that occurred in the different districts will be presented and illustrated with some reported examples.
General Influence Mechanisms
General influence mechanisms are challenging to uncover because they occur early on in the “influence chain” and most stakeholders are unaware of their existence. Nonetheless, they are crucial in triggering other types of influence mechanisms such as cognitive/affective, motivational, behavioral, or relational mechanisms (Mark & Henry, 2004, p. 40) that will be discussed later on. District A is the only district in which two examples of general influence mechanisms were identified. During discussion of results, M&E findings can add “weight” and “credibility” to issues that are raised by the different stakeholders. Put differently, the M&E findings are used to back up and justify the stakeholders own concerns and issues (justification). In District A, for example, the NHIS had received complaints from several clients about the attitude of health staff in health centers, which was also mentioned in SEND-Ghana’s M&E report. The NHIS manager explained that his position became stronger during the discussions because “it’s not like only we [NHIS] who are thinking the way our clients are thinking. But even other stakeholders are saying this” (Interview A4). Similarly, the M&E findings can also be used during discussion to convince or persuade someone else of the occurrence of certain issues on the ground. Again in District A, during the same discussion about the attitude of health staff, the concerned service provider became defensive, but the NHIS was able to use the M&E findings to convince them that this was actually the situation on the ground, and the health provider finally accepted: “then they [health provider] accepted this fault” (Interview A4).
Cognitive/Affective Mechanisms
Cognitive/affective mechanisms “refer to shifts in thoughts and feelings, such as attitude valence” (Mark & Henry, 2004, p. 42). When M&E findings are discussed face-to-face or during stakeholder meetings, the concerned stakeholders have the chance to explain why certain issues did or did not occur, which in turn reduces tension, increases understanding, and has the potential to change certain attitudes or opinions as was the case in Districts B and D. In District B, for example, the monitoring revealed long waiting times at the health centers for NHIS clients. Health staff were subsequently able to explain that those long waiting times were due to the cumbersome compulsory registration procedure of NHIS clients upon attending the health center. A failure to follow the registration procedures correctly would have negative consequences for the health centers in terms of claims settlement. As a result of that explanation, some stakeholders changed their attitude and attributed the root of the problem to the bulkiness of NHIS registration procedures as opposed to staff at the health centers.
M&E findings can provide information about the extent to which a certain issue is present within and across districts. In other words, they can contribute to making local descriptive norms 7 explicit or even challenge them in other cases. For example, in District A, the difference in attitude of health staff toward NHIS clients as opposed to non-NHIS clients was known to be present within the district but the extent to which this behavior was present all across the district only became clear through the monitoring and subsequent sharing of results. When descriptive norms are made explicit, this can further trigger other types of mechanisms such as a change in injunctive norms (motivational mechanism) or changes in individual and collaborative practices (behavioral mechanisms) which are discussed subsequently.
Another important type of cognitive influence mechanism that occurred following M&E is agenda setting, which can occur in a more or less formal way. As visible in Table 3, examples of agenda setting could be identified in all four districts under examination. Issues with quality of prescribed medicines and the exclusion of certain medicines from the NHIS medicine approved list were mentioned in SEND-Ghana’s M&E report and discussed (SEND-Ghana, 2010). In the northern regions, where farming is the predominant livelihood, snakebites are widespread and the exclusion of anti-snake venom (ASV) under NHIS is an important and common concern across the region. Because this issue can only be resolved at the regional or even the national level, many NHIS managers, including those in Districts A–C, have turned this issue into an agenda point to be discussed at higher levels within the NHIS. Other examples of agenda setting have been reported in District D regarding delays in the issue of NHIS membership cards. Not only M&E findings but also the proposed recommendations of SEND-Ghana’s M&E reports can be turned into agenda points, especially when they cannot be implemented at district level.
Motivational Mechanisms
This type of influence mechanism is characterized by human responses to perceived goals, social rewards and punishment (Mark & Henry, 2012). In District B, for example, there was pressure on the District Assembly “to do the right thing, because people were watching and monitoring” (Interview B2). The free registration of PWD under the NHIS by the District Assembly in District B can be partially explained by this type of pressure: “They would advise you to follow-up and do the right thing.” Stakeholders know that if they fail to do the right thing, “it will be in the open, out there” (Interview B3), and the fear of exposure by CSOs, such as SEND-Ghana can create pressure for them to change their attitude and behavior as a consequence.
When M&E findings are discussed in face-to-face meetings or in more formalized discussion spaces, such as the interface meeting, stakeholders try to resolve the issues raised. This reaction to the findings can lead to the formulation of pledges, promises, benchmarks, action points, or commitments. Independently of the term used, they are made in response to perceived (social) goals, rewards, and sanctions. Therefore, a new category “pledges for action” was created under motivational influence mechanisms. 8 The only pledges that were identified in the four districts were those made during the formal interface meetings in Districts A and B. In District A, pledges were made in the following three “broad areas”: The District Chief Executive (DCE) promised to improve transport modalities to facilitate access to health facilities by repairing the ambulance and providing Motor Kings 9 for pregnant women in rural areas; second, the lack of adequate health staff would be tackled by getting an additional doctor in the district, working out an incentive package to retain health staff, and providing sponsoring of medical students and health trainees; and finally, the District Assembly promised to complete the construction of a new NHIS office building. In District B, the DCE pledged to open a new NHIS substation within the district and the District Assembly together with the NHIS promised to write a funding proposal to institutionalize regular district-wide meetings for stakeholders to discuss implementation challenges of the NHIS. Finally, the District Assembly promised to register Livelihood Empowerment Against Poverty beneficiaries and PWDs for free under the NHIS.
Behavioral Mechanisms
Behavioral mechanisms “refer to changes in actions” (Mark & Henry 2004, p. 42). Collaborative changes in practice are changes that occurred or pledges that have been fulfilled following SEND’s M&E. A common collaborative change in practice that was reported across Districts A–C is the capacity to provide gender disaggregated data at the NHIS district level, which was not the case before. However, this change could not be confirmed on the ground and there appears to be some contradiction as to whether or not the new management information system is already functional. Other examples of reported changes that have occurred are the sponsoring of medical students and health trainees (District A), the free registration of PWD under NHIS by the District Assembly (District B), and the improved communication strategy for PWD in health facilities (District D).
However, some type of changes, mostly visible changes, can only take place once and therefore a new category, one-time action, was also created. In District A, for example, the new NHIS office was completed, nurses’ accommodation is under construction as part of the incentive package to retain health staff and a new ambulance has been provided. In District B, some progress has been made toward the establishment of an additional NHIS building by making a plot of land available.
Relational Mechanisms
Relational processes refer to changes in “aspects of ongoing relationships, structures and organizational processes” (Mark, 2011, p. 115). This category of influence mechanisms was added later on to the Theory of Evaluation Influence (Mark, 2006; Mark & Henry, 2012) and has not yet been investigated thoroughly. However, some evidence of relational mechanisms could be found in the interviews although they were difficult to verify. Feelings of empowerment, for example, have been expressed by the NHIS manager in District A: “All those things [discussions] have helped empower us [ … ]. But as the issues go, we are getting more empowered.” In District B, the intention to institutionalize stakeholder dialogues or create democratic forums was expressed but no concrete action has been taken (yet). The DCE allegedly proposed to make the interface meeting with SEND-Ghana compulsory once a year, as a kind of platform for discussion, and there was interest in organizing regular dialogue meetings between the NHIS, service providers, the District Assembly, and the community.
Interface Meeting as a Catalyst of Influence Mechanisms
Regarding the second question, a quick glance at Table 3 reveals a difference between districts with an interface meeting (Districts A and B) compared to those without an interface meeting (Districts C and D) from different vantage points. Taking the number of times a particular influence mechanism has occurred within the district as an indication, the presence of an interface meeting seems to have had a positive effect in terms of triggering outcomes. Motivational mechanisms are far more present in Districts A and B (a total of 10 instances) than in Districts C and D (a total of 1 instance). A similar pattern emerges for behavioral mechanisms with nine instances in Districts A and B versus three instances in Districts C and D. Further, although general and relational influence mechanisms appeared challenging to uncover, they are only present in districts with an interface. Interestingly, this pattern diverges when looking at cognitive/affective mechanisms where the table suggests there is not much difference between the four districts (seven instances for Districts A and B vs. six instances for Districts C and D). From yet another perspective, there are more interpersonal (11 vs. 5 instances) and collective influence mechanisms (15 + 2 potential instances vs. 3 instances) in Districts A and B than in Districts C and D, respectively. The importance of the interface meeting and some implications of these results will be discussed more in depth in the following section.
Discussion
About Influence Mechanisms and Pathways
The aim of the first research question was to provide a map of the different influence mechanisms that occurred following SEND-Ghana’s M&E of the NHIS. The results section has outlined these different mechanisms and has shown the rich variety in intermediary outcomes. The importance of intermediary outcomes has also been highlighted in several other studies. A cross-country evaluation carried out by Rocha Menocal and Sharma (2008) found that social accountability initiatives mostly had an effect at the intermediary level in terms of changes in behavior and practice. Further, Björkman and Svensson (2009), in their study on community-based monitoring in Uganda, suggested that social rewards and sanctions from community members were important tools to induce change at the behavioral level. Similarly, Mark and Henry (2004, p. 42) pointed out that motivational processes could be a key intermediate step toward changing one’s behavior. The results of this study follow the same line by suggesting that SEND-Ghana’s M&E takes place in a context that is fertile ground for the emergence of motivational mechanisms, which in turn can trigger behavioral mechanisms, for example.
Several interviewees referred to the variety of social and political pressures public officials have to deal with when performing their duties. These pressures take an additional dimension in the case of Ghana because of the fierce polarization that exists between the two main political parties (Gyimah-Boadi, 2009; Kopecky, 2011). For example, if the DCE 10 makes a promise to the community members and complies with it, he or she will put his or her party in a favorable light and indirectly “secure” his or her appointment. Especially construction projects, such as the building of the NHIS building in District A, are favored by the DCE and his or her entourage because of “their high visibility to the local electorate” (Crawford, 2009, p. 73). The constant presence of DCMCs who are monitoring several programs and policies in the different communities of their district further enhances those pressures and “puts them [public officials] on their toes” (Interview B3). Thus, when the results of SEND-Ghana’s M&E are communicated publicly, the concerned stakeholders feel compelled to react and make promises or pledges to improve the situation. Those pledges may then result or not in concrete changes in the district. The category pledges for action was created because the author felt that none of the existing influence mechanisms (see Table 1) captured this more politically motivated outcome just described. Pledges can be made on several occasions, but the results indicate that they were only made during the formal interface. The strong interrelation between the pledges and interface is further discussed in the second part of this section.
An additional idea behind the Theory of Evaluation Influence (Henry & Mark, 2003; Mark & Henry, 2004) is that a particular sequence of influence mechanisms can constitute an influence pathway, as explained earlier. The reconstruction of complex influence pathways appeared quite challenging in this research, especially because SEND-Ghana’s M&E of the NHIS took place a couple of years before. Weiss, Murphy-Graham, and Birkeland (2005) experienced the same challenge when trying to study influence pathways in their research on the use of the Drug Abuse Resistance Education program evaluations. Despite the inability to uncover and map complete pathways, the current research was able to point toward two weak links or breaking points: the agenda setting and the pledges for action. The examples of agenda setting that were present in the different districts referred to action points that could not be tackled at district level and were put on the agenda for consideration at a higher level. Because of incomplete decentralization, key decisions about programs/policies in the health sector are still taken at the central level, beyond the sphere of influence of the district stakeholders (Crawford, 2009; Kathyola & Job, 2011). For example, although the ASV was put on the agenda by the different NHIS managers, and later discussed at the regional level, the ASV issue remains unsolved today.
While some of the pledges for action identified in this research have resulted in behavioral changes, others formed a breaking point in the influence pathway for the following reasons. First, concerned officials sometimes make a token promise in order to get SEND-Ghana off their backs without an excessive reputational cost. Public officials know that these pledges are not legally binding and that SEND-Ghana does not have the power to apply sanctions if they fail to fulfill them (weak enforceability). One of the interviewees expressed this frustration: “And we [DCMC + FNGO] push, push, push, but the commitment—it’s easier said than done. They [district level officials] will easily say it, but act to see it on the ground is not always easy” (Interview B1). Second, because of unpredictable budget flows 11 and weak budgeting practices at the district level (Crawford, 2009; Kathyola & Job, 2011; Ministry of Health Ghana, 2011) there are insufficient funds to carry out the planned activities and pledges, also within the health sector. The inability to buy Motor Kings for the rural areas in District A may illustrate this point. Along the same lines, it is not always clear who is responsible for the implementation of the actions that these pledges entail. For example, in District B, the responsibility for the construction of an additional NHIS substation has been tossed back and forth between the District Assembly and the NHIS. Finally, some of the pledges made cannot be fulfilled at the district level and have to be tackled at the regional or even national level. The posting of additional doctors and nurses in the different districts, for example, is a decision taken at the national level with little input from the districts in question (Ministry of Health Ghana, 2011, p. 18).
Analyzing breaking points is not only useful to understand (incomplete) influence pathways but could also help organizations, such as SEND-Ghana, to put in place additional strategies when planning for subsequent M&E rounds. The use of the Theory as a planning tool was first introduced in Henry and Mark (2003), but has not yet been studied in depth. Although the evidence is not conclusive, the findings and the discussion mentioned previously also suggest that a motivational to behavioral pathway is more likely to occur than a cognitive/affective to behavioral pathway, at least for the district-level context in which SEND-Ghana is carrying out its M&E.
Importance of Interface Meeting
The second objective of this article was to examine the potential role of the interface meeting as a catalyst for influence mechanisms. According to Figure 1, the interface meeting would spur general influence mechanisms, which, in turn, would trigger other influence mechanisms. However, with the exception of District A, not many general influence mechanisms were identified in this study. This could be partially explained by memory gaps of the respondents due to the time lag between the research and the occurrence of these early influence mechanisms, or to the inability of the research instrument to capture this dimension. Further research is needed to clarify this point.
The findings that were analyzed from different vantage points in the Results section support the idea of the interface meeting as an important catalyst for the occurrence of influence mechanisms. Interestingly, a divergent pattern was observed for cognitive/affective influence mechanisms, and for a subtype of behavioral mechanisms (collaborative changes in practice), albeit to a lesser extent. Cognitive/affective mechanisms were present in all districts independently of the interface. So interface meetings are not the only space in which SEND-Ghana’s M&E findings are discussed. Indeed, during the M&E data collection phase and the subsequent validation meetings, where preliminary data are presented to public officials, SEND-Ghana and the FNGOs already discuss the M&E findings. As a type of “damage control,” these officials can start looking for solutions within their departments or units and propose agenda points to be discussed internally.
Detailed analysis of behavioral mechanisms shows that collaborative changes in practice have occurred both in districts with and without interface, pointing toward the existence of alternative change processes. The collaborative changes that were identified during the research were mostly as a result of changes in guidelines that had been introduced from the top, such as the improvement in the ID distribution system (District B), or changes that had already been discussed at other levels, such as the sponsoring of health trainees to retain them in the districts (District A). On the other hand, one-time actions were mostly linked to the fulfillment of pledges made during the interface, and visible changes in the district, typically construction works (Crawford, 2009).
Returning to the close intertwining of the pledges for action (motivational mechanisms) and the interface, the following could explain its importance. In principle, pledges can be made in other types of meetings too, but because of the public character of interface meetings, and the involvement of the media at times, there is a spotlight on those public officials and their pledges, for which, in theory, they can be held accountable later on. In such a context, one can ask why officials continue to attend this type of meeting. Several explanations can be advanced. First, given the weak capacity of official M&E units at district level (Kathyola & Job, 2011; Government of Ghana, 2011, p. 31) and the lack of available information (Fiankor & Akussah, 2012), especially in the northern regions, the interface meeting offers an opportunity to share and discuss SEND-Ghana’s M&E findings which become a valuable source of information. Most public officials are aware of the issues raised but do not have data to support them. Different interviewees expressed the crucial role evidence plays as a basis for discussion without falling in the trap of party politics and “witch hunting.” When credible and accepted M&E results are presented, it becomes harder to ignore the problems. Second, because of the nonreading culture (Fiankor & Akussah, 2012), the oral transmission of results during the interface meeting is an appropriate dissemination strategy. The interviews confirmed that most stakeholders obtained the M&E findings from the discussions rather than from reading the report. In those districts where no interface occurred, such as District C, most concerned stakeholders did not recall having read the NHIS report. Finally, some stakeholders are motivated to attend the meetings because of the financial incentives they receive in the form of travel expenses. Financial incentives also play a role at the institutional level, but in a different way. Districts are supposed to organize quarterly review meetings and include CSOs to discuss development issues within the district (Crawford, 2009). Because of financial constraints 12 —accommodation, transport, and beverages/food are supposed to be paid for the participants—not all districts are in the position to organize such meetings on a regular basis. Organized interface meetings, funded by CSOs, then become an interesting opportunity to capitalize on without incurring additional costs for the districts. The meetings offer a platform for them to bring and discuss some of their own grievances.
The fact that stakeholders attend those interface meetings, independently of their underlying motivation for doing so, has positive consequences in terms of creating the right environment for influence mechanisms to occur. The findings about the importance of the interface are also consistent with Cornwall and Coelho’s (2007) work on the creation of new participatory spheres and Valovirta’s (2002) work on argumentative evaluation. Valovirta (2002, p. 76) argues that that the creation of “forums for reasoned and open argumentation” is crucial because evaluations have an important value as inciters of argumentation and debate. The interface of SEND-Ghana is an example of such a forum for discussion, argumentation, and co-construction of interpretation of M&E findings among stakeholders. In addition to the discussion, commitments are made, or in other words, a space is created in which stakeholders are pressured to become more answerable, and hopefully more accountable toward district citizens.
The interface has its limits of course, especially within complex political contexts, where a multitude of actors and change processes converge, independently of SEND-Ghana’s actions. While interface meetings, as a built-in dialogue within SEND-Ghana’s M&E framework, add weight to the pledges made and can, at times, create enough soft pressure to push them over the implementation edge, they are not magic bullets. An in-depth analysis of the different pledges made during the interface meetings revealed that most of them had already been written down in existing plans, guidelines, and budgets or had been made at other occasions, and thus were not entirely new. Some examples include the nurses’ accommodation, the improvement of transport facilities to health centers, and the creation of an incentive system for health staff (see, e.g., Ministry of Health Ghana, 2007). So the discussions based on SEND-Ghana’s M&E results are not triggering new pledges but can, at times, put additional pressure on prioritizing and resolving certain problems that were already present but had not been resolved despite previous promises.
Strengths, Limitations, and Future Research
One of the strengths of this research is its contribution to the small number of empirical applications of Mark and Henry’s (2004) framework. To the best of the author’s knowledge, this study is also the first one that attempts to apply the Theory of Evaluation influence (Henry & Mark, 2003; Mark & Henry, 2004) in a developing context. Moreover, two additional influence mechanisms—pledges for action and one-time action—were proposed that may be linked to this type of CSO-led M&E, with its special attention to advocacy strategies. The identification of variants of existing influence mechanisms, such as one-time action, and new influence mechanisms, such as pledges for action, constitute an important contribution to the “menu of possible influence mechanisms” and, more broadly, to the research on evaluation influence. The current research also illustrates the flexibility and adaptability of the Theory of Evaluation Influence in building a context-specific local theory of influence (Henry & Mark, 2003).
On the other hand, although care was taken to reduce the occurrence of biases as much as possible, the research has some limitations. This type of CSO-led M&E may only occur within certain contexts similar to Ghana, characterized by an open and free press, a consolidated democracy, and an active civil society, despite its socioeconomic and political challenges (Abdulai & Crawford, 2010). The purpose, however, was to increase our understanding of M&E influence (internal validity), rather than generalize the findings to other contexts (external validity). Further, tracing mechanisms that occurred in the past is challenging because of the necessary trade-off between waiting long enough until sufficient time has elapsed for such changes to be observed, but not too long in order that memory/recall biases related to earlier mechanisms may be reduced (see also Weiss et al., 2005, for similar challenges). Finally, as mentioned earlier, influence mechanisms that could have occurred at the level of the community and the organization itself were not investigated. The focus was on a specific portion of the M&E process that occurred at the district level. As a consequence, some aspects of the influence pathways were not identified despite their potential importance.
This current research also brings along new questions and suggestions for future lines of research on evaluation influence. Having discussed the different influence mechanisms and established the importance of the interface meeting a next research project could, for example, investigate which influence mechanisms would be necessary and/or sufficient for change to occur at the district level. Another methodological approach, such as qualitative comparative analysis, could be used to tackle this question. Another line of research could examine the use of the framework as a planning tool, taking into account the implications of this research. Especially CSOs, that work in the same line as SEND-Ghana, may find this line of research relevant to increase the impact of their work.
Conclusion
This article set out to study the consequence of a particular type of M&E carried out by CSOs in a developing context who combine their M&E with advocacy strategies. The experience of SEND-Ghana with the M&E of the NHIS was taken as an example to reach this end. The results suggest that several intermediary outcomes are important, and the role of organized discussion spaces, such as the interface, is also highlighted. Future research is highly desirable to further examine some of the findings and to advance our understanding of this type of CSO-led M&E.
Footnotes
Acknowledgements
The author would like to thank Nathalie Holvoet for her inspirational input as thesis supervisor and the three anonymous reviewers for their valuable comments. Special thanks to all the interviewees and the SEND-Ghana staff for their time and input.
Declaration of Conflicting Interests
The author declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author declared the following financial support for the research, authorship, and/or publication of this article. The field research on which this article is based was carried out with financial assistance from the Research Fund of the Institute of Development Policy and Management, University of Antwerp.