Abstract
Policy research concerning developing countries must compete for scarce resources with alternative development investments, many of which are amenable to quantitative assessment of their impact and economic efficiency. This is especially true for policy research that addresses agriculture, food and rural poverty—rural policy research. This paper draws on existing evaluations of rural policy research to identify good practice in the conduct of impact evaluations in developing countries. While much has been learnt from these evaluations about how rural policy research can influence policies, the impact of the policy changes that may follow, and about methods for conducting such studies, very few have assessed the efficiency or economic benefit of rural policy research investments. The paper concludes that while the current focus on the use of mixed-method evaluations is necessary and sufficient in most cases, in the context of allocating public resources, evaluations that provide plausible estimates of the rates of return to major rural policy research investments, or even rural policy research institutions yield important additional and comparative information for decision makers. However, such quantitative assessments do not replace but depend on the prior conduct of qualitative and mixed-method evaluations.
Keywords
Introduction
Policy research, commonly undertaken by social scientists, is defined here as research intended to result in new or improved policies, regulations, and institutions (and their management) that enhance economic, social, and environmental welfare (Raitzer and Ryan, 2008). Much policy research is publicly funded, and as such is often scrutinized to determine whether it is a useful way to spend public monies, as well as to learn how to increase its impact. This has led to a voluminous literature, based mostly on evidence from rich countries, about the challenges and available methods for assessing the value of policy research, and in some cases, of social science research more generally (e.g. Bastow et al., 2014; Gardner and Brindis, 2017). There have been far fewer attempts to assess the value of policy research in developing countries despite the more stringent financial constraints on policy research imposed by governments and development agencies (DfID, 2014), and a scarcity of trained policy researchers. This is particularly the case with policy research in the fields of agriculture, food policy and rural development—rural policy research (RPR), where the funding of RPR must often compete with spending on scientific agricultural research and investments in other kinds of rural development projects. These latter investments tend to have more direct and visible impacts that can be, and often are, quantitatively assessed, using randomized trials, econometric analyses, and other quantitative methods, whereas the impacts from investments in RPR are more difficult to demonstrate. This can place RPR at a disadvantage when competing for funds, a consideration that has encouraged a growing interest in methods and good practices for evaluating RPR in developing countries. This paper reviews the experience in evaluating RPR so far, concentrating on targeted RPR projects rather than institutions, and draws lessons for future evaluations.
Underlying considerations
Like most policy research RPR typically seeks to help governments identify, understand, and adopt policies that are technically and economically efficient, socially equitable, environmentally beneficial, and politically palatable. Within the broad nexus of policy formation and policy influence, the pathway whereby effective research produces positive impacts is often characterized as follows (Gardner, 2008): A policy problem or issue justifies the outlay of money and manpower (inputs) on a set of research activities. This leads to outputs (research papers, briefs, conferences, and the like) that are ingredients in an interactive and iterative advocacy process involving many stakeholders and unpredictable windows of political opportunity. From this advocacy process emerge outcomes in the form of a sufficient consensus for change and the legal and institutional steps necessary to give effect to the new or changed policy. Last, those outcomes deliver
The linearity in this simple model is not generally found in practice, however, and many other factors besides policy research can affect policy outcomes and impacts (Figure 1). The process within which policy-research outputs contribute to policy outcomes generally depends on a number of variables affecting the formation of advocacy coalitions around a particular policy issue, as well as the efficacy of those coalitions in pursuing their policy goals (Weible et al., 2009). Such coalitions engage in a number of messaging, convening, financing, and lobbying activities designed to influence political outcomes. In addition, deficits in local capacity for both policy interpretation and implementation may require substantial efforts to first be directed at augmenting these skills then enhancing (or creating) the local institutions that employ this additional human capital. A recent paper (DfID, 2014) notes that the capacity to adapt and absorb research results is a stronger factor in economic growth in low-income countries than the production of new research knowledge. Such factors mean that there may well be long time gaps between the delivery of evidence-based policy change and the achievement of consequent welfare gains. Although there are no estimates of the length of these lags for RPR there is evidence from the biomedical and health sciences that the lag can be as long as 17 years on average (Dean et al., 2013).
Impact pathway for policy research.
Neither can the opening of windows of political opportunity be ignored, unpredictable as they might be (Resnick et al., 2018). Such political opportunities are commonly brought about by new leadership, or a sharp change in economic fortunes, such as a fiscal crisis or shock to world prices. Leadership changes may also slam those windows shut. Certain actors or institutions may be involved in both the research process generating policy insights and the political process of making policy changes, for example, donors and government, which in turn may give these actors a disproportionate degree of influence over the policymaking process (Place and Hazell, 2018; Slade and Renkow, 2014).
Policy formulation combines research and other outputs with a range of political considerations that reflect a mix of ideology, vested interests, and institutional inertia. Within the policy formulation process, the means by which research outputs might gain traction include a number of inter-related political, communication, and financing activities. A multitude of organizational variables are important too, not least institutional reputations, which in the case of research institutions center prominently on scholarly impact, and the idiosyncratic leadership skills of those delivering policy messages. While very difficult to quantify, these characteristics of research institutions affect the probability that attempts to influence policy are successful or not (Slade and Renkow, 2014).
The public administration literature raises significant questions about the role of research-based knowledge vis-à-vis other forms of information as a driver in public policymaking. For example, referring to empirical evidence from surveys of policymakers, Weiss (1977) notes “the major use of social research is not the application of specific data to specific decisions. Rather, government decision makers tend to use research indirectly, as a source of ideas, information, and orientations to the world.” In a similar vein, Caplan (1979) refers to decision makers and researchers as members of two distinct “communities,” asserting that a “great divide” exists between producers and consumers of knowledge. Some writers (e.g. Punton, 2014) have sought to classify the ways in which policy research contributes to policy outcomes and impact, distinguishing between ways in which research evidence is used and the types of policy change that result. Such work on pathways to outcome helps to clarify where and how impacts can be evaluated.
Moreover, some political scientists have identified examples of policymakers opting in favor of inefficient programs or non-competitive rent-seeking opportunities that maximize the well-being of specific (loyal) interest groups (Bates, 1998). This is, of course, a view that is strikingly at odds with the conventional view taken by economists of decision makers as social welfare maximizers. Similarly, the widely observed importance of so-called “policy champions” in the advocacy process reinforces the relative importance of advocacy processes vis-à-vis knowledge creation in apportioning credit for a particular policy outcome.
On the other hand, examples abound of high-level decision makers who are well informed and attentive to research findings, particularly if those findings are generated by a well-respected research institution. However, effectively reaching those decision makers generally requires a range of messaging mechanisms beyond research papers. These might include policy briefs, organized conferences, workshops and trust built on personal engagement between researcher and policymaker. Even very high-quality research may have limited influence without effective “marketing.” Thus, determining whether a research project has contributed to a policy outcome requires attention to research outputs beyond journal articles to include these “softer” outputs. Such attention has given rise to a considerable literature seeking to establish how research influences policy and if it does to what degree—its contribution or the attribution share. See, for example, Dean et al. (2013).
It is clear from the foregoing that the measurement of RPR impact is an ex-post activity—impact cannot truly be measured until it has occurred, or enough of it has occurred to allow its future trajectory to be reasonably estimated. There is a growing body of published studies that purport to provide ex-post evaluations of RPR. We first review these studies in the next section and the extent to which they have achieved their overall goals. Then we examine the sequence of steps required for a full impact evaluation, and draw on the available literature to describe the important challenges, available methods, and good practice guidelines for each step in conducting an evaluation. This leads to our conclusions and lessons for future evaluations.
RPR impact assessment: Achievements so far
The vast majority of studies seeking to evaluate the impact of RPR have been conducted by a small number of research organizations, among which the International Food Policy Research Institute (IFPRI), the Consultative Group on International Agriculture Research (CGIAR) and the Australian Center for International Agricultural Research (ACIAR) are the most notable. A basic framework for assessing the economic impact of RPR was set out at a 2001 conference reported in Pardey and Smith (2004). An ex-post evaluation involves two distinct phases: (1) assessing the influence of the policy research on a policy change and (2) evaluating the consequences of the policy change. In the first phase, the primary goal is to document how, and to what extent, research outputs combine with other, non-research based influences to produce policy outcomes. This is largely an attribution problem. The key goal in the second (post-outcome) phase lies in identifying and assessing subsequent welfare effects against a plausible counterfactual. This assessment may range from a qualitative evaluation of the effectiveness and impact of the policy change in achieving its goals through use of mixed methods, to a quantitative assessment of the economic value or efficiency of the underlying research investment using cost-benefit analysis. Both phases are needed for a convincing ex post evaluation of the holistic impact of a policy research investment, yet few studies have successfully completed both phases.
There have been many studies of the influence of RPR within the CGIAR in recent decades, but only a very few have gone beyond the first phase of creating plausible narratives about the influence or contribution of a specific program of research on subsequent policy decisions (Walker et al., 2010). Similarly, different kinds of qualitative evaluation have been carried out by the Canadian International Development Research Center (IDRC), and the UK Overseas Development Institute (ODI), for their multi-sectoral policy research, but none of these have gone on to evaluate the socio-economic impact of the policies that were influenced. While qualitative studies of policy influence are central to learning about policy processes and impact pathways (Place and Hazell, 2018; Raitzer and Ryan, 2008), they cannot demonstrate the economic value of policy research investments.
To help fill this gap, the CGIAR’s Standing Panel for Impact Assessment (SPIA) sponsored a series of RPR impact studies from 2005 to 2010 (Raitzer and Ryan, 2008; Walker et al., 2010). In part, this work was prompted by the World Bank meta-evaluation of CGIAR which identified a striking lack of credible studies analyzing and quantifying the impact of the large historical investments in RPR by the CGIAR system (World Bank, 2003). SPIA conducted a “scoping study” that identified and reviewed 24 ex-post assessments of CGIAR policy-oriented projects (CGIAR Science Council, 2006). These assessments spanned a range of policy domains: trade and market policies, property rights, plant genetic resources, and gender and provided substantial qualitative evidence on how and why RPR findings and outcomes find their way into real-world policy change. The majority of these assessments stopped well short of quantifying the resulting changes in rural development, food security, poverty reduction and environmental sustainability. Only three yielded quantitative estimates of the economic value (or efficiency) of the research investments. Babu (2000) evaluated food policy reforms in Bangladesh, including the abolition of the Rural Rationing Program and the implementation of a Food for Education Program. Ryan (2002) evaluated the impact of IFPRI researched policy reforms on the rice trade in Vietnam, particularly a reduction in the export tax. Ryan and Meng (2004) estimated the impact that policy research and related activities had on the Food for Education Program in Bangladesh. These studies estimated that the research delivered net benefits in the tens of millions of dollars (US$27−US$166 million in the case of food policy reform in Bangladesh, US$248 million for the food for education program, and US$45 million for rice price reform in Vietnam) for a relatively small investment in RPR.
In 2007, as a follow-up to its scoping study, SPIA commissioned seven quantitative impact assessment studies of RPR to augment those done by IFPRI. These evaluations reviewed a wide range of policy interventions—forestry, fertilizer, conditional cash transfers, milk marketing, and pesticide policy. The estimated net present value of the benefits from each of these RPR projects was in the tens or hundreds of millions of dollars (Renkow and Byerlee, 2010; Walker et al., 2010). The impressively high returns to specific RPR projects largely reflect their modest budgets, relatively short gestation periods, and a compressed diffusion process.
A wide-ranging review (Hazell and Slade, 2015) of 30 independent evaluations of IFPRI’s RPR projects concluded that these evaluations provide plenty of plausible narratives about policy impact, while the few more quantitative studies suggested that, subject to some strong assumptions regarding attribution of impact, there were some substantial economic returns. However, most of these evaluations only considered the economic benefits of the research, measured as increases in national income, economic surpluses, lifetime earnings from education, or savings in government costs. Only two studies explicitly considered the impact of IFPRI’s research on the poor and none sought to measure environmental benefits, cross-country spillovers, or regional and global public goods.
Other quantitative studies of the impact of policy research undertaken by the ACIAR also show favorable returns to RPR investments. Several RPR projects have been selected by the ACIAR for impact evaluation, most recently by random sampling. The ACIAR evaluations represent extremes in terms of potential impacts. Two of the studies, Lindner (2011) and Mullen (2010), review RPR related to price and trade policy reforms in Indonesia and China, respectively. The impact of these “macro” reforms was estimated to be in the billions of dollars. However, definitively attributing these policy changes to ACIAR-supported RPR proved to be a bridge too far, even though only a tiny share of the benefits would suffice to cover the entirety of ACIAR investment in the research.
In contrast, several ACIAR evaluations looked at very specific, highly local policy and institutional changes. Aggregate benefits for these policy changes were modest, but the descriptions of the impact pathways from ACIAR-supported RPR to on the ground policy changes were detailed and plausible. For example, in Vietnam, the implementation of a revised schedule for irrigation water developed through RPR, increased crop yields, reduced irrigation costs and provided discounted net benefits of US$13 million, a benefit-cost ratio of 10:1, plus unmeasured environmental benefits (Harris, 2006). In another case, the RPR sought ways of encouraging smallholders to use private sector palm oil processors in Papua New Guinea. The researchers devised an electronic payments scheme for smallholders, an e-payment card for women collecting loose fruit, and model land-use agreements between plantation companies and local communities. The impact evaluation (Fisher et al., 2012) estimated substantially increased smallholder participation with net discounted benefits of $55 million and a benefit-cost ratio of 20:1 for the investment in RPR.
All of the impact studies reviewed so far have been conducted within a particular, country-specific policy environment. Most produced knowledge potentially relevant to policy domains in other countries. However, documentation of such spillovers is difficult, particularly given the inherent, “right time, right place” nature of policy changes. Two studies—Behrman (2010) analysis of IFPRI’s contribution to Mexico’s conditional cash transfers program and Ryan (2002) analysis of IFPRI’s contribution to policy change in Vietnam’s rice sector—quantified these spillovers, both finding that the value of spillovers alone exceeded the project costs.
It is probable that some of the most important impacts of CGIAR’s policy research lie in influencing the global policy agenda, even though these impacts cannot be readily quantified in terms of increasing incomes and reducing poverty. Examples include Bioversity’s role in successfully concluding the International Treaty on Plant Genetic Resources (Gotor et al., 2010), World Agroforestry Center’s contribution to the UN Framework Convention on Climate Change (see http://www.worldagroforestry.org/downloads/Publications/PDFS/RP15435.pdf) and the influence of IFPRI’s research on international trade liberalization in the Doha trade negotiations (Hewitt, 2008).
On a slightly different note, many RPR outputs provide benefits beyond immediate changes in policy. For example, policy research can produce new knowledge and data that influence future research, for example, panel data from representative rural household surveys. With time, this new knowledge may also serve to modify ideological beliefs as well. In a similar vein, RPR conducted in some CGIAR centers has also focused strongly on building policy research capacity in individual countries which may with time help those countries make better informed policy decisions. Evaluating these indirect or lagged benefits is particularly difficult (Kuyvenhoven, 2014).
Given the challenges of quantifying the economic benefits from policy changes—including inter alia that some desired outcomes from RPR involve social, environmental and capacity building goals that cannot realistically be valued in narrow monetary terms and the difficulties of attributing observed changes to specific policy research activities, it is now a commonplace in the evaluation literature to use both qualitative
Overall, the vast majority of the 50 to 60 ex-post impact evaluations of RPR reviewed for this paper made greatest use of a narrower range of qualitative methods. The most commonly deployed qualitative methods were semi-structured interviews, mainly of pre-identified stakeholders especially policy makers and policy advisors. Other methods included documentary analysis, citation analysis, field visits of varying duration and intensity and direct observation. However, all of these qualitative methods although carefully and rigorously used remain open to unknown and unknowable degrees of sampling bias. Moreover, while they can generate valuable insights into the influence of policy research, and the effectiveness and results of consequential policy changes, these methods are unable to draw firm conclusions about the efficiency or economic value of RPR investments.
Evaluating policy research: Breaking down the elements
The extensive literature on RPR provides multiple insights into the benefits that may or may not flow from policy research, as well as practical information on ways to evaluate its impact. These insights can most easily be understood by considering each of the steps needed to undertake an ex-post evaluation of a policy research project. Each step can lead to useful information in its own right, but if all the steps are successfully completed, a quantitative assessment of the economic returns to the RPR investment becomes possible. There are six such steps:
Develop a Theory of Change (TOC) for the project.
Assess the relevance, timeliness, and quality of the policy research outputs.
Determine the policy outcomes.
Assess the extent to which the research outputs influenced the policy outcomes (or changes), in the context of relevant work by other policy researchers and influencers of policy.
Quantify the economic, social and environmental impact of the new or changed policy, and compare it to the impact that would have occurred with a plausible counterfactual policy. The difference between the two measures the incremental impact of the policy change, and should, in principle, be adjusted to include any incremental cost or saving from implementing the changed policy compared to the counterfactual.
Given the quantitative evaluation of the impact of the policy change in 5, an attribution share from 4, and the costs of the policy research, calculate a cost-benefit ratio or some other measure of the efficiency or economic return to the policy research investment.
Each step presents its own challenges and options, as discussed below.
Develop a TOC
There is a logical and broadly predictable pattern in all policy research that runs from concept to impact. A typical research project starts with the development of a clear statement of the problem or topic to be researched and a detailed research design in which the methods of enquiry, resources and analytical method are articulated, thereafter expected outputs are set down, then the expected short-run outcomes and last, ultimate or long-term impact is outlined. This causal chain may be thought of as the TOC. There are many definitions of a TOC, the best known of which is probably that proposed by Weiss (1972, 1995). She said quite simply that a TOC is a theory of how and why an initiative could or did work.
A TOC begins with an objective or goal in mind and is authoritatively defined as “a specific and measurable description of a social change initiative that forms the basis for strategic planning, ongoing decision-making and evaluation” (www.theoryofchange.org). Although similar to a logical framework (log-frame) or an impact pathway, a TOC normally includes a more detailed description of the process(es). A typical impact pathway, as used in RPR indicates the overall goal of the research, and, in reverse order, the expected impact, the required outcomes to achieve the impact, the required outputs to achieve the outcomes, and the required activities to achieve the outputs. A TOC goes further and explains the connections between outputs, outcomes, and impacts, including the assumptions that facilitate those connections, and also identifies any non-linearities along the way. It should also identify possible spillover effects—both positive and negative—and set out contextual factors and external influences that might affect the causal chain or the outcomes of interest (White and Phillips, 2012).
While many stakeholders in the policy process may develop explicit or implicit TOCs for their own missions, the important TOC in the context of evaluating the impact of policy research is the one developed by the research team. Ideally, this TOC will explicitly indicate the policy outcome and its intended impact, the key decision makers involved in the policy outcome, the behavior or attitudes that need to be changed, the partners that need to be engaged, the capacity or skills that need to be strengthened, the evidence that is most useful to the process, and the communication methods best suited to disseminating the evidence. The more explicit the TOC, the easier it is for an evaluator to identify the data to collect, the stakeholders to interview and the questions to ask. However, ex-ante theories of change are often omitted in the design of policy research, leaving them to be retro-fitted by the evaluator. Obviously, such ex-post TOCs are more descriptive than prescriptive. Nevertheless, Belcher et al. (2017) in an instructive paper examine four impact evaluations of RPR: a long-term forest management research program in the Congo Basin; a large research program on forests and climate change; a multi-country research project on sustainable wetlands management, and a research project of the furniture value chain in one district in Indonesia. All were guided by ex-post TOCs. The authors conclude that
Using an explicit TOC as the analytical framework helps identify and test hypotheses about how the research contributed to change.
The evaluations benefited from well-organized evidence bases.
A TOC facilitates learning at the project or program scale, providing the basis for generalizable learning about how research contributes to outcomes and impacts and about ways to render research more effective.
The clear delineation of research outcomes in a TOC helps to manage expectations about the kind and extent of “impact.”
A TOC is also both a valuable learning tool for researchers and research managers and an aid to discussions with funders about research contributions.
Assess policy research outputs
The enumeration of research outputs should include research publications and their quality and use, outreach and communications events, and encounters with decision makers, policymakers, non-governmental organizations (NGOs), private-sector entities, and so on. Identifying the outputs is relatively straightforward if a research team has kept adequate records, but this is not commonly the case. Moreover, overlapping projects and researcher responsibilities within the same institution can sometimes make it hard to assign outputs to specific policy research investments.
The quality and use of research outputs can be assessed through bibliographic analysis, using available web data on citations, downloads, media mentions, use in social networks, and so on (see, for example, Bastow et al., 2014). However, evaluators must make judgments about the salience, timeliness and quality of the outputs in relation to the overall goals of the policy research. This is a bigger challenge when evaluating multi-year research programs than for targeted research projects. A recent paper sponsored by the International Development Research Centre (IRDC) seeks to provide comprehensive and “holistic” guidance on how to assess the quality of research outputs. To do so it recommends that traditional measures of scientific quality be combined with “customizable research quality rubrics” or numeric ratings and other qualitative measures (Ofir et al., 2016).
Determine policy outcomes or changes
Research can inform policy and practice in two ways: it may inform decisions on specific interventions (e.g. what intervention to use in response to a given problem), and it may be used in a more subtle way to inform a decision maker’s understanding of a context and possible need for change.
Documenting change in specific policies or programs is relatively straightforward in terms of what changed, who made the change, and when. What changed can be something simple, such as a tax rate, or something more complex, such as a sector-wide reform involving several institutions and their rules. Such changes can and should be tracked by researchers themselves as they take place, although this is not always done. More challenging is when RPR aims to change a dominant paradigm, such as a government’s view of the role of the private sector in agricultural input markets. This may require more indirect measures of change, such as USAID’s Feed the Future policy indicators (see, www.agrilinks.org/sites/default/files/ftf-indicator-handbook-march-2018-508.pdf). Similarly, it is possible, although more challenging, to document when policy research shows that the government should not change its policy (e.g. moving to higher tariffs from a lower tariff regime).
Assess the influence of the policy research on any policy change
There is an extensive and varied literature on evaluating and attributing the influence of policy research outputs. Given that research outputs are usually only one of several inputs into the policy decision process, one approach is to start with a relevant policy change and then trace events backwards to understand the contribution research and other inputs made to the policy outcome. These so-called “episode studies” (Carden, 2009b) systematically analyze the relative influence of non-research factors on the policy change vis-à-vis the possible influence of any relevant policy research outputs. This type of analysis is useful for understanding the relative contribution of research to policy changes, but is of less value in evaluating all the outcomes from a specific research initiative (which also means it is less useful for conducting a cost-benefit analysis of an RPR project).
However, a variant of episode studies seeks to track the relationship between research and policy in both directions—forwards and backwards. To do so the ODI’s RAPID framework (see, for example, Hooton et al., 2006; Leksmono et al., 2006) uses episode studies of individual policy changes, case-study analysis of specific research projects, and outcome-mapping. These studies are complemented by data collection and triangulation through literature reviews, documentary analysis, participatory workshops, stakeholder interviews and field visits. Not surprisingly they are relatively expensive and time consuming for all involved.
But the most common approach in the literature is to work forward from the outputs of a specific policy research investment to a relevant policy change. This literature spans a wide range of disciplines including ethnography, philosophy, law, sociology, and anthropology. Among the varied ways of determining influence four stand out. They are realist evaluation, 1 the general elimination method, 2 process tracing, 3 and contribution analysis 4 (White and Phillips, 2012). These methods share a common core that requires the delineation of a TOC together with a number of competing hypotheses. Thereafter, evidence is assembled to test and validate, invalidate, or revise the hypothesized explanations and thereby establish or deny causality.
In qualitative evaluations of this kind, four tests have been defined to assess the strength of alternative hypotheses (see Befani et al., 2016). They are (1) “Straw in the Wind” tests that provide evidence for or against a hypothesis, but by themselves cannot confirm or deny it; (2) “Hoop” tests, that if passed, affirm the relevance of a hypothesis but cannot fully confirm it and, if failed, eliminate a hypothesis; (3) “Smoking Gun” tests that confirm a hypothesis if passed, or weaken it if failed; and (4) “Doubly Decisive” tests that confirm a given hypothesis while eliminating any others.
Some authors favor using Bayesian methods to attribute policy outcomes to policy research. This involves eliciting information on the subjective beliefs of decision makers such that a probability distribution of decision maker beliefs can be specified, tracked over time and related to specific policy research (Gardner, 2004; Lindner, 2004; Schimmelpfennig et al., 2006). While conceptually appealing to economists, such approaches have not yet been shown to be practicable.
The links between input, activity, output and outcome are central to understanding how research translates into welfare but are difficult to determine. The challenge is to understand the way “contribution” and “attribution” relate to the outputs and outcomes from the research input and activity. Attribution refers to the share of the outcomes made by the research team and contribution to whether at least some outcomes or impacts have resulted from the research.
The terms influence and contribution overlap conceptually. Influence tends to imply causality while contribution is more ambiguous, implying a weaker notion of shared influence. It is commonly acknowledged in the literature that policy research is not the sole determinant of a change in policy. Others may have worked toward the same end. Research is often collaborative and attributing influence to specific organizations does not serve partnerships well. Joint recognition of the inputs that all stakeholders provide to the policy outcome is needed—hence contribution. Research may also be partly dependent on earlier work or ideas that were produced by different organizations or emerged through other forms of interaction and this too requires recognition. Furthermore, strong claims of influence may not be received well by policymakers who rightly seek to guard against claims of caving in to external interests. A focus on contribution rather than attribution makes the collection of evidence in a policy research evaluation less demanding.
There is a less obvious distinction to be made here about whether the aim is to assess the contribution of an organization or the contribution of the research evidence to an RPR outcome. Both can be important, and both are challenging because multiple organizations or interest groups are often active in promoting pieces of evidence or in advocating a particular policy position. In this context, it is important to distinguish between the new knowledge generated by the research and older knowledge pushed by advocacy groups. However, when the objective is learning and replication by others, the most important element is the evidence. A focus on the evidence rather than the organization or individual offering the evidence is useful in that references and sources embedded in evidence can show transparent links to earlier work or ideas that were produced by different organizations or emerged through professional or even accidental interactions. The question of whether to focus on the organization or the evidence also has implications if counterfactuals are to be used in assessing the contribution of research. If the emphasis is on evidence, the counterfactual is the absence of the evidence and not the organization.
In practice, the influence of research outputs on the policy process will be highly context-dependent or even path dependent, and the approaches required to validate those influences are more historical and rhetorical than statistical. The complexity of tracing influence also increases as one moves from policy research aimed at a defined area (project), to a countrywide, regional or even global problem. This is not only more difficult conceptually, but is more costly to do as well. Studies to assess the impact of a specific body of policy research typically devote a large amount of effort to teasing out context-dependent influences of that research within the advocacy process. For RPR, this has been done mainly via interviews and surveys of key stakeholders and by tracking downloads of working papers and policy briefs. However, describing how research was used, and hence taking a measure of its relative importance, can be challenging. As noted by Kydd (2015), the policymaker’s or other users’ encounter with research-based evidence is rarely observed and is difficult to reconstruct. It may be particularly challenging to measure policy influence or contribution beyond a classification of whether it had a major, moderate, or minor influence. Institutional reputations may also contribute to influence as well as the idiosyncratic leadership skills of those who deliver policy advice (Renkow and Slade, 2013).
Ideally, interactions between researchers and policy makers and users during the research as well as related data (such as baseline surveys) should be recorded as part of a monitoring and evaluation system for each research project. This would lower evaluation costs in several ways and avoid the difficulties that ex post evaluators face in trying to track down key individuals who may no longer be in place, and the loss of information as stakeholders memories fade. It would also reduce the challenges of constructing a counterfactual. However, to ensure the credibility of such information it is important that the monitoring be undertaken (or at least verified) by an independent third-party evaluator engaged for the purpose.
Assess the impacts of the new or changed policy and compare with a plausible counterfactual policy
This step in the evaluation involves first measuring the impact of the new or changed policy and second, doing the same for a plausible counterfactual, and then, by comparing the first with the second, determine the difference, part or all of which can subsequently be attributed to the policy research. The difference should, in principle, also be adjusted to include any cost or saving to government from implementing the new policy compared to implementing the counterfactual. This step can be especially challenging when evaluating a multi-year research program that may have influenced several policies in a country, or which worked in multiple countries, or generated international spillovers or international public goods (IPGs). In many situations, it may be more practical to focus on measuring policy impacts for some selected cases rather than for an entire research portfolio. Measuring the impact of a policy change is of course only worthwhile once the contribution or influence of the relevant policy research has first been established.
A primary challenge in quantifying benefit streams from policy research lies in defining a realistic counterfactual—the alternative state of knowledge that would have existed without the research (Baker, 2000). Place and Hazell (2015) describe three possible situations. In the first situation, the policy would not have changed without the policy research, for example, if the research played an important diagnostic role in identifying problems with the old policy. In this case, the relevant counterfactual is the old policy. A second case arises if there was going to be a policy change anyway, and the research may have led to a more informed change with a better outcome. In this case, the relevant counterfactual is the level of benefit obtained with the policy change that would have occurred in the absence of the research. A special case arises where the policy research may have affected the timing of the policy change but not the policy itself, in which case the relevant counterfactual would be the original timing of the policy change (e.g. Raitzer, 2008; Ryan, 2002). In the third situation, the research convinces policymakers not to make a planned change to the existing policy, thus preventing a worse outcome. In this case, the relevant counterfactual is the new policy that would have been put in place.
There has been an explosion in the application of experimental methods to assess a variety of interventions related to technology adoption and social protection/safety net programs (De Janvry et al., 2011). In the policy realm, a growing body of literature employs randomized controlled trials to assess policies related to health care (Gertler and Vermeersch, 2013), corruption (Olken, 2007), teacher performance (Muralidharan and Sundararaman, 2009), and school vouchers (Angrist et al., 2002). But these methods have so far not proven useful for evaluating whole swathes of RPR that aim to influence sector, country or international policy outcomes. This is because randomization to assess the impact of policy changes in agriculture and rural development is rarely possible. A price change, for example, cannot ethically (or practically) be randomly assigned, and randomization usually requires the impact evaluation to be designed ex ante. Practical difficulties of these kinds abound together with a range of methodological objections (see, for example, Picciotto, 2012; Ravallion, 2018).
Most existing quantitative assessments of the welfare effects of RPR outcomes have used economic models (both general equilibrium and partial equilibrium) to simulate outcomes. Economic surplus models have been used in calculating the net economic benefits from policy change by Kaitibie et al. (2009) to analyze dairy sector policy change in Kenya, and by Ahmed et al. (2010) to estimate the effects of a fertilizer allocation program change. Raitzer (2008) applies an economic surplus model, but also captures the value of environmental services resulting from changes in deforestation as a result of a change in policy on the sourcing of materials for the Indonesia pulp and paper industry. Others have used analyses from large household surveys to help estimate economy-wide benefits accruing to smallholder farmers (e.g. Templeton and Jamora, 2008). Yet others have relied on econometric methods using time series data to estimate the productivity and poverty reducing impacts of changes in agricultural policies or public investments (e.g. Renkow, 2010).
Imposing a range of alternative counterfactuals would seem to be relatively straightforward for policy research that relies on simulation-based methodologies. The Ryan (1999), Babu (2000) and Ryan and Meng (2004) studies cited earlier used country economic models to estimate the gains from bringing forward some planned policy changes in Vietnam and Bangladesh, where the earlier implementation was partly the result of IFPRI’s policy research.
Another possible approach is the econometric analysis of cross-country panel data sets, where differences in country policy approaches to similar problems may serve to provide “natural” experiments. Benin (2016) used this approach to estimate the effect of countries’ implementation of the Comprehensive Africa Agriculture Development Program (CAADP)—predominantly a policy analysis and support framework—focussed on key country outcomes such as agricultural expenditure and productivity, income, and nutrition. In this case the signing of a CAADP country compact was used as the definition of treatment.
One limitation of attempting to quantify impacts is that desired outcomes from RPR may involve social, environmental and capacity building goals that cannot easily be valued in economic terms. So, even when suitable data and methods are available to quantify important aspects of the impact of a policy change against its counterfactual, there may still be need for a supplementary analysis of these additional impacts.
Calculate a benefit/cost ratio for the policy research
Well implemented, the steps outlined above provide valuable information about the effectiveness and impact of a relevant policy change compared to its counterfactual, and in some cases of the share of the impact that can plausibly be attributed to a policy research investment. If those impacts have been successfully quantified then, given also the costs of the policy research, all the pieces are in place to calculate a benefit-cost ratio or other efficiency measure of the return to the policy research (e.g. a rate of return, the reduction in poverty per dollar spent, and so on).
As explained in section 3, there have been very few benefit-cost estimates for RPR, and most of the ones that have been done are for research studies focused on defined policy changes within single countries where it is relatively easy to track influence and impacts. For the most part these studies show positive and substantial returns. The very few studies that have evaluated benefit-cost ratios for RPR that leads to cross-country spillovers or global IPGs suggest even higher returns.
A good example of a benefit-cost calculation is a study by Behrman (2010) who examined the influence and impact of research on Mexico’s social safety-net program, PROGRESA/Oportunidades. Using simulations of the benefit-cost ratio for the research, under very conservative assumptions, Behrman showed that the benefits outweighed the costs substantially—and by much more when the conservative assumptions were relaxed. There were also extensive but unquantified spillover benefits in other countries. This study also illustrates another important point: even when quantification of impacts is possible using rigorous methods like randomized trials, the evaluator must also use qualitative methods for assessing policy influence and attributing a share of benefits to the policy research. This is a tricky step sometimes solved by assuming a percentage “attribution share”— or a range of possible attribution shares—in order to compute rates of return or benefit-cost ratios. Of course, such assumptions are only as convincing as the contextual, key informant-sourced, evidence supporting them. Nonetheless, it is difficult to conceive of an alternative that might be more immune to criticism.
When impact assessments are based on case studies rather than on the entire output of a program of policy research, then it is important to include the full costs of the entire research program and not just those for the policy research leading to the case study result. This is especially important if the case studies are cherry picked, that is, they are known success stories.
A common objection to cost-benefit analyses is that it can easily be manipulated to provide a desired result. This is a greater danger in ex ante evaluations where the analyst must rely on estimates of costs and benefits that cannot be observed, and where the analyst often has a vested interest in justifying the investment. For ex post evaluations, these dangers are greatly reduced as key data are typically observed, especially if the project had a reasonable M&E system in place, and the calculations are undertaken by an independent evaluator. The data, the assumptions and the methods of calculation are also in principle fully transparent, and can be checked by a third party. This is not true of many of the qualitative and mixed methods widely used in impact assessment today. Another objection to cost-benefit analysis is that it is hard to place an economic value on some social and environmental impacts, making it difficult to include them among the calculated benefits from an investment. Clearly, not all types of RPR investments are suitable for a full cost-benefit analysis, and efficiency might more appropriately be measured in terms of the average cost of delivering a unit of key social and environmental benefits (e.g. the average cost of helping a person or household escape chronic poverty, or the average cost per unit reduction in greenhouse gas emissions). But most RPR investments in developing countries have a core economic or technological component aimed at increasing productivity or economic growth, and can be judged on this basis. If there are additional social and environmental effects that cannot be quantified in a cost-effective way, then these can be reported in more qualitative ways as additional effect.
Rather than tackle cost-benefit analysis in impact evaluation some commentators recommend value for money (VFM) analysis (Brown and Tanner, 2019). A major proponent of VFM is the UK Department for International Development (DfID) that defines VFM as “maximising the impact of each pound spent to improve poor people’s lives” (DfID, 2011). The underlying concept is making the best use of available resources to achieve sustainable development outcomes.
A VFM analysis consists of gathering data along the project’s results chain so as to be able to estimate VFM indicators across five dimensions (1) economy, (2) efficiency, (3) cost-efficiency, (4) effectiveness, and (5) cost-effectiveness. These indicators closely mirror the steps in the standard linear log-frame. VFM was developed primarily as an alternative to cost-benefit analysis in assessing investment project performance, hence the strong emphasis on cost-efficiency and cost effectiveness. It is mainly a form of comparative analysis and is therefore less suited to outcome measurement in softer forms of project investment such as research. Similarly, it is not well suited to comprehensively assessing the varied dimensions of impact.
Conclusion
Existing impact studies of RPR have greatly expanded our understanding of how policy research can influence policies and welfare. They have highlighted the importance of networks of influence, messaging (dissemination), windows of opportunity, and the key role-played by participatory processes in the design and implementation of RPR. They have also emphasized the value of close interaction with policymakers, as a way to enhance impact. And, they have led to methodological improvements by using investigation techniques from many different disciplines—in short, mixed methods. However, there has been only modest progress toward quantifying the benefits of RPR, and hence assessing its efficiency or economic value. Quantitative evaluations leading to such calculations face a number of problems, especially the lack of suitable methods for measuring the impacts of many types of RPR. Even when methods are at hand, suitable data may not be available because of insufficient investment in monitoring and evaluation by the research team or its management—a commonly reported problem handicapping cost-benefit analysis in the World Bank (2010). Quantitative evaluations are further complicated by the long lead times involved before some impacts are realized (e.g. strengthened research capacity as a result of collaborative research), and because of the indirect nature of influence of RPR (e.g. RPR outputs may influence other players, such as other researchers and advocacy agents, who in turn have influence and impact on policies), making these diverse impacts impossible to track—indeed the policy research team may not even be aware of them. There may also be cumulative and overlapping effects and synergies from projects that focus on the same policy issue or on the same place. Overlooking these aspects in setting up a reporting framework could lead to distorted assessments of the contribution of RPR projects.
It is clear that impact evaluations of RPR will continue to depend on qualitative and mixed methods to provide compelling narratives connecting policy research to notable outcomes. Indeed, from the perspective of some donors, there is significant demand for “story-telling” vis-à-vis quantification (Place and Hazell, 2015). Consequently, several research groups like the CGIAR now feature links to “outcome stories” on their websites—in part, presumably, to meet such demands. Moreover, much is being learned from these and other qualitative studies about policy processes and impact pathways which in turn may help improve the design of future RPR projects.
However, most qualitative and mixed-method studies do not provide assessments of the efficiency or economic value of RPR investments. However such information is still needed by governments and many development agencies if well-informed economic decisions are to be made about the balance between public spending on RPR and on other forms of rural investment. It is unrealistic to expect that full economic analyses could or should be undertaken for most RPR projects. But some types of RPR projects should be regularly and systematically sampled to provide measures of the relative rates of return for major RPR programs, or even entire RPR institutions. As our six-step approach to an ex post evaluation shows, economic evaluations are not only an optional final step in the process of impact assessment, but also depend on completion of all the preceding steps. Hence, good qualitative and mixed-methods studies are not only valuable for the insights they provide, but are the essential foundation of quantitative economic assessments (Belcher, 2017).
Footnotes
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded independently by the authors.
