Evaluation

In this introductory article, a brief history and introduction to contribution analysis is provided to lay the stage for the articles that follow. At the heart of contribution analysis is the aim to be able to make credible causal claims about the contribution an intervention is making to observed results. The key role that theories of change play is noted, and what a useful theory of change ought to contain is discussed. The article then makes a link between the philosophical discussions on causality and contribution analysis through a discussion of contributory causes. It is argued that such causes, which on their own are neither necessary nor sufficient, represent the kind of contribution role that many interventions play: where there are a number of other influencing events and conditions at work in addition to the intervention of interest. Contribution analysis is an approach to confirming that an intervention is a contributory cause.

Contribution analysis (CA) has been advocated and discussed more than it has been applied. This article draws on five evaluations applying CA in the context of EU policies in the areas of development aid, agriculture, employment, and governance. Over the last five years, the authors have developed their capacity to implement the six steps of CA and especially to draft contribution stories and to make contribution claims. These practical efforts have, in turn, caused them to explore the foundations of the approach and refine and operationalize key concepts such as contribution claim, causal mechanism, and strength of evidence.

This article examines the methodological strengths and weaknesses of contribution analysis. The authors contend that a salient characteristic of contribution analysis is its accounting for influencing factors and alternative explanations. We argue that contribution analysis in its current form needs further methodological and practical elaboration in this respect. Therefore, this article pays particular attention to how the methodology can identify and determine the extent of influencing factors and alternative explanations. The authors propose a corresponding framework – the Relevant Explanation Finder – which can help evaluators account for these elements and thereby create more credible contribution stories.

Contribution analysis is a structured approach to theory-based impact evaluation originally developed in Canada in the context of Results-Based Management (RBM) although there have been few examples of contribution analysis in practice since Mayne’s original paper (2001). We argue that contribution analysis adds value to other theory-based evaluation approaches by providing a more structured and rigorous approach to participatory evaluation planning, data analysis and reporting. It can be applied in the context of participatory strategic planning and performance monitoring as well as impact evaluation. Examples are drawn from Scotland and Canada in the performance context of RBM in Canada and Outcomes-Based Accountability (OBA) in Scotland. The authors argue that, as a participatory process, contribution analysis strengthens both conceptual and practical understanding of planning/managing for outcomes and implementation and change theories, thus helping to build collaborative capacity within and across partner organizations. For public managers, the contribution analysis process has a strong appeal and practical value when faced with the task of demonstrating the contribution of single organizations to addressing complex social issues while working in partnership with other public agencies facing multiple accountabilities.

In recent years, debates on impact evaluation of development cooperation have flourished. There has been a marked increase in statistical impact evaluation exercises, most notably randomized experiments, and at the same time, many evaluations with a focus on impact have relied on alternative methodological approaches. The diversity in methodological approaches can be explained by differences in epistemology, characteristics of the evaluand and of the evaluation context. Broadly speaking there are two approaches: impact evaluations that rely on statistical design and/or multivariate analysis with statistical controls as a basis for attribution; and impact evaluations relying on systematic argumentation guided by some type of causal theory of change. Contribution analysis is an example of a heuristic framework that fits in the latter category and this article relates this approach to an evaluation of prizes awarded by UNESCO (The United Nations Educational, Scientific and Cultural Organization). It discusses the intervention logic underlying UNESCO prizes and the corresponding framework for empirical assessment that in part reflects the institutional context and constraints. Finally, we discuss how the evaluation can provide the basis for a full-fledged impact evaluation of a prize using contribution analysis.

Contribution analysis and theory-based evaluation are linked. This article discusses the contributions of several (recent) developments to further deepen the links between theory-based evaluation and contribution analysis. This will be done by describing how three well-known (and almost classical) problematic situations for impact evaluations can be addressed through the application of insights and tools from theory-based evaluation.

Utilization-focused evaluation involves identifying and working with primary intended users to design and interpret an evaluation. This includes the process of working with primary intended users to render judgments about the extent to which the preponderance of evidence supports a meaningful and useful conclusion about degree to which an intervention has affected observed outcomes and impacts. This is the essence of contribution analysis. Two in-depth examples illustrate this process.

This article discusses the further development of theory-driven evaluation approaches that are informed by contribution analysis. Using an illustrative example of an ongoing dance/physical activity programme for health promotion, a number of challenges are identified when applying a theory-driven evaluation approach. These challenges are reformulated as questions that need to be answered to make further progress with theory-driven evaluation including contribution analysis. Questions include: What is a ‘good enough’ programme theory? How does one arrive at expectations of programme impacts? How does the programme theory incorporate heterogeneous mechanisms that programme recipients might need? What does causality mean for complex interventions? What are structures that can facilitate learning from evaluations? How does the application of theory-driven evaluation approaches help generate an ‘ecology of evidence’? Discussion of these questions leads to a ‘roadmap’ for how contribution analysis might be further tested and refined.