Evaluation

In recent years, evaluation research has taken an increasing interest in the proliferation of evaluation systems. One of the main assumptions being that such evaluation systems can enhance the use of evaluations and their findings. The current article analyses this assumption by empirically studying the workings of a specific evaluation system within the Danish Ministry of Employment. By applying the concept of constitutive effects, the article analyses how the evaluation system affects processes of policymaking within the central government. It is shown how the institutionalisation of an evidence-based evaluation system, at the ministerial level, structures the process of policymaking around a specific understanding of policy content, timeframes and expertise. Based on these findings, it is argued that evaluation systems can both enhance the instrumental use of evaluations and significantly alter policymaking processes.

Olympic impact/legacy analysis as a research area has become a frequently discussed topic in the sociology of sport literature in general and in event management in particular since the 2000s. Although Olympic impact/legacy analysis has increased in volume, many studies still face methodological challenges (e.g. failure to adopt rigorous theoretical frameworks or overlooking additionality). The aim of this article is to unpack the logic of stakeholders in a critical evaluation of two London 2012 legacy programmes, critically reviewing the outcomes sought, and the actions adopted to achieve those outcomes, with reference to the concepts of programme theory, action theory and process tracing. We seek to redress the shortcomings of previous Olympic legacy claims which have failed to link theoretically informed accounts of the nature of Olympic-led/Olympic-inspired outcomes to explanations of the specific causal mechanisms that account for the generation of such outcomes.

The literature on policy design provides insightful resources to improve policy evaluation. We compare how this literature addresses causation, evaluation, instrumentation, and intervention, based on different methodologies. Then, we explain how the realist approach can contribute to theory-based evaluation with multi-methods. We go through the process leading from the adoption of policy aims to the production of a policy outcome by a causal mechanism linking agenda setting to policy formulation, cross-sectorial coordination, political interplays, and this policy’s outcome. This hypothesis is then tested against a typology of policy instruments classified by the State resources. These instruments are hereby treated as expected empirical observations that provide an empirical basis to assess the theoretical process following a Bayesian logic. We eventually present a five-step protocol aimed at making realist policy evaluation easier, which includes process theorization and operationalization, empirical tests design, case selection and justification, congruence analysis, and deep within-case study.

There is a well-documented interest in how insights from the study of complexity can be applied to policy evaluation. However, important questions remain as to how complexity is understood and used by policy evaluators. We present findings from semi-structured interviews with 30 UK policy evaluators working in food, energy, water and environment policy domains. We explore how they understand, use and approach complexity, and consider the implications for evaluation research and practice. Findings reveal understandings of complexity arising from contextual factors, scale-related issues and perceptions of unpredictability. The evidence indicates terminological and analogical use of complexity and its concepts by policy evaluators, but limited evidence of its literal use. Priorities for the future include framing complexity more pragmatically and as an opportunity not a cost. Communicating this up the policy hierarchy is the key to progressing complexity-appropriate evaluation – this can be enabled by strengthening links between policy evaluation and academic communities.

This article discusses the integration of a diagnostic lens in qualitative or mixed methods evaluations, arguing that this will improve quality: in particular, it will improve transparency, credibility and reliability of evaluation findings.We start by unpacking the notion of evaluation quality and pointing out the typical weaknesses of qualitative methods. We then introduce the basic notions of diagnostic approaches and how they relate to theory-based evaluation, process tracing, and Bayesian updating, arguing for the merits of taking a formal Bayesian approach, founded on the confusion matrix, which amongst else reduces confirmation bias and conservatism. This article draws parallels between the process tracing tests and elements of the confusion matrix.

This article describes The Evidence of Impact Model that was recently developed to support the analysis of short- to medium-term impact arising from Participatory Action Research and Developmental Evaluation approaches that were used within the Evidence for Change initiative. Evidence for Change was a team capacity building initiative that supported rapid, bottom-up, evidence informed change in health and social care environments. To explain the need for The Evidence of Impact Model and outline how it was developed, a variety of similar models are explored and the challenges of capturing impact arising from Evidence for Change are discussed. The resulting new model based on the existing concept of micro, meso and macro levels of impact is outlined in detail with supporting qualitative evidence. The authors conclude that The Evidence of Impact Model provides an effective lens for exploring impact at a variety of levels within an emerging and complex environment.

The Full Country Evaluations were Gavi-funded real-time evaluations of immunisation programmes in Bangladesh, Mozambique, Uganda and Zambia, from 2013 to 2016. The evaluations focused on providing evidence for improvement of immunisation delivery in these countries and spanned all phases of Gavi support. The process evaluation approach of the evaluations utilised mixed methods to track progress against defined theories-of-change and related milestones during the various stages of implementation of the Gavi support streams. This article highlights complexities of this type of real-time evaluation and shares lessons learnt on conducting such evaluation from the Zambian experience. Real-time process evaluation is a complex evaluation methodology that requires sensitivity to the context of the evaluation, catering for various information needs of stakeholders, and establishment of mutually beneficial relationships between programme implementers and evaluators. When used appropriately, it can be an effective means of informing programme decisions and aiding programme improvement for both donors and local implementers.

This article outlines how a realist evaluation of dementia care in hospitals used non-participant observation to support the refinement and testing of mechanisms likely to lead to the use of person-centred care. We found that comments and explanations of their actions from hospital staff during observation periods provided insights into the reasoning that generated their actions for care in real time. This informed subsequent data collection and analysis. Two worked examples of mechanisms first identified during non-participant observation demonstrate (1) how they were uncovered, and (2) how this informed research activities for theory refinement. Early, iterative engagement with the analytic process, primarily involving reflection and debate with the research team, maximised the potential of observation data to support surfacing underlying mechanisms, linking them to specific contexts and outcomes.