Evaluation

Global South non-government organizations rely on international funding and aid for continued service delivery. Service evaluation plays a significant role in ensuring compliance and ongoing service funding. Traditional service evaluation approaches could not take place during 2021 due to COVID-19, alternate mechanisms needed to be embraced. This article reports on the benefits and challenges of undertaking service evaluations online during the pandemic and the learnings and possibilities for a post-pandemic world. It emphasizes the importance of translating a relational approach to service evaluation to the online environment. Key learnings include that while some of the benefits of context and in-person connection are lost, taking a relational approach involving careful planning and reflexivity means the online evaluation process can be successful. Undertaking service evaluations online offers possibilities in a post-pandemic world as cost-effective alternatives to the expensive and time-consuming reality of in-person service evaluation across international borders and within development contexts.

Evaluation that supports social, ecological, and governance systems change and transformation raises ethical questions about what and whose worldviews do and should ground evaluative processes. This article illustrates one approach to ethical analysis within a case study of the first phase of an initiative to co-create a monitoring, evaluation, and learning system. The case drew on the principles of Blue Marble Evaluation in partnership with local staff and Indigenous leaders of the Amazon Sacred Headwaters Initiative. The approach uses critical and relational systems thinking to examine the sources of motivation, power, knowledge, and legitimacy that influence and should influence an evaluation system. Results reframe typical early phase evaluation process work from a contractual agreement to a co-created ethical space that engenders the legitimacy of the evaluation process. Contributions include a conceptual framework and process for ethical analysis that could be adapted by others.

This article discusses potential ways of combining two methods of evaluation in single-case studies: the synthetic control method and the process tracing method. Both are designed to examine certain events/programmes that take place in given cases but view these events/programmes from different causal perspectives. Seeing an event/programme as a cause, synthetic control estimates its impact on one or more outcomes. Conversely, starting from a certain outcome, process tracing uncovers the causes responsible. One can start from the causal explanation reached via one of the two methods and then proceed to examine that explanation through the other method. Once the causes of an outcome are traced via a process tracing analysis, that account can be validated by estimating the effects of those causes via synthetic control. Equally, once the impact of a certain event is estimated through synthetic control, causal mechanisms traceable via process tracing can be exploited to refine that impact evaluation.

The paper addresses the challenges of evaluating the impact of business coaching programmes with a varied portfolio of firms working across sectors and countries. Observable indicators of changes in business management practices are rarely relevant across sectors. Therefore, evaluators need to rely on the perceptions of the managers who have received coaching. We designed an online survey to compare the effectiveness of business coaching within a portfolio and across programmes. The survey was applied to the portfolio of two private sector development programmes. We derived so-called ‘contribution scores’ from individuals’ perceptions of how business management practices had changed and their perceptions of the role of business coaching in bringing about these changes. The survey included some features to reflect on response reliability. We show that the tool seems fairly reliable for comparative analysis and helped to identify the types of firms and contexts where business coaching support appears more effective.

Despite broad consensus on the importance of measuring “impact,” the term is not always understood as estimating counterfactual and causal estimates. We examine a type of public sector financing, “Social Impact Bonds,” a scheme where investors front money for public services, with repayment conditional on impact. We examine five cases in four European countries of Social Impact Bonds financing active labor market programs, testing the claim that Social Impact Bonds would move counterfactual causal impact evaluation to the heart of policy. We examine first how evidence was integrated in contracts, second the overall evidence generated and third, given that neither contracts nor evaluations used counterfactual definitions of impact, we explore stakeholders’ perspectives to better understand the reasons why. We find that although most stakeholders wanted the Social Impact Bonds to generate impact estimates, beliefs about public service reform, incentives, and the logic of experimentation led to the acceptance of non-causal definitions.

Policy evaluation literature has stressed the importance of independence of evaluations to guarantee objective evidence collection. The evaluator–client relationship is critical in this respect, since it contains inherent tensions due to the necessity for independent assessments alongside requirements for increased responsiveness to clients’ interests. Despite this distinct relationship, the client perspective has only recently received attention in research. This article presents findings from a survey among US evaluation clients and compares these to existing evidence from Switzerland. Unlike previous studies, we distinguish between constructive and destructive client influences. We show that professional experience and client familiarity with evaluation standards increase the likelihood of constructive influences aimed at improving evaluation results. Nevertheless, the findings indicate that dissatisfaction with an evaluation increases client’s attempts at influence that may be destructive. By discussing both motives behind influence and potential preventive measures, this article seeks to contribute to the increased social impact of policy evaluations.