Abstract
This issue explores evaluator practice around systems approaches and preferences for theories of change (ToC); and delves into specific and arguably under-evaluated contexts such as humanitarian crises, technology aids for older people and water-, hygiene- and sanitation-based development. It considers some familiar evaluation themes such as evaluation use and the role of the evaluator; explores some theoretical boundaries such as actor-network and social network theory; and extends thinking on microfinance evaluations to take account of the trend towards deploying microfinance to improve direct health outcomes.
Cristiano Rossignoli, Alberto Giani, Francesco Di Iacovo, Roberta Moruzzo and Paola Scarpellini advocate the merits of participatory evaluation even in profoundly troubled humanitarian settings. Although evaluating humanitarian action has become commonplace over the last two decades it is inevitably constrained by the disruptions, conflicts, security risks and other challenges that characterize humanitarian crises. Nowhere is this more true than in the Gaza strip – the case that this article uses to build a more general argument for bottom-up participatory evaluation. The authors set out ‘to identify the difficulties as well as benefits applying a participatory process in the Gaza Strip’. The case concerned the health and the management of reproduction among ‘small ruminants’ such as goats. These animals are an important source of nutrition and livelihood for Bedouin communities in Gaza, a key resource where ‘food insecurity and related vulnerability’ are among the most serious problems recognized by UN agencies. As the focus of the evaluation required input from local stakeholders, it confronts one of the characteristics of humanitarian action evaluations where stakeholder engagement is usually low. While even this evaluation was disrupted by renewed conflict, the authors argue that a participatory evaluation yielded positive results even though not all went smoothly. The authors demonstrate why pragmatism is so often the hallmark of a capable evaluation – just because one cannot follow all the steps in a chosen model does not mean that one should not do what can be done. As Rossignoli and colleagues demonstrate, doing what is possible is better than not even trying because of anticipated difficulties.
Emily Gates asks: what are the implications of the widespread contemporary interest in ‘systems approaches’ among evaluators for evaluation practice? She recognizes that there are a ‘myriad concepts, theories, and methods that originate in the systems fields’ which can be ‘broad and ill-defined’ and designed her eight case studies of ‘seasoned evaluators’ to accommodate this diversity. These cases set out to answer two questions: ‘1) What systems approaches are evaluators using in their evaluation practices? 2) How do these systems approaches influence the way these evaluators design and conduct evaluations?’ Gates does not expect her cases to lead to generalizable findings but rather that they ‘provide insights into … potential implications of systems approaches for evaluation practice’. It is striking how different are the understandings of ‘systems-informed evaluation practice’ among these eight eminent evaluators. At the same time, Gates identifies commonalities in the way these evaluators of a systems persuasion conduct their evaluations. She draws together implications for evaluators’ training; the way evaluations are commissioned; and for future evaluation research. However, Gates recognizes that ‘training evaluators in a different way of thinking and being is not an easy task’: a problem perhaps compounded by the evident diversity of the systems approaches that this article surfaces in these fascinating case-studies. I also found myself reflecting on how difficult it is to determine how far reported differences in evaluation practice are or are not the consequence of systems approaches. Some at least can as easily be traced through to other evaluation traditions that these eight eminent evaluators are associated with or even to the kinds of evaluation ‘objects’ that they work with. But a systems orientation would of course lead us to expect such interconnections!
Yuval Ofek observes that there are broadly two variants of ‘theories of change’ (ToC): one programme-oriented and the other actor-oriented. The former follows the course of a programme through to outputs and outcomes while the second is ‘based on the expectations and relationships forged by partner organizations, target populations of the interventions, and other stakeholders’. The author’s interest is in the preferences and expectations of users of evaluation. This is partly a matter of meeting their needs; and partly because in some settings one may be more suitable than another. He questions the assumption that evaluation users will necessarily favour programme-oriented models, even if these will be more familiar and do ‘receive more attention in the literature’. In line with actor-network theory, actor-oriented ToCs assume that ‘actors are individuals with different and often competing agendas and interests, different perspectives on the same issues, and different sets of interactions and associations’. As such, these frameworks may be especially suited to complex programmes where ‘unintended outcomes and nonlinear cause-and-effect relations’ are common. Preferences were investigated with an in-depth survey using vignettes to help respondents better understand the characteristics of each ToC variant. The survey of public sector employees and managers indicated that while there was a small preference for programme-oriented ToC, there was considerable openness to actor-oriented ToCs. In the author’s view, that at least justifies ‘that evaluators and evaluation managers should familiarize themselves with both types of ToCs, including their advantages and disadvantages and the situations when each is appropriate’.
Tim Gomersall, Louise Nygård, Alex Mihailidis, Andrew Sixsmith, Amy Hwang, Annicka Hedman and Arlene Astell are concerned with how to evaluate ‘ambient assisted living technologies’ (AAL). In a world in which the population of wealthier countries especially are ageing, pressures on care and health services are also growing. Attention has focused on AAL as one important ‘solution’ for these problems. Such technologies are seen as both a means of enabling older people to remain longer in their own homes and of ‘enhancing people’s everyday lives’ with greater independence from care and health services. As with any new ‘object’ of evaluation the question arises: ‘how it should be conceptualized?’ The authors’ ‘basic contention is that AAL products and services should be understood as technologies that both depend upon and reshape social networks’. Such networks can, for example, include engineers, service providers as well as family and friends. Gommersall and colleagues argue that ‘evaluations can benefit by moving beyond the individual level, to understand how technology produces effects at the level of the person’s support network, and what kinds of network are needed for the technology to be beneficial to a person’. The authors also argue that as society becomes increasingly ‘saturated with ICTs’ the kinds of methods needed are those that can capture diverse examples of how AAL technologies are used. These are more likely to be ‘capable of unpicking the real-life, emergent impact of technological interventions’ than experimental trials, for example. As the authors’ own research demonstrated, the diversity and creativity of the users of these technologies ensured that there is generally no single ‘intervention’ to sign up to!
Neil McHugh, Olga Biosca and Cam Donaldson discuss how to evaluate the effects of microfinance on the determinants of health. Evaluations of microfinance have tended to focus on indirect health outcomes – such as purchasing health insurance or health services. The authors are interested in more direct effects: ‘the potential of microcredit, in and of itself, to act in the more pervasive role of a public health intervention’. This mirrors recent developments in thinking about public health in the UK and elsewhere. It takes as its ‘starting premise that population health improvement is conditional on a health-promoting societal context characterized by a culture in which healthy behaviours are the norm, with supportive institutional, social, and physical environments’. This perspective on microfinance poses both conceptual and methodological problems. The authors draw on current thinking about the evaluation of complex public health initiatives to put together a proposed ‘conceptual framework’ that aims to capture the multiple ‘pathways that link microcredit and health. They revisit the experience of using RCTs in microfinance evaluations noting that problems already apparent become more severe when the evaluation objects are actual or potential health outcomes.
Stephanie Lu, Susan Elliott and Christopher Perlman take evaluability assessment (EA) into the development issues that confront those working in the water, sanitation and hygiene sector– known as the WaSH sector – in less developed countries. This was ‘an organizational level evaluation of a small, international non-governmental organisation (NGO) in water-based development’. Lu and colleagues followed the now well-recognized framework for EA set out by Thurston and Potvin. The evaluation mainly worked with the NGO’s Board of Directors around the NGO’s strategy and organization. This was important given the changing way water is being regarded in development as the Millennium Development Goals give way to the Sustainable Development Goals; and the challenges of growth that this NGO was facing. The EA was reported to have helped this small NGO to orientate its ‘WaSH work towards international goals in a constructive manner’. The authors conclude that while cases such as this are necessarily limited in the general lessons that can be drawn, this EA ‘contributes to substantive knowledge’ with regard to ‘how EAs can be conducted for resource-limited NGOs as a first step to encouraging evaluative thinking’, and demonstrates how Thurston and Potvin’s EA framework ‘can be applied in a WaSH context’.