Abstract
This issue mirrors the times we live in - challenging for public policy as well as for evaluators! We continue the debate about values and morality in evaluation with another Platform essay; and formally mark the passing of Professor Christopher Pollitt, a distinguished friend of this journal and a member of its Editorial Advisory Board. Articles also consider other current themes such as the media and our polarised societies and evaluating violent extremism; the implications for evaluation of draft EU regulations for the 2021–2027 programming period; integrating narratives and stories with theories of change; and the legitimacy of evaluation designs from a stakeholder and an evaluation commissioner’s perspective.
Anders Hanberger addresses some highly contemporary topics: how democracy is threatened by growing ‘polarisation’ in ‘media saturated’ societies where messages can be ‘slippery’ and mobilised in ways that ‘eventually change the political discourse’. According to Hanberger, evaluation is itself ‘mediatised’ when it adapts to the demands of the media, both ‘new’ and traditional. He illustrates his case with a vignette that describes how official Swedish government reports on crime was used both by the Swedish Democrats and Donald Trump to associate crime with immigration. Hanberger considers five major democratic evaluation ‘orientations’ - ‘elitist’, ‘participatory’, discursive’, ‘progressive’ and ‘market’ based - in terms of their commitment and ability to protect democracy and handle mediatisation. He argues that for the most part ‘democratic evaluation is ill-prepared to help manage current growing polarisation and threats to democracy’ although the orientation, labelled ‘progressive’, is better equipped than others. Hanberger concludes by raising uncomfortable questions that resonate strongly in Sweden now that the Swedish Democrats, a party with extremist roots have edged closer to government: Should an evaluator accept commissions that could be used (or distorted) to undermine democracy? And how might different democratic evaluation orientations be strengthened in these circumstances? Questions many readers of this journal in different parts of the world may also be asking themselves.
Margit van Wessel discusses how to evaluate advocacy in international development. She argues that advocacy that seeks to bring about sustainable change in public and private spheres embodies many of the features of complexity that evaluators tussle with. ‘Causal relations between actions and results are difficult to establish, achievements tend to be largely invisible and hard to trace and influencing often takes place behind closed doors’. The challenge evaluating advocacy is ‘how evaluation can be geared to the nature of interventions and the complexity of the relevant contexts and change processes’. Taking account of the characteristics of the evaluation ‘object’ is of course a more general concern for evaluators as the author acknowledges. van Wessel sets out to meet this challenge by combining theories of change with narrative assessment, i.e. by eliciting and interrogating the ‘stories’ told by those engaged in advocacy. She suggests that stories co-constructed by evaluators and advocates can make it possible to ‘shed light on mechanisms as they are observed and interpreted by advocates as actors’ even though potential bias in these stories needs to be quality assured. According to van Wessel, when these narratives are examined by evaluation specialists in relation to a well specified theory of change, they offer ‘a new direction for conceptualizing rigour’.
Steve Connelly and Dave Vanderhoven consider the ‘legitimacy’ of evaluation designs and consequently of the evidence these designs and associated methodologies produce. The authors are also interested in the role of ‘theory’ in evaluations and how stories and vignettes contribute - some interesting intersections then with both Hanberger and van Wessel. Connelly and Vanderhoven adopt an explicit constructivist framework building on the work of Beetham as well as their experience as advisors for an evaluation of an adult social care integration programme. They understand ‘negotiations over rival methodologies as processes of legitimation and de-legitimation’ during the stages of an evaluation; and, following Beetham, do so in terms of ‘rules’, ‘justifying principles’ and varieties of ‘consent’. Detailed description over three stages of an evaluation chart the varying levels of support among the protagonists in this evaluation for different approaches: ‘stories’, ‘quantitative and qualitative assessments of outcomes’ and ‘theories of change’. The end-result was a ‘patchwork’ of different methodologies and evidence that was nonetheless considered ‘legitimate’ both within the programme being evaluated and by city wide stakeholders. Connelly and Vanderhoven believe that ‘causal explanation is the bedrock of effective evaluation’ and that theories can be used to develop a legitimate ‘patchwork’ of different methodologies. The authors notion that even when not fully implemented, ‘Theories of Change act as regulatory ideals – not in themselves possible, but informing feasible, practically-adequate theorizing’ also echoes some of van Wessel’s thinking. Who knows, maybe moving away from either/or logics in evaluation design is beginning to gain traction!
In the Platform essay in this issue, Nicoletta Stame continues the debate around values and ethics in evaluation. In some ways Stame builds on and responds to the earlier Platform essay on this subject by Thomas Schwandt (see Evaluation 24.3); but in other ways she takes the debate in a different direction. Much of her argument focuses on the consequences of the ‘fact-value dichotomy’ in the context of a notion of evaluation that is about learning for improvement. Stame notes that whilst the principle that evaluators have a responsibility to wider society has been widely accepted - and often embodied in standards and guidelines - this now need to become operational, in terms of ‘an ethical methodological competence’. Stame considers not only the ethical competences of evaluators but also the need to consider whether policies themselves are ‘good’ and how to make such judgements. The importance of judging what constitutes ‘public values’ and ‘public good’ has according to Stame become more salient given the troubled times we live in. (In this regard she echoes some of Hanberger’s concerns.) Having reviewed extant approaches to making such judgments, Stame looks to efforts to develop a ‘moral social science’ for example in the work of Albert Hirschman and Judith Tendler. That part of the debate certainly needs to be taken further - others are welcome to join in……
Claudio Radaelli’s ‘in memoriam’ highlights some ‘lessons and memories’ of the late Christopher Pollitt. This was intentionally solicited from a distinguished political scientist and public management scholar - albeit one with an interest in evaluation who has published in this journal. Like many significant contributors to evaluation scholarship, Christopher Pollitt was a boundary-man, interdisciplinary whilst occasionally venturing in and out of evaluation worlds. Radaelli captures this liminal role well.
Lyn Pleger and Susanne Hadorn observe that research into the independence of evaluators has concentrated on the ‘pressures’ experienced by evaluators: relatively ‘little attention has been paid to the client’s perspective so far’. They situate their concerns within an Evidence Based Policy paradigm that depends on ‘facts’ gathered through ‘rational’ means to improve performance. However as evaluation takes place in political settings, it is unsurprising that pressures to curtail report findings by evaluation commissioners have often been described as a major ethical challenge for evaluators. (It is interesting to read this article alongside that by Connelly and Vanderhoven: despite quite different evaluation orientations and methodologies there are some common themes.) Starting from a Principal/Agent perspective that emphasises the extent to which evaluators and commissioners pursue their interests, the Pleger and Hadorn conducted a survey of Swiss commissioners of evaluation. Some findings are unsurprising: the power imbalance towards commissioners is clear from the way evaluators are selected; and ‘previous experiences with evaluators are decisive for the selection procedure’. At the reporting stage evaluation commissioners similarly request amendments sometimes amounting to ‘content distortions’. The authors claim that ‘the low level of knowledge of clients about the evaluation standards is striking, thus suggesting that clients are often not aware of the ethical problems they are causing through interference in the evaluation process’. In this light Pleger and Hadorn discuss how improvements in communication between commissioners and evaluators or the introduction of ‘mediating bodies’ could be encouraged.
Ben Baruch, Tom Ling, Rich Warnes and Joanna Hofman report on developing a measurement framework in what they call an ‘emerging field’ in evaluation, that of ‘counter-violent extremism’ - CVE. (The authors distinguish CVE including for example ‘radicalisation’ as a distinct area from the more developed field of counter-terrorism research.) Baruch and colleagues use evidence-based healthcare as a model for their endeavours. They regard ‘coherence, stability and agreement about what is measured and how data are interpreted’ as a first step towards enhancing evaluation capacity in the field. This focus on measuring individual or group behaviours is as the authors recognise, open to criticism and they do not hold to the view ‘that violent extremism is best understood solely by understanding individuals’ characteristics and choices’. However for the authors, it is a starting point: ‘[T]his limited but important question is our concern here’ whilst also acknowledging that this would only be part of ‘good quality evaluation which would also require ‘clarity about the underpinning theory of change’.
In our Visit to the World of Practice section Andrea Naldini considers the proposed regulations for the evaluation of EU Cohesion Policy in the next programming period (2021-2027). Naldini whose intention is to encourage a debate about these proposals, describes Cohesion Policy (CP) as ‘the largest and most evaluated policy in the world’. As an evaluation market of great significance to evaluators it is right that the evaluation community should be aware of proposed shape and design of the next generation of CP evaluations. The author reviews the way that the evaluation of EU funds has evolved since 1989 before analysing current proposals, noting the various changes from previous regulations. Naldini ends by suggesting ways in which these proposed regulations might be improved, especially with a view to improve evaluation quality.