Abstract
Every two years the European Evaluation Society (EES) organizes its biennial conference. We are currently in full swing preparing the EES 2018 Conference (www.ees2018.eu) around the theme of Evaluation for more Resilient Societies. It is therefore a particular pleasure to be able to introduce this Special Issue of the journal Evaluation, which contains a fine selection of papers from the previous EES conference, held in 2016, in Maastricht (www.ees2016.eu). The theme of the 2016 conference was: Evaluation Futures in Europe and Beyond. Connectivity, Innovation and Use. The articles found in this issue relate to this subject in various innovative and original ways.
Evaluation has always been a practice-based exercise, borrowing from different branches of the social sciences, more than a theory-based discipline. However – and not only because it is in our statutes – the EES also believes it is important to develop the academic dimension of the evaluation discipline and profession. Evaluation has to establish its own theoretical and methodological canons alongside those it borrows and adapts from elsewhere. Therefore, we strongly support evaluators contributing to the development of theory and new methodological approaches. One way of doing this is to encourage those who present at EES biennial conferences to prepare full scholarly papers, which, providing they are of a sufficiently high standard, will be published in Evaluation. This is intended to keep the balance between practice and theory, and between borrowing from other disciplines whilst continuing to develop our own contribution to the social sciences.
More than 500 abstracts are generally proposed for presentation at an EES conference. These are ‘peer-reviewed’ before being selected. Most of those selected for ‘paper’ and ‘panel’ sessions are then presented verbally backed up by PowerPoint® slides – and sometimes inputs appear in ‘poster sessions’. Growing these presentations into full articles takes time and effort. A first long-list on the basis of the conference abstracts is followed by outlines and drafts which then go through the regular peer review processes of the journal. It is always pleasing to see how many interesting articles emerge after EES conferences following this often-extended process. The first ‘wave’ of articles following an EES conference traditionally appear in a Special Issue of Evaluation but there are always more that cannot fit into a single issue and continue to be published in this journal’s pages and in other publications over the following years.
This Special Issue of Evaluation contains seven interesting and innovative articles proposing new evaluation theories and approaches, insights and findings. What follows is a brief introduction to these articles – enough hopefully to whet our readers’ appetite to read them all in due course.
Francesco Giffoni, Silvia Salini and Emanuela Sirtori contribute to the much-practiced – yet still poorly theorized – evaluation of SME and business support, an important plank in EU Cohesion Policy. The authors note that whilst Theory Based Evaluation (TBE) has begun to feature in the evaluation of business support this has mainly been ‘focused on explaining how a policy contributes to produce certain effects’, rather than on ‘quantitative and statistically significant estimates’ of impact. The article uses Bayesian Network Analysis in combination with TBE as a new mixed-method approach to reveal the mechanisms, [and to explain] how public support leads to changes in a firm’s behaviour and economic performance. This approach is empirically tested in policy interventions within the 2007–13 Cohesion Policy programme in Poland, Spain and Italy. The authors are cautiously optimistic about further developing this set of mixed methods whilst also suggesting that there is potential not only to contribute to evaluation theory and methodology, but also to the economic theory of the firm.
Astrid Brousselle and Jean-Marie Buregeya argue that theory-based evaluation approaches and, in particular, contribution analysis, logic analysis and realist evaluation, should be understood as complementary components of a new theory in evaluation. Following Guba and Lincoln’s multi-generational framing, they suggest that we are seeing the emergence of a fifth generation in evaluation: the explanation generation. Brousselle and Buregeya assess each theory-based approach in terms of ‘knowledge construction, valuing, and use’, the three dimensions used by Shadish, Alkin and colleagues, as the ‘foundational dimensions’ that characterize all evaluation approaches. On this basis the authors argue that a new consolidated theory is possible because the main theory-based approaches ‘share foundational principles and answer complementary, and even overlapping, evaluative questions…’.
James Copestake, together with colleagues from the UK, Ethiopia, Malawi and Ireland, regards impact evaluation as ‘a social process’ that focuses on ‘who gains access to information through the evaluation process and when’. The authors argue that the ‘choreography’ of different protagonists in an impact evaluation has an important influence on the credibility and usefulness of the generated evidence. They relate this choreography to wider debates in international development which are then explored in greater depth in a case study of a ‘cost-effective approach to impact evaluation’. This entailed collaborative design of a qualitative impact evaluation protocol (͚the QuIP ͛) and its pilot use in Ethiopia and Malawi. The authors use an original approach – double blind interviewing – to reduce project-specific confirmation bias, followed by staged ‘unblindfolding’ as a form of triangulation. The article argues that these steps can enhance credibility of evidence, and that ethical concerns associated with them can be addressed by being open with stakeholders about the process.
According to Sebastian Lemire and Christina Christie, as policymakers nowadays routinely demand that policies and programmes be evidence-based, evaluators are also expected to produce an evidence–base of generalizable knowledge. However, the authors argue that we currently over-rely on a single ‘blueprint’ – the use of systematic reviews of experiments and quasi experiments for ‘knowledge construction across individual evaluations’. Lemire and Christie examine the promises and pitfalls of such reviews through the example of a recent meta-analysis of ‘Housing First’ studies. Whilst this analysis produced statistically significant ‘effect–size’ estimates, there were also considerable variations. Furthermore, ‘significant unexplained variation is a commonplace occurrence in systematic review’. The authors unpick weaknesses in the ‘data–bricks’ that contributed to building these results – including different levels of ‘community involvement’; lack of consistency specifying ‘Housing First’; poor consideration of local contexts; and, most importantly, very different definitions of ‘housing tenure’ – the chosen measure of impact and ‘what works’. Lemire and Christie advocate alternative blueprints for knowledge building that either set out to complement meta-analysis with other approaches; or more radically build on the quite different methodological assumptions.
Silke Haarich starts from the observation that, as many complex societal challenges cannot be ‘solved’ by one organization or government, networks and partnerships have become increasingly important. Haarich discusses different understandings and theories of networks and multi-level governance which increasingly feature in contemporary public policymaking and its evaluation. These understandings and theories derive from fields as diverse as institutional economics; meta-governance; Social Network Analysis; actor-centred institutionalism; and public management. The author notes the continuing theoretical as well as practical barriers to evaluating networks and, on this basis, identifies ‘the requirements for a new evaluation tool’. It is on this basis that the author came to design a specific new ‘tool’ intended to support the evaluation of networks as complex governance systems. According to Haarich, this tool – labelled GOCAPASS – has strengths and weaknesses. For example, whilst it ‘supports evaluation of processes and learning in networks, it does not evaluate policy field results that may (or not) have been achieved due to the network’. Arguably the tool (or toolkit) could be expanded in this regard. Although still in an early development stage, it has been tested in regional innovation policy in Chile; and in governance of EU macro-regional strategies.
The 2030 Sustainable Development Goals (SDGs) are integrated and interdependent, confronting evaluators with demands to evaluate a system of multiple rather than single objectives. Evaluation approaches focusing on objectives in isolation are now less relevant than they were in relation to the prior Millennium Development Goals (MDGs). Anne Stephens, Ellen Lewis and Shravanti Reddy were commissioned by the UN Women Independent Evaluation Office to design an evaluation approach suited to evaluation in the SDG context. This takes the form of a new ‘Guide’ that rests on the GEMs framework – an abbreviation derived from:
Peter Milley, Barbara Szijarto, Kate Svensson and Bradley Cousins explore the overlap between evaluation and social innovation (SI) practice. The authors draw on 28 empirical studies of evaluation in SI contexts to describe what evaluation practices look like, what drives those practices and how they affect SI. The reported influences of evaluation on SIs included advancing SI strategies, improving delivery, balancing aggregate and local information needs, and reducing risk. Conflict resolution, the quality of relationships, and availability of time and capacity mediated these influences. Building on the cross-cutting analysis of the 28 different cases, the authors argue that peer-reviewed empirical studies and a broader range of study designs are needed to further explore the overlap between evaluation and SI. This should include research on how evaluations influence SI processes over time, during different innovation phases, in different spatial settings and at different scales of social innovation.
This Special Issue demonstrates the liveliness and relevance of the theoretical and methodological thinking in evaluation today. Exploring the frontiers of the discipline of evaluation remains a mission to which the EES continues to be committed. At the same time, it is to be hoped that several articles in this issue will also contribute to other branches of the social sciences – for example, to our understanding of networks, the theory of the firm and even of ‘intersectional’ analysis.
In conclusion, I would like to thank Riitta Oksanen, former EES President, for her contributions to the present issue and more generally for all the work she has done for the Society during her presidential term. Thank you very much Riitta! And finally, I would also like to greatly thank the editor of Evaluation Elliot Stern and members of the journal’s editorial team – especially Emma Hamilton – for the energy and effort that has gone into ensuring the content and quality of this Special Issue.
On behalf of all who have contributed to making this issue, I trust that you find it an interesting and instructive read – and one that that will impel you even more strongly to join us at the forthcoming EES conference in Thessaloniki Greece this October!