Abstract
This issue of Evaluation follows soon after the death of Carol Weiss, a renowned evaluator, an active member of our Editorial Advisory Board and a good friend to this journal. In News from the community we mark Carol Weiss’s passing with a short remembrance prepared by another member of our Editorial Advisory Board, Nicoletta Stame. In this issue we include articles that are both theoretical and methodological; and both policy- and practitioner-oriented, in line with the way Carol Weiss herself exemplified and encouraged us to favour.
Luísa Veloso, Pedro Abrantes and Daniela Craveiro explore the evaluation system put in place since 2005 in Portugal’s schools. The evaluation system was initiated by Portugal’s education Ministry in line with a tendency internationally towards similar public management innovations. The authors suggest that in Portugal, with its relatively recent transition from dictatorship, school reforms should also be seen in terms of how the State reshapes decentralisation and democratic practices put in place with this transition. The school evaluation system involves ‘strategic choices and ‘ideological assumptions’, however the authors do not see this simply as ‘imposition’ by government. Rather Veloso, Abrantes and Craveiro regard the evaluation system as a ‘social construction’ in which many actors exert influence and help shape the process. An understanding of how different actors participate in the evaluation system is central to the analysis presented. Patterns of participation – of communities, parents and pupils – differ between schools and regions. This follows from different economic, cultural and political traditions as well as rural and urban differences. Such differences are reflected in the kinds of (self) evaluation reports produced by schools. As Veloso and colleagues conclude, it is ‘the democratic participation of the various actors in the education community in the organization of the school [that is] one of the key factors differentiating organizations and the (implicit) criteria that govern the evaluation process’.
Tim Blackman, Jonathan Wistow and Dave Byrne argue that Qualitative Comparative Analysis (QCA) is especially suited to ‘complex’ and ‘wicked’ policy problems characterised by multiple causes and with different causal pathways in different contexts. This is a further example of the increasing visibility of QCA methods in evaluation (see for example, Stefan Verweij and Lasse Gerrits in 19.1). The authors deploy QCA to understand patterns of teenage conception rates across England which have remained stubbornly above policy targets and vary across local authorities. Whilst policy guidance commonly suggests that specific factors or variables are associated with better local performance, Blackman, Wistow and Byrne argue that analyses based on ‘the ‘variable’ only’ is ‘potentially misleadingly if there is wide variation and too much significance is attached to an average that has no reality in any cases’. Using QCA, the authors identified cases where the ‘gap’ between local and national rates of teenage conception narrowed and cases where this did not happen. An interesting feature of the approach adopted was the involvement of practitioners in the study. For example ‘results were shared with practitioners’ which allowed the researchers to ‘incorporate insights from practice into [our] explanations so that these could be grounded in practitioners’ worlds’.
Denise de Souza revisits some of the classical debates in realist social theory to address issues that are also of interest to realist evaluators. The author traces some of the differences in emphasis between realist evaluators who she suggests share Ray Pawson’s ‘cognitive’ critical approach with a more ideological stance of ‘critical’ realist theory. One may or may not be in sympathy with the author’s view that realist evaluators have followed a more ‘social engineering’ rather than the more normative path favoured by critical realists. However it is unusual for writers in this journal and elsewhere to dig back into the roots of realist theory with as much care and attention as has de Souza. The author’s intention is to draw on the ideas of critical realists – following Bhaskar and Archer – to address practical problems identified by realist evaluators, ‘reinstating the critical realist perspective in realist evaluations and syntheses/reviews’. In particular Denise de Souza looks again at the familiar Context Mechanism Outcome (CMO) configuration through a different lens, by placing greater emphasis on cultural as well as structural aspects of ‘context’.
Bob Picciotto argues that the evaluation criteria and tools used in international development to evaluate ‘development effectiveness’ deserve wider consideration and application. Notions of ‘development effectiveness’ have become more important in order to convince taxpayers among others that aid and development transfers are worthwhile, that ‘aid works’. Picciotto suggests that the policy worlds of development and other policies are converging to reflect the ‘emerging global consensus’ that ‘public policy should seek to remedy economic, social, political, cultural, institutional and subjective ills that reach well beyond material deprivation’. Hence the wider appropriateness of evaluation criteria such as ‘relevance’, ‘efficacy’, efficiency’, ‘sustainability’ and ‘impact’ that have been developed within the framework of the OECD’s Development Aid Committee.
Gender equality is a widely shared policy commitment in international development as in domestic policy in many countries. Julia Espinosa discusses gender equality and gender sensitivity in evaluation practice. In particular she ‘aims to contribute to an increase in knowledge about the incorporation of gender issues into the evaluative processes of international-development agencies’. The author provides a useful historical overview of the evolution of gender-sensitive evaluation practice in the international sphere from the Women in Development (WID) approach of the early 1990s through to the Gender and Development (GAD) approach following the Beijing conference of 1995 based on structural understandings of inequalities between women and men. Espinosa reviews methodological issues such as the use of frameworks and indicators, doing so throughout from a committed feminist stance. From this perspective the author identifies barriers to the more effective incorporation of gender equality into development evaluation and what would be needed to overcome these barriers.
Orla Doyle and Claire Hickey consider the issue of ‘contamination’, a critical issue for experimental and quasi-experimental methods. These methods assume that those receiving a service or ‘treatment’ can be distinguished from those who do not – the control or comparison group. If ‘contamination’ occurs between the two groups – that is the intended control receives a similar service or treatment – then the internal validity of an experiment is threatened. The authors are especially interested in the extent of contamination about which relatively little is known. Doyle and Hickey first review the literature and then consider contamination in the specific case of three childhood intervention programmes in Ireland. The authors look to the different ways contamination occurs as well as what can be done to limit it.
In A visit to the world of practice Veronica Gaffey describes new thinking about the way EU Structural Funds should be evaluated. Requirements of regional and cohesion policy in these Structural Funds have been important drivers for the spread of evaluation practice and culture across the EU, especially in the South and East of Europe. Since the mid 1990s, Structural Funds evaluations have been guided by a ‘logic model’ that links both the outputs with the results and impacts; and the short-term with the assumed long-term effects. This logic model has been used to support methods that rely heavily on ‘impact indicators’. Gaffey and colleagues from the EU Directorate responsible for regional policy acknowledge that ‘indicators alone cannot tell the whole story’. In terms of a serious ‘results orientation’, this evaluation approach has proved to be unable to capture results and impacts. Many evaluators have also long been uneasy about the EU’s over-reliance on indicator-led evaluation approaches. They will welcome an intention for the future to ‘abolish ‘impact indicators’ and instead focus on impact evaluation, using different methods – both quantitative and qualitative – counterfactual and theory-based’.