Evidence-based teaching: Rhetoric and reality

Chris Brown, Making Evidence Matter – A New Perspective for Evidence-Informed Policy Making in Education. London: IOE Press, 2014

One of the ironies of current demands for policy-based teaching is that policy makers and politicians themselves show such a cavalier approach to research, either ignoring it or opportunistically cherry-picking studies which confirm their prejudgements. Chris Brown, formerly a senior civil servant and head of research at the Training and Development Agency (TDA), is able to give us an insider’s view.

Rather surprisingly, he avoids discussing the official sidelining of research. We can find examples very easily by reading Robin Alexander’s (2014) blog post From Phonics Check to Evidence Check. In addition to the scandalous sidelining of the carefully researched Cambridge Primary Review (by a Labour government), Alexander points to the more recent National Curriculum revision; new requirements for assessment and accountability; the shift to school-led and school-based initial teacher education; the new teacher professional standards; and the preferential treatment of academies and free schools. We should add the insistence on synthetic phonics ‘first, fast and only’; baseline assessment for 4-year-olds as a way of ‘holding primary schools to account’; and a dogmatic attachment to written exams.

The strength of the book Making Evidence Matter lies, rather, in the author’s theorisation, based on experience, of the not-so-straightforward manner in which policy makers are able to engage with research. He rejects the accusation that researchers are to blame for failing to produce clear reports, along with the assumption that policy makers follow a neutral and reliable research-informed process for developing policy. He warns against naive assumptions of a level playing-field, namely

That the voices of researchers carry equal weight compared . . . with others operating within the policy ‘sphere of influence’ (such as think tanks) [and that it is] the quality of the argument that matters. (p. 2)

Researchers have to compete for attention with special advisors, think tank opinion formers, lobbyists and pressure groups, the media, and constituents and consumers (p. 48). Brown warns that research is often used ‘symbolically’ to support the decisions which politicians have already taken.

Research can legitimately impact on policy in various ways, and not simply as ‘proofs’ of ‘successful interventions’ through RCTs, for example: we should consider the importance of conceptual clarification, or challenges to a dominant discourse and assumptions.

The book exposes Habermasian assumptions of policy-making as open and even-handed participation in decision-making as naive, and draws, rather, on Foucault for models of ‘discursive control’:

When policy makers exercise power, they do not, in the UK context at least, employ direct and brutal coercion; they achieve their goals by dictating what constitutes ‘normative’ reality. (p. 95)

He quotes both civil servants and academic researchers to show how unwelcome views are rubbished.

At the same time, he seeks to persuade academic readers about presentation skills they should use, including careful timing of interventions. Having admitted that politicians have generally already made up their minds, and that civil servants are tasked with securing evidence that suits the party in government, the final chapters seem rather optimistic in their proposals for alternative ways in which civil servants and researchers could work together as a learning community. This would indeed require a ‘culture change’ (p. 145), and one wonders how this might come about, certainly in the English context.

Chris Brown (ed.), Leading the Use of Research and Evidence in Schools. London: IOE Press, 2015

Brown’s other recent book Leading the Use of Research is more of a patchwork, but welcome because of the range of voices it includes. Its chapters are also more engaged with specific issues and fields.

Toby Greany, while listing major initiatives such as the Teaching and Learning Research Programme (TLRP), points to the top-down nature of the New Labour model which shows that ‘evidence does not translate into simple linear changes in practice’. He prefers the term ‘evidence-informed practice’ or ‘knowledge animation’ rather than ‘evidence-based’. He criticises the dismissal of small-scale school- and practitioner-led enquiry, and points out the extreme work pressures which prevent teachers reading research. Although schools are very much in the driving seat under the Coalition and Conservative governments (within tight Governmental control, of course), this presents many challenges of limited capacity. Also schools are still having to second-guess the Ofsted inspectorate.

Tom Bennett’s chapter ‘Evidence and quality’ (a shorter version of his book Teacherproof) argues for firm distinctions between natural science and social science. Quantitative evidence is often problematic in the social realm because human action is less predictable, and subject to a ‘vast array of possible causal factors’. As the utilitarian philosopher Jeremy Bentham discovered, it can be extremely difficult to quantify pleasure! Furthermore, in education, many of the outcome measures are problematic because terms such as ‘literacy’ require careful definition and discussion. The author is less polemical than in his book, where the very notion of a ‘social science’ is ridiculed. It remains a central and extremely important part of his argument that many interventions have been imposed on teachers with scant research evidence to underpin them (e.g. Brain Gym, learning styles) and that much research relies inappropriately on satisfaction surveys rather than real evidence of impact. Unfortunately, there is the suggestion that snakeoil remedies have come from universities, rather than the commercialisation of continuing professional development (CPD) in the context of a systematic shrinking of local education authorities.

Various chapters (e.g. those by Lesley Saunders and Louise Stoll) argue for a more interactive approach to strengthening teachers’ engagement with research. Stoll’s emphasis on developing professional learning communities involves ‘knowledge animation’, promoting enquiry habits of mind, openness and deep learning conversations. Meaningful and productive engagement with research is not a one-way process but involves ‘surfacing’ tacit knowledge and blending it with explicit external knowledge (p. 70): it is about knowledge creation not absorption. Other chapters examine ways of overcoming practical obstacles to practitioner research in schools. Chapter 10, by Carol Taylor and Karen Spence-Thomas, focusing on effective pedagogies, is careful to avoid reductionist models and to suggest ways of attending to the pupil experience.

Lorna Earl writes a thoughtful evaluative summary of the book which is sensitive to the pressures of a high-stakes accountability regime. She warns against a narrow view of evidence as statistics and test scores, and against seeing teachers as passive consumers and implementers of research findings.

Valerie Farnsworth and Yvette Solomon (eds), Reframing Educational Research – Resisting the ‘What Works’ Agenda. Abingdon: Routledge, 2013

This book pursues, in greater depth and drawing more intensively on theoretical frameworks, the question of practitioners’ engagement in and with research. The editors begin by rejecting the implied binary whereby theoretical is seen as unpractical, and consequently practical becomes identified with a crude ‘what works’ approach. They espouse a reciprocal relationship between practice and research. The book arose from a series of seminars drawing on perspectives such as activity theory and CHAT, communities of practice theory, Bernstein and Bourdieu. One particular attraction of this book is the application of such theories to illuminate complex interventions (e.g. Daniels writing about professional development for integrated children’s services, Deignan about dyslexia support in universities).

Criticism of the ‘what works’ agenda includes concerns about naive assumptions of research providing an objective and singular ‘Truth’, which ignores the beliefs and bias which research designers bring to their projects. Similarly, the editors question the privileging of RCTs as the only reliable method. Instead of a simple research-to-practice transmission belt, they argue for an ‘enlightenment’ model of research use, ‘whereby research does not tell teachers what to do, but instead provides resources that practitioners can use to make sense of their situations and behaviours’. Research ‘calls attention to the existing conflicting positions, sometimes elaborates them, and sometimes generates new issues altogether’.

Many of the chapter authors are very well known in their own fields – names such as Harry Daniels, Mel Ainscow, Seth Chaiklin, Julian Williams and others – so it would be futile to attempt to cover all their thoughtful contributions. Chapter 4 ‘How do you make doctors?’ (Pearson, Carroll and Dornan) explores the engagement with knowledge as a kind of identity formation with a strong affective element. It is certainly not a matter of ‘assembling micro-components of medical proficiency’ (p. 55). Drawing on Lave and Wenger’s ‘legitimate peripheral participation’, they examine the ‘craftsmanlike identity formation’, which occurs through supported engagement in practice.

Chapter 6 (Mel Ainscow) looks at the Manchester Challenge, a 3-year regional school improvement initiative which he led. Within the context of politicians dismantling local authority support for schools, this initiative sought to develop new forms of network and partnership between schools. In a statistics-saturated environment, schools were encouraged to supplement quantitative with qualitative data to question why and how things are as they are, and how they came to be that way. The relations between schools and communities were explored, and with an emphasis on ‘developing children as participative citizens in designing the way things are in school’ as part of education for citizenship and aspiration-building. Mel Ainscow reaffirms the importance of democratic local government as ‘the conscience of the system’, and the strength of pluralistic understandings of ‘what works’ rather than one-size-fits-all mechanisms.

One fascinating chapter (‘Not just so stories’, by Anna Sfard) looks at the consequences of essentialist labelling, and argues for a more discursive and narrative approach to research. This connects with postmodern views of knowledge as process and discourse, and late Wittgenstein’s situated understanding of thinking and speaking. Such an approach avoids objectifying terms such as able or learning disabled in order to follow processes of development through participation, problem-solving and meaning-making.

Finally, Julian Williams and Julie Ryan provide a wonderful example of academic experts working with teachers to improve maths teaching. Moving beyond the normal expectations of Lesson Study, they describe how a ‘third space’ emerges when the academic partners step outside of the silent non-participant mode to discuss obstacles and thought processes with students, and when teachers are encouraged to say less and listen more to their students. The academic involvement provided a temporary respite from high-pressure accountability and gave teacher-partners space and time to think and discuss.

Theorising from this, the authors’ argument is that the ‘what works’ agenda places too much emphasis on ‘invariance across context’ – indeed, that this is a tacit assumption of much current policy. This results in alienation, in a Marxist sense:

As the object of policy activity the teachers and learners may be deprived of agency. In this discourse the curriculum is non-negotiable to the learner, the pedagogy is non-negotiable to the teacher, the English ‘three-part lesson’ may be a non-negotiable law for the inspectorate, and performance management may be non-negotiable for the head teacher, governor or middle manager . . . But the paradox is that – by alienating the teachers and learners in the process – ‘what works’ may come to not work after all! This raises the question: can policy be developed that allows for the free and unalienated engagement of the professional? (p. 204)

The conclusion we would like to draw is that the three activities posited, namely, ‘policy/policy making’, ‘research’ and ‘teaching-learning’ become alienated from each other to the extent that they become separate, divided by more or less impermeable boundaries, and controlled. Thus, teachers and learners may have little agency in policy making, or research may remain trapped in its own scholarly work, and policy may remain in a Westminster bubble . . . The danger with ‘what works’ as a policy agenda is that it becomes a separate and alienated activity: a government minister may decide to include long division in the new mathematics curriculum with no reference to the knowledge and expertise of teachers and researchers; or a research team may discover the pattern that single sex schools produce more physicists without revealing the processes that make this happen. But joint activity can be productive in reducing these barriers. (p. 211)

Robert Cassen, Sandra McNally and Anna Vignoles, Making a Difference in Education: What the Evidence Says. Abingdon: Routledge, 2015

This is an important and thoughtful volume which aims to survey research evidence comparing the effectiveness of different practices in key areas of education (early years, good teaching, literacy etc. Not surprisingly there are times at which the authors appear overstretched, but the book is invaluable as a reference to such a wide range of research.

The authors have sought to take ‘a nuanced approach to causality’ and deal in probability rather than certainty – ‘what can reasonably be expected’. They have made every effort to follow through on this intention, and the tone of their writing is careful and thoughtful, often providing prompts to readers to form their own balanced views. Whilst emphasising the value of RCTs as a ‘robust’ form of analysis, they recognise their limitations, and that that there are other valid forms of research evidence.

The authors are renowned educational statisticians whose analysis and systematic reviews have provided some lucid accounts of key issues such as poverty-related underachievement. Even so, there are times at which I sensed the difficulties of attempting to summarise the complex debates within a specialism from the outside.

Selecting some detail, Robert Cassen’s chapter on teaching (parts of which are devoted to teacher training, CPD and performance related pay) is problematic at times. Take, for example, the sentence:

John Hattie… concluded that about 50 per cent of the variation in student outcomes is due to students and their cognitive ability; the next largest source of variation is teaching, at about 30 per cent.

There are considerable assumptions in the notion of students possessing a quantity of ‘cognitive ability’, as there are in Hattie’s original references to ‘bright students’ and ‘not so bright students’, the absence of socioeconomics in his calculations, and so on. We read the caveat that ‘Hattie’s derivation of effect sizes from meta-analyses is methodologically controversial’ but this needs a clearer explanation.

We are told that one way of judging ‘the good teachers’ is by looking at their ‘value-added’, but warned that this should be put alongside other methods. There is overreliance on Hanushek for claims about the measured impact of having a very effective teacher – research which others have questioned. The chapter might usefully have considered controversial debates about the relative effectiveness of different teaching methods; and the problem endemic to teacher effectiveness research that the studies are often distorted by limited forms of assessment (multiple choice tests, word recognition, arithmetic calculation etc.) or indeed derive most of their data from fields where the assessment data is most readily available, such as primary numeracy. There is no mention either of the extensive research studies in the Cambridge Primary Review.

Anna Vignoles’ chapter on early years points us to a hundred research studies and explains very clearly some of the issues at stake. The Perry Pre-school evaluation, among others, is carefully considered, but more use could have been made of the English EPPE study. The author explains clearly that conclusions about the impact of an intervention depend upon the nature of the alternative; in the case of nurseries, what kind of childcare different children might otherwise receive. The claim (based on a single 1988 study) that ‘more structured’ programmes are more effective is somewhat misleading, but there is extensive evidence for the importance of qualifications. The importance of the parental contribution is highlighted, but would benefit from examples of Sure Start children’s centres. Some reservations are expressed about strongly genetic notions of ability, and the author rightly emphasises the importance of education but without pointing to the historic debates around the notion of IQ as genetically fixed intelligence.

Sandra McNally provides a useful summary of the research on school organisation, resources and effectiveness. This is a clear and cautious discussion of debates around budgets, academies, choice and accountability. It includes relevant examples of policy-led evidence, as opposed to evidence-led policy.

For all my reservations about the book’s blind spots, within the book’s parameters it provides a very useful survey of research on the relative effectiveness of different practices and a plethora of citations to follow up. However there are dangers in such volumes for those who regard such a book as a comprehensive guide, whose professional reading is limited or who read uncritically as soon as the word ‘evidence’ appears. Headteachers and policy makers should tread carefully.

John Hattie and Gregory Yates, Visible Learning and the Science of How We Learn. Abingdon: Routledge, 2014

Hattie’s hubris has troubled me for some time, along with the parlous attempts by some of his admirers to regard him as the fount of all knowledge. His claim to have synthesised over 800 meta-analyses and 50,000 separate studies has led to accolades from the TES that Visible Learning (Hattie, 2009) ‘reveals teaching’s Holy Grail’. He is doubtless a large part of the inspiration behind the design of the Education Endowment Fund’s Toolkit for guiding spending intended to close the attainment gap in England’s schools (a kind of league table ranking the most cost-effective ‘interventions’).

There are considerable doubts about the consistency of Hattie’s calculations. In particular, does his ‘hinge point’ of a 0.4 effect size (supposedly the mean of all the interventions he surveyed) signify that anything less is negative impact? Given that effect sizes are calculated in comparison with a control group, how do you construct a control group that receives zero feedback, or zero direct instruction, for example? How do his comparisons work for different ages of students? Doesn’t it matter, for the calculations, whether the intervention lasted just a few weeks or a whole year?

The technical appendix to the EEF Toolkit explains some of the problems, but does not make clear how the Toolkit itself resolves them – even if that must involve taking an arbitrary but consistent decision. This remains the case even though the Toolkit (rather shakily) attempts a translation from effect size to months of progress. The whole venture of comparing ‘interventions’ is fraught, too, in its attempt to calculate an average out of widely divergent contexts, ages, definitions, curriculum areas and so on, let alone how easily effects in experimental situations can be replicated in normal ones. A naive verification might be to add up the additional months of progress calculated for each of the Toolkit’s categories: does implementing all these interventions simultaneously produce 5 or 6 years’ progress within a single school year? does it turn 10 year olds into 16 year olds?

Having let off all this steam, I was pleasantly surprised by this book. One of the contradictions of Hattie’s writing is that, alongside the calculations, he is capable of extremely sensitive and reflective evaluations of pedagogical processes. This particular volume is concerned with how effective learning takes place, rather than with teaching.

There is, inevitably, a problem that learning is not a single entity but depends very much on field and intended outcome: learning as data acquisition/memorisation is rather different than learning within creative activity or learning to solve a practical problem. What pleased me, as a reader, was that the authors do take the trouble to differentiate. In discussing time, for example, they avoid simple ‘time on task’ calculations, recognising that the quality of attention and engagement is what matters. These distinctions enable a more intelligent and nuanced evaluation of statistical data and systematic reviews. For example, the authors cite a study which shows that, when course length is reduced by three quarters, students are able to score just as much on a multiple-choice test requiring rote learning but fail miserably on extended written questions requiring more complex understanding (p. 41).

In a chapter on the ‘recitation method’ (IRE), the authors go well beyond meta-analyses to discuss, from direct experience, how students can pretend to learn and how teachers can easily be deceived into thinking they have students’ attention. This chapter draws together intelligently various conclusions that arise from its selected reviews.

Karen Bjerg Petersen, David Reimer and Ane Qvortrup (eds), Evidence and Evidence-Based Education in Denmark – The Current Debate. Aarhus: Aarhus University, 2014

The most stimulating contribution to the ongoing debate on evidence-based practice actually comes for free, in the form of an open-access download (Issue 14 of the journal CURSIV, http://edu.au.dk/forskning/publikationer/cursivskriftserie/2014-14/). The Danish government has made a particularly strong demand for teachers to use research evidence, including privileging RCT design and setting up the Danish Clearinghouse, though debate from leading researchers has since caused it to back down in various ways including a change in terminology to ‘evidence-informed’. This collection brings together a number of significant critical contributions.

Gert Biesta raises significant philosophical questions about what is meant by evidence. He questions whether we should regard education in a quasi-causal way, and as subject to efficient ‘interventions’ which bring about ‘outcomes’, or rather as an interaction or communication – following Dewey and Mead, a ‘process of meaning and interpretation, not of physical push and pull’ (p. 21).

He argues that cause–effect relationships can be measured and compared only in closed systems, but that these are rare even in the natural world – think of the weather! Education is an ‘open system, a semiotic system and a recursive system’. Although school-based education is basically a system for achieving partial closure (through buildings, classrooms, streaming and setting, curricula, exams), and teachers work to ensure that their students learn particular things, there are serious dangers if this goes beyond a tipping point where control leads to indoctrination:

This tipping point indicates the situation where we try to stop all interactions with the outside world, where we try to completely control the meaning making of our students, and where we also try to completely control the thinking and reflection of the agents within the system – thus removing their agency altogether. (p. 21)

This places at the centre the nature of the outcomes being measured when researchers purport to calculate the most ‘effective’ methods. Biesta pursues his analysis by pointing to three aspects of what is learnt: he distinguishes qualification (the transmission of knowledge, skills and dispositions needed by society), socialisation (initiation into cultures, practices and traditions) and subjectification (identity – qualities such as critical thinking, autonomy, morality, compassion or democracy). A positive impact in one of these domains can have negative impact in another: for example, high attainment in the qualification domain might teach, in the subjectification domain, that the only thing that matters in life is competition.

Finally (as Biesta has argued in more extended form elsewhere), arguments about ‘what works’ should always be subservient to ethical, social and political questions: evidence should never replace judgement about the purposes of our educational activity and the nature of education.

This should give a flavour of the other papers in this collection. Jensen and Kjedsen explore tensions in social work, where the attempt to impose certain methods on the grounds of ‘evidence’ for their effectiveness conflicts with a professional need to be flexible and responsive in relation to client needs. The moral pressure on social workers (‘social pedagogues’, in the European view) to adopt ‘evidence-based’ methods challenges their professional assessment of situations. The authors argue for regaining a stronger professionalism, and reasserting ‘the role of judgment, tacit knowledge, and trial-and-error’ (p. 39). Social work professionals must assert their own science involving ‘action competence’, which also demands a connection between user and research perspectives and calls for ‘cooperative knowledge production’ (p. 41). These writers call for managerialism to be ‘interpreted as a political and moral programme, rather than a collection of scientifically based administrative techniques’, involving the holy trinity of ‘managers, markets and measurement’ (p. 43). By contrast, social workers’ professionalism should be ‘marked by attitudes like offering unconditional help and support for citizens in need’. Competition and efficiency cannot be prioritised ‘at the expense of the freedom to carry out professional judgements’ (p. 43). Professionals must assert their own version of evidence-based practice.

Merete Wiberg has no issue with the use of evidence to ‘ensure causality between pedagogical methods and outcomes’, and certainly does not support work which has limited effect. However, educators have to draw on ‘various kinds of knowledge and experience achieved through personal life and education’, not simply submit to authoritative guidance. While judgments have to be made to reduce uncertainty, a critical stance is needed which does not reduce the complexity of situations. Citing Aristotle on ‘phronesis’ (practical/moral judgement), it is not possible to expect ‘the same degree of precision in all situations’, especially moral dilemmas (p. 56). Rationality is part of action, not isolated in the human mind (Dewey), and inquiry involves:

observation of the detailed makeup of the situation; analysis into its diverse factors; clarification of what is obscure; discounting of the more insistent and vivid traits; tracing the consequences of the various modes of action that suggest themselves; regarding the decision reached as hypothetical and tentative until the anticipated or supposed consequences which led to its adoption have been squared with actual consequences. This inquiry is intelligence. (Dewey, 1957, pp. 163–164)

Professionals should not be deceived by a method ‘dressed in evidence-based clothing’ or spurious guarantees of certainty and predictability. Educational professionalism requires a sense of democracy and responsibility, and a resistance to the development of authoritarian personalities.

Albrechtsen and Qvortrup discuss the need for a broad view of ‘evidence’, reminding us that evidence in a court of law can involve ‘personal experiences, testimonies, documents or archives, artefacts and observations’. We use evidence in many ways in daily and professional lives:

We all find pieces of evidence, make links between them, discover patterns, make generalizations, create explanatory propositions all the time, emerging out of our experience, and this is all ‘empirical’ (citing Thomas, 2004).

Teachers are ‘already employing different kinds of evidence in their teaching as part of an ongoing process, but with different abilities and levels of reflection’. Hence, the argument is not simply ‘a case of either having evidence or having no evidence for something’. Teachers draw on ‘experiential knowledge’ including conventions and norms and the experiences they had as students: all of this reflects ‘what usually works in concrete settings’ and should not be discarded. The authors argue that experiential knowledge should enter into dialogue with research and pedagogical theory. Professional learning also involves noticing. While it is unprofessional to base one’s decisions solely on gut feelings and opinions, it is also unacceptable to make decisions based merely on ‘data’ or ‘evidence’. Intuition has a role, and decisions need to be more ‘care-driven’ than ‘data-driven’.

Thomas Rømer continues the debate, taking issue with those who would rank the blind controlled experiment over personal evaluations by users and professionals. He points to the error of privileging knowledge which is furthest away from practice, and of confusing quantifiable with objective. ‘Evidence’ becomes damaging when it ‘detaches itself from the situated structure of educational knowledge and connects instead to a new system of policy goals that are not related to pedagogy and learning theory at all’. Problems include approving certain methods regardless of subject matter or context; pushing away cultural purposes, that is, the goals and aims of education; emptying educational practices of culture, virtue or politics; and reducing practice to the simple application of evidence-based rules.

Rømer warns us that much of the evidence-based movement is designed to serve the global economy, treating the pupil as a ‘soldier in the competition among nations’;

Education is not about using techniques to maximise a ranking score, but rather about appearing in an effective and energetic culture in full, vibrant memory.