Meta-analyses and the making of an evidence-informed profession

Before entering into a broader discussion about the use of meta-analysis in school decision-making, it’s worth appreciating just how far teaching has progressed towards becoming an evidence-informed profession.

The very fact that teachers and school leaders like me are questioning schools' use of meta-analyses and systematic reviews is incredible. It was only 6 years ago that Ben Goldacre outlined his revolutionary vision for the use of research in education, believing it could learn valuable lessons from medicine where evidence-based practice has led to huge advances in patient outcomes.

Establishing a culture

In a paper entitled Building Evidence into Education Goldacre suggested there was ‘a huge prize waiting to be claimed by teachers. By collecting better evidence about what works best, and establishing a culture where this evidence is used as a matter of routine, we can improve outcomes for children, and increase professional independence.’

To me, the phrase ‘establishing a culture’ is important. It is only by normalising the use of research evidence to inform decision-making in schools that we can learn from what has worked elsewhere, and draw upon these insights intelligently within the context of our own school circumstances.

As Goldacre rightly points out, teaching, like medicine, involves a combination of ‘craft [knowledge] and personal expertise, learnt over years of experience.’ Teachers are more likely to be able to improve outcomes for their students when they combine the best of their own insights with the best insights learned from elsewhere.

And this is where meta-analyses come in. For all its problems, the aggregation of multiple studies - sometimes reported as an effect size or as the number of months' progress – has made it easier for school leaders to understand where best to direct their limited time and resources.

There is much more of a culture of teachers and school leaders using research now than there was back in 2013. This is worth bearing in mind when considering how schools are using meta-analyses.

The evolution of meta-analyses

John Hattie’s Visible Learning and the EEF's Teaching and Learning Toolkit are probably the two best-known examples of meta-analysis in education. Whatever your views about the validity of these attempts at summarising the impact of different strategies on student outcomes, it is hard to deny their growing influence on school improvement activity.

Whilst Hattie may have fallen a bit out of favour – perhaps a consequence of the growing criticism of his approach – the EEF continues to increase in influence. Well over half of school leaders now use the toolkit to inform their decisions, up from just over one in ten in 2012.

To me, this is no mean feat, and evidence of building the kind of culture that Goldacre envisaged, one where research, alongside craft and institutional knowledge, guides decisions about how best to improve student outcomes.

I understand the flaws in combining studies of different scope, design and emphasis – the apples and oranges problem. I also accept that the dashboard style of presenting the findings of synthesised studies is flawed and even potentially misleading. These are fair criticisms that those using the findings of meta-analyses should understand and those putting them together should make clear.

How meta-analyses are used in schools

At one end of the continuum, there will be some school leaders who in their search to make a difference focus only on crude impact figures, and who fail to recognise limitations of the process or nuances behind the findings, such as the details provided in the underlying studies themselves.

Whilst unfortunate, and something that needs to change, this overreliance on meta-analysis is a reflection of the pressures some school leaders feel to get immediate results for their students. This is not really anything new and at least demonstrates schools are attempting to do the right thing!

At the other end of the continuum are more intelligent and well-informed uses of meta-analyses and systematic reviews, such as this example from Megan Dixon where the findings from aggregated research represent the starting point for the shaping of school improvement, alongside more localised forms of knowledge and insight.

The future for meta-analyses

More and more schools and school leaders (like me) are learning how to do this kind of development activity effectively – starting with the big picture provided by meta-analyses and understanding what it can and cannot tell us, and how to dig a little deeper to find out what lies behind the headlines.

The research culture that Goldacre imagined for the profession is growing, and meta-analyses and systematic reviews are playing an important role in connecting school leaders to the evidence base, helping them to make ‘best bets’ for their students as a matter of course.

The very fact that we are having this discussion about the value of meta-analyses in education is, I think, evidence that things are moving in the right direction. A research-informed culture is taking hold.

About the author

Phil Stock @joeybagstock

Phil is deputy head teacher at a large secondary school in South London, and has worked in 3 different schools in different roles, including Head of English. He is currently responsible for teaching, learning and assessment, as well as staff professional development. Phil is also the research lead at his school and has a keen interest in the role of evidence in improving student outcomes.

Find out more:

Follow the Big Evidence Debate on Twitter #TBED2019.