Meta-analysis involves the aggregation of data from existing studies that share key features, in order to create a summary that reflects all the available evidence.
It can trace its early origins to methods developed in astronomy for pooling the results of multiple observations in the early nineteenth century. In the early twentieth century, these methods were developed and applied in areas such as medicine and agriculture, though it was not until the 1970s that they were at all widely used.
Gene Glass’ Presidential Address to the American Educational Research Association in 1976 is seen by many as the birth of the name ‘meta-analysis’ and the start of quantitative research synthesis moving into the mainstream1.
Rigorous methods
In that same year, a young researcher named Larry Hedges was starting graduate school and saw the need to develop rigorous methods for synthesising research findings across studies through meta-analysis.
Hedges was a statistician who went on to develop substantial parts of the theory of the analysis and combination of effect sizes, even having an effect size measure named after him: Hedges’ g. He was co-author of the first major textbook on meta-analysis (in 1985) and of many other textbooks and papers in the decades since. He is an applied statistician who has engaged actively with policymakers, practitioners and researchers in a wide range of fields, not just education.
In 2018, his “scientific approach to improving education for future generations” won him the $4m Yidan Prize for Education Research. Hedges is a colossus who has been at the leading edge of the theory and use of meta-analysis for more than 40 years.
He is coming to Durham on Tuesday 4 June to defend its use.
Controversy
Right from the start, meta-analysis was controversial.
The eminent psychologist Hans Eysenck famously dismissed it as ‘mega-silliness’. Many others have objected to particular uses or methods of meta-analysis.
In the last decade, the approach and influence of John Hattie’s research summaries, based on a synthesis of thousands of meta-analyses, has been particularly controversial. As policymakers and practitioners have been encouraged, entreated or constrained to base educational decisions on the results of these syntheses, so a number critical concerns have grown about the strength of the claims being made and their relevance for practice.
One such critic is Dylan Wiliam, well known to many for his work on formative assessment and for a lifetime’s work exploring the productive interface between educational research and practice. As well as being one of the world’s best educational researchers, Wiliam is at heart a classroom teacher: no one understands better the complexity of what teachers do in classrooms and the potential and limits of the role that knowledge of the results of education research can play in promoting, embedding and sustaining improvements in their practice. In 2016 he wrote:2
“right now meta-analysis is simply not a suitable technique for summarizing the relative effectiveness of different approaches to improving student learning, and any leader who relies on meta-analysis to determine priorities for the development of the teachers they lead is likely to spend a great deal of time helping their teachers improve aspects of practice that do not benefit students.”
Wiliam is coming to Durham on Tuesday 4 June to present his arguments.
The Big Evidence Debate, Tuesday 4 June 2019, Durham.
References
1. Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the health professions, 25(1), 12-37.
2. Wiliam, D. (2016). Leadership for Teacher Learning: Creating a Culture where All Teachers Improve So that All Students Succeed. Learning Sciences International.
