Last month the Education Endowment Foundation published their first set of evaluation reports. The six studies involved 238 schools and 6,800 pupils and represented the first stage in a process that aims to deepen our knowledge of what works in education, with a particular focus on raising the attainment of those eligible for free school meals. Several of the studies were designed as randomised controlled trials (RCT’s) which are widely acknowledged as the most powerful way to determine causality in education research.
Given the clamour for RCT’s amongst the education community – most evident at the ResearchEd event last autumn – it was surprising how little coverage these reports got. Kevan Collins (from the EEF) did his bit with a rather self-congratulatory piece in the TES trumpeting the reports as an example of what can be achieved when schools work together. A couple of other articles commented on the findings in relation to teaching assistants who came out of these studies slightly better than they have done in other research. But on the whole the reaction was muted: nothing groundbreaking had emerged to shake the foundations of the education system. Instead we got a preview of the kind of nuanced findings that these RCT’s are likely to produce over the next few years.
It is no coincidence that the effect sizes detected were relatively small (typically ranging from 0.10 to 0.25). The attraction of RCT’s is that they can do a good job of isolating the impact of a particular intervention. It is hardly surprising, therefore, that the reported gains were modest. Improving attainment is difficult and it would be extraordinary and implausible if relatively small interventions led to huge gains. Hopefully the publication of these reports will lead to a greater questioning about the reliability of much of the existing research. The truth is that many of the studies included in Hattie’s Visible Learning, for example, fail to meet the basic standards of methodological rigour. Reading Recovery is a good example. As the evaluation report from the EEF notes, of 78 existing studies on Reading Recovery only 4 met basic evidential standards. In the coming years we will see an increasing divide between the methodologically sound studies that report modest effect sizes and the less rigorous that will continue to claim implausibly large benefits for certain interventions. It is critical that consumers of education research are aware of that divide and encouraged to look beyond effect size and instead at how robust the finding is. The EEF could help here by getting rid of the conversion they use in the toolkit between effect size and months progress. It is neither rigorous nor particularly illuminating and it promotes a one-dimensional analysis of the evidence.
The other point that stands out from the first round of reports is the difference between evaluation and endorsement. Even where a positive outcome is detected it does not follow from this that the intervention is worth pursuing. The costs – both financial and opportunity – need to be carefully considered. In some cases an intervention that appears effective on the surface might actually be a poor use of time and resources. The Catch-Up Numeracy study, for example, found that although the carefully designed small-group intervention programme was effective compared to equivalent time in the classroom, it was actually less effective than other small-group interventions. Until we have built up a large-number of RCT’s looking at a whole range of interventions we are unlikely to be able to make accurate judgments about what constitutes best practice. In this sense the utility of the toolkit is cumulative: the more reports that are included the more powerful it will become as a guide for decision making in schools.
For now the publication of these initial reports should be welcomed. They will not immediately tell us how to improve the education system but they will start to give us an idea about what works (reading recovery-style literacy catch-up programmes) and what doesn’t (Catch-Up Numeracy). If this knowledge informs decisions that school leaders make about where to invest time and money then we are certainly moving in the right direction.