Dr Lindsey Macmillan is an Associate Professor (Reader) of Economics at UCL Institute of Education, Dr Matt Dickson is a Reader in Public Policy at Institute for Policy Research, University of Bath, and Professor Simon Burgess is a Professor of Economics at University of Bristol.
A HEPI Occasional Paper out today claims that “grammar schools … play a significant role in supporting social mobility”. This is based on two statements in the paper: firstly, that a high proportion of disadvantaged pupils attend grammar schools, and secondly, that areas with selective systems are better at progressing children into elite universities.
The striking claim is in stark contrast to much of the rest of the evidence, including our own. On closer inspection, the report’s claim relies on faulty data and inappropriate statistical methods.
Faulty data
The claim made by the report’s author that ‘45% of pupils in grammar schools come from below median income households’ is based on a Department for Education (DfE) technical report that clearly states “The data and threshold used to define households as below the median income in this analysis should therefore be treated as provisional. Caution should be taken in drawing definitive conclusions from these findings until we have completed further work” (p. 12).
This DfE report presents analysis on the first attempt to match household income data records into the National Pupil Database. As is quite clear on reading the DfE report, the household income data that is matched in is incomplete, particularly for higher earners, meaning that 70% of pupils are defined as below median household income (£24,900) when using a threshold from a national survey.
Using data that dramatically over-estimates the number of below-average-income families is obviously central to how we should view a finding that there are more below-average-income families in grammar schools than we might have thought.
However, the HEPI report does not mention this severe drawback of the data, and makes no correction for it in the analysis. Rather this questionable finding is repeated numerous times throughout the report.
Our own published analysis of the background of pupils attending grammar schools clearly shows that these schools provide very limited opportunities for pupils from both disadvantaged and ‘ordinary working’ families.
Inappropriate statistical methods
The analysis on progression to higher education compares progression in areas with grammar schools (selective areas) to progression in all non-grammar areas (see Table 2, page 27).
This is a naïve comparison, given that we know that selective areas are not representative of the nation as a whole. The areas that chose to keep grammar schools have specific characteristics – they are, for example, generally more affluent and have a higher proportion of degree educated people. These are precisely the sorts of characteristics that support access to elite universities, and so we would naturally expect to see more pupils in those regions attending ‘highly selective higher education institutions’. As such, comparing grammar school areas with all other areas will upwardly bias the grammar school effect.
Other research has shown that even restricting analysis to grammar areas matched with similar looking non-grammar areas does not solve this bias problem, and so we should expect the naïve comparison in the report to contain a non-trivial upward selection bias. This inappropriate statistical approach means that the report’s conclusion is unlikely to be robust.
Incomplete picture
More fundamentally, conclusions about the impact of any policy on social mobility have to consider the impacts at the bottom of the distribution as well as the top. Even if grammar schools increase the chances of disadvantaged children who attend them going on to (elite) HE institutions, the aggregate impact on social mobility depends on whether there is any offsetting impact on the disadvantaged in these areas who do not attend the grammar school. Given that the vast majority of disadvantaged students in selective areas do not attend grammar schools, and the evidence suggests that these students suffer an education and wage penalty compared to their equivalents in comprehensive areas, the conclusion that grammar schools actually enhance aggregate social mobility is unlikely to hold true.
The merits or otherwise of selective versus comprehensive education is a fiercely debated area of education policy and has been reignited by recent government moves to expand the grammar school system. As with all policy analysis, it is very important to ensure that appropriate data and robust statistical methodologies are used before drawing conclusions and making policy recommendations. This is particularly the case when the actual aggregate impacts of a policy may in fact be the reverse of what is being claimed.
Excellent research, should be disseminated widely!
Thank you for engaging. While the Department for Education publication from which the 45% figure is drawn is provisional, it represents the best estimate currently available. It also takes steps to address the issues you raise, notably using the median income in their dataset (a lower figure), not the £24,800 figure, to calculate their results.
More importantly, the key findings on progression to higher education in my report do not depend on the 45% figure and are drawn from independent data sources, as referenced in the report. This include the fact that 39% of pupils in selective school areas progress from state schools to highly-selective universities, compared to just 23% in comprehensive areas, or that England’s 163 grammar schools send 30% more BME pupils to Cambridge than the nearly 2000 comprehensive schools.
Overall, my report acknowledges that less advantaged children are somewhat less likely to attend a grammar school. Notwithstanding this, the benefits of attending, in terms of progression to highly selective higher education, are so dramatic that they continue to be a net benefit to disadvantaged pupils in those areas.
Thank you again for your engagement and I'll respond more fully in a Wonkhe piece on Monday.
Mansfield’s source was also taken out of context for the 2016 Conservative Manifesto in order to mislead the public. https://www.conservatives.com/manifesto Page 50 states, “Contrary to what some people allege, official research shows that slightly more children from ordinary, working class families attend selective schools as a percentage of the school intake compared to nonselective schools.” I wonder who carried out that "official research"?
Another potential feature is that in areas with selective schools the parents may be less likely to use the private sector. Whereas in areas without grammars, such parents may opt for private rather than state schools.
Great blog you've got here.. It's difficult to find high-quality
writing like yours these days. I really appreciate individuals like you!
Take care!!
Hi,
Thanks for this really interesting read. In my undergraduate I studied a lot to do with social mobility and grammar schools looking in particular at "fairness"and equality.
In my research I found that not only is data on GS varied and selective, the GS themselves are geographically selective. E.g. in the south Manchester area there are 4 grammar schools but there are none between this area heading southwards for 30 miles. Therefore not only is academic ability a barrier but also the location and capabilities of students to get there.
Personally, I do not fall with the crowd that tends to criticise GS as I don't think we can successfully assess the instrumental value with existing data or system.
What are your thoughts on this?
Best wishes,
Maddy