Dr Lindsey Macmillan is an Associate Professor (Reader) of Economics at UCL Institute of Education, Dr Matt Dickson is a Reader in Public Policy at Institute for Policy Research, University of Bath, and Professor Simon Burgess is a Professor of Economics at University of Bristol.
A HEPI Occasional Paper out today claims that “grammar schools … play a significant role in supporting social mobility”. This is based on two statements in the paper: firstly, that a high proportion of disadvantaged pupils attend grammar schools, and secondly, that areas with selective systems are better at progressing children into elite universities.
The striking claim is in stark contrast to much of the rest of the evidence, including our own. On closer inspection, the report’s claim relies on faulty data and inappropriate statistical methods.
Faulty data
The claim made by the report’s author that ‘45% of pupils in grammar schools come from below median income households’ is based on a Department for Education (DfE) technical report that clearly states “The data and threshold used to define households as below the median income in this analysis should therefore be treated as provisional. Caution should be taken in drawing definitive conclusions from these findings until we have completed further work” (p. 12).
This DfE report presents analysis on the first attempt to match household income data records into the National Pupil Database. As is quite clear on reading the DfE report, the household income data that is matched in is incomplete, particularly for higher earners, meaning that 70% of pupils are defined as below median household income (£24,900) when using a threshold from a national survey.
Using data that dramatically over-estimates the number of below-average-income families is obviously central to how we should view a finding that there are more below-average-income families in grammar schools than we might have thought.
However, the HEPI report does not mention this severe drawback of the data, and makes no correction for it in the analysis. Rather this questionable finding is repeated numerous times throughout the report.
Our own published analysis of the background of pupils attending grammar schools clearly shows that these schools provide very limited opportunities for pupils from both disadvantaged and ‘ordinary working’ families.
Inappropriate statistical methods
The analysis on progression to higher education compares progression in areas with grammar schools (selective areas) to progression in all non-grammar areas (see Table 2, page 27).
This is a naïve comparison, given that we know that selective areas are not representative of the nation as a whole. The areas that chose to keep grammar schools have specific characteristics – they are, for example, generally more affluent and have a higher proportion of degree educated people. These are precisely the sorts of characteristics that support access to elite universities, and so we would naturally expect to see more pupils in those regions attending ‘highly selective higher education institutions’. As such, comparing grammar school areas with all other areas will upwardly bias the grammar school effect.
Other research has shown that even restricting analysis to grammar areas matched with similar looking non-grammar areas does not solve this bias problem, and so we should expect the naïve comparison in the report to contain a non-trivial upward selection bias. This inappropriate statistical approach means that the report’s conclusion is unlikely to be robust.
Incomplete picture
More fundamentally, conclusions about the impact of any policy on social mobility have to consider the impacts at the bottom of the distribution as well as the top. Even if grammar schools increase the chances of disadvantaged children who attend them going on to (elite) HE institutions, the aggregate impact on social mobility depends on whether there is any offsetting impact on the disadvantaged in these areas who do not attend the grammar school. Given that the vast majority of disadvantaged students in selective areas do not attend grammar schools, and the evidence suggests that these students suffer an education and wage penalty compared to their equivalents in comprehensive areas, the conclusion that grammar schools actually enhance aggregate social mobility is unlikely to hold true.
The merits or otherwise of selective versus comprehensive education is a fiercely debated area of education policy and has been reignited by recent government moves to expand the grammar school system. As with all policy analysis, it is very important to ensure that appropriate data and robust statistical methodologies are used before drawing conclusions and making policy recommendations. This is particularly the case when the actual aggregate impacts of a policy may in fact be the reverse of what is being claimed.
