Mind the data gap

A results day like no other 

The UK education system is complex – in part due to devolution resulting in varying policies between England, Scotland, Wales, and Northern Ireland, and in part due to the mature ecosystem of qualifications that operate in schools and colleges. However, the importance of grades for students’ future progression in education and work is a consistent and defining feature of this time of year. 

It is with good reason that significant attention is paid to results season because it sets the path many young people follow through the rest of their lives. For example, many sixth forms or colleges in England and Wales require students to achieve a good pass at GCSE in Mathematics in order to progress to A level Mathematics. Meanwhile students who do not achieve a grade 4 or higher in Mathematics or English are required to continue studying the subject until they achieve a grade 4 or equivalent, or reach the end of compulsory education at the age of 18. The grades of Highers (the national school-leaving exam in Scotland), A levels, and Level 3 vocational qualifications affect students’ ability to take up university offers.

In lieu of exams, awarding organisations in the UK are generating grades for students based on a combination of teachers’ predicted grades, performance in previous formal exams, the school’s previous performance, and – in an eleventh-hour announcement by England’s Education Secretary – mock exams. Perhaps inevitably, the approach adopted by the various administrations has caused controversy, as can be seen from national reporting (here, here, here, here and here).

Equity and fairness is a key concern for the current cohort of students receiving results. Many individuals and organisations have expressed concerns that disadvantaged students may be unfairly affected by the grade-setting. Similarly, there are concerns about how valid and reliable some of the data feeding into the grade-setting may be, particularly with the now-prominent role mock exams play in grades awarded in England. A 2011 report produced for Ofqual, England’s examination regulator, noted: 

Teachers are widely assumed, and have been shown, to be very capable of rank ordering their own students in the same way that an external test would (Martinez, Stecher & Borko 2009). In other words, correlations between teacher assessments and test results are typically high for individual teachers working within the contexts of their own classrooms. But across classroom, schools, subjects and pupil year groups the association can be inconsistent (see Hoge & Coladarci 1989 for a review of research). It is this type of inconsistency that causes concern where teacher assessment features prominently in regional or national assessment systems (Wijkstrom 2006; Stanley et al 2009), and leads to the need for moderation in some form.

An experiment we may need to be prepared to repeat 

For the time being, the UK education system is still working through the fall-out from the first wave of the pandemic. But let’s look to the future.  

The re-opening of schools for all pupils in September is a stated priority for the government. Recent calls for schools to be the last to close in lockdown indicate the urgent prioritisation of education. However, a recent spate of local lockdowns in reaction to outbreaks of Covid-19 gives some indication of the likely challenges surrounding the return to school. With the no widely available vaccine likely to be available until 2021 at the earliest, the potential for disruption to education remains high. 

To minimise the risk of generational scarring caused by interrupted education and to preserve the validity and reliability of exam results, thought must be given to the role data plays in building a resilient education system.

Despite the similarities between the formulae the devolved education systems have used to arrive at their grade allocations, there are some notable differences in the availability of data between systems and qualification frameworks. For example: 

• The education system in England is relatively light on external assessment. Recent reforms have removed testing at Key Stage 3, stripped coursework and removed modular assessment from academic and general qualifications. Students take SATs in Year 6 (age 10-11) at the end of primary school; the next external national assessment they sit is in Year 11 (age 15-16) when students sit GCSEs. This means that the latest externally set and moderated exam data available for many pupils in the English system is from five years ago. 
• In the last significant reform of the England school system in 2015, AS levels were decoupled from A levels. This means that students in England who sit A level exams are awarded based on their performance at the end of the two-year course, rather than sitting an exam at the end of the first year and an exam at the end of the second year, the results of which are combined to their final A level result. (Some students in England still sit AS levels, although these tend to be students in more affluent areas or better-funded schools because of the additional exam fees associated with double-entry.) Students in Wales, however, still sit AS levels as part of their A level programme, which means that A level students’ 2020 grade will be based on performance in the 2019 exam series.
• Many vocational curricula are assessed through a combination of coursework and exam. This hybrid nature means that students will have submitted work for internally assessed units, which will have been graded and moderated before lockdown, and the awarded grade will take into account this evidence of student performance.

It is clear that there are significant gaps in student data that will influence or exacerbate algorithmically generated results. Unprecedented times call for unprecedented actions. 

But now we have a precedent, albeit a flawed one. To be completely fair, any system trying to square the circle of awarding grades in 2020 is doomed to be imperfect. The models used will be subject to scrutiny and analysis over the coming days, weeks, and months as researchers assess the impact on the attainment gap and equity.  

So, the question for policy makers looking to build resilience into the education system in readiness for any other period of disruption to come is: how do we improve the volume of reliable student assessment data? Could the UK’s next wave of education reform include a radical rethink of what assessment looks like in the 21st Century?