Read on Twitter
I wanted to write a thread about my understanding of the A-level adjustments, partly developed through discussion with @Frances_Coppola, and others.
As a summary, the adjustments, whilst creating a distribution that delivers an appropriate distribution, but unfair outcomes
As a summary, the adjustments, whilst creating a distribution that delivers an appropriate distribution, but unfair outcomes
OfQual have had an incredibly difficult task thrust upon them, so it's not surprising that there are significant problems with the applied algorithm, but these problems could have been (and were) foreseen, and the results should probably have been treated more generously.
Because students could not sit formal exams, their grades were constructed based on:
1. Centre Assessment Grades
2. Historic grade distributions for the Centre
3. The relative prior attainment of the cohort at the Centre, compared with the previous average prior attainment
1. Centre Assessment Grades
2. Historic grade distributions for the Centre
3. The relative prior attainment of the cohort at the Centre, compared with the previous average prior attainment
1. The Centre Assessment Grades (CAGs) created a prediction of the student's attainment, and (more importantly) a ranking of the student in the cohort.
The rankings are important, because the algorithm used creates a distribution of grades, onto which students are aligned
The rankings are important, because the algorithm used creates a distribution of grades, onto which students are aligned
2. The distribution of prior grades from the centre is constructed based on the prior 3 years distributions of grades. This gives an initial guess of the grade that a student would have achieved.
3. This distribution is then adjusted based on whether the current cohort has better or worse prior attainment than previous cohorts.
For example, if the GCSE performance of the current cohort is higher, then the proportion of A*s available increases, and vice versa.
For example, if the GCSE performance of the current cohort is higher, then the proportion of A*s available increases, and vice versa.
Having constructed the distribution, students were mapped onto this revised distribution, so in a cohort of 100 students, if 5% were due to gain A*s, then the 5 top-ranked students would be awarded an A* - note that this is irrespective of the centre assessment grade.
As a note, this is why the ranking was so important - the grade was determined by the rank, not the CAG; as such, these grades may bear no relation, and in reality 39% of all grades awarded were lower than the CAG
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/909368/6656-1_Awarding_GCSE__AS__A_level__advanced_extension_awards_and_extended_project_qualifications_in_summer_2020_-_interim_report.pdf
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/909368/6656-1_Awarding_GCSE__AS__A_level__advanced_extension_awards_and_extended_project_qualifications_in_summer_2020_-_interim_report.pdf
This then becomes important, as if a centre were defined as "small", then the statistical algorithm would be unreliable, so the calculated grade was replaced by either the CAG, or a blend of the CAG and the grade from the algorithm.
So the first takeaway is that students who sat A-levels in small cohorts (either at subject level), or at school/college level, would be more likely to be awarded their CAG, and as such, would be more likely to gain a higher A-level grade.
This can be seen in the big increases in A* proportions in subjects that are relatively small (e.g. German, Classical subjects), or that will have small cohorts within schools (e.g. Further Maths)
This raises the first major issue of potential fairness with the system, in that students will be treated differently, depending on where they are at school. Those in large centres are allocated grades largely based on past cohorts (and their rank)....
... and students in small cohorts are awarded grades based on their teachers assessment of their own ability and potential.
Returning to the algorithm, there is a further sting in the tail, in the adjustment for the prior attainment of the cohort.
The distributions of grades are adjusted by the relative prior attainment of the cohort compared with previous cohorts' prior attainment.
The distributions of grades are adjusted by the relative prior attainment of the cohort compared with previous cohorts' prior attainment.
The aim of this is to adjust the distribution so that, if the current cohort is very high achieving, then the proportion of high grades will be higher. If the current cohort is relatively low achieving, then more low grades will be available.
However, this can cause a skew in the distribution, as if there is a random shock in the year of GCSEs (e.g. a couple of pupils had very bad GCSE results), then this will pull down the distribution of available A-level grades.
This means that, if, say, pupil A gained bad GCSE results, due to illness, for instance, this would pull down the distribution of grades, and if pupil B is low ranked in the cohort, pupil B may gain a lower A-level result, because of pupil A's GCSE results!
The way the prior attainment is included is by constructing a probability distribution of A-level grades nationally, based on their average performance at GCSE.
The algorithm does take account of the fact that older pupils (within year) perform better at GCSE...
The algorithm does take account of the fact that older pupils (within year) perform better at GCSE...
... however, it doesn't take account of if there are students who might have had one or two GCSEs in which they had performed poorly. Since the metric (appears to be) an average across all GCSEs taken (rather than, say, the best 8), this causes comparison issues.
Then, the grades need to be applied to pupils.
Is, say, the standardisation algorithm applies that 5% of students gain an A*, 15% gain an A, 12 % gain a B, then:
The top 5% of the ranks of students gain an A*, the next 15% gain an A, and so on.
Is, say, the standardisation algorithm applies that 5% of students gain an A*, 15% gain an A, 12 % gain a B, then:
The top 5% of the ranks of students gain an A*, the next 15% gain an A, and so on.
However, it is rare that these sorts of distribution exactly match the number of students in a cohort.
(e.g. If there were 19 students in the cohort, you would have to make a decision about whether the 4th student gained an A or a B).
(e.g. If there were 19 students in the cohort, you would have to make a decision about whether the 4th student gained an A or a B).
To calculate this, the percentile is calculated based on the mid-point of the student's rank, and compared with the distribution. The largest grade that satisfies the inequality below is awarded.
In this example, we would look at (4-0.5)/19=0.184
In this example, we would look at (4-0.5)/19=0.184
Remember, the top 5% would be awarded an A*, the next 15% an A, and so on.
So, in this case, 20% of students would be awarded an A or above. In this case, the percentile is 18.4%, so the largest grade to be awarded is an A, so the student would be awarded an A.
So, in this case, 20% of students would be awarded an A or above. In this case, the percentile is 18.4%, so the largest grade to be awarded is an A, so the student would be awarded an A.
Where this raises an issue is if, for instance, there are 19 students (as in the example above, but, say, 3% of students were due to be awarded a U.
Even with a CAG of an A*, the 19th ranked student would be awarded a U, as the percentile would be 97.3.
Even with a CAG of an A*, the 19th ranked student would be awarded a U, as the percentile would be 97.3.
