Read on Twitter
Why could the A-level algorithm never work (THREAD)
The Algorithm that was used essentially tried to calculate the expected number of A*, A, B and so on grades within each school, and then fitted students to these, according to their ranking in the school.
The Algorithm that was used essentially tried to calculate the expected number of A*, A, B and so on grades within each school, and then fitted students to these, according to their ranking in the school.
However, this ignores the fact that within a sample (from a population), each separate sample will include variation. So for instance, even if, on average, a school produces 10% of students at A*, in one year, the value might be 5%, and in another, the value might be 15%.
This is what is known as sampling variation (and is well understood in statistics).
The algorithm, as written, paid no attention to this sampling variation, and so was always going to award an incorrect grade to a significant number of students.
The algorithm, as written, paid no attention to this sampling variation, and so was always going to award an incorrect grade to a significant number of students.
Had the algorithm started with the CAG as an initial hypothesis, and then looked to see if there were statistically significant differences in the proportion of the students within the school gaining an A*, or an A and above, (and so on), then it would have been better...
Of course, this assumes that it is even possible in all cases to calculate an expected value (and the sampling variation that surrounds it).
In many cases, because the historical data wasn't available, the expected values were impossible to calculate.
In many cases, because the historical data wasn't available, the expected values were impossible to calculate.
The sampling variation effect is magnified when you have (relatively) small cohorts.
In a cohort of 16, a single additional student gaining an A* would be an increase in the proportion by 6.25%.
In a cohort of 16, a single additional student gaining an A* would be an increase in the proportion by 6.25%.
Using the algorithm, if the expected proportion of students to gain an A* was 3%, it would be impossible for a student in that school to gain an A* - even though we know that there is certainty that students will have achieved A*s in the past!
Returning briefly to sampling variation. If any school were to produce exactly 10% of A*s year in, year out, with no variation, then this could be seen as an indication that something had gone very badly wrong, and the grading was unfair.
