Read on Twitter
Did Ofqual use "professional" statisticians with experience of the type of question they were asked to answer?
Or was this a bunch of folks drawn on for their academic skill and purist instincts with little experience of real world modelling?
A thread.
Or was this a bunch of folks drawn on for their academic skill and purist instincts with little experience of real world modelling?
A thread.
The more I learn of how the Ofqual algorithm worked...the less confidence I have in the folks responsible for it.
But hey! What do I know? I only worked as a "professional statistician" for 20+ years developing predictive statistics.
Only I don't have Professor in my name!
But hey! What do I know? I only worked as a "professional statistician" for 20+ years developing predictive statistics.
Only I don't have Professor in my name!
Suspicions over what this bunch of chambers did starts with their banging on about fitting for accuracy.
Which is important. But not the be-all and end-all of the job. As I used to try to explain in a Prediction 101 talk I gave to business people.
Which is important. But not the be-all and end-all of the job. As I used to try to explain in a Prediction 101 talk I gave to business people.
Because, sure: "accuracy" is important criterion. But so, too, is consequence.
If you build a risk predictor, whether for credit scoring or child abuse, you must always deal with a trade-off.
If you build a risk predictor, whether for credit scoring or child abuse, you must always deal with a trade-off.
Roughly: the lower you set the risk of erring on the plus side (minimise false positives), the more likely you are to e're on the down side (maximise false negatives).
And vice versa.
Also, not all "accurate outcomes" are equivalent.
And vice versa.
Also, not all "accurate outcomes" are equivalent.
The latter I explained by reference to staticians fave measure, which is to opt for predictive accuracy of 95%. Which is a great measure if you're looking at mailing response or insurance risk.
Less so if a mortgage. And disastrous if playing advanced Russian roulette.
Less so if a mortgage. And disastrous if playing advanced Russian roulette.
Cause 95% accuracy implies 5% of the time - one time in 20 - you get it wrong. I don't mind odds like that if I'm betting a tenner on the grand national.
I mind them very much if the matter being predicted is life threatening.
I mind them very much if the matter being predicted is life threatening.
And so to Ofqual. Fitting results accurately to a range of grade outcomes A to F plus U is a simple academic issue. But the practical consequences fluctuate widely according to the specific outcome.
In particular, the awarding of a U.
In particular, the awarding of a U.
Because changing grades from AAA to AAB MAY spell the end of a specific university outcome. But at least the individual emerges with three qualifications that will count in future.
Whereas taking any single result down from a graded result to a U may well be life-changing...
Whereas taking any single result down from a graded result to a U may well be life-changing...
For starters, it means you only have 2 qualifications as opposed to three. This has knock-on effects in terms of not just uni places, but also apprenticeships.
Your entire set of life options is suddenly limited...
Your entire set of life options is suddenly limited...
A professional Ofqual team wd have put in additional moderation/checking for changes to predicted grades deemed to be life-changing.
Specifically, they'd have had special procedures to review AT INDIVIDUAL LEVEL any result demoted to U.
Probably also any demotion of 2+ grades.
Specifically, they'd have had special procedures to review AT INDIVIDUAL LEVEL any result demoted to U.
Probably also any demotion of 2+ grades.
The fact they didn't suggests strongly that Ofqual relied on the wrong sort of modellers: a bunch of academics obsrssed with an accuracy measure, to the exclusion of trickier conclusions on outcome.
That is all.
/fin
That is all.
/fin
