Read on Twitter
Okay, finally got a chance to read this WIRED piece on the IB fiasco (now that #acl2020nlp is done). https://www.wired.com/story/algorithm-set-students-grades-altered-futures/
Full disclosure: my older son finished his IB diploma last year, in a program that let him complete it in 10th & 11th grades. So in other words: but for that program, he would have been caught up in this this year. So, he wasn't directly affected, but it was a near-miss.
I think the most valuable thing about the WIRED article is the quotes from @geomblog:
“All this points to what happens when you try to install some sort of automated process without transparency,” he says. “The burden of proof should be on the system to justify its existence.”
“All this points to what happens when you try to install some sort of automated process without transparency,” he says. “The burden of proof should be on the system to justify its existence.”
Also: "Using data from other schools—as IB did for schools with little track record—is a “red flag,” he says, because it would mean some students’ grades were calculated differently than others."
Another striking thing about this article is people (not @geomblog ) quoted as saying "the model is flawed", which I think misses the point:
The flaw lies in using a model to solve the problem (students can't take tests) that attempts to predict what students would have scored.
The flaw lies in using a model to solve the problem (students can't take tests) that attempts to predict what students would have scored.
IB faced a difficult problem: the tests they use to determine student scores in their program couldn't be administered in the usual way and yet they needed to determine who gets the diploma and with which scores.
Looking to machine learning to "fill in" those scores, even if the model is "validated" on past data, is not an appropriate deployment of machine learning.
On top of that, doing it without even being transparent about both the model and the data they trained it on makes it even worse. (Transparency wouldn't fix the underlying flaw in deploying ML this way, but it would help with accountability.)
At a meta-level, I'm really curious how @iborganization came to this decision: whose idea was it to turn to ML? Was this an internal suggestion, or something that was sold to IB? >>
Does anyone at @iborganization have sufficient understanding of ML to be able to evaluate this idea, and were they listened to if they raised concerns? >>
Where to go from here? I think @iborganization needs to make this right. Even if *on average* the scores look a little better than usual, there are actual people whose educational and career goals have been derailed. >>
And *even if* some IB students in a normal year end up with scholarships/college acceptances rescinded because of their actual scores on actual tests, that doesn't mean it's fair that the same happened because of imaginary tests this year. >>
And also: I think there's a lesson to be learned here about what expertise is needed in-house. Any business thinking about buying ML services from a third-party needs to have access to someone (employee, consultant) who can advise them on appropriate v. risky deployment.
