Read on Twitter
Hey #NLProc (and AI folks working on language), can we just ... not?
https://www.neowin.net/news/microsofts-ai-model-has-outperformed-humans-in-natural-language-understanding
https://www.neowin.net/news/microsofts-ai-model-has-outperformed-humans-in-natural-language-understanding
The Microsoft blog post that this links to is slightly less bad, only because the "natural language understanding" part is buried in text, not the headline:
https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/
https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/
Back to the puff piece, there's a bit where they say: "It is important to note that ..." and I thought maybe there's be something sensible coming like: these results should be treated with caution because, but no.
"... this is not the first model to surpass human baselines."
"... this is not the first model to surpass human baselines."
This is just pure #AIhype and it does harm in the world. Everytime you tell the public that "AI understands human language" (ahem, English), then that lends credibility to all of the AI snake oil be sold for surveillance, exam scoring, exam score interpolation, etc. >>
Also, it looks ridiculous, too. If you think that scoring high on SuperGLUE shows understanding, show me what your machine is doing that actually amounts to understanding.
But before anyone @s me with their claim to how their LM is understanding something, please read what I've written about this (or at least watch our talk video) (w/ @alkoller ) https://www.aclweb.org/anthology/2020.acl-main.463/
