Read on Twitter
GPT-3 won a best paper award at #NeurIPS2020! Congratulations to that team, it truly is an incredible piece of work, and has changed the way many of us think about what massive LMs can do. But we should also talk about inequality in the research community -- that work couldn't...
have been done almost anywhere other than OpenAI. With an estimated cost to reproduce of millions of dollars in just compute (not including researcher time and salaries), no academic lab could have won that award for that work. Best paper awards are some of the most...
prestigeous items in our field; they drive promotions, hiring, etc. and focus attention on the work, so it's worth talking about. "Best paper" is not well-defined, and here we're rewarding work that is empirically not reproducible, which doesn't provide low-budget experiments...
so future researchers can compare against it without training a comparable model for millions of dollars to make comparisons "fair". There IS value in answering scientific questions that can only be answered with millions of dollars, but it's unclear if we should consider such...
work in the same tracks (and thus, eligible for the same awards) as research which promotes ideas which can be evaluated by scientists with smaller budgets too.
