Read on Twitter
I don't get the argument made against training bigger models from the environmental perspective. Given breakthroughs like AlphaFold, do people really think they can weigh the opportunity cost of n-th order outcomes due to training bigger models?
Can we really bet that the auxiliary knowledge we get from training bigger models will not transfer to say climate modeling or other potentially beneficial use cases?
I think many underestimate how more trial & error is a precursor to positive black swans.
I think many underestimate how more trial & error is a precursor to positive black swans.
Given how there are other direct ways of bigger impact on the carbon footprint (like going vegan or reducing air travel or campaigning for renewable resources), using climate change to argue that research in larger models is misdirected, seems vacuous and potentially regressive.
This reminds me of how even the Indian government's minuscule spending in space research gets criticized because India has other pressing problems. It ignores positive n-th order effects and black swans from of such research.
And all this is not to say research in smaller models/ Green AI is not important. I think the work to improve efficiency of smaller models can set good constraints for innovations. But just that it doesn't make research in bigger models misdirected.
