Against Reproducibility: A thread. https://twitter.com/MPC_HMGU/status/1362213277216948227

One of the most annoying things about the reproducibility crisis industry is the default position that labs getting different results doing the “same” (but is it) experiment is a problem.

It’s not a problem—it is an opportunity to learn what factors influence complex biological systems. It's a critical driver (perhaps the critical driver) of discovering unknown unknowns, the most important kind of unknown.

And it's a problem that can be provisionally ignored much of the time, because the conclusions of each study should be bounded by the effects they measured relative to appropriate controls and…

…with *as a given* that it is under the conditions, animals, food, housing, reagents, instruments that were used in a particular study.

In biology, we seldom know what input variables are important and can only control a subset of them.
I can’t count all the times something interesting has been discovered that was motivated by the question, “Why are we getting a different result from them?”

So it would IMO be extremely progress-thwarting to demand that we put some huge effort into eradicating variability in experimental conditions in discovery science.

The way to find out which factors are important is not to demand that we reduce variability, it is to encourage documenting the variability in experimental conditions and to not characterize (expected) variability in outcomes as a “failure” or a “crisis.”

The current study is interesting because they tried to reduce some of the variability (strains used, measurements), and saw that results still varied a lot. A lot a lot. This is a good thing to do, but I hate that that result is framed as a “reproducibility” problem.

Because look:

But also look:

The conditions are not standardized AT ALL! This is not a “reproducibility” problem, this is a RESULT about how food relates to metabolism. Why would we want to FIX this by demanding conditions be standardized to the point that we discover less? /fin