Read on Twitter
It looks bad at first glance...
...but the implied counterfactual literature with a normal distribution around 0 would only happen if we studied associations at random. Instead, scientists often opt to study things where there is prior evidence to suggest a real effect. https://twitter.com/fmg_twtr/status/1334884184675012609
...but the implied counterfactual literature with a normal distribution around 0 would only happen if we studied associations at random. Instead, scientists often opt to study things where there is prior evidence to suggest a real effect. https://twitter.com/fmg_twtr/status/1334884184675012609
You can get close to that figure from an innocuous literature with these rules: if effect is known to be small, rarely study it. If studying, study effect once. If you get significant effect, study more. Here's a small, simulated literature; looks bad, but it's perfect science:
Now lets implement p-hacking: imagine any researcher who gets a z-score close to the critical value (within 10%) can find some way to game statistics to get a significant result. That's a lot more like the figure from the paper:
That version of p-hacking is, at worst, fudging a value of .078 to something that is significant at a=0.05. Is it incorrect and possibly unethical? Yes. Is it leading to a literature of mostly incorrect results? No. It leads to a plot that looks bad *ONLY* when focused on p<0.05
The literature implied by the awful looking plot of Z-scores is consistent with a *nearly* flawless literature. We all know it's not flawless, but we should also reject the *vacuous* idea that this means the literature is largely incorrect. (ahem https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
This underscores the point that obsessing around statistical significance means losing sight of vastly larger range of results that should be considered in a continuum, rather than as binary decisions about whether an effect exists or not.
The authors of this paper give a careful, nuanced result. Resist the Twitter hot take from the plot that makes it seem like there's a vast literature of garbage science.
