Chapter 2 of #ScienceFictions is ostensibly about the Replication Crisis but goes much further afield than psychology. The chapter offers a succinct case for why we are right to talk about a wider ‘crisis’ in science rather than just cincerning results for some specific theories.

The opening quote of the chapter is a sentiment really wish more people took to heart. Published means ‘survived peer review’ it does not mean ‘proven to be correct’. This should be self evident but is too often ignored, including by people who should know better.

The chapter begins with a whirlwind tour of some of the most high profile failures in psychology and ties them all together by suggesting that they provide the foundation for what became the ‘replication crisis’. The summary provided is great & while brevity means some details...

...are not discussed, you do get a good overview of the problems with things like headline catching priming studies & TED talk generating ‘power posing’ research. The problems with the classic Stanford Prison experiment are also highlighted, illustrating how deep the issues run.

It’s important to consider here just how many of these effects were promoted through pop psychology books or things like popular TED talks. Key ingredients include a counterintuive finding, a focus on implicit cognition, and the promise of simple life changing interventions.

And the problem isn’t just restricted to self-promotional researchers hawking some fashionable new theory, we have giants in the field like the noble prize winning Kahneman, who published papers warning about overinterpreting low powered studies that fall prey to these issues.

It’s hard not to wince when reading quotes like the one below, especially when considering that subsequent research has largely confirmed that many of the high profile results Kahneman is touting were cherry picked ‘statistical flukes’.

As Stuart notes Kahneman later (admirably) fully acknowledged his mistakes but the same cannot quite be said for Zimbardo and Amy Cuddy both of whom remain wedded to defending their research. See: https://www.vox.com/science-and-health/2018/6/28/17509470/stanford-prison-experiment-zimbardo-interview and https://www.forbes.com/sites/kimelsesser/2018/04/03/power-posing-is-back-amy-cuddy-successfully-refutes-criticism/#7d9de5033b8e.

In any case the chapter moves on to highlight that the concerns raised in psychology’s ‘Replication Crisis’ are not actually unique and we find similar kinds of problems in fields like economics, neuroscience, evolutionary biology, and marine biology.

These are not the fields people typically express concerns about but that may be in many cases because there have not yet been the same level of internal auditing that is currently going on in psychology. I think it’s also worth considering how little attention such concerns...

...are given by people who condemn the lack of rigor in fields like critical studies or gender studies. I’m not arguing for some false equivalence here. Just that if one of your stated concerns is the accumulation of reliable scientific information in academia, these issues...

...should be concerning, rather thanjust the ones that relate to the endless culture wars. Indeed, most concerning is when Stuart moves on to discuss how the same issues with low quality research & a lack of focus on replication impact medical research, including cancer research.

There is a lot of rather terrifying information presented, including how few biomedical studies even report enough details to make replications possible. Similarly, a list of recommended treatments that were later found to be ineffective, or worse harmful, makes for grim reading.

Most of this was not news to me (though I had no idea about bispectral index monitors & how useless they turned out to be) but when presented collectively it comes across as quite damning, which is likely part of why ‘alternative’ medicine advocates can seem so convincing.

A point to make here though, that Stuart emphasizes later, is that these discoveries of ineffective treatments and recognition of poor quality evidence are being made by other scientists/researchers. This is part of what makes science unlike many other approaches.

So while I agree that the high level of Cochrane reviews that report insufficient evidence for various treatments (45%!). It’s also important to celebrate that such reviews are being undertaken and negative or ambivalent findings reported.

The chapter ends on a semi-depressing note by noting that researchers, such as John Ioannidis, had been issuing warnings that went largely ignored. His classic paper ‘Why Most Published Research Findings are Wrong’ should be a staple of undergraduate courses.

The paper shows how it’s possible to ‘prove’ that listening to certain kinds of music makes you physically younger in years lived, by simply exercising researchers degrees of freedom and selective reporting. Sadly, Ioannidis himself has been in the headlines recently for some...

...hyperbolic and methodologically questionable papers published on the coronavirus epidemic. An important illustration of how everyone is fallible, including critics and those you might like to venerate as scientific heroes. Nobody is perfect.