What happened to the replicability crisis? Should you believe famous social scientists?

Are all the pages in these best selling books you're reading any use?

Or have you just purchased expensive and uncomfortable toilet paper? Thread below

1/ In 2010, Amy Cuddy and her colleagues published a paper on Power Posing. In it, she demonstrates that “high-power, non-verbal displays” cause changes in body chemistry that increase testosterone and decrease cortisol.

2/ By assuming “two simple 1-min poses” A person can, according to Cuddy, “ instantly become more powerful”. Cuddy’s 2012 TED talk based on this work is now one of the most watched talks in history.

3/ Yet subsequent work that has attempted to replicate these findings have found no effect whatsoever. In 2014, Ranehill et al. find “no effect on hormones and risk tolerance in a large sample of men and women.”

4/ Who is right? Who is wrong? Is Cuddy a charlatan peddling unrigorous, p-hacked results? Are Ranehill and her colleagues jealous researchers determined to take down a rival? What does it mean that other academics have not been able to replicate these findings?

5/ Questions like these have captured the imagination of the media and the public. Behind the technical discussions of p-values, degrees of freedom and boundary conditions lies ancient anxieties.

6/ How do we know what we know? Can we trust the results that that Science hands down from above?

7/ Science set off the academic equivalent of a hand grenade in 2015 when it published the Open Science Collaboration Estimating the Reproducibility of Psychological Science.

8/ In it the authors, find that only one third of studies published in top psychology journals were replicable.

9/ The idea that attempts at replication bring us closer to the truth is as old as the scientific method. Every replication is at its core an attempt at falsification.

10/ Each failed replication falsifies a theory that is to be discarded. Over time this process leaves us only with the best theories that are best suited to understanding the world around us.

11/ By this logic, Cuddy’s work should be condemned to the dust heap of history alongside bloodletting and phrenology.

12/ This is a process that works well in the natural sciences. In fact, thinkers like Popper would go as far as to demarcate the difference between science and non-science by this criterion.

13/ Much of the media conversation follows this logic. But does this logic match the realities of research in psychology and OB?

14/ In reality, true replication is nigh impossible in psychology or OB. Unlike in chemistry or physics where the relevant factors of an experiment can be controlled with relative ease (e.g. temperature, air pressure, mass),

15/ human behavior is mediated by so many different factors that it even the best intentioned researcher can inadvertently introduce crucial differences in methodology that dramatically alter results.

16/ The media’s coverage of the replication crisis is predicated on a simplistic view of research. Studies are either right or wrong. Researchers are either saints or charlatans.

17/ Power posing either always works or never works. These Manichean dichotomies lead to sloppy thinking.

18/ Academics have the unenviable job of reconciling contradictory results by dissecting how replications differed from original studies. This is backbreaking, unsexy work unlikely to earn anybody tenure.

19/ A new approach of “adversarial collaboration” championed by @PTetlock and Daniel Kahneman may be academia’s salvation.

20/ Researchers who disagree should design and run a new experiment together, instead of just attacking each other in the literature using old data generated from different experimental conditions.

21/ Latham & Locke have already applied this “antagonistic joint design of experiments” to great effect in the goal setting literature. https://psycnet.apa.org/buy/1989-10539-001

22/ In the meantime us hoi polloi should be wary of theories of human behavior that claim to be simultaneously simple, general and accurate.

23/ The Greek poet Archilochus wrote nearly three thousand years ago: “The fox knows many things, but the hedgehog knows one big thing.” Those of us who bet on ideas for a living should embrace the uncertainty, contradiction and complexity in the social sciences.

24/Be the fox, not the hedgehog.

Thanks to @AdamMGrant and Jillian Li for their help with this research. Mistakes are my own