I’m very excited to share the latest Many Labs project! In Many Labs 5, we examined whether adding more expertise to replication designs could increase replicability. Here’s what we did/found (begin thread):

It seems (at least to me) like replicability rates in psychology have room for improvement. There are lots of great ideas on how to achieve that. One thought is that maybe replicators don’t have the necessary expertise to conduct a solid replication. Adding expertise could help.

We took 10 replications from the Reproducibility Project: Psychology (RP:P; OSC, 2015) that were labeled “not endorsed” and conducted two new sets of replications. The first set used the methods employed in RP:P.

The second set used methods that were revised through formal pre-data collection peer review. We revised the RP:P protocols based on any correspondence we could find from RP:P. Then we asked original authors/other experts for feedback on our designs.

So what did we find? In aggregate, the results from both sets of replications were very similar. Depending on how you analyze it, the Revised protocols produced effect sizes either r = .002 or .014 bigger than the RP:P protocols.

That’s in aggregate though. Looking at the individual studies, there are a few where the added review might have led to stronger effect sizes. Maybe that’s chance, but if I were to rerun those studies, I’d use the revised protocols (and big samples).

We also ran prediction markets. Unlike past markets, our forecasters had a really tough time predicting these studies. Maybe it was down to the small number of markets, but it’s also possible that it’s hard to guess which revisions will help.

So is expertise irrelevant for replicability? Of course not. There’s a reason I didn’t lead the eye tracking study in this sample – I don’t even know how to turn the equipment on. Researchers need some baseline knowledge to run a replication.

The question here is, if researchers seem to have that baseline knowledge, does adding more expertise improve replicability? This project didn’t find evidence for that. I’m sure it’s possible in some cases but we didn’t see much movement here.

There are tons of people to thank for this effort, too many to list here. I’ll break them into three groups. First, thank you to all the expert reviewers. They gave so much time, thought, and energy for this project; I’m really indebted to them.

Second, thank you to everyone at AMPPS, particularly @profsimons. The intervention here was adding peer review. That wouldn’t be possible without the journal and its editor. This was a huge lift for them and I’m very thankful.

Finally, thank you to the entire ML5 team. They made this project possible. I’m so lucky that I had the privilege of working with so many bright, talented, and generous researchers. They are the best part of these projects.

The project is reported across 11 articles in the new issue of AMPPS. You can see the preprint for the overarching report here: https://psyarxiv.com/sxfm2 . Methods, data, scripts, etc. can be found off of this landing page: http://osf.io/7a6rd/  (end thread)
.