Received reviews for our first @RegReport at @CollabraOA and I want to share my experience #openscience

Ready?

Thread 1/

Our RR contained a reanalysis of published data and a proposal for a (conceptual) replication. Our motivation was based on the belief that the data analysis of the original paper erroneously brushed over clusters of data (details aren't relevant). 2/

So we ran a reanalysis taking these clusters into account and the original effect either vanished or was at least much much smaller. We wanted to follow up with a high power replication. 3/

We received three thoughtful reviews and a long evaluation of the editor. All of them were generally positive, written in a collegial tone, constructive, and encouraging. The editor did a great job in weighing reviewer requests and raising additional points. 4/

Two (out of three) reviewers self-identified (!) The author of the original study also raised a super important point regarding our argument and thus our justification for the paper. @TraversEoin made a similar point when we shared our RR on Twitter 5/

We went back to the drawing board and ran some computational models to formally check our assumptions about the data generation process. And you know what?! Our assumptions didn’t hold up. The clusters in the original data can result from a unimodal data generation process. 6/

Sooo our study rational had a fatal flaw! And it was picked up during the @RegReport peer review process PRIOR to data collection. The result: We will not submit a revision. There is nothing to revise in a meaningful way. 7/

Consequently, we will not spend 100s of hours collecting data, we will not spend money on this research, and we will no pollute the literature with a narrative that is inherently flawed. 8/

Of course, sunk-cost-fallacy and all, it sucks a little that our work won't lead to a "visible" outcome. But we are also super grateful for this experience. I’ve advocated for @RegReports for a while now. This experience shows me again how well this model can work. 9/

Side note: We were really happy with the entire process at @CollabraOA. It’s an awesome journal, its #openaccess, feedback was quick and constructive. Submit your work there #openscience

/end