Read on Twitter
= Replication publications in reputable psychology journals =
Let's talk about replication in psychology. We think we've been doing better in the last decade, but have we improved?
Thread below.
Let's talk about replication in psychology. We think we've been doing better in the last decade, but have we improved?
Thread below.
A recent paper by @BrianNosek etal. discussed this progress, details multisite collaborations, & outlines challenges.
Preprint tweet: https://twitter.com/BrianNosek/status/1359118772972507143?s=20
Preprint tweet: https://twitter.com/BrianNosek/status/1359118772972507143?s=20
Yes, progress has been made, these mass collaborations are amazing achievements for our field, yet replications have not gone mainstream, far from it.
Unfortunately, multi-site mass replication articles set expectations so high, it created barriers for single replication papers.
Unfortunately, multi-site mass replication articles set expectations so high, it created barriers for single replication papers.
Consider this mind-blowing tweet from @RogertheGS, editor of JESP, from last year 2019: "JESP received roughly 670 manuscripts in 2019. Of these, 12 had "replicat*" in the title". JESP, Elsevier ethical concern aside, is flagship open-science outlet. https://twitter.com/RogertheGS/status/1222847696140218369?s=20
How did JESP do in 2020? Not much better. Listen to @RogertheGS discuss this in SPSP a few hours ago:
"In JESP we rarely get more than 10 replications a year, out of 700".
"In JESP we rarely get more than 10 replications a year, out of 700".
Let that sink in for a second...
10 replications... out of 700... (!)
that's... less than 2%, and we're not even talking about accepted for publication, just submissions.
that's just horrible. we must do better, much better.
10 replications... out of 700... (!)
that's... less than 2%, and we're not even talking about accepted for publication, just submissions.
that's just horrible. we must do better, much better.
Our team submitted 7 replication manuscripts to JESP in 2020.
Yet, we keep facing difficulties, not related to rigor but to issues of "contribution".
We then try & move those to Meta Psychology & Collabora:Psychology asking for streamlined process without additional review.
Yet, we keep facing difficulties, not related to rigor but to issues of "contribution".
We then try & move those to Meta Psychology & Collabora:Psychology asking for streamlined process without additional review.
Like @RogertheGS said earlier in his video: "we're going to need to think how to reward these activities because it's not part of the normal..."
Are we rewarding replication work? Are we evaluating replication work correctly?
Are we rewarding replication work? Are we evaluating replication work correctly?
We appreciate JESP, @RogertheGS, & editorial team, yet even with that team we keep getting "contribution" rejections for replications.
Some explained this by pressures of journal reputation & expected acceptance rates.
I'm no editor, yet we need transparency & consistency.
Some explained this by pressures of journal reputation & expected acceptance rates.
I'm no editor, yet we need transparency & consistency.
Our success rate with JESP is less than 50%.
No, rejections were not related to rigor or methods, but based solely on "contribution" and "value".
So, if so few replications get submitted, and even those face inherent biases, what does this mean for our field?
No, rejections were not related to rigor or methods, but based solely on "contribution" and "value".
So, if so few replications get submitted, and even those face inherent biases, what does this mean for our field?
If the expectations we have for replications are set higher than for other studies looking for mass collaborations....
If we demand replications to adhere to old-school "contribution" & "value"...
If even flagship open-science supportive JESP rejects 50%+...
Are we changing?
If we demand replications to adhere to old-school "contribution" & "value"...
If even flagship open-science supportive JESP rejects 50%+...
Are we changing?
To balance all of that I do want to say that our team has been able to publish better than I ever expected, but this mainly thanks to specific open-science supportive editors & reviewers, sometimes even unexpected journals.
Change happening locally, yet we need it systematic.
Change happening locally, yet we need it systematic.
I am hopeful mainly due to:
Registered Reports
Revolutionary open-science journals like @Meta_Psy
Open-science reviewers who push editors to emphasize rigor & transparency
Open-science editors who push against old-school system & promote change.
Registered Reports
Revolutionary open-science journals like @Meta_Psy
Open-science reviewers who push editors to emphasize rigor & transparency
Open-science editors who push against old-school system & promote change.
I can already imagine cynicism/snark/pushback. I get those sometimes when I tweet about our "contribution" replication rejections.
As ECR, I also worry about engaging editors, not just for me, but for our team.
Yet if we don't raise those, we won't improve.
We'll keep trying.
As ECR, I also worry about engaging editors, not just for me, but for our team.
Yet if we don't raise those, we won't improve.
We'll keep trying.
