Read on Twitter
I’ve always been baffled by reports that large proportions of reviews contain offensive or demeaning language. I know it happens and it is never ok. But I’ve seen 1000s of reviews and very few that clearly crossed the line. So I looked into this claim. [Thread] https://twitter.com/dsquintana/status/1320613686566883328
First, here is the Nature column Dan was quoting from (very sorry to pick on you, Dan!):
https://www.nature.com/articles/d41586-020-02512-0
The author of this column doesn’t provide a link to the study mentioned in the quoted passage, so I looked it up. /2
https://www.nature.com/articles/d41586-020-02512-0
The author of this column doesn’t provide a link to the study mentioned in the quoted passage, so I looked it up. /2
I think it’s this article (which the author of the Nature column is also a co-author on):
…https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-020-00096-x
3/
…https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-020-00096-x
3/
So let’s see if 10-35% of peer reviews contain “demeaning language”.
First, this from the abstract (pic).
12% contain an unprofessional comment, and 41% contain incomplete, inaccurate, or unsubstantiated critique.
Already I’d say 12% is the better figure for “demeaning”
/4
First, this from the abstract (pic).
12% contain an unprofessional comment, and 41% contain incomplete, inaccurate, or unsubstantiated critique.
Already I’d say 12% is the better figure for “demeaning”
/4
Then, how did they operationalize “unprofessional comment” (the category with 12% hit rate)?
Two things:
1. Any reference to authors’ sex, gender, age, race, place of origin, or native language, or comments that could be interpreted as demeaning.
Sounds reasonable.
5/
Two things:
1. Any reference to authors’ sex, gender, age, race, place of origin, or native language, or comments that could be interpreted as demeaning.
Sounds reasonable.
5/
But the 2nd thing that counted as an “unprofessional comment” was a reviewer talking about questionable research practices (pic). What! That’s not inherently “unprofessional” nor is it “demeaning”. (Of course it can be done in a demeaning way, but so can almost any comment).
6/
6/
And that’s not getting into the other category (incomplete, inaccurate, or unsubstantiated critiques), where the 43% comes from, which has some less-than-ideal operationalizarions as well (eg, “we counted instances where reviewers viewed common methods as critical flaws”).
7/
7/
Finally, some of the reviews were coded from case studies of reviews submitted by people contacted through “through informal collegial discussion” about this research. Six out of 327 people contacted provided reviews. Could they be ppl who got especially nasty reviews?
8/
8/
I can’t quite tell from Table 1, but it looks like a LOT of the reviews that were flagged for unprofessional comments were from the case studies, not the (I think probably more representative?) sample of reviews from Publons.
9/
9/
If I’m doing the math right, in the Publons dataset, 23 out of 1,491, or 1.5%, of reviews contained “unprofessional comments” and 17 out of 1,491 contain demeaning language, or 1.1%
A much better summary of this study, imo, is “1.1% of reviews contain demeaning language.”
/10
A much better summary of this study, imo, is “1.1% of reviews contain demeaning language.”
/10
To be clear, that is 1.1% too many. There’s no excuse for demeaning language. But it seems strange to me to go from these pretty clear data to a claim like the one in the Nature piece that paints a very different picture.
/end (with apologies for the unprofessional content)
/end (with apologies for the unprofessional content)
Addendum: Offensive, demeaning behavior happens all the time in academia. We need to find it & root it out. I don’t think peer review is where most of it happens. A lot of it happens in labs (often by advisors), at conferences, under the guise of “mentoring”, etc.
Second addendum: this is an important point about Publons. So neither subsample is likely to be representative. Seems better not to talk about absolute levels at all then. https://twitter.com/ingo_keck/status/1320652771130036225
Another correction: I don’t think I’ve seen thousands of reviews. That was flase. My best guess is I’ve seen something like 750-1,000.
