Read on Twitter
I’ve been thinking a lot about a role of editors lately (for absolutely no apparent reason, wink wink). But no matter how I look at this, I come back to the same conclusions, which will definitely deserve a place in #UnpopularOpinionSeries. This may be long, so bear with me. 1/
There is a lot of information out there which explains what the editors should or should not do and how they should or should not behave. Some of the most comprehensive are of course @C0PE guidelines: 2/ https://publicationethics.org/files/COPE_G_A4_SG_Ethical_Editing_May19_SCREEN_AW-website.pdf
And @WileyNews ( @latwec) also provides a very detailed outline. No matter where you look though, most of this is focused on mechanics: you need to be fair & honest in assessment, you need to be prompt in seeking expert opinions… 3/ https://authorservices.wiley.com/ethics-guidelines/index.html
…you need to have systems & mechanisms to deal with post-publication criticisms, you need to ensure that research you publish is done ethically & that the publication process itself is robust. 4/
Many of these guidelines and practice descriptions – as well as other imaginings of what editorship is elsewhere in the interwebs – talk about editing as #gatekeeping. And it is how we often see editors, as gatekeepers… 5/
…and we bemoan this responsibility – or power (depending on your point of view) – that they have/yield. Rarely though do we ask or try to define what this actually means. We speak of this as ability to prevent cr*p from being published… 6/
…(spoilers – it does not); as ability to destroy people’s careers (that may be *sometimes* true, but the picture is much more complex than not getting that paper in CSN); as ability to stymie research (remember the old adage about science progressing one funeral at a time);… 7/
…as ability to influence how research is done & where $$$ goes (may be true for a small # of journals, but also a bit of a vicious circle); as ability to shape who in science have a say, to either increase or prevent diversity (either is likely true for specific scenarios). 8/
And yes, it is a lot of power – and responsibility – to shoulder and whatever your personal experience with individual editors, on a whole we do take this seriously and do our best to treat everyone fairly within the constraints of the framework we work with. 9/
For the most part tho, we seem to perceive gatekeeping through selectivity lens: editors select what is interesting enough for their journal. It is a minor part of what we do, but one we get most grief for. Some go as far as to call this malpractice: 10/ https://twitter.com/chrisdc77/status/1136884112789258240
(NB: I wouldn’t call this malpractice, because is not the role of publishing industry to ensure no publication bias. It doesn’t mean this isn’t bad, and I am all for journals not selecting for interest. Mind @chrisdc77, as he will be back in this thread.) 11/
On the other hand, ppl expect editors to reject papers that are not super conclusive, that are descriptive, that report only associations etc. We tend to expect ‘higher profile’ journals to not publish what we considered to be weak, narrow in scope, not generalisable etc. 12/
That puts editors in an impossible position: we want them to gatekeep, but also we don’t want them to gatekeep. We also want to have a say in what journal considers of appropriate interest. It’s where reviewers proclaiming “I rejected this paper but it was accepted” come from.13/
Is where comments like “how did this get published in *this* journal” originate. Is why people bemoan that journal X has fallen so much. We make those comments without recognising what they mean: that we don’t have concerns about publication itself, we have concerns… 14/
…that the work found place in a venue for which we don’t deem it worthy. Here is a question then: who decides what is worthy to be published, who decides where, and who decides why? 15/
Or in other words: if the study is sound within the defined framework that is clearly outlined, do we have the right to call for #retraction because we disagree with the authors’ interpretation of the data, or because we don’t like what it showed? 16/
Even more, do we have the right to tell editor it is their ethical or moral duty to retract such paper? And can the editor choose to retract based on crowdsourced opinion on whether the messaging of the paper is insulting and to whom? 17/
And the answer is, strictly speaking – NO #OverlyHonestEditor (Though occasionally these things happen, see @ivanoransky & @armarcus:) 18/ https://www.wired.com/story/science-journals-are-purging-racist-sexist-work-finally/
By now you must know this thread was inspired by kerfuffle around the @NatureComms mentorship paper 19/ https://www.nature.com/articles/s41467-020-19723-8
…and @pollyp1’s Open Letter to the journal’s EiC. In what follows – and really, please bear with me – I will explain why I don’t think this paper should be retracted… 20/ https://twitter.com/pollyp1/status/1329455882481131524
…how to call for retractions in general (and how not to do it too), and what the journal and authors could’ve done to prevent this mess we’re in (but remember that hindsight is always 20/20). 21/
First, the paper. I have now read it. While I don’t pretend to understand all of the methods, I thought that when taken at face value it is fine. Following intervention of the referees it manages to mostly avoid causal language 22/ https://static-content.springer.com/esm/art%3A10.1038%2Fs41467-020-19723-8/MediaObjects/41467_2020_19723_MOESM2_ESM.pdf
In a rather speculative discussion it retained an unfortunate statement that upset a lot of people. I don’t disagree with the general sentiment here, but must point out that this is not the first, nor the last paper to speculate in the discussion. 23/ https://twitter.com/millerjm86/status/1328765654338441216
I do recommend you read the whole thread, as it is a good intro to the paper. I am also aligned with @millerjm86 & others who commented on this, that this whole paragraph was not necessary and is going too far… 24/
…in particular that it makes recommendations to policy makers which don’t seem justified by data in the paper. However, this isn’t because the paper is wrong. It is because it has a very narrowly defined scope. Which is where a lot of disagreement & critique stems from. 25/
For example: some note paper is not about mentorship but rather co-authorship. This is up for debate, as there doesn’t seem to be one strict definition of mentorship, & authors define exactly what they mean by this. But this, in general, is a red herring in as much that… 26/
…if the paper was about co-authorship, it would still be showing association between co-authoring with male/female colleagues and impact on future publications. Which is another critique here, and that is: what is impact. 27/
Authors define it through citations, and this is a massive limitation of the paper. We all know that citation metrics are poor proxies for a lot of things – esp. something as ephemeral as impact. And yet everyone uses them that way. 28/
Readers are also upset about just the fact that authors even dared to look at F/M differences. This seems to imply they set out to somehow undermine #WomenInSTEM. It’s a weird argument, & I always caution against ascribing motives to other people (Hanlon’s razor & all that). 29/
But we should also ask: should they not have done this? Surely, if we are to be able to increase #DiversityAndInclusion, these are the type of analyses that we must be doing. Should they not have reported the results then? Of course not. 30/
Social media pundits say: I would’ve rejected based on these reviews. I have read the reviews. If I had been the handling editor of this paper, I would’ve not rejected it: the reviewers were critical, but offered a clear way to address many of the shortcomings of the paper. 31/
I see nothing in the reviews that would be a fatal flaw – which is perhaps where editors often go against the crowd (researchers tend to consider everything they think is wrong as a major flaw, even if it’s completely inconsequential to the conclusions #OverlyHonestEditor) 32/
Moreover, I disagree with critics proclaiming that this was published against reviewer recommendation and without addressing criticisms. It clearly wasn’t and they clearly were. We would all do well to remember that the fact that the authors did not address… 33/
…some points the way we may wanted them to, does not mean they weren’t addressed. You can throw a stone if you never ‘respectfully disagreed’ with a reviewer, or chose to slightly change the wordage of your paper instead of doing extra work. 34/
Frankly, is not clear to me where these comments come from, but I think we are witnessing one problem with how open review is implemented in many journals – that we don’t see what the previous versions of the paper were. 35/
So, short of someone actually outlining methodological issues that are not covered by the paper’s framework, what we’re left with in the call for retraction is our displeasure that authors dared to express an opinion that many found hurtful. 36/
And before you say “but this will affect policymakers”, can we please be fair to policymakers and trust they don’t make policy based on a single paper? If we don’t, we’re as offensive to people who work in policy as you think this paper is offensive to you. 37/
(NB Please don’t come with Reinhart-Rogoff paper at me. I know some politicians will use single papers to support their policies, but opportunists will brute force policies regardless of whether there is 0, 1, or 100 papers supporting them #OverlyCynicalEditor) 38/
Back to call for retraction. @C0PE guidelines for when the papers should be retracted are quite clear and robust. The one reason applicable here is: 39/ https://publicationethics.org/files/cope-retraction-guidelines-v2.pdf
Note that hurt feelings aren’t one of the reasons, nor is potential for the paper to cause harm if misused. There is a reason for this, which brings me back to what the role of editors is and isn’t, and what editors can and cannot do. 40/
It is as simple as that: editors are not thought or tone police. We may disagree with some opinions, and in cases when they are overtly racist, sexist, misogynist, xenophobic, chauvinist, etc we can decline to publish. Most of the time, we must allow for opinions to be aired. 41/
Now, readers may disagree with this, and this is where post-publication processes come in. And while in some cases these may involve retractions, calling for one should be something that is done rarely and not without ample evidence it is justified. 42/
Bear in mind another important point from the @C0PE guidelines: retractions are not a punitive measure, and are primarily a tool to correct scientific record in cases where this cannot be achieved with a correction. So ask yourself: would a correction suffice here? 43/
And when you put all emotions aside, and think about the paper without guessing how it may or may not be interpreted or used, do you still think this cannot be corrected? 44/
And so, I stand by what I said earlier: unless a new major methodological issue emerges, I will be surprised if this is retracted as this would be done purely to limit reputational damage and satisfy Twitter outrage. 45/
Now, I take umbrage with how the call of retraction happened here. I am rather surprised that it was directed at a journal too, as this bypasses what would be the normal process: talking to the authors and letting them decide whether they want to retract the paper themselves. 46/
There is a reason we tend to go down this route. Is because we are all human. We all make errors, and we all make statements that may appear tone-deaf: this is often a result of a tunnel vision and why it is a good idea to ask a friend to read your paper before you submit… 47/
…because we are often married to our work, to words that we spend so many hours crafting to send specific message that we failed to recognise it sends other messages too. /48
There is power dynamics that comes into play when we call for retraction. Writing a letter to EiC of the journal is like going to CPS to tell them your brother calls you names. Writing such letter without talking to junior authors, when you are a person of authority… /49
…well, there is something deeply and sadly ironic about that too. So here is my short list of what not to do and what to do when calling for retractions:
Don’t go with nuclear option first. It leaves no space for discussion. /50
Don’t go with nuclear option first. It leaves no space for discussion. /50
Don’t appeal to ethical obligations of editors. We know what they are & they’re not just to you.
Don’t gang up. Retractions don’t happen just bc there is a mob. They happen, bc there is a valid argument presented.
Don’t imply motivations. You’re not in the authors’ head. /51
Don’t gang up. Retractions don’t happen just bc there is a mob. They happen, bc there is a valid argument presented.
Don’t imply motivations. You’re not in the authors’ head. /51
Do talk to the authors. That you disagree with them does not mean you cannot be civil. Be mindful of power differential.
Do explain why. No, “people on Twitter raised issues” is not a valid argument if there is no details. /52
Do explain why. No, “people on Twitter raised issues” is not a valid argument if there is no details. /52
Do stick to facts and merits. Comment on robustness of methods and accuracy of statements in the context of the paper (i.e. don’t impose your definitions to prove authors are wrong).
By gods, do explain which part of a paper you think is so wrong, it justifies retraction. /53
By gods, do explain which part of a paper you think is so wrong, it justifies retraction. /53
Finally, consider where – emotionally – the call for retraction is coming from. Remember retractions are not meant to be punitive. Remember papers can be corrected and allowed to stand. /54
Consider if a letter to editor – and actual letter/comment, not an outraged demand – is a better way of putting study in context. I don’t often advocate for letters, I hate them in fact. I think they are mostly misused by editors who want to avoid retractions… /55
But this, this seems to be a perfect example of a scenario when they should be used. So use them. #OverlyExasperatedEditor /56
So could this have been handled differently from the start? Probably: I see an obvious option for both editors and authors. These may not have been possible for many reasons, but are tools that you – as an editor and an author – should consider, if you face similar situation. /57
For editors (putting aside that terrible, terrible para in the discussion): this would have really benefited from having an editorial or commentary published at the same time (not clear to me that the journal does this, is more common in smaller outlets). /58
Would this change the paper? Unlikely. Could this have made the argument about how this research should or should not be interpreted and used and address the issue of potential misuse in policymaking? Most definitely. /59
For authors: this is a prime example of a study that could’ve been both amazing and much less exposed to criticism, had it been conducted in the format of @RegReports. It's a pity in fact, since @NatureComms introduced the format in the summer (but paper is older than this). /60
(So here is a space, 50 tweets later, for @chrisdc77 to come back and preach to whomever lasted long enough to get to this point.) /61
So that’s it. Thank you for reading. A few final caveats:
I’m won’t say don’t @ me. By all means do. Can’t promise I’ll engage, and on some things we may need to agree to disagree.
I don’t endorse the paper, I just read it for what it is. I recognise it was hurtful to many…/62
I’m won’t say don’t @ me. By all means do. Can’t promise I’ll engage, and on some things we may need to agree to disagree.
I don’t endorse the paper, I just read it for what it is. I recognise it was hurtful to many…/62
…and for the little that it’s worth, the text *elsewhere* in the paper suggests that authors are well aware of the uphill battle for #WomenInSTEM. So let’s judge them for all of it, not for the one opinion we didn’t like. /63
We’d all do well if we took the paper not as the final, but as the starting point for the discussions on how inclusion in science should look like. This paper may irk you, but it showed one way to study diversity – and many ways to improve how such studies can be done… /64
I truly hope to see many follow up papers that do better job by building on the authors’ work and not by tearing it apart; I wish they give us hope and clear directions how to build and be better as a community. /65
This thread is borne out of many social media frustrations, so while the mentorship paper is its focal point, what I say about: being editor, how we perceive editors’ duties, & retractions, can likely be generalised. But don’t take my word for it. Go and study it. 66/66
