Do you think peer reviewers should be paid for their work? If yes, welcome to The Movement (yes, another one) @450Movement

This seems set up as giving publishers the middle finger, and I feel it lacks nuance. And so, here is my completely #UnsolicitedOpinion on the matter 1/

First, I want to say I unequivocally support referees being remunerated for the work they do. I said it before & will keep saying it. There is no question in my mind that current system work as slavery propped by the notion of greater good 2/ https://twitter.com/Edit0r_At_Large/status/1083899535674535937

However, this is presented as a false us vs them dichotomy and is very much a stick it to the man rebellion without considering what sticking it actually means and who is the man 3/

There's often a notion that refs do a 'job' which is why they should be paid. A corollary is that if we paid reviewers, the quality of reviews would be so much better. So let me repeat again: payment doesn't guarantee quality (take it from someone who's seen paid reviews). 4/

So what you need to bear in mind is that if we decide to treat reviewers as contractors, the expectation of review will also change: no more two line summaries followed by a list of typos. There should be terms of service; reviewers should be expected to indeed do a job 5/

I saw @jamesheathers talk earlier this year about why review often does not catch even the most glaring problems. He said this thing that I really like, because it goes to the core of the problem: it happens, because referees do not *interrogate* the work they review. 6/

So if we treat reviewers as contractors, you better believe that interrogation goes into my ToS:

I do want you to look over Western blots carefully & question why one loading control is used for 20 different proteins. 7/

I do want you to query that physics paper, where between Eq. 4 and 5 the authors say - there is a proof for this transition, but we had no space to explain it. I want you to tell them: unless you're a freaking Fermat, show me the proof 8/

I do want you to ask why sequencing data, the deposition of which is now the field standard, is not deposited. I want you to do the same for crystallography, metabolomics, chemical structures, new species registration, fossil specimens, behavioural data 9/

I want you to ask for the code, and if feasible: check it can be installed, run, tested. Is authors' responsibility to make this work and provide test datasets, but it should be reviewer's role to then check this 10/

I want you to look at the data and the stats. You may not be a statistics expert, but as RIVETS, GRIM & SPRITE have proven (chapaeu bas @sTeamTraen & @jamesheathers), it is often not needed to figure out the paper is trash 11/

I want you to at least spot check the cited literature. I want you not to assume the paper is from good lab & so the authors know what they are doing, or that they are experts so they know the field (and are unbiased about it) 12/

I want you not to say: I read the first page but it was so bad it made my eyes bleed, so I won't waste more of my time (true story). Or: English is so bad I cannot help you (it really, really most of the time is not) 12/

And then I want you to write up all of this in the most constructive, unoffensive way possible, without suggesting that authors are incompetent or stupid. Would help if you tried not to be misogynist or racist about it too #OverlyExasperatedEditor 13/

And I know y'all lovely social media pundits will probably say: but don't all reviewers do this? Well, first ask yourself a question do *you* always do it (you don't have to tell us, just be honest with yourself). If you do, well done, you get a gold star, but... 14/

...even if every keen science advocate on TT does, think about it this way: there is several million papers published a year at the moment. These need on average two reviews. Of the, say, 5 million reviews performed yearly, how many do you think are that thorough? 15/

So yes, the referees should be paid. But just remember that you will be paid for the work you do and what you do now may not be that work #OverlyHonestEditor

This is not to say it would be a bad thing. I would kill to get reviews like this for every paper I handle 16/

But there is more. There is still the us vs them problem. The set up is: commercial publishers skin us alive, they can afford to pay. This may be true & I know some people say they'd still review for free, as community service, for non-commercial (e.g. society) publications 17/

What it doesn't account for: society publishers worse than many commercial (ACS anyone?); community journals published by commercial publishers (who merely provide a platform); commercial publishers who give APC waivers right & left to help LMIC authors 18/

The final issue is: who will saok in the cost. I grant you, it should not be reviewers answering this, just as we shouldn't ask any employee how their employer plans to pay them. But is nonetheless the question that needs asking 19/

Because even if you want to run a diamond journal on a shoestring, you'd still need to pay for 3-4 reviews (across multiple rounds) per paper. That is $1350-1800 just for reviews, not counting admin cost. This $$ has to come from someone. Who, do you think, it is going to be? 20/

Which brings us to the major problem: we think about this as publishers exploiting reviewers. But *nobody* is willing to pay. Universities don't make it a part of your job or evaluate you for it. Funders don't say: we'll give you $2K extra for APC so that refs are fairly paid 21/

A minor aside: many reviews are performed by postdocs for their sups (formally invited to review). I want to trust they would get their pay for this, but do you want to be that postdoc who get screwed even if publishers cash out for reports? 22/

The bottom line, to me, is this:

refs are exploited in the most f*cked up way. Not by publishers; by the collective scientific enterprise. And as this collective, we need to figure out how to change this.

Or be rid of peer review altogether, & just watch the world burn. 23/23

I probably missed a lot here, and also simplified some stuff. I realise that part of the thread may seem like dunking on reviewers. This is not intended like that (but I do see equally many bad reviews as I see bad papers)...

... and I really cannot emphasize enough how much I agree with the notion that people should be paid for the work they do (also: unpaid internships, blurgh).

I just don't think this is as one-dimensional as many people seem to think.