Read on Twitter
If you want to critique a scientific paper well
Step 1: summarize the paper fairly & clearly
Step 2: articulate the portions you find problematic
Step 3: articulate how those portions would bias the result (and in which direction)
[thread]
Step 1: summarize the paper fairly & clearly
Step 2: articulate the portions you find problematic
Step 3: articulate how those portions would bias the result (and in which direction)
[thread]
Step 1: Summarize the paper as if you were an author.
Please don't caricaturize it.
Please don't incorrectly summarize it.
It is really clear to onlookers when you are not fair to the original argument
Be fair!
Please don't caricaturize it.
Please don't incorrectly summarize it.
It is really clear to onlookers when you are not fair to the original argument
Be fair!
Step 2: What methodology is problematic; What specifically about the results, methods or discussion do you disagree with?
Be very precise!
Be very precise!
Step 3: Why is it problematic? In which direction will it bias the paper.
"The sample has a selection bias"- that's not good enough
"The bias is towards enrolling healthier pts on average, which would distort the result upward" -- better, better
But how and why...
"The sample has a selection bias"- that's not good enough
"The bias is towards enrolling healthier pts on average, which would distort the result upward" -- better, better
But how and why...
Now the extra steps
Step 4: Do not be angry
You have to purge yourself of emotion if you want to do it well. (we all err, occasionally)
Step 4: Do not be angry
You have to purge yourself of emotion if you want to do it well. (we all err, occasionally)
Step 5: It is ok to raise documented or undocumented (but verifiable conflicts of interest), but don't lead with this. Prove the paper is wrong first, and then raise this issue.
Step 6: If the conflict is weak or unverifiable, I would not raise it at all.
If it is an intellectual conflict... ha, good luck separating beliefs that we hold rightly from ones we hold wrongly.
If it is an intellectual conflict... ha, good luck separating beliefs that we hold rightly from ones we hold wrongly.
Step 7. Stay focused, don't be distracted. Someone will try to steer the argument towards why and how you are criticizing. Stay laser focused on the issues
Now, if I might make an observation.
I find these principles are often violated
Folks often present distortions of the original argument
I find these principles are often violated
Folks often present distortions of the original argument
They fill the air with smoke, but don't point to the fire.
They use buzz words that people use with problematic paper, but don't specifically articulate what is wrong
They use buzz words that people use with problematic paper, but don't specifically articulate what is wrong
They don't point to the direction it is wrong.
In fact, half the time, the paper is wrong the opposite direction they think.
Here, they are arguing the estimate is too high, but the error they find would lead to underestimation.
In fact, half the time, the paper is wrong the opposite direction they think.
Here, they are arguing the estimate is too high, but the error they find would lead to underestimation.
Too much emotion/ distraction
Start with the conflicts, but don't say if anything is wrong
Start with the conflicts, but don't say if anything is wrong
