Read on Twitter
OK. The culmination of a year-plus, um, argument-like thing is finally here, and it's clearly going to get discussed on Twitter, so I'll post a thread on the affair for posterity & future links about my stance on this entire thing.
A long time ago, in a galaxy far away, before any of us had heard of COVID19, some surgeons (and, it must be noted for accuracy, a PhD quantitative person...) wrote some papers about the concept of post-hoc power.
I was perturbed, as were others. This went back and forth over multiple papers they wrote in two different journals, drawing quite a bit of Twitter discussion *and* a number of formal replies to both journals.
The second of their papers, the one which really got me worked up, may be found here: https://www.journalofsurgicalresearch.com/article/S0022-4804(19)30185-4/fulltext
Note that this paper was published on April 27, 2019. I wrote several Twitter threads in the aftermath, and also submitted (at the EIC's invitation) a formal reply to the journal.
Here we are, and the response letter has finally been published - over a year later (almost sixteen months).
My response to their work may be found here: https://www.journalofsurgicalresearch.com/article/S0022-4804(20)30502-3/fulltext
Their response to my response may be found here: https://www.journalofsurgicalresearch.com/article/S0022-4804(20)30500-X/fulltext
Note: in this thread, I'll refrain from discussing the post hoc power issue further. I've said everything useful that I can say in the letter posted above.
It does sadden me a bit that my best efforts at explanation were not sufficient to convince them of the fallacy of their work. But I really don't see what else I can say that will go anywhere.
Rather, the most striking thing about this response is the incredible closing passage calling for professionalism in scholarly discourse and taking a couple of shots at the idea of using Twitter for such discussion.
Sure, conversations on social media happen differently than they happen inside the realm of journals. There are downsides to that, but there are also upsides to it.
One thing that clearly seems important to highlight here is the timing. The flawed initial work was published in April 2019. My critical letter was submitted within weeks of its appearance. It did not appear in the journal until August 2020.
I do understand that social media has its own troubles to grapple with as a medium of scientific discussion. Some discussions turn nasty. Sometimes a pile-on starts by people that don't have all the facts right.
So I'm not going to pretend that discussion on SoMe is perfect either, not by a long shot, but we have *got* to stop with this fallacy that the *only* way to discuss science is by submitting formal letters to the editor.
Even in *good* responses, like this, the editor may agree to publish your concerns after what is often a lengthy delay. In many cases you don't even get that response - often there is no opportunity for a formal journal-published response at all.
Honestly, at this point I'm kind of out of steam for this argument, but I find several comments in their screed about professionalism to be, at best, wildly disingenuous.
Suggestions that their critics were hidden behind pseudonyms (nearly all that I saw have named Twitter accounts, and many of us wrote letters to both journals *with our names on them*) or that it is in fact *the professional statisticians* that don't know what they're doing...
This passage is too incredible not to mention:
Sure, a bunch of professional statisticians said the work lacked credibility and incorrectly applied the main statistical principles, but we'll go ahead and present it as though *they* are the ones who lacked the skills to do the work or interpret the results correctly.
It doesn't seem that any minds are being changed here. Oh well. I'm sure they're just as annoyed (or probably more) with me and my fellow critics as we are with them at this point. So we will consign ourselves to another paper living in the literature promoting...
...a long-debunked and fallacious practice. Especially humorous to me that the authors conclude their response letter with a call for "collaboration and brainstorming" because...
I mean, I think we had a pretty good brainstorm here on @PubPeer:
https://pubpeer.com/publications/4399282A80691D9421B497E8316CF6
https://pubpeer.com/publications/4399282A80691D9421B497E8316CF6
So if the authors are really serious about "collaboration and brainstorming" solutions, it seems curious that they showed minimal interest in understanding why, ah, the entire field of statistics (estimating here...) finds their proposal lacking merit.
Anyways, I think I'm done discussing the substance of the proposal and why it fails. I think I've said everything useful I can say on that score in the published paper linked above. I wish I was better at explaining this, or that it was simpler to explain.
Also, if anyone knows who "Mr Griffith" is or finds the letter by "Mr Griffith" that they are responding to, I would love to see it.
I have been sent a copy of the letter by Mr Griffith! Hopefully it, too, will soon be available on the journal website.
