Read on Twitter
Inspired by @IrisVanRooij, I want to express some concerns that may be controversial and even outrageous to some but I feel we at least should have a discussion. I'm wondering if statistics in psycholinguistics could use a rethink. It feels like the tail now wags the dog.
We spend an extraordinary amount of time setting up models, simplifying models, and hoping for each one to converge. And waiting for days for a model to finish running. Like many labs, we now use dedicated machines for running statistical analyses.
Our papers are held up for weeks due to that one model that won’t converge as we try, simplify, run, wait, repeat, over and over. Many are experiments with a 2x2 within-S design that you used to be able to analyze with an HP calculator. Students' work is delayed getting out.
We spend more time in lab meetings and on Slack talking about what statistical approach might work and what X said in an article but Y challenges in a preprint than on substantive theoretical issues related to language processing. Stats is sucking up all the intellectual oxygen.
We’re now running 2x2 experiments with 100s of subjects because our complex models require that large an N to test for possible effects we’re not the slightest bit interested in. These Ss have to be paid for or recruited as "volunteers".
And when we finally get to the manuscript review stage, at least two reviewers are guaranteed to assert that the analyses that were run are absolutely wrong and that A,B, and C must be done instead. Of course, those two reviewers will completely contradict each other...
...and yet both will be equally convinced they’re right. So it’s back to running analyses, now to handle the revision stage, with fingers crossed that one of the reviewers will be happy and the other will back down.
I’m not saying we haven’t improved many of our stats practices over the last 15-20 years. I don't think I'm hankering for the good old days. What I am saying is that this part of our job has become a major focus rather than being merely a tool, and that seems problematic.
Finally, I expect some of the things I've said here are wrong or based on faulty assumptions. If so, I'm happy to get educated, all in the spirit of having an open, scientific discussion!
