Read on Twitter
On the "stats in psycholinguistics debate" (aka "back to ANOVA?"), I get the frustration. It's totally annoying to have to work to understand the vagaries of Stan rescaling and convergence when all you want to do is your 2x2. On the other hand...
Probably the biggest methodological story in cognitive modeling in the past ten years is the move from optimal cognitive models to descriptive, data analytic modeling. Here's a (dated) blogpost I wrote on work by Tauber, @djnavarro, @AndyPerfors, Steyvers: http://babieslearninglanguage.blogspot.com/2015/09/descriptive-vs-optimal-bayesian-modeling.html
This paper describes a strategy that has become critical in my lab's work (and that of many many others, e.g., @todd_gureckis, who schooled me a bit early on): pose a cognitive model with linking functions that generate the observed experimental data!
Ex: @EricaJYoon and @mhtessler (who BTW really taught my lab these tools) used RSA pragmatics models as the "glue" between the data from disparate behavioral experiments, with Bayes Factors to compare between related but theoretically different variants: https://www.mitpressjournals.org/doi/full/10.1162/opmi_a_00035?mobileUi=0
Here's another, even more ambitious one, doing this with children's data (by @elManuBohn and @mhtessler): https://psyarxiv.com/2wgfb/ - again, the data analysis is done with a Bayesian cognitive model, but one that has regression modeling as part of the package!
OK, so how does this relate to the "stats" question? Back in ANOVA days there was not an elegant way to make tight connections between models and data. The ANOVA software doesn't even report coefficients by default, sometimes, so all you see is the p-value, not the estimates.
Moving to MLM (lmer) allows you to start inserting model-based predictors - trial level, stimulus level, participant level - in a principled way. So you start to predict variance but you still don't link "regression model" uncertainty to "cognitive model uncertainty."
This is critical - you can't know *how much you don't know* about which model is right: if you regress on model predictions, you assume that the model predictions are *correct.* In contrast, if the regression is one part of a broader data analysis model, uncertainty is linked.
This allows a quantitative comparison between hypotheses, meaning you can actually compare the evidence that the data gives with respect to two competing models. That to me is a huge advance!
So when we say we want to stick with F1xF2, that's fine if we only want factorial designs and no quantitative theory. But if we want to do experiments that are used as the foundation for quantitative theories, then we need more.
And as painful as that is in the short term, it means learning how Bayesian regression models are fit and interpreted. In principle I was trained in this stuff, though I'm not a statistician and I still think it's a pain... but it's worth it!
Bonus! Your data analysis is better in a bunch of the ways that @tmalsburg mentioned: https://twitter.com/tmalsburg/status/1341330781336039426?s=20
Bonus part 2: Here's my take on how Bayesian LMMs help you get the random effects you want: http://babieslearninglanguage.blogspot.com/2018/02/mixed-effects-models-is-it-time-to-go.html and http://babieslearninglanguage.blogspot.com/2019/05/its-random-effects-stupid.html
Just for transparency, I'm tagging @victorf13 since I appreciated his posts (and hist honesty about frustration) and am responding to them.
