Read on Twitter
A Monte Carlo simulation paper on mixed-effects models by Park, Cardwell, and Yu (PCY) has just appeared, which offers suggestions for random-effects specification for complex designs. It seems to argue for model selection of random effects. My thoughts... https://meth.psychopen.eu/index.php/meth/article/view/2809
In Barr, Levy, Scheepers, & Tily, 2013 (henceforth BLST), we only considered a design with a single binary predictor. More complex designs require complicated random effects structures that make convergence less likely, and so can be challenging to fit. https://www.sciencedirect.com/science/article/pii/S0749596X12001180
In another 2013 paper, I looked at Type I error & power for different random effects structures in 3-factor designs. When testing a predictor what is most important is including its 'critical slope'; other slopes may be sacrificed as needed for convergence https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00328/full
Park et al. considers designs having up to 3 binary predictors. They do not look at Type I error or power but are mainly interested in convergence and in using model selection to find the 'true' underlying model
A primary issue in interpreting Monte Carlo simulation data is how well the data generating process (DGPs) reflects real-world processes. Sometimes unrealistic DGPs are used to illuminate properties of a modeling approach, but they shouldn't form the basis for any recommendations
The DGP in BLST assumed that individual differences around an non-null effect may be small but are never exactly zero. If you have a priming effect of 30 ms, one subject may have 27 ms, another 33 ms, and a third 30 ms. You would never have everyone showing *exactly* 30 ms
Real-world psychological processes never give rise a population of individuals varying in their overall response tendencies but with null variance around any non-zero effect of an intervention
In their simulations, Park et al. consider three different DGPs: a random intercepts only DGP, a zero-correlation DGP, and a 'maximal' DGP. They find that AIC/BIC are good at finding the 'true' DGP *except in the case of the maximal DGP* where a simpler model is usually selected
See the problem? Random-intercepts only and zero correlation models do not exist as real-world DGPs. If the underlying generative process is typically variance around a non-zero effect, AIC/BIC almost never select that model. I read this as: model selection = bad
BLST showed that model selection can inflate Type I error rates if not done with extreme care. When testing a fixed effect of critical interest it's best to include the random slope for that term even if the data does not fully support it /fin
