Read on Twitter
A longer thread on replicability vs asymptotics of network psychometric methods: Recently, I've been seeing both tweets and articles which discuss if network psychometric methods are "replicable". I wanted to explore this topic, as it is a very important issue to get right.
First, the question that a method, defined here as a statistical procedure, is replicable, is a tad strange to me. Results replicate, not methods. That being said, statistical methods ideally have properties that make replication of results possible ...
Specifically, most statistical methods are asymptotically consistent (I am blending methods, models and estimators here, but I think the spirit of this statement is correct). As sample size increases, if the assumptions of the model are correct ...
...then (in a frequentist perspective), your model will converge to the "true" model as you increase sample size.
How does this apply to network psychometric models? Take a commonly used way of constructing networks, partial correlations ...
How does this apply to network psychometric models? Take a commonly used way of constructing networks, partial correlations ...
A partial correlation matrix is 1 to 1 with the correlation matrix (take the inverse of a correlation matrix, and do some standardization and sign switching). Correlations are consistent estimators of the linear relation between two variables, so partial correlations are too...
If partial correlations are not "replicable" than neither are correlations.
One issue that has arose in the conversation about network psychometric models is the observation that partial correlations have greater standard errors than correlations at the same sample size...
One issue that has arose in the conversation about network psychometric models is the observation that partial correlations have greater standard errors than correlations at the same sample size...
This is true, they do, but that is just because the conditioning inherit in the partial correlations reduce the degrees of freedom, which increases the SE. You have to pay the noise piper for using more information in your calculation...
@wdonald_1985 has an excellent, extensive explanation of this here: https://psyarxiv.com/fb4sa
So, this implies that, if all the assumptions about the generative process are correct, then network psychometric statistical models are going to be replicable (in that, large samples...
So, this implies that, if all the assumptions about the generative process are correct, then network psychometric statistical models are going to be replicable (in that, large samples...
... from the same population should have approximately the same network, and that differences between samples should decrease with sample size). I ran some quick simulations to verify this, and this is indeed the case (however, regularization lead to lower "reliability" at...
...finite sample sizes, in that the variance of the estimates is inflated. LASSO is not a consistent estimator (by design, actually), and in fact regular LASSO is not guaranteed to converge on the true model (it lacks the oracle property that adaptive LASSO has)...
But from an overall model fit standpoint, you get increasingly better "replication" as sample size increases even with regularization).
That being said, concern about the replicability of network psychometric results is a valid one, and should be investigated! ...
That being said, concern about the replicability of network psychometric results is a valid one, and should be investigated! ...
@MiriForbes has some excellent work laying out these concerns ( https://psyarxiv.com/bvu84/ ), and while I have the above issue with applying "replicability" to a statistical model, concerns like these need to be taken seriously, particularly if we take them in light of the applied...
...question, what is the structure of psychopathology. Because if the math is right, which it is, these networks should be replicable. I have two concerns I'll discuss here...
The first, which Dr. Forbes and Co raise in the above preprint, is about the interpretation of individual edges. If an edge between insomnia and fatigue doesn't replicate between two samples, that does lead to meaningful interpretation differences...
...even if overall the fit is approximately the same. The fact that the math works in no way undermines this critique, as this is an issue of theoretical "replicability", not statistical "replicability" (terms used colloquially)...
I would argue that this sort of "unreplicability" is a result of the granularity of the analysis. Because you are not summarizing multiple bivariate relations in single parameters (like in CFAs), you just need more power to detect these relations...
The second concern is, to me, far more of a damning question: If the math is right, then these networks should replicate. If there are issues with replicability, that means the math is wrong (it isn't) or the underlying assumptions of the math are violated...
For cross-sectional network analyses to replicate, this assumes that the dynamical process underlying the symptoms for each person is a) the same between people and b) the same within person for all time (i.i.d and stationary)...
These are fairly strict assumptions, and I somehow don't think they hold all the time in psychopathology. So what are the consequences? I am working on elaborating on this issue, but I will leave with a fairly chilling thought...
If the reason why network psychometric models don't replicate is due to a violation of these assumptions, then any cross-sectional dimensional analysis of psychopathology data (looking at you, factor analysis) will have the same issue...
If that is the case, I think the only reason why latent variable models replicate is that low-rank representations, because they blend relations together, have much more wiggle room on what counts as replicable...
But they don't necessarily tell you anything meaningful, if those assumptions are violated. Molenaar's seminal 2004 paper ( https://www.tandfonline.com/doi/abs/10.1207/s15366359mea0204_1) is a great exploration of this exact problem.
Long thread, I suppose the tl;dr is: Most models fail to replicate when the assumptions underpinning them are violated. Network and/or factor models are no exception.
