Read on Twitter
Psychometric validity: A thread. I am sitting here feeling enormous frustration that virtually without fail, every time I partner on a quant/survey project w community members, I give “the talk” (all the reasons why ideally we’d use “validated” measures) and then... 1
They point out all the problems with the measures I dig up (language that is offensive or misleading, perspectival biases, items that do not apply in the intended context, failure to cover issues considered essential by the community, etc etc). 2
While imagining the remonstrations of reviewer 2 (from real life: “the use of novel items is not just, as the authors admit, a limitation, but rather a fatal flaw”) I agree with my collaborators. I almost always find myself ultimately removing nearly all validated measures... 3
Of late I’ve also been involved in several different ‘data harmonization’ projects. Almost always inclusion criteria require prior validation, ideally across multiple studies. The deeper question of construct validity rarely comes up—when it does, what can anyone do? 4
In many cases, even ostensibly co-produced measures (which are few and far between!) still suffer from problems ostensibly tied to the relatively narrow range of views represented (for example, service users in just one region of one country) 5
And yet, it goes without saying, measurement is the backbone of mental health services research. 6
