Read on Twitter
New paper now out online in @jnnp with @jonstoneneuro, @craig_ritchie68 and @AlanCarson15
This is a systematic review of performance validity test performance in clinical populations
(Thread 1/12) http://jnnp.bmj.com/cgi/content/full/jnnp-2020-323776?ijkey=D0MHaBdIqpr5oJy&keytype=ref
This is a systematic review of performance validity test performance in clinical populations
(Thread 1/12) http://jnnp.bmj.com/cgi/content/full/jnnp-2020-323776?ijkey=D0MHaBdIqpr5oJy&keytype=ref
People kept asking me whether performance validity tests (PVTs) are useful in diagnosing functional cognitive disorders (spoiler: no). I wanted to better understand what we were actually looking at with these tests so had a closer look (2/12)
PVTs (well-known ones are the TOMM, and the Word Memory Test) are designed to detect invalid performance in neuropsych tests. They used to be called effort tests so a lot of people still think that they detect poor effort, which isn’t really how they’re designed (3/12)
Neuropsychologists are experts in interpreting the results of PVTs and in considering the various reasons someone might ‘fail’ or underperform. For the rest of us, it can be hard to understand what it means when someone fails PVTs (4/12)
There are problems with the PVT literature. A lot of studies look at PVT performance in people who are involved in litigation, or in very mixed clinical groups. Different studies use different cutoffs. Most importantly there is no gold standard to assess the tests against (5/12)
We systematically reviewed the literature to find descriptions of PVT failure rates in people with defined clinical conditions who were not (as far as we know) involved in litigation. Our search identified 69 valid studies (6/12)
We found surprisingly high rates of PVT failure in a whole range of conditions: MCI, mild and moderate dementia, mild and moderate-severe brain injury, psychosis, and others. Failure rates were *no higher* in functional disorders than in other conditions (7/12)
So - are all these people not trying hard? Are they trying to fail? Seems unlikely. We speculate in the paper that there are many reasons to perform in the invalid range in PVTs. The tests themselves *do not* tell us what these reasons are (8/12)
But while we don’t really know what we are measuring with PVTs, what IS clear is that PVT failure means the results of other neuropsych tests are invalid. Like when someone moves inside the MRI scanner - doesn’t matter why - the result is that the pictures come out fuzzy (9/12)
Our finding that PVT failure is actually pretty common suggests a bigger problem lurking in the background (10/12)
That is - if PVT failure is common, invalid neuropsychological test results are also common.
And if we rely excessively on neuropsychological tests in research and clinical practice without being mindful of this, we are going to make mistakes (11/12)
And if we rely excessively on neuropsychological tests in research and clinical practice without being mindful of this, we are going to make mistakes (11/12)
Thanks as always for support from my supervisors @jonstoneneuro @AlanCarson15 @craig_ritchie68,(after a tricky few months), from @EdinUniBrainSci, and from @BaillieGifford, who fund my research (12/12)
