Only evidence for accuracy of UK-Rapid Test Consortium Covid-19 AbC test is flawed

Over a week ago with @sarahboseley I asked for transparent reporting of RTC study that says test has 99.40% accuracy

But NOTHING has been made available or explained

1/16 https://www.theguardian.com/commentisfree/2020/aug/27/covid-19-antibody-test-coronavirus-corners-being-cut

This is the totality of information about the study that is public - from Abingdon Health’s website - its three sentences.
https://www.abingdonhealth.com/uk-covid-19-rapid-antibody-tests-approved-for-professional-use/

But three sentences are enough to see the flaw.

2/16

Selecting only EuroImmun +ves biases sample to those with easy to detect antibodies. EuroImmun assay only has ~70% sensitivity - those with low or no antibodies are highly likely to have been excluded.

Then it's as easy as ABC to get a good estimate of sensitivity!

3/16

But that estimate is going to be WRONG and will not tell you what the false negative rate will be when used in clinical practice.

Those false negative on EuroImmun are more likely to be false negative on the AbC test as well (unless it is really good).

4/16

Actually @guardian were told by @Abingdon_Health that selection bias is worse than that.

Quote from email

“The 203 positive samples were taken from those with symptoms and all 203 samples tested positive for Covid-19 by two PHE approved methods (Roche and EuroImmun) ... ”

5/16

(I've waited a week to see if @Abingdon_Health would tell us all that).

So they had to be positive on BOTH EuroImmun and Roche antibody tests to get into the COVID-19 group.

Frankly it is remarkable that there were any false negatives at all!

6/16

And the same applied for the test negatives.

@guardian were told by @Abingdon_Health

“100% of the negative samples tested negative by two PHE approved methods (EuroImmun, Roche)”.

So estimates of specificity will be biased upwards in the same way.

7/16

Given that 30% of the COVID-19 group could have been dropped the potential magntiude of this bias is substantial. This is far more than a "theoretical concern"

This is spectrum bias. It is not new - NEJM reported on it when I was at primary school. https://www.nejm.org/doi/full/10.1056/NEJM197810262991705

8/16

SO ....

Stating that the sensitivity and specificity of these tests are 98.03% and 99.56% for detecting previous SARS-CoV-2 infection is a

MISLEADING AND A DANGEROUSLY WRONG ANSWER

9/16

AND ...

Stating that the sensitivity and specificity of these tests are 98.03% and 99.56% for detecting antibodies is a

MISLEADING AND A DANGEROUSLY WRONG ANSWER

10/16

AND ...

Stating that the sensitivity and specificity of these tests are 98.03% and 99.56% without saying what they are actually detecting is

A MISLEADING ANSWER

as it will be misinterpreted to mean one of the above which will be WRONG.

11/16

All you can CORRECTLY say is that

In people who with suspected COVID-19, who have already tested positive using both the EuroImmun and Roche tests, 98.03% of them also test positive with the AbC test

12/16

AND

In people who never could have had COVID-19, and who have already tested negative using both the EuroImmun and Roche tests, 99.56% of them also test negative with the AbC test

13/16

Nobody meets these criteria

(and if they did, they probably wouldn’t want a third antibody test)

so these findings are POINTLESS.

Certainly they are not enough for @MattHancock to sign £M purchasing deals. Other tests have much better and convincing evidence

14/16

Are @MHRAgovuk carefully checking IFU and marketing claims?

*make sure target condition is "antibodies" (not previous infection)

*that population are "people already tested +ve or -ve twice"

But really this shouldn't be approved without a proper clinical accuracy study

15/16

When @DHSCgovuk have funded excellent groups like the CONDOR-PLATFORM ( https://www.condor-platform.org/ ) to support industry doing these studies it is appalling to see such a flawed effort.

We need to get real about the importance of good clinical accuracy studies.

16/16