Read on Twitter
What is the most important factor in #COVID19 diagnosis in radiology?
All answers were right (w assumptions), but *by far* the most important factor is location.
So here is a little tweetorial on how the covid pandemic broke medical statistics.
https://twitter.com/DrLukeOR/status/1280700132837584897?s=20
1/17
All answers were right (w assumptions), but *by far* the most important factor is location.
So here is a little tweetorial on how the covid pandemic broke medical statistics.
https://twitter.com/DrLukeOR/status/1280700132837584897?s=20
1/17
Background: covid has various mimics on medical imaging. A nice review article came out a few days ago, with the conclusion: "The typical imaging features of COVID-19 have low specificity due to their overlap with a number of other conditions."
https://pubs.rsna.org/doi/10.1148/radiol.2020202504
2/17
https://pubs.rsna.org/doi/10.1148/radiol.2020202504
2/17
The idea is sound but this statement is a great segway, because the claim that imaging has low specificity is wrong.
And right. And wrong.
First up, specificity tells us "of negative cases, how many were called negative?"
It is the reciprocal of the false positive rate.
3/17
And right. And wrong.
First up, specificity tells us "of negative cases, how many were called negative?"
It is the reciprocal of the false positive rate.
3/17
Sp is one of the "big two" metrics in diagnostic medicine. Why?
Because it is *prevalence invariant*. This means the value should be similar across locations, which is super important so we can compare tests!
(Don't start about spectrum effects, btw. We'll get to that)
4/17
Because it is *prevalence invariant*. This means the value should be similar across locations, which is super important so we can compare tests!
(Don't start about spectrum effects, btw. We'll get to that)
4/17
So what is the specificity of imaging for covid?
Well, this systematic review says between 25% and 33%. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7227176/
But! This is based on two studies, both from Wuhan. Specificity is harder to measure, so most studies only report sensitivity.
5/17
Well, this systematic review says between 25% and 33%. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7227176/
But! This is based on two studies, both from Wuhan. Specificity is harder to measure, so most studies only report sensitivity.
5/17
Another study, albeit beset with design flaws, claims that radiologists *can* distinguish covid from viral pneumonia, the most common mimic!
In this cohort, the specificities per radiologist range between 90 and 100, with two outliers below 30.
https://pubs.rsna.org/doi/full/10.1148/radiol.2020200823
6/17
In this cohort, the specificities per radiologist range between 90 and 100, with two outliers below 30.
https://pubs.rsna.org/doi/full/10.1148/radiol.2020200823
6/17
Many said these results were implausible.
I disagree! The results were biased to some extent, but it is *entirely possible* to have specificity >90% when diagnosing covid on CT or Xray.
In fact, the responses were more wrong by claiming "we know specificity is 20-30%".
7/17
I disagree! The results were biased to some extent, but it is *entirely possible* to have specificity >90% when diagnosing covid on CT or Xray.
In fact, the responses were more wrong by claiming "we know specificity is 20-30%".
7/17
(not just because other papers have reported higher specificities - for example this paper which was less flawed and found an Sp of ~70% - https://www.medrxiv.org/content/10.1101/2020.04.22.20070441v1)
8/17
8/17
Because ...
specificity is not prevalence invariant in covid!
WHAAAT?
9/17
specificity is not prevalence invariant in covid!
WHAAAT?
9/17
So, spectrum effects 
I hate the claim that se/sp are not prevalence invariant d/t spectrum effects. Spectrum effects are when different populations contain different cases, causing your measurements to vary.
This is not the same as the value changing with prevalence!
10/17
I hate the claim that se/sp are not prevalence invariant d/t spectrum effects. Spectrum effects are when different populations contain different cases, causing your measurements to vary.
This is not the same as the value changing with prevalence!
10/17
The causal pathway is:
different population -> different prevalence
different population -> different measurements
**NOT**
different prevalence -> different measurements
There is literally no mathematical way for prevalence of a disease to affect se or sp...
Except...
11/17
different population -> different prevalence
different population -> different measurements
**NOT**
different prevalence -> different measurements
There is literally no mathematical way for prevalence of a disease to affect se or sp...
Except...
11/17
It does in covid! Albeit through one weird trick.
It is the *relative* prevalence of covid vs it's mimics that matters here. Pandemics are really the only time this ratio changes drastically.
The thing is, covid mimics are rare. A hospital might get a few per day.
12/17
It is the *relative* prevalence of covid vs it's mimics that matters here. Pandemics are really the only time this ratio changes drastically.
The thing is, covid mimics are rare. A hospital might get a few per day.
12/17
So, if you are in South Australia where I live, we have had no covid for weeks. For every mimic that comes in, there is a chance you will flag it as possible covid. This means the specificity is 0%.
But if you are in Texas, where there were 10,000 new cases yesterday...
13/17
But if you are in Texas, where there were 10,000 new cases yesterday...
13/17
Then the maybe few dozen covid mimic cases in the state are nothing at all. Your specificity is ~100%.
So measuring and reporting the specificity of imaging in this pandemic is problematic. It depends on where and when you did the experiments.
14/17
So measuring and reporting the specificity of imaging in this pandemic is problematic. It depends on where and when you did the experiments.
14/17
This raises a thorny question: if where and when you are matters more than the findings on the study ... is imaging worth it?
Well, not really. At least, not to diagnose covid. All the societies agree, imaging is not for primary diagnosis.
https://lukeoakdenrayner.wordpress.com/2020/03/23/ct-scanning-is-just-awful-for-diagnosing-covid-19/
15/17
Well, not really. At least, not to diagnose covid. All the societies agree, imaging is not for primary diagnosis.
https://lukeoakdenrayner.wordpress.com/2020/03/23/ct-scanning-is-just-awful-for-diagnosing-covid-19/
15/17
What does all this mean? Should we stop reporting specificity?
Maybe? We certainly can't claim that the specificity of xray or CT will be a single value, and any meta-analysis should not meta-analyse specificity, which would be meaningless).
16/17
Maybe? We certainly can't claim that the specificity of xray or CT will be a single value, and any meta-analysis should not meta-analyse specificity, which would be meaningless).
16/17
If nothing else, it is interesting though.
If you have been wondering why results on specificity are all over the shop, then look no further. Covid broke specificity.
17/17
If you have been wondering why results on specificity are all over the shop, then look no further. Covid broke specificity.
17/17
