Read on Twitter
The following study is being widely shared as definitive evidence that 76% of the population of Manaus, Brazil, became infected with COVID-19.
Quick thread on placing this study in the proper context (and why outliers are so popular...)
1/
https://science.sciencemag.org/content/early/2020/12/07/science.abe9728
Quick thread on placing this study in the proper context (and why outliers are so popular...)
1/
https://science.sciencemag.org/content/early/2020/12/07/science.abe9728
First, note that this was not a random population survey but a prevalence study done on a convenience sample of blood donors.
To decide whether the IFR is 1% or .1%, this may not be so important; to decide whether the AR is 75% or 35%, it matters much more!
2/
To decide whether the IFR is 1% or .1%, this may not be so important; to decide whether the AR is 75% or 35%, it matters much more!
2/
In particular, the questions we have about HITs and attack rates are sensitive to much smaller multiplicative errors than other questions we answer with these surveys.
Second: this study never measured prevalence above 44% even in the study population. Higher infection rates...
Second: this study never measured prevalence above 44% even in the study population. Higher infection rates...
were projected by correcting for waning antibodies, etc. These corrections may be appropriate, but they have not been routinely applied in all AB studies, and could lead, for example, to downward revision of IFRs if the methodology was consistently used. 4/
One way in which might hope to validate the choice of convenience samples in this study is if a random population survey was done at any time this study was underway, in the same location.
While this sounds optimistic, such a study was in fact done! 5/
While this sounds optimistic, such a study was in fact done! 5/
As the authors noted in the preprint version of this article, a random population survey found 3x less the prevalence they were finding in blood donors from the same city at the same time.
https://www.medrxiv.org/content/10.1101/2020.09.16.20194787v1.full.pdf
They explain this away by suggesting that... 6/
https://www.medrxiv.org/content/10.1101/2020.09.16.20194787v1.full.pdf
They explain this away by suggesting that... 6/
the sensitivity of the test may have been miscalibrated for tests using blood from finger-pricks.
But in fact (we are getting into the weeds here), the population survey did validate the sensitivity of their test with blood from finger pricks. 7/ https://twitter.com/WesPegden/status/1308139416037228546
But in fact (we are getting into the weeds here), the population survey did validate the sensitivity of their test with blood from finger pricks. 7/ https://twitter.com/WesPegden/status/1308139416037228546
The published version of the article seems to have dropped discussion of the random survey.
What's the upshot?
There have been a number of randomized population surveys done in South America; the continent seems to be producing more good work of this type than anywhere else.8/
What's the upshot?
There have been a number of randomized population surveys done in South America; the continent seems to be producing more good work of this type than anywhere else.8/
But the paper that made it into Science and thus is now widely being shared is not a study from this continent which did the best science, but the study which reported the biggest number (even though it was incompatible with other science which used a better methodology!) 9/
Another important point is that small errors from non-random samples matter for estimates of attack rates and epidemic sizes. If the random survey that found 3x less prevalence was used to "correct" the results of this survey, the result would be an attack rate under 30%, 10/
even after the adjustments the authors make for waning antibodies etc.
It is also important to emphasize and remember that outlier reports are rarely the way to make a strong case for a general phenomenon. 12/12
It is also important to emphasize and remember that outlier reports are rarely the way to make a strong case for a general phenomenon. 12/12
Correction:
The Science paper does still reference the population study; it just references the June results instead of May, which are 14% instead of 12%.
Again, note that this is roughly 3x below what is found in Science paper using blood donors. 13/14 https://twitter.com/alchemytoday/status/1336836268521558017
The Science paper does still reference the population study; it just references the June results instead of May, which are 14% instead of 12%.
Again, note that this is roughly 3x below what is found in Science paper using blood donors. 13/14 https://twitter.com/alchemytoday/status/1336836268521558017
Note that the Science paper gives two explanations for why the random study should be discarded.
1) It had a small random sample of 250 people [actually, pooling May+June, there are 500 samples]
2) The test may have low sensitivity [as noted above, sensitivity was tested].
14/15
1) It had a small random sample of 250 people [actually, pooling May+June, there are 500 samples]
2) The test may have low sensitivity [as noted above, sensitivity was tested].
14/15
There may well be some unknown problems with the random survey; on the other hand there are known problems with convenience samples.
Note that the 95% confidence interval on 13% positive from the random sample of 500 people would be just a few percentage points. 15/15
Note that the 95% confidence interval on 13% positive from the random sample of 500 people would be just a few percentage points. 15/15
