Read on Twitter
1/
My appraisal of the Danish mask study (DANMASK-19): This was a negative trial - masks were not shown to prevent #COVID19.
Could chance or bias make this outcome more likely?
Does this mean we don’t need to wear masks?
Let’s take a deeper look.
#HowIReadThisPaper
(Thread)
My appraisal of the Danish mask study (DANMASK-19): This was a negative trial - masks were not shown to prevent #COVID19.
Could chance or bias make this outcome more likely?
Does this mean we don’t need to wear masks?
Let’s take a deeper look.
#HowIReadThisPaper
(Thread)
2/
Before beginning, if you have not already done so, I implore you - read the paper!
https://www.acpjournals.org/doi/10.7326/M20-6817
Then, come back, and please comment, add what I have missed, and correct me where I am wrong.
Critical appraisal is a group effort.
Before beginning, if you have not already done so, I implore you - read the paper!
https://www.acpjournals.org/doi/10.7326/M20-6817
Then, come back, and please comment, add what I have missed, and correct me where I am wrong.
Critical appraisal is a group effort.
3/
First, let’s agree on what was tested:
The hypothesis was that the recommendation to wear masks (added to other public health measures) would reduce the incidence of COVID19 among wearers from 2% to 1% over 1 month, in a setting where mask use was uncommon (Denmark).
First, let’s agree on what was tested:
The hypothesis was that the recommendation to wear masks (added to other public health measures) would reduce the incidence of COVID19 among wearers from 2% to 1% over 1 month, in a setting where mask use was uncommon (Denmark).
4/
6,024 adults were randomized; half were told to wear masks when out of the house.
The primary outcome was a composite of a self-reported positive #SARSCoV2 home antibody test, OP or nasal PCR, or hospital-based diagnosis at 1 month.
The funding source is not worrisome.
6,024 adults were randomized; half were told to wear masks when out of the house.
The primary outcome was a composite of a self-reported positive #SARSCoV2 home antibody test, OP or nasal PCR, or hospital-based diagnosis at 1 month.
The funding source is not worrisome.
5/
Next, let’s estimate the likelihood of finding a difference in the primary outcome.
Finding a large effect size (a 50% relative reduction in incident COVID19 among mask wearers), when the outcome is already rare, is very difficult.
How do we know the outcome was rare?
Next, let’s estimate the likelihood of finding a difference in the primary outcome.
Finding a large effect size (a 50% relative reduction in incident COVID19 among mask wearers), when the outcome is already rare, is very difficult.
How do we know the outcome was rare?
6/
Only ~2% of patients were diagnosed with COVID19 during the study period.
As @DrTomFrieden notes in the editorial, this rate was much lower than the US & the UK: https://www.acpjournals.org/doi/10.7326/M20-7499
Even if that was an underestimate of transmission in the community…
Only ~2% of patients were diagnosed with COVID19 during the study period.
As @DrTomFrieden notes in the editorial, this rate was much lower than the US & the UK: https://www.acpjournals.org/doi/10.7326/M20-7499
Even if that was an underestimate of transmission in the community…
7/
...surveillance data from Denmark confirm that the daily number of new confirmed cases (left) and the total number of patients hospitalized with COVID19 (right) were both decreasing during the study period:
https://ourworldindata.org/coronavirus/country/denmark?country=~DNK
https://www.sst.dk/en/english/corona-eng/status-of-the-epidemic/covid-19-updates-statistics-and-charts
...surveillance data from Denmark confirm that the daily number of new confirmed cases (left) and the total number of patients hospitalized with COVID19 (right) were both decreasing during the study period:
https://ourworldindata.org/coronavirus/country/denmark?country=~DNK
https://www.sst.dk/en/english/corona-eng/status-of-the-epidemic/covid-19-updates-statistics-and-charts
8/
How do we know that a 50% effect size is large?
For comparison, 50% is the lower limit of vaccine efficacy the FDA recommended for COVID vaccines seeking licensure:
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/development-and-licensure-vaccines-prevent-covid-19
This study tested whether a recommendation provided at least that much protection.
How do we know that a 50% effect size is large?
For comparison, 50% is the lower limit of vaccine efficacy the FDA recommended for COVID vaccines seeking licensure:
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/development-and-licensure-vaccines-prevent-covid-19
This study tested whether a recommendation provided at least that much protection.
9/
On the basis of these observations, failure to find a benefit of masks seems likely.
Next, let’s investigate sources of chance.
The primary outcome point estimate (OR = 0.82) suggests an 18% reduction in the odds of infection, but the 95% confidence interval spans 0.54-1.23.
On the basis of these observations, failure to find a benefit of masks seems likely.
Next, let’s investigate sources of chance.
The primary outcome point estimate (OR = 0.82) suggests an 18% reduction in the odds of infection, but the 95% confidence interval spans 0.54-1.23.
10/
The width of the confidence interval is larger than the magnitude of the effect estimate itself.
This indicates that there is a lot of uncertainty in the estimate of the effect, and is another possible explanation for the failure to demonstrate a >50% relative reduction.
The width of the confidence interval is larger than the magnitude of the effect estimate itself.
This indicates that there is a lot of uncertainty in the estimate of the effect, and is another possible explanation for the failure to demonstrate a >50% relative reduction.
11/
This is also apparent in the subgroup analyses.
On visual inspection, we see the point estimates for most subgroups were to the left of unity, but had CIs that crossed one.
This is compatible with a modest (<50%) effect size, estimated with low precision (due to low power):
This is also apparent in the subgroup analyses.
On visual inspection, we see the point estimates for most subgroups were to the left of unity, but had CIs that crossed one.
This is compatible with a modest (<50%) effect size, estimated with low precision (due to low power):
12/
Although there were no significant interactions, the point estimates suggest a stronger effect in those who:
-Wore glasses (↑ protection)
-Were outside longer (↑ exposure)
-Lived in the capital region (↑ prevalence)
If true, these would be consistent with what we expect.
Although there were no significant interactions, the point estimates suggest a stronger effect in those who:
-Wore glasses (↑ protection)
-Were outside longer (↑ exposure)
-Lived in the capital region (↑ prevalence)
If true, these would be consistent with what we expect.
13/
Next, let’s examine sources of bias.
There are at least two sources of bias towards underdiagnosis of COVID19 in this trial:
One is self-report of the primary outcome. Figure 1 shows that ~20% of people in both groups did not report outcome data.
Next, let’s examine sources of bias.
There are at least two sources of bias towards underdiagnosis of COVID19 in this trial:
One is self-report of the primary outcome. Figure 1 shows that ~20% of people in both groups did not report outcome data.
14/
Another source of bias towards underdiagnosis is the time lag of seroconversion.
We know it takes ~2 weeks to generate Ig’s.
For a study lasting 1 month, this means any infections in the last two weeks could have been missed if patients hadn't yet formed Ig's.
Another source of bias towards underdiagnosis is the time lag of seroconversion.
We know it takes ~2 weeks to generate Ig’s.
For a study lasting 1 month, this means any infections in the last two weeks could have been missed if patients hadn't yet formed Ig's.
15/
In addition, there are at least two more sources of bias towards a null finding in this study:
-Low adherence to the mask recommendation
-Low positive predictive value (PPV) of serology
Let’s examine each of them.
In addition, there are at least two more sources of bias towards a null finding in this study:
-Low adherence to the mask recommendation
-Low positive predictive value (PPV) of serology
Let’s examine each of them.
16/
Fewer than half of people in the mask group reported wearing masks as recommended (46%).
Low uptake of the intervention diminishes our ability to detect any benefit of masking that may exist.
Fewer than half of people in the mask group reported wearing masks as recommended (46%).
Low uptake of the intervention diminishes our ability to detect any benefit of masking that may exist.
17/
Regarding serology and PPV: table 2 reveals that most (~75%) infections were diagnosed via a positive IgM or IgG.
This is problematic because the false-positive rate of any test increases rapidly when outcomes are rare and specificity is much lower than 100%.
How rapidly?
Regarding serology and PPV: table 2 reveals that most (~75%) infections were diagnosed via a positive IgM or IgG.
This is problematic because the false-positive rate of any test increases rapidly when outcomes are rare and specificity is much lower than 100%.
How rapidly?
18/
At a prevalence of 2% and a sens/spec of 82.5% / 99.5% for the antibody assay used in this study, the PPV is 77%.
However, using the specificity reported by the FDA of 97.5%, just 2% lower, the PPV falls to 40%.
https://www.accessdata.fda.gov/cdrh_docs/presentations/maf/maf3285-a001.pdf
At a prevalence of 2% and a sens/spec of 82.5% / 99.5% for the antibody assay used in this study, the PPV is 77%.
However, using the specificity reported by the FDA of 97.5%, just 2% lower, the PPV falls to 40%.
https://www.accessdata.fda.gov/cdrh_docs/presentations/maf/maf3285-a001.pdf
19/
A PPV of <50% means we would expect the majority of all infections diagnosed by serology to be false positives.
False positives are likely to be distributed evenly, thus biasing towards the null.
For more on PPV pitfalls, see this thread: https://twitter.com/rbganatra/status/1266701605258383360?s=20
A PPV of <50% means we would expect the majority of all infections diagnosed by serology to be false positives.
False positives are likely to be distributed evenly, thus biasing towards the null.
For more on PPV pitfalls, see this thread: https://twitter.com/rbganatra/status/1266701605258383360?s=20
20/
Let’s close with summarizing what this study adds and some suggestions for how these issues could be addressed in future studies.
Replication in a higher-transmission setting & powering for a more modest effect size are good starting points.
But...
Let’s close with summarizing what this study adds and some suggestions for how these issues could be addressed in future studies.
Replication in a higher-transmission setting & powering for a more modest effect size are good starting points.
But...
21/
~15% of people in the mask group reported experiencing adverse reactions from others because of their masks, and 1/3 of those who did not wear masks as directed cited their work as the reason.
Social & behavioral incentives were not aligned with mask use in this study.
~15% of people in the mask group reported experiencing adverse reactions from others because of their masks, and 1/3 of those who did not wear masks as directed cited their work as the reason.
Social & behavioral incentives were not aligned with mask use in this study.
22/
This raises the Q of whether an individual-level RCT is the best way to study a population-level recommendation.
An argument made eloquently by @MonicaGandhi9 and others is that a community-level cluster RCT could better align social & behavioral incentives with mask use.
This raises the Q of whether an individual-level RCT is the best way to study a population-level recommendation.
An argument made eloquently by @MonicaGandhi9 and others is that a community-level cluster RCT could better align social & behavioral incentives with mask use.
23/
The authors note this study did not test masks for source control, which is likely to have an even greater impact on transmission.
That impact, coupled with low risk of harm from masks, is the basis for @CDC's continued recommendation for their use:
https://www.cdc.gov/coronavirus/2019-ncov/more/masking-science-sars-cov2.html
The authors note this study did not test masks for source control, which is likely to have an even greater impact on transmission.
That impact, coupled with low risk of harm from masks, is the basis for @CDC's continued recommendation for their use:
https://www.cdc.gov/coronavirus/2019-ncov/more/masking-science-sars-cov2.html
24/
Bottom line: Rare outcomes, powering for a big effect, & multiple sources of chance & bias increased the likelihood of a negative study.
DANMASK-19 didn’t show a large benefit for wearers, but a meaningful public health benefit likely still exists. Keep wearing masks.
(End)
Bottom line: Rare outcomes, powering for a big effect, & multiple sources of chance & bias increased the likelihood of a negative study.
DANMASK-19 didn’t show a large benefit for wearers, but a meaningful public health benefit likely still exists. Keep wearing masks.
(End)
