Read on Twitter
Because I am a glutton for punishment, here's a tiny thread about why causal inference and policy is so tricky re: "masks and COVID-19"
Disclaimer #1: Mute this thread if this topic harms your mental health
Disclaimer #2: Masks help prevent asymptomatic transmission
Disclaimer #1: Mute this thread if this topic harms your mental health
Disclaimer #2: Masks help prevent asymptomatic transmission
To solve problems we have to properly define them.
Well-intentioned debates have been driven by two distinct problems:
A) Whether there is sufficient evidence to adequately estimate policy effects
B) Whether there is sufficient evidence to act
First (A):
Well-intentioned debates have been driven by two distinct problems:
A) Whether there is sufficient evidence to adequately estimate policy effects
B) Whether there is sufficient evidence to act
First (A):
1) Defining the treatment matters:
While "masks" as a concept are causal for *something*... we can't estimate their effects unless we define the proposal more clearly; do what with masks? what kind of mask? for how long?
Not defining these terms was at the heart of many debates
While "masks" as a concept are causal for *something*... we can't estimate their effects unless we define the proposal more clearly; do what with masks? what kind of mask? for how long?
Not defining these terms was at the heart of many debates
Note that specificity is a continuum.
Perhaps the Rx is: Everyone in the US uses a hodgepodge of whatever kinds of masks they have available for the next two months.
This is more helpful than "masks!" but more difficult to estimate than "HCWs in an ICU wear N95s vs nothing"
Perhaps the Rx is: Everyone in the US uses a hodgepodge of whatever kinds of masks they have available for the next two months.
This is more helpful than "masks!" but more difficult to estimate than "HCWs in an ICU wear N95s vs nothing"
2) Defining the population matters:
Who is using them, where and when is also important. Even a well-defined treatment does not have one common effect (though we are best served estimating averages). Background risk, other interventions, compliance, timing all important
Who is using them, where and when is also important. Even a well-defined treatment does not have one common effect (though we are best served estimating averages). Background risk, other interventions, compliance, timing all important
Researchers must draw "rules" for models and estimates based on past/recent population contexts with little similarity to where they are being applied, e.g. widespread constant use of surgical masks during winter vs. "novel" introduction of home fabric masks in spring/summer
3) Defining the outcome matters:
A focusing on reducing individual risk was driven by a primary concern for health care workers and the available studies. High constant exposure from symptomatic patients & proper use meant effects were largely irrelevant to the public context
A focusing on reducing individual risk was driven by a primary concern for health care workers and the available studies. High constant exposure from symptomatic patients & proper use meant effects were largely irrelevant to the public context
A shift in target effect of reducing community spread means different studies become relevant e.g. mask permeability, but simultaneously, meant fewer studies with relevant data to analyze.
Debates on available evidence often failed to note this distinction
Debates on available evidence often failed to note this distinction
Thus, the unprecedented nature of this pandemic (in nature & responses) means data to evaluate 1, 2, and 3 are lacking and we are mostly likely unable to directly estimate effects for everyone, let alone relevant local effects
It is important not to undersell this.
Now (B):
It is important not to undersell this.
Now (B):
However, in the face of this, decisions and recommendations must still be made. And they must balance potentially competing priorities and resource constraints.
Since direct policy evaluation evidence is not available, the role of data is more qualitative, more indirect.
Since direct policy evaluation evidence is not available, the role of data is more qualitative, more indirect.
Recommendations now, for example to use cloth masks, are not based on rigorous estimates of policy effects but by weighing needs of HCWs and indirect physiologic and population evidence.
And that's the right thing to do.
And that's the right thing to do.
Changing opinions based on available data and context are expected. They would be expected even if we could rigorously estimate effects.
However, I worry conflict in/outside the research community are from the belief that empirical findings do/ought to do more than are warranted
However, I worry conflict in/outside the research community are from the belief that empirical findings do/ought to do more than are warranted
We often fault policymakers for oversimplifying research findings.
But there is counterpoint that some of us feel the right policy decision must be clinched by empirical evidence.
IMO both must be comfortable recommending under honest uncertainty.
But there is counterpoint that some of us feel the right policy decision must be clinched by empirical evidence.
IMO both must be comfortable recommending under honest uncertainty.
When we lack clarity on the limits of evidence, try to lean too hard into it, or try to hide uncertainties, they backfire and lead to even more noise.
I've use masks as an acute example, but there are obviously much thorough reporting of the stakes here:
https://www.theatlantic.com/health/archive/2020/04/coronavirus-pandemic-airborne-go-outside-masks/609235/
and here: https://www.nytimes.com/2020/03/17/opinion/coronavirus-face-masks.html
https://www.theatlantic.com/health/archive/2020/04/coronavirus-pandemic-airborne-go-outside-masks/609235/
and here: https://www.nytimes.com/2020/03/17/opinion/coronavirus-face-masks.html
