Read on Twitter
Some movement to announce here: JAMA Open have now corrected this paper 2 months after it was published
Unfortunately, it has gone from an error-filled useless analysis to a slightly less error-filled useless analysis
Some more peer-review on twitter 1/n https://twitter.com/GidMK/status/1327872367893176320
Unfortunately, it has gone from an error-filled useless analysis to a slightly less error-filled useless analysis
Some more peer-review on twitter 1/n https://twitter.com/GidMK/status/1327872367893176320
2/n The updated paper is here https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2772834
And you can read @ikashnitsky and my original commentary on the paper here https://osf.io/9yqxw/
And you can read @ikashnitsky and my original commentary on the paper here https://osf.io/9yqxw/
2.5/n Important to note that this is a very influential paper. It has been in >100 news stories, and has been cited by the EU and WHO (!)
Worrying that until recently it was openly wrong
Worrying that until recently it was openly wrong
3/n In brief, the original paper made a number of obvious errors in analysis, and conducted a largely meaningless analysis using years of life lost (YLL) to compare the harms of COVID-19 and school closures
4/n So, what's changed?
Well, the main structure of the study is the same. They've taken an estimate of the impact of school closures from Argentinian children in the 70s, and applied it directly to children learning from home during the COVID-19 pandemic
Well, the main structure of the study is the same. They've taken an estimate of the impact of school closures from Argentinian children in the 70s, and applied it directly to children learning from home during the COVID-19 pandemic
5/n Previously, this estimate entirely relied on an incorrect meta-analysis that showed that every extra year of schooling reduced your risk of death by 25% over your entire lifespan
This has been removed
This has been removed
6/n (If I was of a skeptical bent, I'd guess that this meta-analysis was removed instead of corrected because simply correcting the model entirely reverses the main findings of the paper)
7/n Instead, the authors have completely arbitrarily divided their analysis into two sections - that based on 'US' and 'European' studies
This gives them some...interesting results
This gives them some...interesting results
8/n If we take these results at face value, closing schools during COVID-19 in the US either killed ~10x more or 5x less people than had they stayed open, depending on whether the analysis is based on 'US' or 'European' studies
9/n So, what are these studies?
Well, here's the first obvious error that the authors have left in. The two US studies you can see in the table from the supplementaries below
Well, here's the first obvious error that the authors have left in. The two US studies you can see in the table from the supplementaries below
10/n Problem is, Mazumder (2008) is simply a re-examination of THE SAME DATA (1919 and 1939 schooling law changes in the US) as Lleras-Muney (2005)
It's simply incorrect to just blindly chuck these estimates into the same model as if they aren't intrinsically related
It's simply incorrect to just blindly chuck these estimates into the same model as if they aren't intrinsically related
11/n And it gets worse. As we pointed out in our critique, the MAIN CONCLUSION from the Mazumder (2008) paper was that SCHOOLING HAD MINIMAL OR NO IMPACT ON MORTALITY
So how did the authors instead estimate a 35.2% reduction in relative risk????
So how did the authors instead estimate a 35.2% reduction in relative risk????
12/n It appears that the authors have cherry-picked a single result from one table of the paper that was actually just recreating Lleras-Muney's results from the original paper using the second methodological approach, rather than using the actual results from the paper
13/n It gets worse. In 2010, Mazumder published an erratum to the paper which I will quote here:
"there is little compelling evidence suggesting a causal link between education and mortality based on Census data and compulsory schooling laws"
"there is little compelling evidence suggesting a causal link between education and mortality based on Census data and compulsory schooling laws"
14/n As ever, I am not here to litigate intentions, but it is amazingly strange to pick the wrong number from a table of a paper that flatly contradicts your main assertion and then lump it in with the paper ~that it was correcting~ as if this wasn't scientifically flawed
15/n I mean, using a paper that says "there is no relationship between missed schooling and mortality" to support your argument that there is a relationship between schooling and mortality is...something
16/n But this brings us back to the results. Either missed days of schooling cost 0.8 (0.1-2.4) million YLL or 13.8 (2.5-42.1) million YLL
Why the massive divergence?
Why the massive divergence?
17/n Well, if you use the incorrect estimates from Mazumder/Lleras-Muney, you get an average of a 45% reduction in RELATIVE RISK OF DEATH AT EVERY AGE per year of schooling
This is hilariously implausible
This is hilariously implausible
18/n For context, this would mean that adding 3 years of schooling for every child would reduce their risk of death to literally 0% for the rest of their lives, effectively rendering them immortal
19/n The authors justify this with some dubious language about how US studies are more similar than Nordic ones, despite the fact that the US studies are based on legal changes in 1919 and 1939, at which time the US was (for example) still segregated
20/n It is also very strange that the authors are happy to use evidence from an Argentinian study on children markedly different from those in the US but balk at studies on European children
It seems an obvious contradiction
It seems an obvious contradiction
21/n Nevertheless, what we now have is a study that, weirdly, says that either school closures have cost very few YLL or a wildly absurd overestimate, depending on whether you limit your analysis to only the studies that the authors prefer arbitrarily to use
22/n Many of the other obvious flaws in the paper remain uncorrected, and I'd urge you to read our full critique if you're interested: https://osf.io/9yqxw/
23/n That being said, it is worth saying that the authors and journal have at least taken ~some~ action here, and correcting the mathematically impossible model is a good first step for this paper
24/n My hope is that now we can further correct the other obvious mistakes and issues, and come to a more realistic estimation, because this paper is currently impacting policy on an international scale
And it is still simply wrong
And it is still simply wrong
25/n I also think it's worth noting the process it took to get this paper corrected even this far
The original response from the lead author and journal editor was, to quote exactly, "you are not right just because you think you are"
The original response from the lead author and journal editor was, to quote exactly, "you are not right just because you think you are"
26/n This was after @ikashnitsky and I pointed out that the paper was MATHEMATICALLY IMPOSSIBLE
Quite a worrying way to respond
Quite a worrying way to respond
27/n After we published our preprint critique, and it was reported on in the Guardian, we were told to submit a comment on the piece as soon as possible online and they'd get back to us https://www.theguardian.com/science/2020/dec/08/coronavirus-study-that-found-us-school-closures-cut-life-expectancy-criticised-by-epidemiologist
28/n Two months after the initial emails, and over a month after we submitted the comment, we have this correction published
Unfortunately, the study has already had an enormous impact, and changed lives across the world
Unfortunately, the study has already had an enormous impact, and changed lives across the world
29/n You will probably be interested @stephaniemlee @MelissaLDavey @apsmunro @DrZoeHyde @devisridhar @DFisman
30/n Overall, what we have is a paper where the mathematically impossible results have been removed, but is still flawed in numerous ways and useless as evidence for decision-making
31/n The sad fact is that the approach the authors took, if implemented correctly, probably would not have found that school closures cost more YLL than COVID-19
32/n This DOESN'T MEAN that school closures are a good thing, necessarily, but YLL is a measure inherently geared towards measuring people who have already died, and it's just not likely that closing schools has cost so much of this metric
