Read on Twitter
Ok, so I've had a read of this paper (which has been all over the news) that says quite explicitly that closing schools will probably cause more years of life lost than leaving them open in a pandemic
Let's do some peer-review on twitter! 1/n https://twitter.com/GidMK/status/1327791987014074369
Let's do some peer-review on twitter! 1/n https://twitter.com/GidMK/status/1327791987014074369
2/n Paper is here. It's actually very simple, basically the authors took an estimate of how many days of school kids lost due to closures during COVID-19, an estimate of how much that impacts their years of life, and multiplied out https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2772834
3/n Specifically, they said:
1. Children miss days of school
2. Days of school cause decreased educational attainment
3. Decreased educational attainment causes less life
1. Children miss days of school
2. Days of school cause decreased educational attainment
3. Decreased educational attainment causes less life
4/n They then compared this with estimates of the number of years of life lost due to COVID-19 already, and found that the loss of educational attainment caused more death (with 98.1% probability!)
5/n Now, that sounds very robust. 98.1% probability is extremely certain!
It's also, as far as I can tell, an incredibly meaningless number. NONE OF THIS IS THE SLIGHTEST BIT CERTAIN
It's also, as far as I can tell, an incredibly meaningless number. NONE OF THIS IS THE SLIGHTEST BIT CERTAIN
6/n Firstly, the paper that all of this extrapolation is based on - reference 14. This is where the authors got the relationship between days of schooling missed and overall educational attainment
7/n This is a long and fascinating examination of the association between teachers' strikes in Argentina and the eventual earnings that children achieved when they grew up
8/n Note there that I said ASSOCIATION. This is a key point. The authors of this paper WARN AGAINST USING IT CAUSALLY because, among other issues, the data for children in Argentina in the 70s and 80s (especially schoolkids) was not great
9/n Indeed, the main analysis of this paper explicitly excludes the impact of missing secondary education because their data was incomplete and many Argentinian children in the cohort at the time did not attend secondary school
10/n It should also be noted that the impact of TEACHERS' STRIKES on ARGENTINIAN CHILDREN may not mean much when we're talking about American children, some % of whom could learn at home and thus were not explicitly missing class
10.5/n (Also, whether the impact of teachers formally striking on children in the 70s is the same as missing days of in-person education during a pandemic in 2020...I would say probably not)
11/n Next, the authors did something odd. They decided to compare this to the DIRECT mortality from COVID-19. This means that they are only counting people who have died as a direct result of the disease
12/n But, if we're comparing the total impact of missed education on children over a lifetime, this makes no sense. COVID-19 has many other harms, after all, some of which are disputed but many of which are easily attributable
13/n Where are the years of lost life due to ICU stays? Hospitalizations? Disability, mental health issues, avoided care due to full hospitals, etc etc etc???
14/n (This is also a problem for the YLL calculation for children missing school, FYI, it is FAR more complex than just missing educational attainment. So many potential things that are not included here!)
15/n Ultimately, what we get in the end - the "98.1% probability" figure - makes almost no sense at all. Even if the Argentinian study was applicable to the US situation, which it clearly isn't, the counterfactual is just inappropriate
16/n It may be possible to do a rigorous cost/benefit of school closures, but I really don't see how this can be considered one
17/n At best, what we've got here is a very rough inference about what the impact of school closures could possibly have been (as long as US kids in 2020 are the same as Argentinians in 1977), compared to an undercount of the impact of what COVID-19 probably was
18/n I should add - I am not saying that closing schools is necessarily a positive in this thread. What I AM saying, however, is that we could not possibly know whether it was or not from this paper
19/n Many further criticisms of this paper that I missed in this also excellent thread. It appears that almost none of the assumptions in the paper make any sense at all https://twitter.com/ikashnitsky/status/1328121853970436100?s=20
20/n Another point I've been thinking about - the authors fail to consider the impact of an ongoing epidemic on school attendance. Lots of cases in the local area probably impacts schooling as well!
21/n Also, what's the impact of parents/carers getting sick and dying? Presumably this will have an outsized impact on the children it affects, which would cause (based on this model) large increases in YLL
22/n Ok, there's another HUGE problem with the study
The findings are entirely based on the idea that an extra year of schooling reduces your risk of death by 25%, a figure reached by aggregating studies in a meta-analysis model
The findings are entirely based on the idea that an extra year of schooling reduces your risk of death by 25%, a figure reached by aggregating studies in a meta-analysis model
23/n This number is the ENTIRE BASIS for the paper. If schooling reduces mortality by less than 25%, then the number of deaths caused by missed school days will drop as well
And we have a problem, because this 25% figure is...wrong
And we have a problem, because this 25% figure is...wrong
24/n Firstly, we have these top 2 papers. Because they're from the US, they are more heavily weighted than the others (this is arbitrary, but whatever)
Except the values for the second paper are flatly incorrect
Except the values for the second paper are flatly incorrect
25/n Mazumder (2008) did not find a relative risk of 0.65 (0-1.3), it found an RR of 0.89 (0.74-1.04)
The paper is here. I've read it very carefully and this appears to be the main finding from the author https://www.chicagofed.org/publications/economic-perspectives/2008/2qtr2008-part1-mazumder
The paper is here. I've read it very carefully and this appears to be the main finding from the author https://www.chicagofed.org/publications/economic-perspectives/2008/2qtr2008-part1-mazumder
26/n So the number is wrong, but this brings us to another issue - the two papers at the top that were double-weighted...are ON THE SAME DATASET
This is a massive issue
This is a massive issue
27/n You see, Lleras-Muney compiled an estimate based on a number of factors on the reduction in risk from 1 year of extra schooling. Mazumder, three years later, took that same database and re-analyzed it considering additional factors, and estimated a lower reduction
28/n However you slice it, it's inappropriate to just bung these two estimates together into a model and treat them as separate. I would argue that the Mazumder paper is probably a better estimate, but either way what they've done here is wrong
29/n If I re-run the meta-analysis they've purported to use with the correct numbers, we end up with this graph instead. A relative reduction in risk of death of 5% for every year of schooling
This is A QUARTER of the estimate used in the study
This is A QUARTER of the estimate used in the study
30/n So AT BEST assuming there are no other errors, the estimate of the years of life lost due to school closures should be divided by 5
