Read on Twitter
1/ Did the Sturgis bike rally cause 266,796 new cases of COVID-19? Probably not. Lesson- Beware viral studies that confirm your pre-existing beliefs so satisfyingly. (Long) thread: https://slate.com/technology/2020/09/sturgis-rally-covid19-explosion-paper.html @FutureTenseNow @EricTopol @slate @govkristinoem @DearPandemic #Sturgis
2/ Like most people working on COVID-19, I am of the strong belief that mass gatherings during a pandemic are a bad idea. When this paper came out, the huge figures immediately hit the "I Told you So!" button in my & many people’s brains. https://www.iza.org/publications/dp/13670/the-contagion-externality-of-a-superspreading-event-the-sturgis-motorcycle-rally-and-covid-19
3/ The first red flag is the huge number itself-it doesn't pass the sniff test.
4/ The rally had an estimated 350-460K attendees, spread over 10 days. To reach such a huge # of 266K over 4 weeks, a LOT of transmission would have to occur both at the rally & in home counties (where cases were counted). No transmission model is mentioned, however.
5/ It requires some mental gymnastics to imagine a scenario where over 200K event attendees became infected at the rally itself. Even with bleak assumptions of 1 % of attendees already infectious (spread over 10 days), yet well enough to ride a motorcycle to South Dakota...
6/ & all of them were “superspreaders,” passing their infection along to another 10 people, back-of-the-envelope math makes it hard to get in the ballpark of this number of infections that could have happened at the rally.
7/ What about the attendees spreading the infection upon returning home? Recall the authors measured increased cases in *home* counties (defined by cell-phone pings pre-Sturgis). This was a motorcycle rally, with some "high inflow" counties far away like CA, NV & FL.
8/ Many attendees likely rode their bikes home & the lure of the open road in August after months of worldwide lockdown may have even induced many riders to take a meandering path home. (Some may have flown-I've since learned there are bike transport services for such events).
9/ Even accounting for people leaving early or going directly home, this leaves v. little time for so many infected riders to get home, infect others, incubate, get tested (with delays),& have these infections show up in county statistics by Sept. 2, just 2 weeks after the rally.
10/ How could such seemingly implausible numbers be estimated? The authors don't track individuals or contacts, but use county-level case data from home counties, & employ a "diff-in-diff" analysis--challenging under the best of times, more so when modeling epidemic spread.
11/ See the @Slate article above for more on the challenges of diff-in-diff, as well as this insightful thread(s) from @RexDouglass: https://twitter.com/RexDouglass/status/1303379252742479872?s=20
12/ Suffice to say the paper's own figures don't inspire confidence in the assumption of parallel trends:
13/ Even taking the model at face value, the precision of the estimates suggests some caution is warranted with bold headline numbers as well:
14/ It will be hard to know exactly what went wrong with the analyses until others replicate, which I understand from @thehauer is proving a challenge: https://twitter.com/thehauer/status/1304166170652745729?s=21
15/ But I'm most troubled that the #s themselves didn't raise flags w/ the authors especially compared to the raw data. They estimated an increase of 177-195 cases for the host Meade county, when the county saw an increase of *29 cases* Aug 3-Sep 2. https://covidactnow.org/us/sd/county/meade_county/?s=1025217
16/ Meade county only had 74 cases *total* up to Sep 2. If someone can explain how a synthetic control model could spit out a result that requires the control counties to have had negative cases, I'm all ears...what am I missing #econtwitter #epitwitter #poptwitter?
17/ . @ashishkjha noticed a similar problem with what's seen in the raw data for the full county analysis--no spikes in counties where the authors say the rally attendees came from, increasing the mystery of where the 266,796 cases could have taken place. https://twitter.com/ashishkjha/status/1303536487259148291
18/ The results show most increased cases in the last 1-2 weeks (especially last week), consistent w/ lag times. Still, the authors state that their estimate of 266,796 cases represents “19% of the 1.4 million cases in the United States between August 2nd and September 2nd.”
19/ In reality, the 1.4 mill. cases over the month is not the right denominator. If most Sturgis induced cases were in the last week, 266K cases represents 45% of cases 2 weeks prior to Sept 2 or *90%* of US cases in the week prior to Sept 2nd.
20/ Does this mean the rally was harmless? Definitely not. Basic transmission knowledge tells us mass gatherings & close contact are risky. Contact tracing reports have identified cases and deaths linked to the event, & this may grow over time. https://www.nbcnews.com/news/us-news/weeks-after-sturgis-motorcycle-rally-first-covid-19-death-reported-n1239088
21/ More broadly, I share @RexDouglass's skepticism that we can learn much from the propagation of aggregate level data analyses in pandemic times. We need individual data from contact tracing & prospective surveys to better understand transmission dynamics. Science is hard!
22/ Finally, at @DearPandemic we are committed to a critical lens regardless of whether it fits our priors. We should be *especially* skeptical of "extraordinary" claims when they fit our existing beliefs, lest we undermine the integrity of the science. https://dearpandemic.org/
