"Policy evaluation in COVID-19: A graphical guide to common design issues" is out!

Dream team of coauthors: Emma Clarke-Deelder ( @Emma_C_Clarke), Joshua A Salomon ( @SalomonJA), Avi Feller ( @AviFeller), and Elizabeth A Stuart ( @Lizstuartdc)

https://arxiv.org/abs/2009.01940 

THREAD !

We have lot of papers asking how well did/do all of those non-pharmaceutical intervention (NPI) policies work for COVID-19 (i.e. what is their causal impact?).

As y'all know, causal inference is hard™, COVID-19 makes it a bit harder, and not all those analyses are great.

Lots of great work on how to DO policy evaluation, but what about how to recognize common issues in EXISTING policy evaluation papers?

Some kind of easy to understand guide for decision-makers and reviewers with a checklist and graphics and such, perhaps?

Don't worry, we got you.

This paper is guidance on how common policy evaluation designs are supposed to work and how to find many all-too-common pitfalls for COVID-19, focusing mainly on pre/post, interrupted time-series, and difference-in-differences based designs.

There's a lot of stuff to look out for! Concurrent changes in behaviors and other policies, time-lag effects, linear or other functional form modelling, tigers, bears, OH MY!

This paper breaks it down and gives you the most important bits you need to know for each design.

As a teaser, let's talk about interrupted time-series (ITS).

The idea of ITS is to make your counterfactual (i.e. what would have happened had the intervention not happened) from what happened in the past before the intervention, by projecting it forward.

Simple right? I wish.

The assumed model of that projection is really important. Most commonly, people will use some sort of linear projection (i.e. a straight line) continuing from before the intervention the outcome.

ITS depends on how reasonable that model is. And that's super tricky for COVID-19.

For infectious disease outcomes, a linear projection (scale dependent) might not be a good choice.

Think about phrases like "flatten the curve," "exponential growth," and "s-curves." Those all describe NON-linear dynamics.

So, is the model justified or justifiable?

What about timing?

Think about a typical policy. First, the policy happens, then people change behaviors, then the behavior change causes change in infection rates, then people become symptomatic, then they get a test, then they/we get compiled results.

That's a LOT of time!

Those time periods aren't fixed; they tend to be spread out. They can also go the opposite way (e.g. people changing behavior in anticipation of a policy). Date of implementation may not be the right date of expected impact.

Tricky stuff, but the next one's maybe even trickier.

What about concurrent changes? Is the policy of interest the ONLY thing influencing the outcome (say, infection rate)? What about all the other behavioral changes, economic changes, other policies, etc?

LOTS of stuff was happening all at once, and it all likes to party together.

And here it all is, organized into a convenient checklist in addition to the graphical representation.

But what about, say a difference-in-differences style analysis? Well, you're just gonna have to read the paper :).

Most importantly, this is just a sniff test; it doesn't deal with data quality issues, statistics, etc. That all needs to check out too, but this will get you pretty far without having to dig very deep.

But first, get sniffing, because it's ain't all roses out there.

I also can't say enough about my coauthors here. This was a fantastically fun paper to work on, with everyone putting a lot of themselves into it and pulling it all together in an extremely short time.

Really and truly: this was AWESOME to work on.

But what about you, dear reader?

I want you to tear this apart, and critique the ever living whatever out of it. Is something unclear? Wrong? Missing? We want to know!!

This is getting submitted next week, but there's time for edits. Would LOVE your feedback (e-mail in paper)!

.... and folks, I promise there are errors to be found (I just noticed two, whoops), and lots of improvements to be made.