Did someone say controlled direct effects? This is my jam.

Jonathan does a good job highlighting issues with the linked paper, but I wanted to shed light on their main assumption under the best possible scenario for the authors. https://twitter.com/jonmummolo/status/1275790509647241222

Basically, they want to say that we are interested in the effect of, say, prosecutor-perceived race for which it is ok to control for arrest because arrest is pretreatment to that perception. Fine, that’s an interesting model to investigate.

Let’s make the cleanest possible DAG for this and assume that prospective-perceived race (Z) is randomly assigned conditional on officer-perceived race (D). We assume D affects arrest (M) which obviously affects the final outcome, charging by the prosecutor (Y)

If you could condition on officer perceived race here, you’d be done. Great. But you don’t have that information so you condition on arrest. If this DAG is correct, then you are still fine. You blocked the back door path from treatment to outcome. But....

What if there is an unmeasured confounded between arrest and being charged? This, to me sounds very plausible! But this would violate the paper’s main identifying assumption (subset ignorability) and will bias the effect of prosecutor-perceived race.

Why, friends, does this cause bias? Because by conditioning on arrest you’ve opened the back door path

Z D M U Y

And, boom, you’ve got some M-bias. In short, potential outcomes are correlated with Z conditional on M under this DAG, so subset ignorability can’t hold.

And again, this doesn’t even require posttreatment bias. It’s the kind of example that has fueled the DAG vs potential outcomes debate for years. The proofs in Gaebler et al work because the key assumption rules out this type of confounding.

The implicit assumption in subset ignotability is that X is sufficient to control for confounding for prosecutor perceived race and the outcome **and** sufficient to control for any confounding between arrest and the outcome. Otherwise, subset ignorability doesn’t hold.

So, I’m not sure this statement from one of the authors can be sustained. You do need to know something about the arrest-charging confounding to make all of this work. https://twitter.com/5harad/status/1275931536710230017

You could make the argument that those unmeasured confounders aren’t plausible and maybe we’d all disagree about that, fine. But subset ignorability isn’t as innocuous as it seems on first glance, even if Z doesn’t affect M.

What’s the bottom line? Causal inference with obsevational data is very difficult and it gets even harder when you are forced to condition on a variable (posttreatment or not). Also, DAGs have their uses!

PS, it’s late, it’s quarantine, and this was all quickly written, so take it with a grain of salt. Maybe I missed something in the paper. But hopefully this helps some folks think through these arguments.