I thought I'd do a short thread on why I think "Regression Discontinuity Design" (RDD) is a poor method, especially for noisy data in economics.

I'm going to generate a "time series" where the only changes are pure noise. Here, the underlying variable x is constant in time (mean x = 1), but has normally distributed noise with a standard deviation of 1.

We'll start with a "0th order" regression using the first and second halves of the data. We find a delta that I'll call ΔRDD.

Let's do this 1000 times with randomly generated x data and compare the distribution of ΔRDD to that of a single data point.

When we get that distribution we can see why there are stats tests that compare means — the distribution of differences between those means (red line) is tighter than the distribution of a single data point (gray dashed).

Let's bump the regression order up to linear — this is a typical RDD that can be found in the econ literature.

What's the distribution of ΔRDD?

The distribution of ΔRDD for linear regression is wider than the 0-order regression — in fact, in this case it's close to the same distribution of the error in each individual data point!

If we go to higher order, we can make the effect measured with an RDD look even bigger.

At 2nd order (quadratic regression), the distribution of ΔRDD is actually equal to the distribution of the error in the individual data points in this particular case.

Of course, if you have more data points you can start to shrink that distribution back down.

With 2x the number of data points, we've made the 2nd order regression distribution of ΔRDD look more like the 1st order.

But that's the issue. In econ, you often can't just go get more data — especially in cases of natural experiments. If N = 40, it'll stay N = 40.

This is why I don't believe RDDs using limited data sets.