Regression Discontinuity fails again

2020 July 5
by Daniel Lakeland

Regression Discontinuity analysis is not a failure as an idea, it’s just a failure as a practical way to learn about the world in most cases. The problem is that most situations in which it’s being applied are noisy human / social studies where the effects are much smaller than the level of noise.

Andrew Gelman picks apart another one of these here.

I’ve been teaching myself to use Julia for all my future data analysis projects. It’s just a fabulous language. So here’s the graphs I came up with:

Years lived post election against percentage point margin.
With LOESS fits using 0.2 0.1 and 0.05 bandwidths

What this shows is … NOTHING. As you decrease the bandwidth you can detect more rapid variation in the function value at the expense of more noise in the estimate. By the time you’re down to bandwidth 0.05 you’re using only 5% of the data to fit any given location in the fit. Right at margin = 0 you can see using the orange or red curves that the estimates are extremely noisy, and certainly nowhere near a 5-10 year bump in longevity moving from left of 0 to right of 0.

Science is broken. Here is Erik defending his study.

As we can clearly see in the raw data, there is no discernible signal, none. So whatever signal is supposedly there, if it’s there it just happens to be *exactly* hidden by the offsetting effect of whatever covariates he’s adjusting for. It just so happens basically that people who won elections didn’t live longer on average than people who lost elections, but if they hadn’t won those elections we have somehow strong evidence that they would have died earlier because they were all sicker people than the losers which we can determine from their covariates…

Whatever. Here’s what it looks like when you have an actual signal… Here I’ve used the same x coords, and then generated random y coordinate Normal(0,s) noise, and then added a signal to it… Three different kinds of signals. In the first graph is a step function that steps up by 5 units as we pass x = 0. The second one adds a little wavelet that decreases right before zero and increases right after zero (negative of the derivative of a gaussian) and the last one is the same as the second one, except confined only to x > 0.

No comments yet

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS