Eugene Volokh quotes extensively from a new paper by Cass Sunstein and Adrian Vermeule that presents an argument for the death penalty based on recent studies that find it has a deterrent effect on potential murderers. In particular:

Disaggregating the data on a state by state basis, Joanna Shepherd finds that the nation-wide deterrent effect of capital punishment is entirely driven by only six states … [The states] showing a deterrent effect are executing more people than states that do not. In fact the data show a “threshold effect”: deterrence is found in states that had at least nine executions between 1977 and 1996. In states below that threshold, no deterrence can be found. This finding is intuitively plausible. Unless executions reach a certain level, murderers may act as if the death is so improbable as not to be worthy of concern. Her main lesson is that once the level of executions reaches a certain level, the deterrent effect of capital punishment is substantial.

This is an elegant idea, but trouble with it is that only few states execute anyone in a given year. Most execute no-one. A tiny few—notably Texas—kill a lot of people in some years. As a result, evidence for a threshold deterrent effect depends on a very small number of observations. In a nice analysis of state-level data from 1977 to 1997, Richard Berk shows that just eleven state-year observations out of a thousand drive the deterrent effect. It’s possible to mess around with the specification a bit to get a less strongly skewed measure (by standardizing the number of executions by the number of death sentences, say) or making the data more fine-grained so that you have more observations (using county-quarters as a unit, for instance), but in the end its hard to escape the worry that about 1 percent of the observations are behind the results.

We’re probably witnessing the birth of a dubious stylized fact about deterrence and the death penalty. I don’t doubt that the Sunstein and Vermeule paper raises a bunch of interesting questions, but the empirical results they rely on just don’t seem that robust. This is a bit ironic given their argument that “The widespread failure to appreciate the life-life tradeoffs involved in capital punishment may depend on cognitive processes that fail to treat ‘statistical lives’ with the seriousness that they deserve.” One of these processes is the tendency to latch on to a cool finding a bit too quickly. Negative results (like the ones reported in Berk’s paper) are just not as interesting, unfortunately.