Another important paper that I’d have expected to receive more attention and directed effort from methods reform crowd. The piranha problem. This is a theoretical paper so I won’t reproduce the proofs here but I’ll give some highlights below. http://www.stat.columbia.edu/~gelman/research/unpublished/piranhas.pdf

The paper studies the following question: How many explanatory variables can have large effects on an outcome?

When put this way, it may not sound too exciting or relevant because it makes you think of individual papers. Hint: Think of the literature as a whole.

Consider an example like below (however outrageous some of these sound). These papers claim that seemingly trivial or irrelevant factors have large and consistent effects, and this runs into the problem of interactions.

The paper provides proofs regarding an inevitable consequence of having many explanatory variables with large effects: the explanatory variables must have large effects on each other.

Piranha theorem: If there is some outcome on which a large number of studies demonstrate an effect of a novel explanatory variable, then either some of the claimed effects are smaller than claimed or some of the explanatory variables are essentially measuring the same phenomenon.

The paper gives reason to be suspicious of certain types of results. Groups of studies that claim to have found a number of important explanatory variables for a single outcome should be scrutinized, esp. when the dependencies among these variables have not been investigated.

The paper formalizes and proves piranha theorems for correlation, regression, and mutual information. These theorems illustrate the general phenomena at work in any setting with multiple causal or explanatory variables.

Overall these results rule out the possibility of multiple large effects or “piranhas” among a set of random variables. They raise suspicions about a model of social interactions in which many large effects are swimming around, just waiting to be captured in quantitative studies

I find this an important conclusion and a fruitful direction for future research. We see countless papers on a single outcome variable singling out individual factors at a time but not a whole lot of concerted effort at making sense of the collection of such results.

We have many “constructs” potentially serving as piranhas in a small pond but they don’t paint a meaningful picture. We need more sense-making research (beyond meta-analyses) and fewer such effects at this point.