When I teach cross validation in my Intro to Statistical Learning course, I literally spend a class on “potential pitfalls of CV” and this is the main error I talk about. Happens all the time in published biology literature- not just for methylation data

1/ https://twitter.com/jmschreiber91/status/1291161574393221123

This error is particularly stressful to me as a statistician, because it means that my data analysis can be totally wrong due to data pre-processing that may have been performed before I ever saw the data and that I don’t know about

2/

For example, what if my collaborator measured data on 400k features. But before giving me the data they screened features based on correlation with response, and then gave me only 40k features most correlated with response

3/

Now any supervised analysis that I do to the data will have awesome-looking cross validation error.... but this is just an artifact of the fact that the features were screened USING THE WHOLE DATA SET before I saw it

4/

This is the stuff of nightmares— it means my whole analysis is garbage. And the scariest part is that I, as a statistician, have no way to know that this screening happened before I got the data. (This is why good communication with scientific collaborators is so important!)

5/

Bottom line is: if you are going to eventually use CV (or validation set approach) to estimate test error, you need to perform ALL your data analysis that involves the response Y using **just the training set folds**. This is harder than it sounds, but critically important.

6/

And this error is virtually unidentifiable to reviewers because Methods sections usually don’t get into this level of detail. And they make papers MORE likely to be published since they lead to (false) positive results.

7/

So it’s up to you to do good science for the love of good science

... and to avoid giving your statistician friends (and @jmschreiber91) nightmares

8/8