Read on Twitter
I wanted to give a little more context to our ( @KatrinaVelle
@MullinsLab @FritzLaylin) paper about #SuperPlots. So here's a late tweetorial. 1/19
Preprint: https://arxiv.org/abs/1911.03509
Published in JCB: https://doi.org/10.1083/jcb.202001064
@MullinsLab @FritzLaylin) paper about #SuperPlots. So here's a late tweetorial. 1/19
Preprint: https://arxiv.org/abs/1911.03509
Published in JCB: https://doi.org/10.1083/jcb.202001064
I’ve noticed a promising trend away from bar graphs in the cell biology literature. That’s great, because reporting simply the average and standard deviation or standard error of the mean or an entire dataset conceals a lot of information. 2/19
So it’s nice to see column scatter, beeswarm, violin, and other plots that show the distribution of the data. Folks like @T_Weissgerber have helped spearhead this with her great paper "Beyond Bar and Line Graphs." 3/19 https://twitter.com/T_Weissgerber/status/1192694904603992064
But a concerning outcome of this trend is that, when authors decide to plot every measurement or every cell as a separate datapoint, it seems to trick some people into thinking that each cell is an independent sample. The result is beeswarm plots like this with tiny P values.4/19
Clearly, two cells in the same flask treated with a drug are not independent tests of whether the drug works: the drug has only been applied once, and there are many reasons the cells in one particular flask might be slightly different from those in another flask. 5/19
To really test a hypothesis that the drug influences the cells, one must repeat the drug treatment multiple times and check if the observed effect happens repeatably. We all know this, so we repeat our experiments. 6/19
But when we go to calculate standard error of the mean and P values, if we use the number of cells as the "n" instead of the number of independent experiments, the result is tiny error bars and ridiculously small p values. 7/19
Each cells as an "n" wrong for two reasons. 1) Cells are not independent tests. A small p value means the data doesn't fit the model well. Part of the model is the null hypothesis, but a small P value may also indicate that another assumption is violated, like independence. 8/19
2) Large sample sizes are likely to reject the null hypothesis even with very small effects. If that effect is from some subtle bias (e.g. data collection was not blinded) or some other artifact, the P value can still be very small if we collect thousands of data points. 9/19
(That's not to say that large sample sizes are bad, simply that we need to be that much more careful in our experimental design when trying to pull out very small effect sizes.) 10/19
Note that all the different experimental scenarios on the right—including random flask-to-flask variability in D—can yield the same plots and tiny error bars in A when we treat each cell as an "n": 11/19
Treating dependent samples as independent is sometimes called "pseudoreplication," and there is ample literature on this problem. 12/19
https://doi.org/10.2307/1942661
https://doi.org/10.1186/1471-2202-11-5 https://doi.org/10.1371/journal.pbio.2005282
https://doi.org/10.2307/1942661
https://doi.org/10.1186/1471-2202-11-5 https://doi.org/10.1371/journal.pbio.2005282
The solution is to either pool dependent samples together and get one average value, or to perform hierarchical statistical analyses. Both approaches avoid false positives; hierarchical analyses are more powerful, but pooling is very simple. 13/19 https://doi.org/10.1523/JNEUROSCI.0362-10.2010
What hasn't been well addresses is a simple way to display both dependent and independent samples in one graph. We often do care about the range of behaviors observed in individual cells. So condensing hundreds of cell measurements to one dot is correct, but unsatisfying. 14/19
Variability cell-to-cell is not unimportant! The fact that some cells in a flask react dramatically to a treatment and others carry on just fine might have very important implications in an actual body. 15/19
So we proposed #SuperPlots, which superimpose sample-to-sample summary data on top of the cell-level distribution. It's a simple way to convey both variability of the underlying data and the repeatability of the experiment. Here's pooled averages overlaid on a violin plot. 16/19
We also propose color-coding each cell-level datapoint by the experimental run it came from, so the reader can see if there if the cell variability is clustered or dispersed. 17/19
On the simplest level, you can paste two (or more!) plots in Illustrator and overlay them. Play around with colors and transparency to make it visually appealing, and you’re done! (We also give a tutorial on how we made SuperPlots in R, Python, and Graphpad Prism.) 18/19
Thanks for reading! Please check the citations in our paper for a lot more detail about statistical analysis and experimental design. I'm no expert, but we cite many experts who have delved into this topic. And see also: https://twitter.com/samjlord/status/1260720989844893696 19/19
