Read on Twitter
New paper by myself and David Robertson. This follows on from some previous work I've done on multiple testing in multi-arm trials. https://onlinelibrary.wiley.com/doi/full/10.1002/pst.2059#.XzVVhqC_0jQ.twitter
Generally there are two main views on multiple testing correction for confirmatory trials testing several distinct treatments vs control.
1) No multiple testing is necessary. If you were planning a trial then you wouldn't be expected to adjust your significance level for other trials ongoing in the same area.
2) The maximum chance of incorrectly recommending an ineffective treatment (the family-wise error rate, FWER) should be controlled. If we have a K-arm trial and only one arm is significant with p-value, say, 0.02, then we'd be suspicious as K gets bigger and bigger.
We propose and investigate a more intermediate approach: controlling the false discovery rate (FDR). The FDR is the proportion of treatments recommended that are actually ineffective.
FDR control is similar to FWER control when most treatments are actually ineffective but more like not correcting when most treatments are actually effective.
Instead of considering type I error and power, we consider positive predictive value (PPV) and negative predictive value (NPV). PPV is probability of a recommended treatment actually being effective. NPV is probability of non-recommended treatment actually being ineffective.
We find that the FDR control has nice properties in terms of PPV and NPV, especially as the number of arms goes up.
