Read on Twitter
RCTs tell us 'what works', right? They give us unbiased causal estimates, right?
New paper with @jakeanders @mjinglis & Hugo Lortie-Forgues: https://bit.ly/32VdEvL
New paper with @jakeanders @mjinglis & Hugo Lortie-Forgues: https://bit.ly/32VdEvL
It's increasingly recognised that RCTs are often underpowered (too small) to detect effects of the size we find in education: https://bit.ly/3pDeiYA
It's less well known that underpowered RCTs also lead to exaggerated effect sizes (ES) among RCTs with promising results...
It's less well known that underpowered RCTs also lead to exaggerated effect sizes (ES) among RCTs with promising results...
We ask: If an RCT yields 'promising' results (eg, ES=0.15, p<0.05), what does that really tell us?
Our answer: ES from such trials are exaggerated by 50% on average, and the probability that the true ES=0 is surprisingly large - certainly much larger than 5%.
Our answer: ES from such trials are exaggerated by 50% on average, and the probability that the true ES=0 is surprisingly large - certainly much larger than 5%.
Besides these empirical findings, our paper tries to provide a clear explanation of how this happens in low powered trials - including nice diagrams 
You can read the @CEPEO_UCL working paper here: https://repec-cepeo.ucl.ac.uk/cepeow/cepeowp20-16.pdf
You can read the @CEPEO_UCL working paper here: https://repec-cepeo.ucl.ac.uk/cepeow/cepeowp20-16.pdf
This is a problem for edu research:
- Clearinghouses/toolkits of effective practices often rely on results from single promising RCTs. Are they reliable?
- Exaggerated ES are a recipe for RCT findings that don't replicate. Are we misallocating funding to scale-up trials?
- Clearinghouses/toolkits of effective practices often rely on results from single promising RCTs. Are they reliable?
- Exaggerated ES are a recipe for RCT findings that don't replicate. Are we misallocating funding to scale-up trials?
So what should we do?
1) Increase RCT power (hard but essential)
2) Use other designs (e.g. matching) alongside RCTs
3) Use 'design analysis' when planning and when reporting results from RCTs: https://bit.ly/3fapnv2
1) Increase RCT power (hard but essential)
2) Use other designs (e.g. matching) alongside RCTs
3) Use 'design analysis' when planning and when reporting results from RCTs: https://bit.ly/3fapnv2
