Read on Twitter
A few key take-aways while @susan_athey presents.
- Adaptive experiments (changing the design as you go) might be a great option for piloting studies.
- In full-scale studies this is much trickier, but there are a few things that can be done. https://twitter.com/MoritzPoll/status/1285968310014812161
- Adaptive experiments (changing the design as you go) might be a great option for piloting studies.
- In full-scale studies this is much trickier, but there are a few things that can be done. https://twitter.com/MoritzPoll/status/1285968310014812161
Basically if you compare several treatment arms and you spread your sample equally, you get moderate precision on all of them. In an adaptive experiment, we identify early on which arms don't work and assign fewer participants to these arms and gain precision on the "winners".
Main issues familiar from the Inference on Winners literature: https://www.nber.org/papers/w25456 Strongly recommend this read for practitioners.
In general, changing your experiment/analysis based on the data you collect is a _no no_ as it opens the door to specification search, p-hacking etc. In fact as the paper above shows, the statistical inference would get ridiculously complex.
One critical problem in practice is that outcomes are often not even measured quickly enough before assigning the next batch. But for now, let's assume that away.
Crucial of course is that timing of study participation is random.
The operations research literature has caught on to this a lot earlier than econ. But econ has a strong edge on structural modeling and the statistical problems surrounding selection problems which comes in handy.
Ok, let's get to the crucial stuff: By reducing assignment to a bad treatment arm, you overweigh the early data which made this arm a "bad" arm in the first place. That's where the bias is coming from.
Some version of propensity weighting can help us out here. IMHO that's where this will become very tricky to implement for people without a strong statistics background. Great place for the authors to offer ready-made code. Susan mentioned a #rshiny app earlier in the talk...
There will be a recording and @RDMetcalfe who as always did a fantastic job moderating can probably point us to it soon.
