Read on Twitter
Running evaluations for policy makers, as opposed to grumpy seminar peeps or the dreaded Reviewer #2? In a new @IDinsight working paper (w/ Torben Fischer and @dougj8921), we argue that this calls for diff methods. Frequentist and Bayesian both! 
1/ https://medium.com/idinsight-blog/informing-specific-decisions-with-rigorous-evidence-5b5be1452b7a
1/ https://medium.com/idinsight-blog/informing-specific-decisions-with-rigorous-evidence-5b5be1452b7a
"Standard" approaches to impact evaluations assume a status quo of doing nothing and a skeptical (probably academic) audience hence the null of no effect and relatively high standards to reject null (P<.05). But policy makers frequently have a different decision framework! 2/
Also standard approaches can break down in certain common situations. For instance, what if a policy-maker wants to scale a program if its effect is above X? Or if it just simply "not worse" than another status quo program? Setting up relevant hypothesis tests can be tricky. 3/
We first provide suggestions to tweak the frequentist approach to map to complicated decision frameworks. For instance, when to use one-sided tests, multiple tests, or lower levels of certainty when determining "success." 4/n
But the real benefit of mapping to a decision-maker is that it allows the researcher to conduct Bayesian analysis, incorporating the priors of the decision-maker. We provide an introduction to Bayesian impact evaluation, along with practical tools. 5/n
Even better, Bayesian analysis allows the researcher to make really clear statements to a policy-maker like: "We think there is a 92% chance the your intervention increased farmer yields by at least 1 ton/ha." Your (frequentist) stats teacher would yell at you for this! 6/n
Other Bayesian goodies: we lay out an approach to sample size calculations, and show how they are affected by priors. We show how to define priors and provide example code (R and Stan) to help Bayesian newbies (like me!) get up and running. 7/n
We hope this paper is helpful to practitioners and happy to receive feedback! Tagging a few people who have similar work and might be interested: @dmckenzie001 @economeager @maxkasy @evavivalt 8/8
Other people that may be interested in this thread are #baysiananalysis and #EconTwitter 9/9
Link to working paper here: https://www.idinsight.org/reports-2/informing-specific-decisions-with-rigorous-evidence
