Read on Twitter
Govts around the world are trying school management reforms. In a new paper using a large-scale RCT in India ( https://bit.ly/35WHq50 ) @singhabhi & I find no evidence of impact from several "best practices"
Yet, the program was scaled up to >600,000 schools (w/ no effect)! 1/
Yet, the program was scaled up to >600,000 schools (w/ no effect)! 1/
The study is based on 7 years of work w several partners:
2 years of a large RCT w/ 1774 treated schools
2 more years evaluating the scaled-up program (w a matched pair design)
& qualitative interviews with officials, head-teachers, teachers to understand what happened 2/
2 years of a large RCT w/ 1774 treated schools
2 more years evaluating the scaled-up program (w a matched pair design)
& qualitative interviews with officials, head-teachers, teachers to understand what happened 2/
What happened?
The school ratings were done and were meaningful, detailed school improvement plans were prepared & uploaded.
But there was no impact on teacher/student attendance, classroom practices, learning.
Paperwork was completed - but nothing substantive changed. 3/
The school ratings were done and were meaningful, detailed school improvement plans were prepared & uploaded.
But there was no impact on teacher/student attendance, classroom practices, learning.
Paperwork was completed - but nothing substantive changed. 3/
Did the program improve during scale-up?
In a matched-pair design (w parallel trends) across 10 districts, we continue to find no evidence of test score gains
Administrative compliance was high & documented in copious paperwork, but there was still no impact on learning
4/
In a matched-pair design (w parallel trends) across 10 districts, we continue to find no evidence of test score gains
Administrative compliance was high & documented in copious paperwork, but there was still no impact on learning
4/
Why do governments adopt and scale ineffective programs?
One possible explanation: institutional isomorphism (DiMaggio & Powell 1983)
Orgs mimic successful peers & "adopt `innovations’ to enhance their legitimacy, to demonstrate they are at least trying to improve.” 5/
One possible explanation: institutional isomorphism (DiMaggio & Powell 1983)
Orgs mimic successful peers & "adopt `innovations’ to enhance their legitimacy, to demonstrate they are at least trying to improve.” 5/
Our experiences also highlight the divergence between perception and reality of program effectiveness
One reason could be that official perceptions are driven by paperwork and administrative compliance, with no visibility on actual impact, as documented by @AkhilGuptaUCLA 6/
One reason could be that official perceptions are driven by paperwork and administrative compliance, with no visibility on actual impact, as documented by @AkhilGuptaUCLA 6/
Why does this matter for research?
As shown by Abadie (2020), well-identified and well-powered null results are especially valuable in settings with widely-held positive priors of impact (seen in this case by the ubiquity of similar programs - see Appendix A) 7/
As shown by Abadie (2020), well-identified and well-powered null results are especially valuable in settings with widely-held positive priors of impact (seen in this case by the ubiquity of similar programs - see Appendix A) 7/
Why does this matter for policy?
There are many policy "reforms" designed by experts & funded by donors - that look excellent on paper & are assessed on metrics such as “people reached” but w no measure of actual impact
High-quality evaluations can discipline such efforts 8/
There are many policy "reforms" designed by experts & funded by donors - that look excellent on paper & are assessed on metrics such as “people reached” but w no measure of actual impact
High-quality evaluations can discipline such efforts 8/
Special kudos to the Govt of MP & @EPG_Edu for being open to the evaluation & possibility of failure, and the stellar @JPAL_SA team of RAs and field staff who held it all together: @GhazalGulati @AditiBhowmick18 @r_sabareesh @krishanu_c @gone_seagull https://twitter.com/Fihi_maFihi/status/1009809583265771520
