Without taking sides, a few ideas intended to improve the quality of standardized test score debates on my Twitter feed: a thread

To pro-testing folks: I don't see anyone disagreeing that rec letters and grades can be biased. But to convince the anti-testing side that this is a good argument against getting rid of standardized tests, you'd need to convince them that these biases are greater than gaps...

on standardized tests. At least for the SAT they're not; see, e.g., @rothstein_jesse here: https://eml.berkeley.edu/~jrothst/publications/rothstein_sat_jmetrics2004.pdf

Also to pro-testing folks: The quasi-exp lit on policies that make test scores play a smaller role in undergrad admissions indicates that some of the things the anti-test folks hope will happen have happened. Here's a nice summary: https://www.chalkbeat.org/2020/5/18/21262897/coronavirus-sat-act-university-california-research-equity

Some of the relevant arguments on both sides are about what will happen in practice under different counterfactuals. Thus, I think that it's important to read the relevant ed policy literature and not just stick to arguments based on correlation matrices and fairness.

To anti-testing folks: I see too little discussion about what new equilibrium will emerge if these tests are eliminated. Quasi-exps are great for estimating immediate impacts of changes to admissions policies and should be widely read. But what would a reasonable skeptic say?

My best shot: 1) If the changes are going to have a small influence on the composition of incoming cohorts, perhaps we should still make them, but I wish my Twitter feed had more discussion of alternative or supplementary policies. Why might effects be smaller than you'd think?

All sorts of reasons. e.g., in their SAT-skeptical report, 3 UC faculty including Rothstein encouraged the UC system to consider replacing the SAT with state ach test scores, which show gaps but are curriculum based and are free to California students:
https://edpolicyinca.org/sites/default/files/2020-04/uc_regents_admissions_analysis.pdf

2) If we find a way to make changes to incoming cohorts really big, is the quasi-exp evidence an externally valid way of approximating all the ways universities will change? Big changes can have all sorts of consequences I hear little talk of, e.g.:
https://www.journals.uchicago.edu/doi/abs/10.1086/706050?af=R&mobileUi=0&journalCode=jole

Neither is a good reason not to try something different than the status quo. I don't find any of these to be slam dunk arguments for either side. I know thoughtful folks on both sides. I hope they'll try to seek out the strongest arguments from people with whom they disagree.