Read on Twitter
I'm kind of surprised I haven't seen an article in @FiveThirtyEight or somewhere similar using the Total Survey Error framework as a tool for categorizing potential sources of survey error and tests for those hypotheses as election results get finalized.
When survey researchers hear about "Shy Trump" voters, we hear it as measurement error, and there's good evidence that it's vanishingly small. But I think the broader public might also be including non-response error as part of how they understand that term
As we unpack the sources of survey error, it's worth keeping our eye on some patterns. For instance, this comparison of survey averages to projected results by @gelliottmorris shows a correlation between 2016 and 2020, but also an intercept shift
Putting out some hypotheses now that can be tested as vote counts get finalized (still not done counting!) and pollsters look back at their own data
1) Everything @davidshor says here about trust and non-response: https://www.vox.com/policy-and-politics/2020/11/10/21551766/election-polls-results-wrong-david-shor
It's tricky to test for, because surveys aren't asking about trust and we don't have great national benchmarks either. That said ...
It's tricky to test for, because surveys aren't asking about trust and we don't have great national benchmarks either. That said ...
... I'll be interested to see patterns of non-response and comparisons of crosstabs to vote shares by urbanicity / rurality, as well as by socioeconomic breakdown of vote share once we have reliable academic post-election surveys
In some ways, the education weighting that was widely adopted this cycle may have been a proxy for trust, but also it seems to have been insufficient (if within each education cohort low trust people both respond at different rates and vote differently than high trust people)
2) Coverage. It seems from preliminary analysis in the press that Trump may have overperformed polls heavily in lower-income areas. I wonder about match rates to the voter file from RBS polling among these voters, since residential mobility makes this data matching harder.
This is going to be just one of many cases where I wish we still had http://Pollster.com 's averages where you could construct custom average by pollster, to compare RDD and RBS as a start
(To be clear, RDD has it's own coverage issues; limiting to a cell phone area code associated with a state both misses people with out-of-state cell phones, and includes people who have moved out of the state since getting that phone number)
3) How researchers handled early voting. One thing I'm looking for is the relationship between poll error and % voting early.
Given the new partisan split in early vote this year, if the % reporting having voted early overstates the actual share of early votes, this could over-represent the proportion of Democrats in the projected electorate
This could happen in a few ways. Some people could have thought they early voted, but their ballot was not accepted (we know this happened for at least some people). Some people could have said they early voted to try to end the survey faster (this happens).
Or more of the people who said they were likely to vote (but not certain) and support Trump could have ended up casting a ballot, but had that uncertainty about turnout screen them out of inclusion in likely voter samples.
4) (Fine, we'll talk about it). Measurement error, specifically Shy Trump voters. This strikes me as unlikely, in part because of @aecoppock's excellent list experiment last cycle that found no Shy Trump voters then https://alexandercoppock.com/coppock_2017a.pdf
I'd also want to compare results by mode, with the notion that web and IVR are more anonymous that live interviewer phone (again, pour one out for http://Pollster.com 's averages). But as far as I understand, there wasn't a big mode effect this year in presidential polling?
So yes, (some) polls were off in 2020, for sure. But not all polls were off equally everywhere for all portions of the electorate, and looking at that variation can help understand the sources of polling error <end>
