Analysis: How to think about polling after 2020
For some, this might mean we shouldn’t pay attention to polling.
To me, to do so would be a mistake. Polling is still one of the best tools for understanding public opinion. While pollsters likely have some work to do to improve accuracy, those who ask for precision from polling are simply asking too much.
A look at the long arc of history suggests that the polling errors of the last few years should not be too surprising. In some ways, you would be right to call them average.
Let’s start with what I think is the most important error of the 2020 election: the national polling. Yes, elections are ultimately state affairs, but most polls we use are national surveys. They help us understand where Americans stand on electoral issues and well as those outside of this realm.
Obviously, we’d prefer the polling to be better. It’s not good that the polling in two elections in a row underestimated Trump.
But we should be clear about the extent of the error we’re talking about here. The polls (when allocating undecideds evenly and concentrating on the major party candidates) had Biden winning approximately 54% to 46%. He won on the order of 52.25% to 47.75%.
Most issues, however, are not so evenly divided.
If you were to construct a 95% confidence interval for the “true margin of error” since 1936, it would be about +/- 10 points. The national error this year wasn’t anywhere close to that.
Historically, polling errors were greatest from 1936 to 1952, when scientific polling was first getting started. The polls also seemed to consistently overestimate one side in that era (Republicans).
Pollsters probably need to and will likely figure out how exactly to deal with those issues.
For now, analysts, journalists and consumers have to recognize that merely repeating the sampling margin of error produced on major polls likely isn’t likely going to cut it. There are other sources of error not incorporated by the sampling margin of error that can rear their head.
This is especially true on the state level where the state polling left much to be desired in 2020, as it did four years ago.
The 2020 state polling was nearly the worst on record since 1972. Only 1980 with an average error of 5.6 points was worse.
More worrisome, the direction of the errors (Republican underestimation) was the same in both 2016 and 2020.
This is absolutely something that must be examined more, and it’s likely to be by pollsters and their critics.
Still, these errors shouldn’t be shocking.
Since 1972, the 95% confidence interval for state polling averages in competitive states has been a little more than +/- 9 points. We had two competitive states (Montana and Utah) exceed that, and both by less than 2 points. We should have had about one based on history.
So what does this mean for consumers?
When one side is ahead of another by more than a few points, that advantage is probably real, even on the state level. It’s not until that lead is closer to double-digits that we know that edge is real outside of rare circumstances.
Perhaps the best way to think about the state polling (and polling at large) is to borrow a phrase from the Trump era: take it seriously, but not too literally.
