08.26.24
Public Polling Has It All Wrong (Again)
If the November election was held today, public polling would drastically underestimate Trump’s performance and, to a lesser extent, the Republican Party’s as a whole.
I’m not a poll truther. On the contrary, I’m a Republican campaign consultant and pollster who has consumed thousands of pages of data. My biggest takeaway? The survey environment is plagued by a large response bias problem that low-budget public surveys refuse to fix.
The basic science behind a poll is simple: if we interview 800 Pennsylvania voters, each answer should be within three points of the whole universe 95 percent of the time. For example, if a survey of 800 Pennsylvania voters showed Trump at 48%, then his actual support could be as low as 45% or as high as 51%.
But this foundation assumes every respondent is equally likely to take a given survey, which is false. Pollsters have two tools to help them adjust for this reality.
The first is quotas. Pollsters dictate how many individuals of a certain group they want in their survey. I want X percent men and Y% women. I want X percent white people and Y% Black. Nearly every pollster uses quotas for geography, age, and race. The major debate is about setting quotas for things like education level and, most controversially, political party affiliation.
The argument in favor of quotas relies on historical voter turnout to model future turnout. Opponents of restrictive quotas argue that a poll may miss changes in group dynamics or the collapse of a particular group. This debate was less of an issue 20 years ago when everyone had landlines and response rates across all groups were fairly high. Today, with extremely low response rates, the response bias has become very pronounced.
For example, in a recent internal statewide survey in a large state, people with a graduate degree were five and half times more likely to answer the survey than those without a degree, while people with a bachelor’s degree were three times more likely.
This doesn’t take into account a fascinating response bias developing around population density. Pollsters often divide geography into urban, suburban, and rural segments by population density. In recent surveys, there has been a pronounced drop-off in rural, white, working-class responses, with some of the quotas being filled in by suburban and urban white working-class voters who are much more likely to be Democratic in orientation.
Requiring quotas on education helped to fix the polling challenges that arose in 2016 and made public polls more reliable. But Kamala Harris’s appointment to the Democratic nomination has supercharged a key group of voters that can have a big impact on polling: wealthy, educated, white Democratic voters. These voters are crawling across broken glass to respond to political surveys.
In the past, top-level quotas set for the whole survey might help to mitigate this problem but many public pollsters cannot afford to set stratification quotas for all the necessary subgroups. For example, top-level quotas will produce a survey group with 43% Democrats and 40% with a college degree, but you really need to set quotas for how many college-educated people are in the Democratic subgroup or you will not fix the response bias problem. This sort of nuanced stratification is very expensive and most public pollsters aren’t willing or able to pony up.
With Kamala Harris as the nominee, politically engaged, wealthy, educated, white voters are taking up too many spots in the Democratic quotas, pushing out downscale, lower-turnout Democrats who are much more likely to be undecided or Trump voters. Whereas a college-educated Democrat might be 95% for Harris, a non-college one might be 88%. That seven-point gap matters and is not reflected in public polling.
Don’t take just my word for it. POLITICO reported that even Democratic pollsters are admitting their internal (read expensive) surveys are much less optimistic than public polling, and they are also worried about this blue mirage.
Fixing this problem is not easy or cheap, but one possible solution is to look at vote history. Looking at prior high-turnout elections, we can estimate how much of the electorate will be made up of reliable voters who have voted in 100 percent of the last four general elections. For example, in Pennsylvania, we might expect 52 percent of the electorate to be composed of these voters, but polling samples following the Biden-Harris switcheroo show an electorate with 60 percent-plus of these high-frequency voters.
This matters because of an oddity in party and vote history. Right now, support for Harris has a slight correlation to prior vote history where high-turnout Democrats have a higher incidence rate of support for her than low-turnout Democrats. Conversely, the correlation on the GOP side for Trump support is somewhat flat if not the inverse. Very reliable and consistent voters (who tend to be more educated Republicans) are less likely to support Trump while less reliable voters are more likely.
This trend existed previously, but it was supercharged after Joe Biden dropped out and Kamala Harris assumed the Democratic mantle. The result is a blue polling mirage that is more Democratic than Election Day will be.
This brings us to the second tool pollsters use, which is weighting a survey. When you weight a survey you treat interviews unequally to make the data more representative of the expected electorate. For example, if the survey electorate contains only 40 percent of people without a college degree but you expect likely turnout at 51%, you can increase the value of the non-college responses and decrease the value of the college responses to accommodate the disparity.
While weighting can be useful on the margins, it has significant limitations because you are using a very small group to extrapolate to a larger group. The classic example of this is Black voters. African Americans tend to be underrepresented in Pennsylvania statewide surveys if quotas are not used. If you weight 50 Black interviews with a margin of error of 13.5 percent to equal 80 interviews, you end up with a lot of risk and an unreliable result.
Many public polls right now are likely oversampling highly educated Democrats, and very high likelihood to turnout voters. They are not capturing the full electorate and while this may fix itself with rising response rates throughout the fall as we approach the election, it is a real problem for the polling industry to address.
So as the fall progresses, keep an eye on the public polling because it is likely once again to significantly understate the support for President Trump and Republicans.
Mark Harris is a founding partner at ColdSpark. You can follow him on X here.