What's the Matter With the Polls?

By Steven Shepard, Politico

04 November 16

The run-up to Election Day might be exposing key flaws in public polling.

n the final days of the presidential race, the public polling has settled into a mostly consistent equilibrium: Hillary Clinton leads Donald Trump both nationally and in most key states, but by a narrower margin than last month, and not by enough to be assured of victory next week.

It’s been a bumpy road to that consensus, however, one marked by wildly careening results in recent days. Campaign professionals say the run-up to the election is exposing key flaws in the public polling conducted by news organizations and academic institutions.

The most obvious flaw is that the polls are overstating sudden fluctuations to the American people, say campaign pollsters whose own data is far more stable — even amid news events that rival the most creative political fiction.

“One of the reasons that I know the polling doesn’t move this much,” said Democratic pollster Nick Gourevitch, “is because any time we do a project internally using many, many interviews — the polling doesn’t move this much.”

It's not a new phenomenon: After the 2012 election, President Barack Obama's campaign bragged about its internal polling, which was far more stable relative to public polling.

While the public polls are almost uniformly more volatile than private data, the ABC News/Washington Post tracking poll — which over the past two weeks resembles a seismograph during a magnitude-8 earthquake — is taking it to another level.

The poll, which has been releasing new results daily, has swung wildly from its launch immediately after the final Clinton-Trump debate to the present time.

In the first iteration of the ABC/Post poll, conducted Oct. 20-23, Clinton led by a commanding 12-point margin, 50 percent to 38 percent. But in the following four-day rolling sample, conducted Oct. 24-27, the race had changed drastically: Clinton led Trump only by 2 points, 47 percent to 45 percent. That’s where the race stood Thursday — Clinton at 49 percent and Trump at 47 percent. At one point earlier in the week, Trump had a 1-point advantage for a day.

Other polls also differed dramatically in October, particularly those in contested states, with some pollsters showing results that swung from survey to survey. A University of New Hampshire survey in mid-October gave Clinton a 15-point lead, but that was down to 7 points in a UNH poll two weeks later.

Now, two polls out Thursday show a neck-and-neck race in the smallest battleground on the swing-state map.

It’s a similar story in Virginia — another battleground state where Clinton has led for most of the campaign, but where new polls this week vary greatly between a solid Clinton lead and a slight Trump edge.

“The state-by-state public polls are just sort of all over the place,” said Robert Blizzard, a pollster at the Republican firm Public Opinion Strategies.

Campaign pollsters point to a handful of factors to explain the volatility of public polling, compared with their own private data.

First, public pollsters don’t apply controls to their samples for partisan inclination. That’s a point of contention: Public pollsters argue that partisanship isn’t a fixed trait and using it as a demographic parameter could predetermine the result the poll is designed to measure.

But campaign pollsters argue that this is a cause of much of the volatility — especially in states where historic turnout rates of registered partisans are relatively consistent. And when partisan inclination bounces around from poll to poll — especially around major news events — it’s mostly a function of voters choosing not to respond. In other words, when there’s bad news about Donald Trump, like his poor debate performances and the leaked audio of Trump describing how he gropes women, Republicans are less likely to participate in polls.

On the other hand, when the news is dominated by stories about Clinton’s email scandal, Democrats don’t want to respond to pollsters.

“Attitudes in high-profile races don’t shift as much as a lot of the polls show. The big question for pollsters is what do you do about that?” asked Gourevitch, the Democratic pollster, who argued against letting partisanship float from poll to poll. “I think you’re in a world where that fundamentally doesn’t work. You have to bring some assumptions to the table.”

There are also differences in sampling, or how phone pollsters assemble the pool of people they call. Campaign pollsters now, almost uniformly, conduct internal polls using lists of registered voters and sometimes filter according to vote history to restrict the survey to more reliable voters.

Some public pollsters, like Monmouth University and Siena College, are also using what is called registration-based sampling. Still, many public pollsters instead use random-digit dialing, contacting Americans by randomly calling phone numbers and then applying a model to discern which respondents are most likely to cast ballots.

Many of those models are based on how enthusiastic voters say they are to support their preferred candidate. When the ABC/Post tracking poll launched, it was the day after the final debate, in which Trump wouldn’t commit to honoring the results of the election, win or lose.

By the time FBI Director James Comey informed Congress his agency was investigation additional Clinton emails, enthusiasm had swung toward Trump, and more of his voters were making it through the likely voter screen.

That’s similar to the nonresponse bias that could result in even more wild swings in the final days. With both parties investing so much to turn out voters in key states — Clinton and Democrats have the more sophisticated operation, by most accounts — even many of the voters who say they are somewhat less enthusiastic might cast ballots on Election Day or in early voting.

“I’m very skeptical of using enthusiasm or strength of support for any kind of likely voter screen,” said Gourevitch.

Ultimately, with the volatile national polling and inconsistent state polling, most practitioners are focused more on their own private data — even though public polling can shape coverage of the race in the final week.

Gourevitch, the Democratic pollster, said the most recent polls were conducted in the wake of Comey’s letter to Congress, when Democrats were likely less motivated to participate. The next wave of polling this weekend may show any effect of Comey’s letter ebbing, or it may show hardening perceptions of both candidates. “I would pay attention to field dates,” he cautioned.

Meanwhile, Blizzard, the Republican pollster, is largely ignoring national polls like the ABC/Post tracking poll, or the one-off CBS News/New York Times poll.

“I don’t really pay as much attention to the national stuff because it’s mostly directional,” Blizzard said, pointing to trends that tend to coincide with movements in the states. “This is not truly a national election.”