Sunday, November 11, 2012

Which Polls Fared Best (and Worst) in the 2012 Presidential Race


NYT


As Americans’ modes of communication change, the techniques that produce the most accurate polls seems to be changing as well. In last Tuesday’s presidential election, a number of polling firms that conduct their surveys online had strong results. Some telephone polls also performed well. But others, especially those that called only landlines or took other methodological shortcuts, performed poorly and showed a more Republican-leaning electorate than the one that actually turned out.
Our method of evaluating pollsters has typically involved looking at all the polls that a firm conducted over the final three weeks of the campaign, rather than its very last poll alone. The reason for this is that some polling firms may engage in “herding” toward the end of the campaign, changing their methods and assumptions such that their results are more in line with those of other polling firms.
There were roughly two dozen polling firms that issued at least five surveys in the final three weeks of the campaign, counting both state and national polls. (Multiple instances of a tracking poll are counted as separate surveys in my analysis, and only likely voter polls are used.)
For each of these polling firms, I have calculated the average error and the average statistical bias in the margin it reported between President Obama and Mitt Romney, as compared against the actual results nationally or in one state.
For instance, a polling firm that had Mr. Obama ahead by two points in Colorado — a state that Mr. Obama actually won by about five points — would have had a three-point error for that state. It also would have had a three-point statistical bias toward Republicans there.
The bias calculation measures in which direction, Republican or Democratic, a firm’s polls tended to miss. If a firm’s polls overestimated Mr. Obama’s performance in some states, and Mr. Romney’s in others, it could have little overall statistical bias, since the misses came in different directions. In contrast, the estimate of the average error in the firm’s polls measures how far off the firm’s polls were in either direction, on average.
Among the more prolific polling firms, the most accurate by this measure was TIPP, which conducted a national tracking poll for Investors’ Business Daily. Relative to other national polls, their results seemed to be Democratic-leaning at the time they were published. However, it turned out that most polling firms underestimated Mr. Obama’s performance, so those that had what had seemed to be Democratic-leaning results were often closest to the final outcome.
Conversely, polls that were Republican-leaning relative to the consensus did especially poorly.
Among telephone-based polling firms that conducted a significant number of state-by-state surveys, the best results came from CNN, Mellman and Grove Insight. The latter two conducted most of their polls on behalf of liberal-leaning organizations. However, as I mentioned, since the polling consensus underestimated Mr. Obama’s performance somewhat, the polls that seemed to be Democratic-leaning often came closest to the mark.
Several polling firms got notably poor results, on the other hand. For the second consecutive election — the same was true in 2010 — Rasmussen Reports polls had a statistical bias toward Republicans, overestimating Mr. Romney’s performance by about four percentage points, on average. Polls by American Research Group and Mason-Dixon also largely missed the mark. Mason-Dixon might be given a pass since it has a decent track record over the longer term, while American Research Group has long been unreliable.
FiveThirtyEight did not use polls by the firm Pharos Research Group in its analysis, since the details of the polling firm are sketchy and since the principal of the firm, Steven Leuchtman, was unable to answer due-diligence questions when contacted by FiveThirtyEight, such as which call centers he was using to conduct the polls. The firm’s polls turned out to be inaccurate, and to have a Democratic bias.
It was one of the best-known polling firms, however, that had among the worst results. In late October, Gallup consistently showed Mr. Romney ahead by about six percentage points among likely voters, far different from the average of other surveys. Gallup’s final poll of the election, which had Mr. Romney up by one point, was slightly better, but still identified the wrong winner in the election. Gallup has now had three poor elections in a row. In 2008, their polls overestimated Mr. Obama’s performance, while in 2010, they overestimated how well Republicans would do in the race for the United States House.
Instead, some of the most accurate firms were those that conducted their polls online.
The final poll conducted by Google Consumer Surveys had Mr. Obama ahead in the national popular vote by 2.3 percentage points – very close to his actual margin, which was 2.6 percentage points based on ballots counted through Saturday morning.
Ipsos, which conducted online polls for Reuters, came close to the actual results in most places that it surveyed, as did the Canadian online polling firm Angus Reid. Another online polling firm, YouGov, got reasonably good results.
The online polls conducted by JZ Analytics, run by the pollster John Zogby, were not used in the FiveThirtyEight forecast because we do not consider their method to be scientific, since it encourages voters to volunteer to participate in their surveys rather than sampling them at random. Their results were less accurate than most of the online polling firms, although about average as compared with the broader group of surveys.
We can also extend the analysis to consider the 90 polling firms that conducted at least one likely voter poll in the final three weeks of the campaign. One should probably not read too much into the results for the individual firms that issued just one or two polls, which is not a sufficient sample size to measure reliability. However, a look at this broader collective group of pollsters, and the techniques they use, may tell us something about which methods are most effective.
Among the nine polling firms that conducted their polls wholly or partially online, the average error in calling the election result was 2.1 percentage points. That compares with a 3.5-point error for polling firms that used live telephone interviewers, and 5.0 points for “robopolls” that conducted their surveys by automated script. The traditional telephone polls had a slight Republican bias on the whole, while the robopolls often had a significant Republican bias. (Even the automated polling firm Public Policy Polling, which often polls for liberal and Democratic clients, projected results that were slightly more favorable for Mr. Romney than what he actually achieved.) The online polls had little overall bias, however.
The difference between the performance of live telephone polls and the automated polls may partly reflect the fact that many of the live telephone polls call cellphones along with landlines, while few of the automated surveys do. (Legal restrictions prohibit automated calls to cellphones under many circumstances.)
Research by polling firms and academic groups suggests that polls that fail to call cellphones may underestimate the performance of Democratic candidates.
The roughly one-third of Americans who rely exclusively on cellphones tend to be younger, more urban, worse off financially and more likely to be black or Hispanic than the broader group of voters, all characteristics that correlate with Democratic voting. Weighting polling results by demographic characteristics may make the sample more representative, but there is increasing evidence that these weighting techniques will not remove all the bias that is introduced by missing so many voters.
Some of the overall Republican bias in the polls this year may reflect the fact that Mr. Obama made gains in the closing days of the campaign, for reasons such as Hurricane Sandy, and that this occurred too late to be captured by some polls. In the FiveThirtyEight “now-cast,” Mr. Obama went from being 1.5 percentage points ahead in the popular vote on Oct. 25 to 2.5 percentage points ahead by Election Day itself, close to his actual figure.
Nonetheless, polls conducted over the final three weeks of the campaign had a two-point Republican bias overall, probably more than can be explained by the late shift alone. In addition, likely voter polls were slightly more Republican-leaning than the actual results in many races in 2010.
In my view, there will always be an important place for high-quality telephone polls, such as those conducted by The New York Times and other major news organizations, which make an effort to reach as representative a sample of voters as possible and which place calls to cellphones. And there may be an increasing role for online polls, which can have an easier time reaching some of the voters, especially younger Americans, that telephone polls are prone to miss. I’m not as certain about the future for automated telephone polls. Some automated polls that used innovative strategies got reasonably good results this year. SurveyUSA, for instance, supplements its automated calls to landlines with live calls to cellphone voters in many states. Public Policy Polling uses lists of registered voters to weigh its samples, which may help to correct for the failure to reach certain kinds of voters.
Rasmussen Reports uses an online panel along with the automated calls that it places. The firm’s poor results this year suggest that the technique will need to be refined. At least they have some game plan to deal with the new realities of polling. In contrast, polls that place random calls to landlines only, or that rely upon likely voter models that were developed decades ago, may be behind the times.
Perhaps it won’t be long before Google, not Gallup, is the most trusted name in polling.

State-by-State Probabilities

Electoral Vote Distribution

The probability that President Obama receives a given number of Electoral College votes.
150
210
270
330
390
5%
10%
15%
20% probability

Tipping Point States

The probability that a state provides the decisive electoral vote.
1 Ohio 49.8%
6 Wis. 5.9%
11 Mich. 0.3%
2 Va. 12.3%
7 Pa. 3.3%
12 Ore. 0.3%
3 Nev. 9.9%
8 N.H. 3.2%
13 N.M. 0.2%
4 Iowa 6.6%
9 Fla. 1.2%
14 N.C. 0.2%
5 Colo. 6.4%
10 Me. Dist. 2 0.3%
15 Minn. 0.1%

Return on Investment Index

The relative likelihood that an individual voter would determine the Electoral College winner.
1 Nev. 13.2
6 Colo. 3.5
11 Ore. 0.2
2 Ohio 11.8
7 Wis. 2.7
12 Fla. 0.2
3 N.H. 6.1
8 Me. Dist. 2 1.2
13 Mich. 0.1
4 Iowa 5.8
9 Pa. 0.7
14 N.C. 0.1
5 Va. 4.4
10 N.M. 0.4
15 Neb. Dist. 2 0.1

Scenario Analysis

How often the following situations occurred during repeated simulated elections.
  • Electoral College tie (269 electoral votes for each candidate) 0.2%
  • Recount (one or more decisive states within 0.5 percentage points) 6.4%
  • Obama wins popular vote 86.2%
  • Romney wins popular vote 13.8%
  • Obama wins popular vote but loses electoral college 0.6%
  • Romney wins popular vote but loses electoral college 5.3%
  • Obama landslide (double-digit popular vote margin) 0.3%
  • Romney landslide (double-digit popular vote margin) <0 .1=".1" span="span">
  • Map exactly the same as in 2008 0.1%
  • Map exactly the same as in 2004 <0 .1=".1" span="span">
  • Obama loses at least one state he carried in 2008 99.6%
  • Obama wins at least one state he failed to carry in 2008 4.2%

State-by-State Probabilities

Electoral Vote Distribution

The probability that President Obama receives a given number of Electoral College votes.
150
210
270
330
390
5%
10%
15%
20% probability

Tipping Point States

The probability that a state provides the decisive electoral vote.
1 Ohio 49.8%
6 Wis. 5.9%
11 Mich. 0.3%
2 Va. 12.3%
7 Pa. 3.3%
12 Ore. 0.3%
3 Nev. 9.9%
8 N.H. 3.2%
13 N.M. 0.2%
4 Iowa 6.6%
9 Fla. 1.2%
14 N.C. 0.2%
5 Colo. 6.4%
10 Me. Dist. 2 0.3%
15 Minn. 0.1%

Return on Investment Index

The relative likelihood that an individual voter would determine the Electoral College winner.
1 Nev. 13.2
6 Colo. 3.5
11 Ore. 0.2
2 Ohio 11.8
7 Wis. 2.7
12 Fla. 0.2
3 N.H. 6.1
8 Me. Dist. 2 1.2
13 Mich. 0.1
4 Iowa 5.8
9 Pa. 0.7
14 N.C. 0.1
5 Va. 4.4
10 N.M. 0.4
15 Neb. Dist. 2 0.1

Scenario Analysis

How often the following situations occurred during repeated simulated elections.
  • Electoral College tie (269 electoral votes for each candidate) 0.2%
  • Recount (one or more decisive states within 0.5 percentage points) 6.4%
  • Obama wins popular vote 86.2%
  • Romney wins popular vote 13.8%
  • Obama wins popular vote but loses electoral college 0.6%
  • Romney wins popular vote but loses electoral college 5.3%
  • Obama landslide (double-digit popular vote margin) 0.3%
  • Romney landslide (double-digit popular vote margin) <0 .1=".1" span="span">
  • Map exactly the same as in 2008 0.1%
  • Map exactly the same as in 2004 <0 .1=".1" span="span">
  • Obama loses at least one state he carried in 2008 99.6%
  • Obama wins at least one state he failed to carry in 2008 4.2%

State-by-State Probabilities

Senate Seat Distribution

The probability that the Republican or Democratic party controls the senate by a given number of seats.
+10
Rep.
Tie
+10
Dem.
5%
10%
15%
20%
25% probability

Tipping Point States

The probability that a state provides the Senate seat that leads to a majority.
1 Mass. 14.9%
6 N.M. 7.6%
11 W.Va. 2.8%
2 Va. 13.8%
7 Ind. 7.4%
12 Mont. 2.2%
3 Ohio 12.7%
8 Conn. 7.0%
13 Me. 0.8%
4 Wis. 10.1%
9 Pa. 4.9%
14 Nev. 0.5%
5 Mo. 9.8%
10 Fla. 3.9%
15 N.J. 0.5%

Democratic Bang for the Buck

The relative amount that a $2,500 contribution to the senate candidate would increase the Democratic Party's chances of winning the seat.
1 Ind. 4.6
6 Nev. 1.7
11 Va. 1.2
2 Ariz. 4.5
7 Conn. 1.7
12 Tex. 1.0
3 N.D. 3.7
8 Wis. 1.5
13 Ohio 0.9
4 Neb. 3.0
9 Mont. 1.5
14 Fla. 0.8
5 N.M. 1.9
10 Pa. 1.4
15 Hawaii 0.8

Republican Bang for the Buck

The relative amount that a $2,500 contribution to the senate candidate would increase the Republican Party's chances of winning the seat.
1 Neb. 3.9
6 Ariz. 1.9
11 Fla. 1.2
2 W.Va. 3.6
7 N.D. 1.9
12 Pa. 1.1
3 Wis. 2.7
8 N.M. 1.8
13 Me. 1.0
4 Ind. 2.6
9 Mont. 1.8
14 Va. 1.0
5 Mo. 2.3
10 Nev. 1.3
15 Ohio 0.9

State-by-State Projections

Detailed polling analysis and projections for the statewide presidential vote, along with recent polls where available.

Colorado

FiveThirtyEight Projections Dem Rep Margin
Polling average 48.6 46.7 Obama +1.9
Adjusted polling average 49.1 46.6 Obama +2.5
State fundamentals 48.8 46.7 Obama +2.1
Now-cast 49.1 46.6 Obama +2.5
Projected vote share ±3.0 50.8 48.3 Obama +2.5
Chance of winning 80% 20%
Polls 538 WT. Date Dem Rep Margin
Ipsos (online)
11/5 48.0 47.0 Obama +1.0
Ipsos (online)
11/4 48.0 48.0 Tie
Keating Research
11/4 50.0 46.0 Obama +4.0
PPP
11/4 52.0 46.0 Obama +6.0
YouGov
11/3 48.0 47.0 Obama +1.0

Florida

FiveThirtyEight Projections Dem Rep Margin
Polling average 47.6 48.3 Romney +0.7
Adjusted polling average 48.3 48.1 Obama +0.2
State fundamentals 46.2 49.3 Romney +3.1
Now-cast 48.2 48.2 Tie
Projected vote share ±2.7 49.8 49.8 Tie
Chance of winning 50% 50%
Polls 538 WT. Date Dem Rep Margin
Ipsos (online)
11/5 47.0 48.0 Romney +1.0
PPP
11/4 50.0 49.0 Obama +1.0
InsiderAdvantage
11/4 47.0 52.0 Romney +5.0
Ipsos (online)
11/4 46.0 46.0 Tie
YouGov
11/3 47.0 48.0 Romney +1.0

Iowa

FiveThirtyEight Projections Dem Rep Margin
Polling average 48.6 46.0 Obama +2.6
Adjusted polling average 49.1 45.8 Obama +3.3
State fundamentals 49.2 46.3 Obama +2.9
Now-cast 49.1 45.8 Obama +3.3
Projected vote share ±3.2 51.1 47.9 Obama +3.2
Chance of winning 84% 16%
Polls 538 WT. Date Dem Rep Margin
PPP
11/4 50.0 48.0 Obama +2.0
American Research Group
11/4 48.0 49.0 Romney +1.0
YouGov
11/3 48.0 47.0 Obama +1.0
Selzer
11/2 47.0 42.0 Obama +5.0
Grove
11/2 47.0 44.0 Obama +3.0

Nebraska CD-2 District 2

FiveThirtyEight Projections Dem Rep Margin
Polling average 44.7 49.1 Romney +4.4
Adjusted polling average 47.5 51.0 Romney +3.5
State fundamentals 45.6 50.2 Romney +4.6
Now-cast 46.0 50.4 Romney +4.4
Projected vote share ±3.8 47.5 51.8 Romney +4.3
Chance of winning 13% 87%

New Hampshire

FiveThirtyEight Projections Dem Rep Margin
Polling average 49.1 46.5 Obama +2.6
Adjusted polling average 49.4 46.3 Obama +3.1
State fundamentals 50.9 44.3 Obama +6.6
Now-cast 49.6 46.1 Obama +3.5
Projected vote share ±3.5 51.4 47.9 Obama +3.5
Chance of winning 85% 15%
Polls 538 WT. Date Dem Rep Margin
New England College
11/4 50.0 46.0 Obama +4.0
PPP
11/4 50.0 48.0 Obama +2.0
UNH
11/4 50.0 46.0 Obama +4.0
Rasmussen
11/4 50.0 48.0 Obama +2.0
American Research Group
11/4 49.0 49.0 Tie

North Carolina

FiveThirtyEight Projections Dem Rep Margin
Polling average 46.7 48.6 Romney +1.9
Adjusted polling average 47.2 48.5 Romney +1.3
State fundamentals 45.5 50.1 Romney +4.6
Now-cast 47.0 48.7 Romney +1.7
Projected vote share ±2.6 48.9 50.6 Romney +1.7
Chance of winning 26% 74%
Polls 538 WT. Date Dem Rep Margin
PPP
11/4 49.2 49.4 Romney +0.2
Gravis Marketing
11/4 46.0 50.0 Romney +4.0
YouGov
11/3 47.0 49.0 Romney +2.0
PPP
10/31 49.0 49.0 Tie
High Point University
10/30 45.0 46.0 Romney +1.0

Virginia

FiveThirtyEight Projections Dem Rep Margin
Polling average 48.2 46.9 Obama +1.3
Adjusted polling average 48.8 46.7 Obama +2.1
State fundamentals 48.2 47.4 Obama +0.8
Now-cast 48.8 46.7 Obama +2.1
Projected vote share ±2.5 50.7 48.7 Obama +2.0
Chance of winning 79% 21%
Polls 538 WT. Date Dem Rep Margin
Ipsos (online)
11/5 48.0 46.0 Obama +2.0
PPP
11/4 51.0 47.0 Obama +4.0
Ipsos (online)
11/4 47.0 46.0 Obama +1.0
YouGov
11/3 48.0 46.0 Obama +2.0
Ipsos (online)
11/3 48.0 45.0 Obama +3.0
* Poll indicates a three-person matchup between President Obama, Mitt Romney and Gary Johnson

0 Comments:

Post a Comment

<< Home