Which Polls Fared Best (and Worst) in the 2012 Presidential Race
NYT
By NATE SILVEROur method of evaluating pollsters has typically involved looking at all the polls that a firm conducted over the final three weeks of the campaign, rather than its very last poll alone. The reason for this is that some polling firms may engage in “herding” toward the end of the campaign, changing their methods and assumptions such that their results are more in line with those of other polling firms.
There were roughly two dozen polling firms that issued at least five surveys in the final three weeks of the campaign, counting both state and national polls. (Multiple instances of a tracking poll are counted as separate surveys in my analysis, and only likely voter polls are used.)
For each of these polling firms, I have calculated the average error and the average statistical bias in the margin it reported between President Obama and Mitt Romney, as compared against the actual results nationally or in one state.
For instance, a polling firm that had Mr. Obama ahead by two points in Colorado — a state that Mr. Obama actually won by about five points — would have had a three-point error for that state. It also would have had a three-point statistical bias toward Republicans there.
The bias calculation measures in which direction, Republican or Democratic, a firm’s polls tended to miss. If a firm’s polls overestimated Mr. Obama’s performance in some states, and Mr. Romney’s in others, it could have little overall statistical bias, since the misses came in different directions. In contrast, the estimate of the average error in the firm’s polls measures how far off the firm’s polls were in either direction, on average.
Conversely, polls that were Republican-leaning relative to the consensus did especially poorly.
Among telephone-based polling firms that conducted a significant number of state-by-state surveys, the best results came from CNN, Mellman and Grove Insight. The latter two conducted most of their polls on behalf of liberal-leaning organizations. However, as I mentioned, since the polling consensus underestimated Mr. Obama’s performance somewhat, the polls that seemed to be Democratic-leaning often came closest to the mark.
Several polling firms got notably poor results, on the other hand. For the second consecutive election — the same was true in 2010 — Rasmussen Reports polls had a statistical bias toward Republicans, overestimating Mr. Romney’s performance by about four percentage points, on average. Polls by American Research Group and Mason-Dixon also largely missed the mark. Mason-Dixon might be given a pass since it has a decent track record over the longer term, while American Research Group has long been unreliable.
FiveThirtyEight did not use polls by the firm Pharos Research Group in its analysis, since the details of the polling firm are sketchy and since the principal of the firm, Steven Leuchtman, was unable to answer due-diligence questions when contacted by FiveThirtyEight, such as which call centers he was using to conduct the polls. The firm’s polls turned out to be inaccurate, and to have a Democratic bias.
It was one of the best-known polling firms, however, that had among the worst results. In late October, Gallup consistently showed Mr. Romney ahead by about six percentage points among likely voters, far different from the average of other surveys. Gallup’s final poll of the election, which had Mr. Romney up by one point, was slightly better, but still identified the wrong winner in the election. Gallup has now had three poor elections in a row. In 2008, their polls overestimated Mr. Obama’s performance, while in 2010, they overestimated how well Republicans would do in the race for the United States House.
Instead, some of the most accurate firms were those that conducted their polls online.
The final poll conducted by Google Consumer Surveys had Mr. Obama ahead in the national popular vote by 2.3 percentage points – very close to his actual margin, which was 2.6 percentage points based on ballots counted through Saturday morning.
Ipsos, which conducted online polls for Reuters, came close to the actual results in most places that it surveyed, as did the Canadian online polling firm Angus Reid. Another online polling firm, YouGov, got reasonably good results.
The online polls conducted by JZ Analytics, run by the pollster John Zogby, were not used in the FiveThirtyEight forecast because we do not consider their method to be scientific, since it encourages voters to volunteer to participate in their surveys rather than sampling them at random. Their results were less accurate than most of the online polling firms, although about average as compared with the broader group of surveys.
We can also extend the analysis to consider the 90 polling firms that conducted at least one likely voter poll in the final three weeks of the campaign. One should probably not read too much into the results for the individual firms that issued just one or two polls, which is not a sufficient sample size to measure reliability. However, a look at this broader collective group of pollsters, and the techniques they use, may tell us something about which methods are most effective.
The difference between the performance of live telephone polls and the automated polls may partly reflect the fact that many of the live telephone polls call cellphones along with landlines, while few of the automated surveys do. (Legal restrictions prohibit automated calls to cellphones under many circumstances.)
Research by polling firms and academic groups suggests that polls that fail to call cellphones may underestimate the performance of Democratic candidates.
The roughly one-third of Americans who rely exclusively on cellphones tend to be younger, more urban, worse off financially and more likely to be black or Hispanic than the broader group of voters, all characteristics that correlate with Democratic voting. Weighting polling results by demographic characteristics may make the sample more representative, but there is increasing evidence that these weighting techniques will not remove all the bias that is introduced by missing so many voters.
Some of the overall Republican bias in the polls this year may reflect the fact that Mr. Obama made gains in the closing days of the campaign, for reasons such as Hurricane Sandy, and that this occurred too late to be captured by some polls. In the FiveThirtyEight “now-cast,” Mr. Obama went from being 1.5 percentage points ahead in the popular vote on Oct. 25 to 2.5 percentage points ahead by Election Day itself, close to his actual figure.
Nonetheless, polls conducted over the final three weeks of the campaign had a two-point Republican bias overall, probably more than can be explained by the late shift alone. In addition, likely voter polls were slightly more Republican-leaning than the actual results in many races in 2010.
In my view, there will always be an important place for high-quality telephone polls, such as those conducted by The New York Times and other major news organizations, which make an effort to reach as representative a sample of voters as possible and which place calls to cellphones. And there may be an increasing role for online polls, which can have an easier time reaching some of the voters, especially younger Americans, that telephone polls are prone to miss. I’m not as certain about the future for automated telephone polls. Some automated polls that used innovative strategies got reasonably good results this year. SurveyUSA, for instance, supplements its automated calls to landlines with live calls to cellphone voters in many states. Public Policy Polling uses lists of registered voters to weigh its samples, which may help to correct for the failure to reach certain kinds of voters.
Rasmussen Reports uses an online panel along with the automated calls that it places. The firm’s poor results this year suggest that the technique will need to be refined. At least they have some game plan to deal with the new realities of polling. In contrast, polls that place random calls to landlines only, or that rely upon likely voter models that were developed decades ago, may be behind the times.
Perhaps it won’t be long before Google, not Gallup, is the most trusted name in polling.
State-by-State Probabilities
Electoral Vote Distribution
Tipping Point States
1
Ohio
49.8%
|
6
Wis.
5.9%
|
11
Mich.
0.3%
|
2
Va.
12.3%
|
7
Pa.
3.3%
|
12
Ore.
0.3%
|
3
Nev.
9.9%
|
8
N.H.
3.2%
|
13
N.M.
0.2%
|
4
Iowa
6.6%
|
9
Fla.
1.2%
|
14
N.C.
0.2%
|
5
Colo.
6.4%
|
10
Me. Dist. 2
0.3%
|
15
Minn.
0.1%
|
Return on Investment Index
1
Nev.
13.2
|
6
Colo.
3.5
|
11
Ore.
0.2
|
2
Ohio
11.8
|
7
Wis.
2.7
|
12
Fla.
0.2
|
3
N.H.
6.1
|
8
Me. Dist. 2
1.2
|
13
Mich.
0.1
|
4
Iowa
5.8
|
9
Pa.
0.7
|
14
N.C.
0.1
|
5
Va.
4.4
|
10
N.M.
0.4
|
15
Neb. Dist. 2
0.1
|
Scenario Analysis
- Electoral College tie (269 electoral votes for each candidate) 0.2%
- Recount (one or more decisive states within 0.5 percentage points) 6.4%
- Obama wins popular vote 86.2%
- Romney wins popular vote 13.8%
- Obama wins popular vote but loses electoral college 0.6%
- Romney wins popular vote but loses electoral college 5.3%
- Obama landslide (double-digit popular vote margin) 0.3%
- Romney landslide (double-digit popular vote margin) <0 .1=".1" span="span"> 0>
- Map exactly the same as in 2008 0.1%
- Map exactly the same as in 2004 <0 .1=".1" span="span"> 0>
- Obama loses at least one state he carried in 2008 99.6%
- Obama wins at least one state he failed to carry in 2008 4.2%
State-by-State Probabilities
Electoral Vote Distribution
Tipping Point States
1
Ohio
49.8%
|
6
Wis.
5.9%
|
11
Mich.
0.3%
|
2
Va.
12.3%
|
7
Pa.
3.3%
|
12
Ore.
0.3%
|
3
Nev.
9.9%
|
8
N.H.
3.2%
|
13
N.M.
0.2%
|
4
Iowa
6.6%
|
9
Fla.
1.2%
|
14
N.C.
0.2%
|
5
Colo.
6.4%
|
10
Me. Dist. 2
0.3%
|
15
Minn.
0.1%
|
Return on Investment Index
1
Nev.
13.2
|
6
Colo.
3.5
|
11
Ore.
0.2
|
2
Ohio
11.8
|
7
Wis.
2.7
|
12
Fla.
0.2
|
3
N.H.
6.1
|
8
Me. Dist. 2
1.2
|
13
Mich.
0.1
|
4
Iowa
5.8
|
9
Pa.
0.7
|
14
N.C.
0.1
|
5
Va.
4.4
|
10
N.M.
0.4
|
15
Neb. Dist. 2
0.1
|
Scenario Analysis
- Electoral College tie (269 electoral votes for each candidate) 0.2%
- Recount (one or more decisive states within 0.5 percentage points) 6.4%
- Obama wins popular vote 86.2%
- Romney wins popular vote 13.8%
- Obama wins popular vote but loses electoral college 0.6%
- Romney wins popular vote but loses electoral college 5.3%
- Obama landslide (double-digit popular vote margin) 0.3%
- Romney landslide (double-digit popular vote margin) <0 .1=".1" span="span"> 0>
- Map exactly the same as in 2008 0.1%
- Map exactly the same as in 2004 <0 .1=".1" span="span"> 0>
- Obama loses at least one state he carried in 2008 99.6%
- Obama wins at least one state he failed to carry in 2008 4.2%
State-by-State Probabilities
Senate Seat Distribution
Rep.
Dem.
Tipping Point States
1
Mass.
14.9%
|
6
N.M.
7.6%
|
11
W.Va.
2.8%
|
2
Va.
13.8%
|
7
Ind.
7.4%
|
12
Mont.
2.2%
|
3
Ohio
12.7%
|
8
Conn.
7.0%
|
13
Me.
0.8%
|
4
Wis.
10.1%
|
9
Pa.
4.9%
|
14
Nev.
0.5%
|
5
Mo.
9.8%
|
10
Fla.
3.9%
|
15
N.J.
0.5%
|
Democratic Bang for the Buck
1
Ind.
4.6
|
6
Nev.
1.7
|
11
Va.
1.2
|
2
Ariz.
4.5
|
7
Conn.
1.7
|
12
Tex.
1.0
|
3
N.D.
3.7
|
8
Wis.
1.5
|
13
Ohio
0.9
|
4
Neb.
3.0
|
9
Mont.
1.5
|
14
Fla.
0.8
|
5
N.M.
1.9
|
10
Pa.
1.4
|
15
Hawaii
0.8
|
Republican Bang for the Buck
1
Neb.
3.9
|
6
Ariz.
1.9
|
11
Fla.
1.2
|
2
W.Va.
3.6
|
7
N.D.
1.9
|
12
Pa.
1.1
|
3
Wis.
2.7
|
8
N.M.
1.8
|
13
Me.
1.0
|
4
Ind.
2.6
|
9
Mont.
1.8
|
14
Va.
1.0
|
5
Mo.
2.3
|
10
Nev.
1.3
|
15
Ohio
0.9
|
State-by-State Projections
Colorado
FiveThirtyEight Projections | Dem | Rep | Margin |
---|---|---|---|
Polling average | 48.6 | 46.7 | Obama +1.9 |
Adjusted polling average | 49.1 | 46.6 | Obama +2.5 |
State fundamentals | 48.8 | 46.7 | Obama +2.1 |
Now-cast | 49.1 | 46.6 | Obama +2.5 |
Projected vote share ±3.0 | 50.8 | 48.3 | Obama +2.5 |
Chance of winning | 80% | 20% |
Polls | 538 WT. | Date | Dem | Rep | Margin |
---|---|---|---|---|---|
Ipsos (online) | 11/5 | 48.0 | 47.0 | Obama +1.0 | |
Ipsos (online) | 11/4 | 48.0 | 48.0 | Tie | |
Keating Research | 11/4 | 50.0 | 46.0 | Obama +4.0 | |
PPP | 11/4 | 52.0 | 46.0 | Obama +6.0 | |
YouGov | 11/3 | 48.0 | 47.0 | Obama +1.0 | |
Florida
FiveThirtyEight Projections | Dem | Rep | Margin |
---|---|---|---|
Polling average | 47.6 | 48.3 | Romney +0.7 |
Adjusted polling average | 48.3 | 48.1 | Obama +0.2 |
State fundamentals | 46.2 | 49.3 | Romney +3.1 |
Now-cast | 48.2 | 48.2 | Tie |
Projected vote share ±2.7 | 49.8 | 49.8 | Tie |
Chance of winning | 50% | 50% |
Polls | 538 WT. | Date | Dem | Rep | Margin |
---|---|---|---|---|---|
Ipsos (online) | 11/5 | 47.0 | 48.0 | Romney +1.0 | |
PPP | 11/4 | 50.0 | 49.0 | Obama +1.0 | |
InsiderAdvantage | 11/4 | 47.0 | 52.0 | Romney +5.0 | |
Ipsos (online) | 11/4 | 46.0 | 46.0 | Tie | |
YouGov | 11/3 | 47.0 | 48.0 | Romney +1.0 | |
Iowa
FiveThirtyEight Projections | Dem | Rep | Margin |
---|---|---|---|
Polling average | 48.6 | 46.0 | Obama +2.6 |
Adjusted polling average | 49.1 | 45.8 | Obama +3.3 |
State fundamentals | 49.2 | 46.3 | Obama +2.9 |
Now-cast | 49.1 | 45.8 | Obama +3.3 |
Projected vote share ±3.2 | 51.1 | 47.9 | Obama +3.2 |
Chance of winning | 84% | 16% |
Polls | 538 WT. | Date | Dem | Rep | Margin |
---|---|---|---|---|---|
PPP | 11/4 | 50.0 | 48.0 | Obama +2.0 | |
American Research Group | 11/4 | 48.0 | 49.0 | Romney +1.0 | |
YouGov | 11/3 | 48.0 | 47.0 | Obama +1.0 | |
Selzer | 11/2 | 47.0 | 42.0 | Obama +5.0 | |
Grove | 11/2 | 47.0 | 44.0 | Obama +3.0 | |
Nebraska CD-2 District 2
FiveThirtyEight Projections | Dem | Rep | Margin |
---|---|---|---|
Polling average | 44.7 | 49.1 | Romney +4.4 |
Adjusted polling average | 47.5 | 51.0 | Romney +3.5 |
State fundamentals | 45.6 | 50.2 | Romney +4.6 |
Now-cast | 46.0 | 50.4 | Romney +4.4 |
Projected vote share ±3.8 | 47.5 | 51.8 | Romney +4.3 |
Chance of winning | 13% | 87% |
New Hampshire
FiveThirtyEight Projections | Dem | Rep | Margin |
---|---|---|---|
Polling average | 49.1 | 46.5 | Obama +2.6 |
Adjusted polling average | 49.4 | 46.3 | Obama +3.1 |
State fundamentals | 50.9 | 44.3 | Obama +6.6 |
Now-cast | 49.6 | 46.1 | Obama +3.5 |
Projected vote share ±3.5 | 51.4 | 47.9 | Obama +3.5 |
Chance of winning | 85% | 15% |
Polls | 538 WT. | Date | Dem | Rep | Margin |
---|---|---|---|---|---|
New England College | 11/4 | 50.0 | 46.0 | Obama +4.0 | |
PPP | 11/4 | 50.0 | 48.0 | Obama +2.0 | |
UNH | 11/4 | 50.0 | 46.0 | Obama +4.0 | |
Rasmussen | 11/4 | 50.0 | 48.0 | Obama +2.0 | |
American Research Group | 11/4 | 49.0 | 49.0 | Tie | |
North Carolina
FiveThirtyEight Projections | Dem | Rep | Margin |
---|---|---|---|
Polling average | 46.7 | 48.6 | Romney +1.9 |
Adjusted polling average | 47.2 | 48.5 | Romney +1.3 |
State fundamentals | 45.5 | 50.1 | Romney +4.6 |
Now-cast | 47.0 | 48.7 | Romney +1.7 |
Projected vote share ±2.6 | 48.9 | 50.6 | Romney +1.7 |
Chance of winning | 26% | 74% |
Polls | 538 WT. | Date | Dem | Rep | Margin |
---|---|---|---|---|---|
PPP | 11/4 | 49.2 | 49.4 | Romney +0.2 | |
Gravis Marketing | 11/4 | 46.0 | 50.0 | Romney +4.0 | |
YouGov | 11/3 | 47.0 | 49.0 | Romney +2.0 | |
PPP | 10/31 | 49.0 | 49.0 | Tie | |
High Point University | 10/30 | 45.0 | 46.0 | Romney +1.0 | |
Virginia
FiveThirtyEight Projections | Dem | Rep | Margin |
---|---|---|---|
Polling average | 48.2 | 46.9 | Obama +1.3 |
Adjusted polling average | 48.8 | 46.7 | Obama +2.1 |
State fundamentals | 48.2 | 47.4 | Obama +0.8 |
Now-cast | 48.8 | 46.7 | Obama +2.1 |
Projected vote share ±2.5 | 50.7 | 48.7 | Obama +2.0 |
Chance of winning | 79% | 21% |
Polls | 538 WT. | Date | Dem | Rep | Margin |
---|---|---|---|---|---|
Ipsos (online) | 11/5 | 48.0 | 46.0 | Obama +2.0 | |
PPP | 11/4 | 51.0 | 47.0 | Obama +4.0 | |
Ipsos (online) | 11/4 | 47.0 | 46.0 | Obama +1.0 | |
YouGov | 11/3 | 48.0 | 46.0 | Obama +2.0 | |
Ipsos (online) | 11/3 | 48.0 | 45.0 | Obama +3.0 | |
0 Comments:
Post a Comment
<< Home