Public Opinion Quarterly, Vol. 75, No. 5, 2011, pp. 962–981

THE EVOLUTION OF ELECTION POLLING IN THE

UNITED STATES

D. SUNSHINE HILLYGUS

*

Abstract Public opinion polls have long played an important role in the

study and conduct of elections. In this essay, I outline the evolution of poll-

ing as used for three different functions in U.S. presidential elections: fore-

casting election outcomes, understanding voter behavior, and planning

from presidential approval to celebrity outfits and sports predictions—but they

remain especially fundamental to the conduct and study of elections. Elections

and polling are so intertwined that it is hard to imagine one without the other. Poll

numbers provide fodder for media coverage and election predictions, they shape

candidate and voter behavior, and they are the basis of interpreting the meaning of

election outcomes. Public Opinion Quarterly was founded in January 1937 on

the heels of the advent of modern scientific polling in U.S. presidential elections.

The first issue included an essay, ‘‘Straw Polls in 1936,’’ explaining how George

GallupÕs quota-controlled survey of a few thousand triumphed over the Literary

DigestÕs straw poll of millions in correctly predicting the election outcome

since the 2000 election. In the 2008 election, there were an estimated 975

D. SUNSHINE HILLYGUS is Associate Professor of Political Science and Director of the Initiative

on Survey Methodology at Duke University, Durham, NC, USA.

*

Address correspondence to

D. Sunshine Hillygus, Department of Political Science, Box 90204, Duke University, Durham,

NC 27708, USA; e-mail: [email protected].

doi: 10.1093/poq/nfr054

Ó The Author 2011. Published by Oxford University Press on behalf of the American Association for Public Opinion Research.

All rights reserved. For permissions, please e-mail: [email protected]

1

There has also been a significant evolution in the nature of election polling.

For decades, polls were typically conducted by telephone, using live inter-

viewers, on behalf of media organizations or political candidates. Today, In-

ternet surveys and IVR polls are increasingly common, and polls are often

initiated by entrepreneurial pollsters conducting them not for a client, but

for self-promotion (Blumenthal 2005). The dissemination of poll numbers

has also changed, with many polls now being reported directly on blogs

and polling aggregation websites rather than by the traditional media. Journal-

ists are no longer the formal gatekeepers determining if a given poll is of suf-

everyoneimmediatelysharedhisview,however.Itwasnotuntilthe1960sand1970s

that surveys became a fixture of political campaigns (J.Converse1987). Early skep-

ticism that a sample of respondentscouldsay anything aboutthe opinions of millions

gave way to a belief in the scientific basis of probability samples. Today, however,

nonprobability samples—typically opt-in Internet surveys—are increasingly com-

mon, and probability samples are experiencing significant methodological chal-

lenges, such as increasing nonresponse and cell-phone-only households. We

now hear near constant questioning of the motivation and methods of pollsters, often

instigated by partisan bloggers and pundits dissatisfied with the results of a poll.

There is, once again, a haze of skepticism surrounding the entire industry.

tion outcomes, understanding voter behavior, and planning campaign strategy.

1. Gallup Poll, November 2008. Retrieved May 19, 2011, from the iPOLL Databank, Roper Center

for Public Opinion Research, University of Connecticut.

2. For a comprehensive history of polling, see Herbst (1993). Jacobs and Shapiro (2005) edited

a terrific special issue of Public Opinion Quarterly that covers election polling in the new infor-

mation and media environment.

3. Although not addressed in this essay, there is a growing body of research examining the nature

and role of polling outside the United States (e.g., Durand, Blais, and Larochelle 2004) and in non-

presidential races (e.g., Bafumi, Erikson, and Wlezien 2010).

tion polling, unscientific straw polls date to at least the 1824 presidential elec-

tion, when informal trial heat tallies were taken in scattered taverns, militia

offices, and public meetings (Smith 1990). As the first U.S. presidential election

to be decided by popular vote, it was inevitable that people would try to gauge

public opinion, however imperfectly. Fast-forward to today, and each new elec-

tion cycle brings a wave of horserace polling numbers feeding the insatiable

appetite of media, bloggers, and political junkies trying to predict the election

outcomes.

the 1936 election despite successful predictions from 1916 to 1932 (Squire

1988). More recently, John Zogby, labeled the ‘‘prince of pollsters’’ after

nailing the 1996 election prediction, saw his reputation tarnished by poor

predictions in subsequent years, with NY Times election blogger Nate Silver

more recently calling him ‘‘The Worst Pollster in the World’’ (Silver 2009).

Of course, the very task of assessing accuracy raises questions about how

best to measure it. Should a forecast be called ‘‘accurate’’ if it correctly predicts

the winner, the winnerÕs vote share, or the margin of victory? In 1996, CBS

News correctly predicted Clinton as the winner, but they forecast an 18-percent

margin of victory over Dole, rather than the 8 percent that he actually received.

tion results as well (Wand et al. 2001).

Traugott, and Kennedy (2005) have added a ninth. Most commonly used are

‘‘Mosteller Measure 3,’’, the average absolute error on all major candidates

between the prediction and the actual results, and ‘‘Mosteller Measure 5,’’

the absolute value of the difference between the margin separating the two

leading candidates in the poll and the difference in their margins in the actual

vote. Increasingly, scholars are also using Martin et al.Õs predictive accuracy

measure, which is based on the natural logarithm of the odds ratio of the out-

come in the poll and the outcome in the election. By all measures, 2008 was

considered a banner year for the polling industry and, by some metrics, could be

defeat Hillary Clinton in the New Hampshire Democratic primary. Predicting

election outcomes is a difficult and high-stakes business, so it is important to

understand why some polls get it right and some get it wrong.

Like any survey, the quality of predictions can be affected by sampling error

and nonsampling errors, including coverage error, nonresponse error, measure-

ment error, processing error, and adjustment error (Groves et al. 2009). It is

most recognized that random sampling error can produce fluctuations in polling

estimates based on chance alone, simply because a poll includes a sample of

respondents rather than the full population. Such error is expressed with the

margin of error that is typically reported alongside polling estimates, and

the fact that the election itself is not a census, but an application of the sampling

principle. Every poll is therefore a sample of a sample’’ (p. 25). In other words,

it is an unknown population to whom pollsters are trying to generalize because

we do not know who will show up on Election Day. Different elections can have

different cross-sections of voters, a point highlighted in 2008 when an unusu-

ally large proportion of minorities and young people turned out to vote for

Barack Obama. As such, one of the most consequential methodological

decisions made by the pollster or analyst is the selection of likely voters.

Forecasts can vary wildly based on the particular method used to define the

scholarly research on political participation. For instance, pollsters typically

use a single likely-voter model for the entire country, but political science research

has shown that state-level factors such as registration requirements, early voting

Once an assumption is made about likely voters, the task of the pre-election

poll is to predict how voters will cast their ballots. The standard polling question

asks, ‘‘If the election were being held today, for whom would you vote?’’ In

making a forecast, pollsters or analysts must make a decision about how to deal

with the respondents who say they are undecided, a pool of voters that varies

based on the timing of the poll and the methodology used (Fenwick et al. 1982).

respondents might not give incomplete or untruthful answers to the vote choice

questions, such as privacy concerns or respondent competence.

Polling predictions can also be jeopardized by individuals changing their

minds about their turnout and vote intention between the time of the survey

interview and Election Day. Although scholarly research often emphasizes

the stability of vote intention, panel data has found that more than 40 percent

of respondents change their vote intention at least once during the campaign

(Hillygus and Shields 2008). There remains debate, however, about the source

of these individual-level dynamics. Gelman and King (1993) argued that move-

dissipated quickly, but there were smaller, persistent shocks late in the cam-

paign. There remains much to be learned about who in the electorate is most

likely to change their minds, when they are most likely to do so, and in response

to what stimuli, and such findings will have clear implications for election fore-

casting. Voter instability is considered the primary explanation for the polling

debacle of 1948. Pollsters called the election for Dewey weeks before the elec-

tion, but a sizeable chunk of voters changed their vote and turnout intention in

the final weeks, and they overwhelmingly supported Truman (Crespi 1988). To

minimize sources of error, pollsters now continue to do election polling as late

as the night before the election, and it is this final poll that is used in the post-

of published pre-election polls, available at http://www.ncpp.org.

formation about how surveys are conducted. Too many researchers do

not know how to document their work, or are reluctant to disclose their

methods for fear of criticism by non-transparent competitors. Too many

significant questions about survey practice remain unaddressed because

getting information about how studies are done is onerous or impossible.

Too many members of the public have become cynical about survey re-

search because they do not understand how different methods underlie

conflicting findings.

we would expect to occur over an election campaign’’ (p. 500). Pooling polls

improves the precision of polling estimates—a larger sample size has a smaller

margin of error. But there are multiple methods for aggregating polls, and it is

not yet clear which one is best.

Some aggregations simply take the average of all available poll numbers.

Yet, naı

vely pooling across polls ignores house effects, all of the methodolog-

ical decisions made by a particular survey firm. Unfortunately, this approach

might not improve accuracy. Simple averaging assumes that the various sources

of bias in the individual poll numbers will cancel each other out, but if biases

them according to some criteria, for example excluding partisan polls or those

deemed to be lower quality. Fivethirtyeight.com weights each poll based on the

‘‘pollsterÕs historical track record, sample size, and recentness of the poll.’’

to evaluate national pre-election polling accuracy remain applicable and infor-

mative.

For example, should accuracy assessments be made based on the final, Elec-

tion Eve polls? It is well established that the accuracy of poll-based predictions

improves closer to Election Day, as the number of undecided voters decline and

vote preferences stabilize. For example, early polling in the 1992 election

showed George H. W. Bush with a healthy lead, but polls converged in pre-

dicting Clinton the winner by the end of the campaign. But it is perhaps less

interesting to get the forecast correct just before the actual outcome is known.

11. Realclearpolitics.com began tracking state polls in the 2004 election, making aggregation sites

the difference in eligibility across states, and the absence of a party identification cue for voting all

affected turnout in the primaries, which in turn affected polling accuracy (Traugott et al. 2009).

tion files—mandated by the 2002 Help America Vote Act—are the cornerstone

of these databases. These files typically include a personÕs name, home address,

turnout history, party registration, phone number, and other information, and

are available to parties and candidates (and, in most states, anyone else who

wants it). Consumer, census, political, and polling data are then merged into

these files to better predict who is going to turn out, what their beliefs and

attitudes are and, ultimately, how they are going to vote.

This development has influenced not only how candidates communicate with

citizens, but also whom they are contacting and what they are willing to say.

ferent messages to different segments of the electorate (Jacobs and Shapiro