How The Downballot reports on and analyzes polls Polling can be very useful, but we have strict requirements for what we will—and won’t—include David Nir Sep 8 From our earliest days more than two decades ago, election polls have always been of interest to us at The Downballot, since they can tell us a great deal about a given race and the electorate at large. But they also have their limitations, which is why we have strict requirements before we include a poll in our coverage. These are requirements we’ve adhered to for a long time and have written about before. The world of polling has changed a good deal since the last time we visited the topic in depth, though, so we felt an update was warranted. Below, we discuss each of our requirements, as well as other factors we take into account to best derive meaning from survey data. Requirements Pollster name Most pollsters have reputations, some good and some bad. Without knowing who conducted a poll, we have no way to place it in the context of that pollster’s other work. And if a pollster is new to us and doesn’t yet have a track record, that’s something we want to know, too—and convey to our readers. Put another way: If a pollster isn’t proud enough to put their name on their work, what does that say about the quality of the work? By the same token, campaigns and political organizations sometimes try to leak polling data without mentioning who produced it. That’s not acceptable to us. There’s also the problem of outright fake polls. Fortunately, these are rare, but sniffing them out is part of our job, and if we encounter an unfamiliar name, we always investigate that outfit’s bona fides before referencing their work. Pollster’s partisan affiliation If a pollster is a partisan outfit—in other words, if it works for either Republican or Democratic clients—we want to know that, because studies show that partisan polls that become public tend to lean in favor of their own side. We’re not accusing partisan pollsters of stacking the deck (though it’s possible some might), since most generally want to get an accurate read of the races they’re surveying. But unlike independent pollsters, data from partisan outfits is only ever released selectively by clients, who, of course, have every incentive to share only numbers they believe are favorable to themselves. Client, if any Some pollsters conduct and release polls independently—that is, they’re paid for by the pollster themselves. These can include universities and, sometimes, partisan pollsters. Many polls that get released publicly, however, are conducted for clients. These clients vary widely. Some are nonpartisan news organizations whose goal is to educate the public about where an election stands. Others, by contrast, are internal polls—those conducted for a client who is either directly involved in the election (like a candidate or super PAC) or has a rooting interest. We accept internal polls, but it’s always important to know when someone might have skin in the game, which is why we strongly prefer when a client’s name is disclosed. Sometimes, however, it is not. In such cases, our next preference is for a general description of who the client is (or isn’t). Whenever we have the necessary information, we will explain what a client’s interest in a race is or might be. If, however, we can’t determine whether or not there was a client for an internal poll, or who that client might be, we’ll always say so, so that our readers can deploy an appropriate measure of skepticism. When campaigns or those with a rooting interest in a race release internal polls, they almost always do so to support a particular narrative, to boost fundraising, or both. Always be on the lookout for such possibilities. Sample size If a pollster has contacted too few voters, the results are more apt to be off-base—and also more apt to fluctuate without cause from poll to poll. In the polling industry, a sample size of 300 is often considered the bare minimum acceptable for a poll, and it’s therefore the minimum we require to include a poll in our coverage. Human respondents Polls must test the opinions of actual human beings. We do not accept simulated responses created by artificial intelligence or other tools. Field dates Knowing when a poll was actually conducted is crucial. Is the data recent? Or is it old? Naturally, we strongly prefer fresh numbers, but campaigns and other organizations will often release dusty polls that may no longer reflect the state of play. In those cases, we often will decline to write them up—they’re old news. We don’t have a hard and fast cutoff, but generally speaking, if a poll is more than a month old, we’re unlikely to include it. The further away we are from an election, the more flexible we’ll be. Conversely, as we get closer to Election Day, this window will shrink. In addition, sometimes a major development will take place while a poll is in the field, or immediately afterwards. By knowing the field dates, we can be on the lookout for any news that might have affected the numbers that the poll might not have been able to account for due to its timing. If a poll is in the field too long, that can render the data suspect. It increases the chance that voters’ views may have shifted from the start of the field period to its end, especially if there’s been a noteworthy intervening event. If a poll has been in the field for more than 14 days, our practice is not to write it up. Likewise, if a poll is in the field for just a single day, that can reduce its chances of reaching a representative sample of voters. We do not set a minimum field period for including a poll; this is just an issue to be aware of. Toplines It’s not enough to just know a candidate is ahead by 3 points. There’s a world of difference between a race where one candidate leads another 51-48, where there aren’t many voters left undecided, and one where a candidate leads 43-40, where a sizable proportion of voters are still up for grabs. We therefore need to know the actual percentage of the vote each candidate is getting in the poll in question, not just the margin between them. (All too often, press reports will only supply the latter.) Undecided voters One of the most important numbers in a poll is how many voters are still undecided between candidates. If a pollster does not allow voters to say they’re still making up their minds and instead forces them to choose a side, they’re leaving out a critical piece of information about the state of the race. We therefore do not publish polls that do not offer some form of “undecided” as an option. Voter screen A pollster should always indicate whether they’ve spoken with registered voters (that is, those whose names simply appear on voter registration rolls) or likely voters (those who, by using some type of filter, the pollster thinks are actually likely to vote in the election in question). While these are the two most common ways of screening voters, pollsters will sometimes use more customized models of the electorate. In any case, pollsters should explain their methodology. Sometimes pollsters will provide results for more than one model. We always report data for likely voters, or whichever model comes closest to approximating a traditional definition of likely voters. Occasionally, a pollster will release a poll surveying only “adults” rather than voters. We don’t report on these sorts of polls because they don’t reflect the electorate (many adults are not registered to vote). Candidate names We only report polls that identify candidates by name (excluding generic congressional ballot or generic legislative ballot questions). In other words, a survey that asks whether respondents support “the Republican, John Smith, or a Democrat” without naming the latter will not pass muster with us. Initial ballot A proper horserace poll asks respondents which candidate they support before sharing any information about any candidates. This is known as the “initial ballot,” and it’s the only type of question we’ll accept. Many surveys—almost always internal polls—also share information about the candidates, whether positive, negative, or both. These are known as “informed ballot” questions, and they can be very useful tools for campaigns and their allies seeking to understand which messages will be most powerful when communicated to voters. However, they describe a hypothetical world in which respondents have just been exposed to information that may or may not wind up reaching the broader electorate. For instance, a candidate might trail by 10 points on an initial ballot question but lead by 10 points on an informed ballot question. The latter reflects an idealized situation that can never be fully replicated in real life. We do not report the results of informed ballot questions. However, when necessary, we may note their existence to help explain why someone might release an internal poll that might otherwise appear unfavorable. Unfortunately, unscrupulous campaigns and their supporters sometimes release informed ballot tests in ways that make them appear to be initial ballot tests. This is a shameful practice, but it’s one we’re always on guard for. Partisan identification of candidates In partisan elections (which represent the overwhelming majority of races we cover), many voters make decisions about which candidates to support based on their partisan affiliation. Therefore, pollsters should identify candidates to respondents by the party they’ll be identified with on the ballot. If a pollster doesn’t include this information, then they’re leaving out important details and failing to accurately mimic the way voters will make their choices when they actually cast their ballots. We therefore do not accept polls that do not provide respondents with candidates’ partisan affiliations. If someone publishing a poll doesn’t make the above information publicly available or adhere to the practices we describe, then we will not include it, barring rare exceptions. Other things we look at In addition to the requirements above, there are a number of other things we look at when examining every poll we come across. Not all of these apply to every single poll, but the questions below are always important to think about. Did the pollster push “leaners”? Sometimes, pollsters will ask undecided voters which candidate they “lean” toward. This practice is known as “pushing leaners” and usually yields useful information about voter preferences. If a pollster provides numbers with and without leaners, we will report the numbers that include leaners. What kinds of questions did the pollster ask before getting to the “horserace” (i.e., the head-to-head matchups between candidates)? Pollsters sometimes ask issue-related questions, and best practices dictate they should be asked after the horserace. That’s because these kinds of questions can “prime” voters to lean one way or the other, especially if they’re on contentious topics or asked in ways that suggest the pollster has an axe to grind. How did the pollster ask about an office holder’s job approval rating? The best way is to ask whether voters simply approve or disapprove (potentially including whether they “strongly” or “somewhat” approve or disapprove). Sometimes, however, pollsters will ask along a four-point scale, such as “excellent,” “good,” “fair,” or “poor.” That “fair” (or sometimes “just fair”) is a very tricky phrase. To some voters, that might be closer to “good”; to others, closer to “poor.” Because of that ambiguity, we generally do not report on approval ratings when they’re asked this way. Did the pollster include every credible candidate in the race? Sometimes, pollsters will leave out one or more credible candidates. For primary polling, this can make the numbers significantly less useful. (By contrast, sometimes a candidate will have dropped out or entered the race after a pollster has conducted a survey but before it releases the data. This is a problem, too.) Judgment, however, is often necessary. Who qualifies as “credible” is something every pollster struggles with, especially in races with large fields of candidates. Asking respondents to sit through a question featuring a dozen or even 20 candidates risks making it harder to obtain completed interviews. However, if we feel a notable candidate has not been included but should have been, we likely won’t cover the poll, though if we do, we’ll be sure to note the omission. In addition, public releases of internal polls conducted before primaries often do not include all potential general election matchups between credible candidates. This doesn’t call into question the accuracy of the poll, but it does leave open the question as to whether other candidates fare better. (There’s a good chance that other names were tested but just not included in such a release.) Conversely, a candidate may include primary rivals when releasing data on the general election, often to argue that they would fare best. Data on such matchups merits skepticism since the candidate who paid for the poll has an incentive to want their primary opponents to look worse. What languages did the pollster interview respondents in? The majority of the time, English alone will suffice. But in some jurisdictions with large Latino populations, it’s hard—if not impossible—to get an accurate read on the electorate without also conducting interviews in other languages, most frequently Spanish and/or Chinese. Has the pollster also provided a breakdown of its sample composition and/or its crosstabs? In an ideal world, pollsters would always tell us the basics about their sample, including the proportion of respondents by gender, party affiliation, age, race/ethnicity, and other demographics. Unfortunately, many don’t, which requires us to trust that they’ve produced a sample that’s likely to reflect the electorate (and therefore requires us to exercise greater skepticism). Similarly, crosstabs allow us, for instance, to see the proportion of men vs. women who say they’ll vote for a particular candidate. That gives us richer data to analyze, and also helps us evaluate the accuracy of a poll. But always be careful: Sub-samples are smaller than the total sample, so they are correspondingly less likely to be accurate. For instance, a national poll of 500 people might include only 60 Black respondents. While that’s likely an appropriately representative proportion, examining the views of Black voters on the basis of such a small sample is something we avoid. How has the poll been weighted? After a pollster has contacted respondents, they’ll almost certainly wind up with a sample that does not reflect the electorate they expect to see on Election Day because various demographics tend to respond to polls at a higher rate than other populations. Therefore, pollsters need to “weight” their sample so that it resembles the electorate they expect to see, giving more weight to under-represented groups and vice-versa. There are many different criteria by which a poll can be weighted, including demographic (such as race, gender, age, and educational attainment) and political (such as party affiliation, ideology, or how a person says they’ve voted in the past). There’s no one recipe for achieving accuracy, and over the years, pollsters have continually modified their approach. For instance, weighting by education was once less common but became largely standard after the 2016 elections, when education levels emerged as a meaningful political dividing line. More recently, it’s become more common for pollsters to rely more heavily on political characteristics in their weighting, such as recalled vote. Such metrics can pose complications—people don’t always remember how they voted, for instance, or may not tell the truth, and how they voted in one election may not correspond with how they vote in the next one. Many pollsters, though, believe that weighting in this manner will improve accuracy. What is the source of the polling data? The quality of the actual presentation of polling data can vary widely. Polls from independent pollsters (such as colleges and universities) and those conducted for media outlets, for instance, typically include all of the information we require and more, presented in a transparent way. However, internal polling can surface in a variety of ways. In the best cases, sponsors share formal polling memos that include the information we need. Other releases are more fragmentary and may emerge in a news article or social media post. In such cases, we will only rely on data shared by reliable outlets. We do not accept polls from low-quality, anonymous engagement farms on social media that often share data without providing any links. Has the pollster professed any beliefs that reflect conspiracy theories about elections or election denialism? We take a heterodox approach to polling—there are many ways to get it right, and no one has a monopoly on the truth. But the truth is what we all must seek. Excluding polls is not something we do lightly, but when a pollster espouses beliefs about elections that are demonstrably false, we are unable to conclude that such a person does in fact believe in seeking the truth. Such firms include the Trafalgar Group, McLaughlin & Associates, and Rasmussen Reports. We remain vigilant in looking out for any other such outfits. We are especially grateful to Mary Radcliffe and G. Elliott Morris of FiftyPlusOne, who generously offered valuable feedback for this piece. We also consulted FiftyPlusOne’s own methodology statement and drew several helpful ideas from it.