I have a very broad question: how do polls work?
I see polls in the news all the time. I also know that companies use polls, and that people post polls online, and so on. However, I feel like I’ve never taken a “real” poll in my entire life. I’ve answered a few online, and filled out surveys on Facebook and stuff, but I’ve never been called by CNN or another news station and asked my opinion about anything. So, who is taking all these polls? Can they really be accurate at all? It seems to me that they’re wrong often, like during the election.
So, I guess I’m asking two things: should I trust polls? Why, or why not?
Polls are indeed everywhere in the media, and elsewhere, as well. We see all kinds of polls from various sources, and on different types of subjects. One may tell us what flavor of ice cream Americans like best, while another might claim to represent who will or won’t vote for a candidate in an election. Can they be trusted?
The answer is yes; however, there are two important caveats. Let’s cover those, then move on to explain how polls and surveys are conducted, and how that helps us know whether we should trust them.
The first caveat is an obvious one: not all polls are trustworthy. We’ll talk a bit more about poll methodology in a moment. However, for now, suffice it to say that it’s not easy to set up a reliable and accurate poll. A poll hosted on Twitter won’t give you an accurate sense of what people around the country think of something. It may not even give you an accurate sense of what people on Twitter think! Funded polls, or those performed by groups with a vested interest in the results should be viewed with suspicion, too. It’s easy to mess up a poll, either by accident or on purpose, for us to say that all polls should be trusted.
The second caveat is perhaps even more important, and it may be a bit less obvious. It’s this: when we view a poll, we need to interpret it, not just "trust" it. What does this mean? Well, polls provide us with a certain amount of information, but they don't argue anything themselves. What we get out of a poll is some raw data, and from there it is on us to interpret things in the correct way. A poll tells us what respondents said, and a good one tells us what the margin of error is for the data. It’s our responsibility to remember key things, like that margin of error and variables external to the poll, and to interpret things with care.
Our short answer, then, is that polls are "trustworthy" if they are conducted in the proper fashion, and if we ourselves are responsible about the way we interpret the data. So, let's talk next about how polls work, so we can learn how to make better use of our ground rules for trusted polls.
Polls and surveys are governed by laws of statistics, a branch of math that deals with odds and percentages. At first glance, this might seem odd. After all, if you were asked your opinion in a poll, you'd give it: no hedging or percentages needed.
The reason that polls need to use statistics is that it's not feasible to poll everyone you need to poll to get an accurate result. For instance, if you wanted to know what residents of New York City thought of something, asking all eight and a half million of them for their thoughts would not be an efficient way to go about it.
However, it is possible to get a good idea of what New Yorkers think by using a sample of the population. You can ask some of those New Yorkers. If you choose a good sample, you can reasonably expect that the breakdown of opinions you find is similar to the breakdown you'd get if you could ask all eight and a half million people.
However, what makes a "good sample" is the key to understanding proper polling, and this is where many polls go wrong. First and foremost, there’s the sample size: larger is better. If you could poll every New Yorker, that would be the most accurate! However, experts know how to calculate the right sample size to get efficient results, without going broke polling countless people.
Imagine for a moment that you wanted to design this poll. What would you do? How would you conduct the poll?
There are many potential pitfalls. If you chose to call people on their land-line phones, you'd end up with a sample that looked older than New York City as a whole does. Put it online, and your sample would be too young. Go door-to-door, and your sample might skew too much to one neighborhood or another. To get an accurate look at what New Yorkers think, your sample would need to be an accurate reflection of New York City as a whole. The percentages of young people, bankers, immigrants, and residents of Queens would all have to be more or less the same.
Perhaps you could ignore some variables. For example, you might not need to ask each respondent their height, for instance, because we might expect short and tall New Yorkers to have the same opinion. However, if our poll is about how high ceilings should be on subway platforms, then that's a variable to which we need to pay attention!
You can see, of course, that good polling isn't easy to do. It also gets more complex. Once we have our data, we need to do the math. We need to tell people who read our poll how big our margin of error is, but we may also need to break down our demographic numbers and adjust our sample. There are powerful tools that help pros do this, say the experts at Survey King, and academics and political pollsters work hard to get things right. However, there’s no denying that polling is tricky, and that some less trustworthy sources fail to get it right.
Once all that is done, and assuming it is done right, we have a poll that is quite accurate. However, we also know how to read it. If our poll says a candidate is ahead by three percentage points, with a margin of error of five, we know the race is still a toss-up. If our poll says someone is ahead in a presidential race by a certain number of points, we need to remember how to interpret that. Our poll tells us important things about the popular vote, but presidents are not elected by the popular vote! If you need any proof that polls are often misunderstood, just look at the 2016 election in which the much-maligned polls were quite accurate, and the folks who were wrong were the ones reaching conclusions the data did not support.
Polls are tools, and they help us understand things. They give us raw data, not opinions. We can trust polls from reputable sources to serve their intended purpose, and we can eliminate small bias errors by averaging poll numbers from trusted sources. However, we must never forget our own responsibilities! It's on us to remember that polls have margins of error, and that they can go out of date as circumstances change. It's on us to interpret polls and figure out what they are, and aren't, telling us.
“And I’m very proud of the 50,000 poll workers and election officials who delivered a free and fair election.” - Ken Blackwell