A Deeper Look at Polls
Jones, Terry, St. Louis Journalism Review
It's political polling season and, for campaign junkies, there is an ample supply of statistics to feed their habits. Here are some tips for sorting out the numbers.
TIP 1: Despite how journalists describe it, the sampling margin of error is a slope, not a cliff. Most media accounts faithfully provide the sampling error. This number (say, 4 percent) means that, if an infinite number of surveys were taken, 95 percent of the results would fall within plus-or-minus 4.0 percent of the proportion (say, 48 percent for Bush) found for the survey being reported. But any number between 48 percent plus-or-minus 4 percent, that is 44 percent-to-52 percent, is not equally likely to occur.
The distribution, your old friend the normal curve, slopes downward steadily on each side of the 48 percent. The next most likely outcome is plus-or-minus 1 percent (49 percent or 47 percent), plus-or-minus 2 percent (50 percent or 46 percent) is considerably less likely, and so forth down both slopes.
Most reporters do not convey this subtlety. When the St. Louis Post-Dispatch/KMOV survey showed Bush leading Kerry 49 percent-to-42 percent in Missouri, the Sept. 17 article cautiously noted that "the president's lead is at the edge of the poll's margin of error of 3.5 percentage points, which means that any individual number could be that much higher or lower."
The message conveyed is that the Bush lead is statistically shaky. But--and this is the central point--any individual number for the Bush estimate between 52.5 percent (plus 3.5 points) and 45.5 percent (minus 3.5 points) is not equally likely. In fact, with a seven-point lead in a survey with a plus-or-minus 3.5 percent sampling error, the odds that Bush is really ahead are about 99 in 100, a safer bet than the 95 percent confidence level.
Here's a rule of thumb. If the sample size is between 500 and 1,000, treat any lead of 5 percentage points or greater as real. For those over 1,000, consider any advantage of 4 points or more to be meaningful.
TIP 2: For an incumbent, there are two equally important poll numbers: one, the difference between the incumbent and the challenger and the other, the difference between the incumbent and 50 percent. Take the Bush 49, Kerry 42 result. The good news for the GOP is that the president has a seven-point lead.
The not-so-good news is that the incumbent president still falls short of a simple majority. After three-and-a-half years in office, he has not sealed the deal with a winning majority (assuming no third-party candidates).
In a some respects, a Bush 52 percent and Kerry 47 percent result--a five point lead having Bush over 50 percent--is better for the Republicans than is Bush's 49 percent and Kerry's 42 percent--a seven point lead but with the incumbent under 50 percent.
TIP 3: Pay attention to whom the results apply: all registered voters or all likely voters? Few political polls include all voting-age adults, screening respondents so that only those reporting that they are registered are included. But, of course, even a substantial fraction of registered voters will not cast a ballot in November.
To adjust for this, many polls use one or more questions to estimate those who are most likely to turn out and then provide estimates both for them ("likely voters") as well as the entire sample ("registered voters"). …