Rasmussen Poll on Wisconsin Dispute May Be Biased

Nate Silver of the Times' FiveThirtyEight blog

We’ve noted before that the automated polling firm Rasmussen Reports has had problems with bias in a statistical sense: in the election last fall, its polls overestimated the standing of Republican candidates by roughly 4 percentage points on average.

A somewhat different issue arises today in a poll the firm conducted on the dispute in Wisconsin between Gov. Scott Walker and some of the state’s public-employee unions.

The poll, which included people that Rasmussen deemed to be “likely voters” from across the country, found that 48 percent of respondents agreed more with Mr. Walker in the dispute, while 38 percent agreed more with “the union for teachers and other state employees.”

That question, though, was the fourth one Rasmussen asked in the survey — and the questions that came before it may have biased the responses.

According to the firm’s statement of question wording, these were the first four questions Rasmussen asked in the poll:

1: How closely have you followed news reports about the Wisconsin governor’s effort to limit collective bargaining rights for most state employees?

2: Does the average public employee in your state earn more than the average private sector worker in your state, less than the average private sector worker in your state, or do they earn about the same amount?

3: Should teachers, firemen and policemen be allowed to go on strike?

4: In the dispute between the governor and the union workers, do you agree more with the governor or the union for teachers and other state employees?

There is nothing wrong with the first question, which simply asks people whether they have been following events in Madison. But the second and third questions are arguably problematic.

The issue is clearest with the third question, which asked respondents whether “teachers, firemen and policemen” should be allowed to go on strike. By invoking the prospect of such strikes, which are illegal in many places (especially for the uniformed services) and which many people quite naturally object to, the poll could potentially engender a less sympathetic reaction toward the protesters in Wisconsin. It is widely recognized in the scholarship on the subject, and I have noted before, that earlier questions in a survey can bias the response to later ones by framing an issue in a particular way and by casting one side of the argument in a less favorable light.

The Rasmussen example is more blatant than most. While many teachers have been among the protesters at the State Capitol in Madison, obliging the city to close its schools for days, there have been no reports of reductions in police or fire services, and in fact, uniformed services are specifically exempted from the proposals that the teachers and other public-sector employees are protesting. So bringing in the uniformed services essentially makes No. 3 a talking point posed as a question.

As an analogy, imagine a survey that asked respondents whether they believed the Democrats’ health care overhaul included “death panels” before asking them whether they approved or disapproved of the bill over all.

The second question in the Rasmussen poll found that 36 percent of respondents believe that public-sector employees earn more than private-sector workers in their state, while 21 percent thought public sector workers earned less, and 20 percent thought they earned about the same amount.

In fact, according to an analysis by USA Today, state employees earn about 5 percent less than comparable employees in the private sector, on average, although federal employees receive significantly (20 percent) more.

A poll is not a pop quiz, and the respondents in the survey are not to blame for giving the “wrong” response. Also, the question posed by Rasmussen, which did not consider the type of work performed and asked simply about average salaries in the respondent’s home state, was not exactly the same as the one studied by USA Today, which covered the whole country and took account of the the type of work done. Still, to the extent that this misperception about pay levels is widely held and casts public employees in a less favorable light, a survey question that reminds respondents of it could bias responses to later questions.

Until we have another survey that designs its questions more carefully, there is no good way to predict how the responses to this one might have turned out differently.

Another automated poll by the survey firm We Ask America, which is a subsidiary of the Illinois Manufacturers’ Association and uses methodology similar to that of Rasmussen Reports, found that a majority of respondents disapprove of Mr. Walker’s budget plan — but it was a survey of Wisconsin residents, not the entire country, as Rasmussen’s was.

Because of the problems with question design, my advice would be simply to disregard the Rasmussen Reports poll, and to view their work with extreme skepticism going forward.