Evaluating Surveys

March 28, 2011

Objectives

• Evaluate a survey by examining sample size, margin of error, question wording, dates or interviews, and sampling methodology.

• Evaluate a survey by detecting whether it may be prone to systematic error.

• Evaluate polls for their moral consequences.

Accuracy of Polls

• Many people have argued that polls are problematic due to perceived inaccuracy.

• This is potentially an issue in survey research and one of the easiest ways to combat that fact is to pay close attention to the description of the survey.

• Things to look for:• Sample size• Margin of error• Question wording• Dates of interviews• Methodology of sampling and weighting

Anchorage Daily News 8-9-07

• For which of the following, if any, would you be willing to give up your right to vote? The trade would be anonymous. Source: Opinion Research Corporation, margin of error +/- 3%.

Anchorage Daily News 8-9-07

Sources of Polls

• One major thing to consider is who sponsored a given poll. While, excluding insta-polls, most major news agencies use respectable standards for surveying other interest, trade groups and random polling firms may not.

• Controlling the poll plays a major role in how the results will come out.

• When in doubt try to seek similar results from reputable agencies.

Consider

• Question wording• Interest group favoring national health

care• Do you believe that federal government

should provide health insurance to those than cannot afford it?

• Trade group representing health insurance companies• Would you be in favor of government-run

health care if it meant limiting your choice of doctor and raising taxes?

Other factors that can affect results

• Timing• Social desirability bias• Appropriate sample • Question wording and choices• Question repeating• Refusal rate• Surveyor oversight

Errors

• Remember the reported margin of error is only one type of error. In practice there are loads of places for error to crop in surveys. • Random error• Systematic error

Morality of Polls

• Does polling have a negative effect on the political process? Electoral process?• Polling on election night, west coast 1980• Bandwagon effects?• Reduce turnout?• ‘Spiral of Silence’? Noelle-Neumann• 1980 pay to poll debate poll• Could lead to stigmatization• Ginsberg’s weakening of public opinion

Polls

• What is gained by polling?• What could be lost or damaged by

polling?

For Next Time

• Read WKB chapter 17.• Answer question 2 on page 364.