Jeffrey henning april lecture series - 2014

Jeffrey Henning, Researchscape, April Lecture Series 2014

Improving the Representativeness

of Online Surveys

Jeffrey Henning

Researchscape International

Event Sponsors


Respondent Selection Issues

Sampling Error

Coverage Error

Nonresponse Error at

Unit

Response Accuracy Issues

Nonresponse Error at Item

Measurement Error due to Respondents

Measurement Error due to Interviewers

Survey Administration Issues

Post-Survey Error

Mode Effects

Comparability Effects

Total Survey Error


Niche Survey

Topline Survey

Probability Survey

Mode Online Online Telephone

Target > 5% incidence

> 20% incidence

General population

Respondents 100 400 400

Length 15 questions 25 questions

5 minutes

Cost/response $5 $5 $20

Price $495 $1,995 $7,995

Comparing Prices


Probability sampling

Probability online panels

Open online panels

Weighting Quota

sampling Sample

matching

River sampling

Intercept samples

Practical ramifications

Agenda


2.0% 3.3% 4.1% 4.7% 5.0% 5.3% 6.4% 6.4%

12.0%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

A B C D E F G H I

Average Absolute Errors

Source: Yeager, Krosnick, et al, 2011

Probability Non-probability


6.0

3.6

2.9 2.6 2.4 2.3

1.9 1.9

1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

A B C D E F G H I

Accuracy = Value

Base = worst performing survey’s average absolute error



9.6% 11.7%

13.2% 13.7% 15.3% 15.6% 16.0%

18.0%

35.5%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

A B E F H G D C I

Largest Absolute Errors




9.6% 11.7%

13.2% 13.7% 15.3% 15.6% 16.0%

18.0%

35.5%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

A B E F H G D C I



Largest Absolute Errors


Key Elements of

Probability Sampling

Coverage

• Known non-zero chance of selecting any member of the target population

External selection

• Random selection of members of the population to participate in the survey


Robustness?

Any method with a low response rate is not a random probability sample. We can’t assume a known and non-zero chance of selection. This is true of telephone, so for most studies the gold standard is not a practical option, even if money were no object. – Ray Poynter, director, Vision Critical, 2013

What about the vast majority of research that has 90% opt-out rates? Do we decide that those people weren’t part of the population to begin with? ...I’m just having a hard time understanding the ongoing push to prove we are using probability samples. – Annie Pettit, Research Now, 2010


Robustness?

Any method with a low response rate is not a random probability sample. We can’t assume a known and non-zero chance of selection. This is true of telephone, so for most studies the gold standard is not a practical option, even if money were no object. – Ray Poynter, director, Vision Critical, 2013

What about the vast majority of research that has 90% opt-out rates? Do we decide that those people weren’t part of the population to begin with? ...I’m just having a hard time understanding the ongoing push to prove we are using probability samples. – Annie Pettit, Research Now, 2010

Response rates were positively associated with demographic representativeness, but only very weakly... In general population RDD telephone surveys, lower response rates do not notably reduce the quality of survey demographic estimates. – Holbrook, Krosnick, Pfent, 2008