Does the Sample Source Matter? A comparison of some leading sample sources

October 3, 2011

Sample is the soil and water of research. The quality of it determines the value of what you produce.

We compared SBA to four other sample

sources that VC regularly uses

We conducted a five minute survey Nov 4 to 7, 2011 with a sample of about 500 people each, using sample from Vision Critical’s Springboard America, and four of the leading sample providers that we have anonymised as “Panel B”, “ Panel C”, “Panel D” and “River E”. The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI)

Sample Quality: Dropout rate during the survey

4%

8% 10%

7%

16%

0%

5%

10%

15%

20%

SBA Panel B River C Panel D River E

Dropouts as a percent of all who started the survey

Data quality: straight-lining

0.5%

6.5%

7.9%

3.9%

7.8%

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

SBA Panel B River C Panel D River E

% who gave the same answer for all 13 trust ratings

Data quality: use of “don’t know”

Average percent who choose the option "don’t know", "can't choose", "not sure", "prefer not to say".

4%

6% 6%

4%

6%

0%

1%

2%

3%

4%

5%

6%

7%

8%

SBA Panel B River C Panel D River E

Average % "Don't Know"

Differences between panels in how people

answered all questions

5%

11%

4%

6% 7%

-1%

1%

3%

5%

7%

9%

11%

13%

15%

SBA Lightspeed SSI Research Now Fulcrum

Percent of times panel differed from the others

Panel C Panel D River

Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible

Expected by chance

Panel B

Differences between panels in how people

answered snack purchase questions

Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible

2%

13%

2%

6% 6%

-1%

1%

3%

5%

7%

9%

11%

13%

15%

SBA Panel B River C Panel D River E

Purchase behaviour: Percent of times panel differed from the others

They were less likely to buy snacks

Differences between panels in how people

answered concept test questions

Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible

11%

26%

6%

16%

13%

0%

5%

10%

15%

20%

25%

30%

SBA Lightspeed SSI Research Now Fulcrum

Concept Test: Percent of times panel differed from the others

Panel B Panel C Panel D River

Differences between panels in how people

answered the purchase intent question

Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible

13% 6%

9% 12% 8%

21%

15%

23%

28%

21%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

SBA Panel B River C Panel D River E

Definitely will Probably will

Conclusions

Sample matters. A lot. Springboard America has fewer drop-outs, notably better data quality, and is “right down the middle” when it comes to comparison to the other panels on the findings, unlike some panels which show real variation—particularly on the concept test.