Post on 04-Dec-2014
description
Does the Sample Source Matter? A comparison of some leading sample sources
October 3, 2011
Sample is the soil and water of research. The quality of it determines the value of what you produce.
We compared SBA to four other sample
sources that VC regularly uses
We conducted a five minute survey Nov 4 to 7, 2011 with a sample of about 500 people each, using sample from Vision Critical’s Springboard America, and four of the leading sample providers that we have anonymised as “Panel B”, “ Panel C”, “Panel D” and “River E”. The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI)
Sample Quality: Dropout rate during the survey
4%
8% 10%
7%
16%
0%
5%
10%
15%
20%
SBA Panel B River C Panel D River E
Dropouts as a percent of all who started the survey
Data quality: straight-lining
0.5%
6.5%
7.9%
3.9%
7.8%
0%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
SBA Panel B River C Panel D River E
% who gave the same answer for all 13 trust ratings
Data quality: use of “don’t know”
Average percent who choose the option "don’t know", "can't choose", "not sure", "prefer not to say".
4%
6% 6%
4%
6%
0%
1%
2%
3%
4%
5%
6%
7%
8%
SBA Panel B River C Panel D River E
Average % "Don't Know"
Differences between panels in how people
answered all questions
5%
11%
4%
6% 7%
-1%
1%
3%
5%
7%
9%
11%
13%
15%
SBA Lightspeed SSI Research Now Fulcrum
Percent of times panel differed from the others
Panel C Panel D River
Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible
Expected by chance
Panel B
Differences between panels in how people
answered snack purchase questions
Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible
2%
13%
2%
6% 6%
-1%
1%
3%
5%
7%
9%
11%
13%
15%
SBA Panel B River C Panel D River E
Purchase behaviour: Percent of times panel differed from the others
They were less likely to buy snacks
Differences between panels in how people
answered concept test questions
Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible
11%
26%
6%
16%
13%
0%
5%
10%
15%
20%
25%
30%
SBA Lightspeed SSI Research Now Fulcrum
Concept Test: Percent of times panel differed from the others
Panel B Panel C Panel D River
Differences between panels in how people
answered the purchase intent question
Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible
13% 6%
9% 12% 8%
21%
15%
23%
28%
21%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
SBA Panel B River C Panel D River E
Definitely will Probably will
Conclusions
Sample matters. A lot. Springboard America has fewer drop-outs, notably better data quality, and is “right down the middle” when it comes to comparison to the other panels on the findings, unlike some panels which show real variation—particularly on the concept test.