© 2005 Knowledge Networks, Inc. Presented at the 2005 Annual Meeting of the American Association of...
-
Upload
cuthbert-conley -
Category
Documents
-
view
212 -
download
0
Transcript of © 2005 Knowledge Networks, Inc. Presented at the 2005 Annual Meeting of the American Association of...
© 2005 Knowledge Networks, Inc.
Presented at the 2005 Annual Meeting of the American Association of Public Opinion
Research
Tom Smith, NORC
Mike Dennis, Knowledge NetworksRick Li, Knowledge Networks
Comparing Knowledge Networks’ Web Enabled
Panel with In-Person 2002 General Social Survey
2
Background
• Examine differences and similarities between in-person and web-based surveys with emphasis on the optimal treatment of the ‘don’t know’ response option.
• Knowledge Networks Experiment 2000 and General Social Survey (GSS) 2000.
Differences in Presentation of “Don’t Know” options. Differences in results attributed to presentation and format of “Don’t
Know” options. Questions can be re-designed to minimize differences in results.
• Knowledge Networks Experiments 2002 re-designed presentation of “Don’t Know” options.
•Basic issue is that there is not a way to capture ‘volunteered don’t knows’ in the web mode.
3
GSS 2002 and KN 2002 Experiments
GSS 2002 KN 2002
Data collection organization
NORC Knowledge Networks
Mode of Data Collection
In-person Web-based
Field period Feb-June 2002 April-June 2002
Question Items
Entire GSS ‘National Priority’ items
Sample Size 2765 1655 (across four experiments)
4
“Don’t Know” Treatments
Experiment Conditions Sample size
GSS “Don’t know” option is not read by interviewers but volunteered by respondents
2795
KN A Respondent instruction placed on first screen to skip questions to indicate “Don’t know/No opinion”; “Don’t know option is not presented on the screen
655
KN B “Don’t know” option presented on screen 333
KN C Respondents who answered “Don’t know” first are prompted to choose answer on the screen that follows.
334
KN D Instruction on all screens to skip questions to indicate “Don’t know/No opinion”
333
5
Question Wording for the Tested GSS Items
We are faced with many problems in this country, none of which can be solved easily or inexpensively. I am going to name some of these problems, and for each one I’d like you to tell me whether you think we are spending too much money on it, too little money, or about the right amount.
First, are we spending too much, too little, or about the right amount on ______?
• Too much• Too little• About the right amount• Don’t know / no opinion
6
Spending Priority Items I
Standard Wording Variant Wording
The space exploration program Space exploration
Improving and protecting the environment The environment
Improving and protecting the nation's health Health
Solving the problems of the big cities Assistance to big cities
Halting the rising crime rate Law enforcement
Dealing with drug addiction Drug rehabilitation
Improving the nation's education system Education
Improving the conditions of Blacks Assistance to Blacks
The military, armaments and defense National defense
Foreign aid Assistance to other countries
Welfare Assistance for the poor
7
Spending Priority Items II
Standard Wording Variant Wording
Highways and bridges N/A
Social Security N/A
Mass transportation N/A
Parks and recreation N/A
Assistance for childcare N/A
Supporting scientific research N/A
8
Instruction for KN Treatment A
9
Summary of “Don’t Know” Rates
4% 5% 4%3% 4% 4%
16%
20%
17%
11%
5% 5%
11%12%
11%
0
5
10
15
20
25
11 Standard Items 6 Standard Items 11 Variant Items
GSS KN A KN B KN C KN D
% of “Don’t Know” Respondents
10
Scatterplot of “Don’t Know” Rates for KN A vs GSS
0%
2%
4%
6%
8%
10%
12%
0% 2% 4% 6% 8% 10% 12%
GSS Don't Know Rates
KN
Exp
erim
ent
A D
on
't K
no
w R
ates
11
Substantive Differences
5% 5%5%
7%
4%
7%
4%4%
5%
11%
7%
6%
0
2
4
6
8
10
12
KN A KN B KN C KN D
Too much About the same Too little
Average Percentage Points Difference Between KN and GSS (Across All 28 Spending Priority Items)
12
Scatterplot of “Too Little” Rates: KN A vs GSS
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
GSS 'Too Little' Rates
KN
Exp
erim
ent
A 'T
oo
Lit
tle'
Rat
es
13
Scatterplot of “About Right Amount” Rates:KN A vs GSS
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
GSS 'About Right Amount' Rates
KN
Exp
erim
ent
A 'A
bo
ut
Rig
ht
Am
ou
nt'
Rat
es
14
Scatterplot of “Too Much” Rates:KN A vs GSS
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
GSS 'Too Much' Rates
KN
Exp
erim
ent
A 'T
oo
Mu
ch' R
ates
15
Conclusions
• KN Experiment A produced most similar results to GSS. Across all 28 spending priority items:
0.6 percentage points difference on “Don’t Know”. 4.8 percentage points difference substantively.
• Some spending items show more significant differences substantively. For instance:
Dealing with drug addiction / Drug rehabilitation Improving conditions of blacks / Assistance to blacks Welfare / Assistance to the poor