© 2005 Knowledge Networks, Inc.

Presented at the 2005 Annual Meeting of the American Association of Public Opinion

Research

Tom Smith, NORC

Mike Dennis, Knowledge NetworksRick Li, Knowledge Networks

Comparing Knowledge Networks’ Web Enabled

Panel with In-Person 2002 General Social Survey

2

Background

• Examine differences and similarities between in-person and web-based surveys with emphasis on the optimal treatment of the ‘don’t know’ response option.

• Knowledge Networks Experiment 2000 and General Social Survey (GSS) 2000.

Differences in Presentation of “Don’t Know” options. Differences in results attributed to presentation and format of “Don’t

Know” options. Questions can be re-designed to minimize differences in results.

• Knowledge Networks Experiments 2002 re-designed presentation of “Don’t Know” options.

•Basic issue is that there is not a way to capture ‘volunteered don’t knows’ in the web mode.

3

GSS 2002 and KN 2002 Experiments

GSS 2002 KN 2002

Data collection organization

NORC Knowledge Networks

Mode of Data Collection

In-person Web-based

Field period Feb-June 2002 April-June 2002

Question Items

Entire GSS ‘National Priority’ items

Sample Size 2765 1655 (across four experiments)

4

“Don’t Know” Treatments

Experiment Conditions Sample size

GSS “Don’t know” option is not read by interviewers but volunteered by respondents

2795

KN A Respondent instruction placed on first screen to skip questions to indicate “Don’t know/No opinion”; “Don’t know option is not presented on the screen

655

KN B “Don’t know” option presented on screen 333

KN C Respondents who answered “Don’t know” first are prompted to choose answer on the screen that follows.

334

KN D Instruction on all screens to skip questions to indicate “Don’t know/No opinion”

333

5

Question Wording for the Tested GSS Items

We are faced with many problems in this country, none of which can be solved easily or inexpensively. I am going to name some of these problems, and for each one I’d like you to tell me whether you think we are spending too much money on it, too little money, or about the right amount.

First, are we spending too much, too little, or about the right amount on ______?

• Too much• Too little• About the right amount• Don’t know / no opinion

6

Spending Priority Items I

Standard Wording Variant Wording

The space exploration program Space exploration

Improving and protecting the environment The environment

Improving and protecting the nation's health Health

Solving the problems of the big cities Assistance to big cities

Halting the rising crime rate Law enforcement

Dealing with drug addiction Drug rehabilitation

Improving the nation's education system Education

Improving the conditions of Blacks Assistance to Blacks

The military, armaments and defense National defense

Foreign aid Assistance to other countries

Welfare Assistance for the poor

7

Spending Priority Items II

Standard Wording Variant Wording

Highways and bridges N/A

Social Security N/A

Mass transportation N/A

Parks and recreation N/A

Assistance for childcare N/A

Supporting scientific research N/A

8

Instruction for KN Treatment A

9

Summary of “Don’t Know” Rates

4% 5% 4%3% 4% 4%

16%

20%

17%

11%

5% 5%

11%12%

11%

0

5

10

15

20

25

11 Standard Items 6 Standard Items 11 Variant Items

GSS KN A KN B KN C KN D

% of “Don’t Know” Respondents

10

Scatterplot of “Don’t Know” Rates for KN A vs GSS

0%

2%

4%

6%

8%

10%

12%

0% 2% 4% 6% 8% 10% 12%

GSS Don't Know Rates

KN

Exp

erim

ent

A D

on

't K

no

w R

ates

11

Substantive Differences

5% 5%5%

7%

4%

7%

4%4%

5%

11%

7%

6%

0

2

4

6

8

10

12

KN A KN B KN C KN D

Too much About the same Too little

Average Percentage Points Difference Between KN and GSS (Across All 28 Spending Priority Items)

12

Scatterplot of “Too Little” Rates: KN A vs GSS

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

GSS 'Too Little' Rates

KN

Exp

erim

ent

A 'T

oo

Lit

tle'

Rat

es

13

Scatterplot of “About Right Amount” Rates:KN A vs GSS

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

GSS 'About Right Amount' Rates

KN

Exp

erim

ent

A 'A

bo

ut

Rig

ht

Am

ou

nt'

Rat

es

14

Scatterplot of “Too Much” Rates:KN A vs GSS

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

GSS 'Too Much' Rates

KN

Exp

erim

ent

A 'T

oo

Mu

ch' R

ates

15

Conclusions

• KN Experiment A produced most similar results to GSS. Across all 28 spending priority items:

0.6 percentage points difference on “Don’t Know”. 4.8 percentage points difference substantively.

• Some spending items show more significant differences substantively. For instance:

Dealing with drug addiction / Drug rehabilitation Improving conditions of blacks / Assistance to blacks Welfare / Assistance to the poor