1

Dependent Interviewing: Evidence from Field Tests

The SIPP Methods PanelNancy Bates and Joanne Pascale

U.S. Census Bureau

Seminar on Dependent Interviewing Univ. of Essex, September 17, 2004

2

Acknowledgements

• Pat Doyle, Jeff Moore, Joanne Pascale, Julia Klein Griffiths, Anna Chan

• Heather Holbert, Elaine Hock, Johanna Rupp, Aniekan Okon, Ceci Villa, Judy Eargle

• U.S. Census Bureau Field Division

3

Methods Panel Research Project

• 4-yr research project to study and refine recommendations– Extant data and research, cognitive assessments,

large-scale field experiments, interviewer assessments

• 3 Field Experiments– Treatment group—experimental instrument– Control group—SIPP production instrument– 1000 households per treatment– Conducted in 6 Regional Offices– CAPI interview (CASES language)

4

Methods Panel Research Project

• 3 Field Experiments

– Experiment in 2000 — Wave 1– Experiments in 2001 & 2002 — Waves 1 and 2– Interviewer assessments, data analysis– Response Rates from 83% - 92%

• Produced SIPP 2004 instrument

5

Cognitive Assessments

• Wave 1: 34 households interviewed

• Wave 2: 4 months later, 23 households re-interviewed

• Cognitive interview followed by debriefing

• Reactions to dependent interviewing techniques:– Sharing of data with other members– Reaction to Respondent Identification Policy (RIP)

6

Cognitive Assessments• RIP Question: “We re-contact households every 4 months

to update information. If we talk to someone else in your household next time, instead of you, is it OK if we use your answers as a starting point?”

• D.I. did not elicit privacy concerns• Respondents expected previous information to be

stored• Few respondents declined RIP request

(misunderstood?)

7

Interviewer Debriefings• Pencil and paper debriefing after each wave

(Wave 1 n=152 ; Wave 2 n=131)

• Section in Wave 2 about D.I. Techniques: – Use of “Same as last time” reminders– Dependent questions vs. “from scratch”– Dependent question nonresponse follow-ups– Correcting mis-reported breaks in receipt

• Rated on 5 point scale (Strongly disagree – Strongly agree)

8

Interviewer Debriefings

• D.I. preferred over “from scratch” (assets)• D.I. perceived as easier for respondent; smoother

interview; easier to administer; more accurate (earnings, unearned income, assets, & health insurance)

• D.I. not perceived more accurate in correcting seam bias (not sig. different from control).

• D.I. perceived more accurate regarding health insurance coverage

9

Attrition

• Response Rates (Unit-level attrition)

• “Learning Curve” effect (year)

• D.I. effect (treatment)

10

Attrition: Unit-Level Nonresponse by Year and Treatment by Wave

0%

10%

20%

2000 W1 2001 W1 2002 W1 2001 W2 2002 W2

% n

onre

pons

e

Control

Test

11

Attrition

• On surface, little evidence D.I. impacted attrition• No significant difference in test/control

comparisons• Closer look at Wave 1 across years and treatment

groups• Logistic regression

– Significant (p<.10) main effect for year (negative)– Borderline (p=.11) year*treatment interaction – Nonresponse decline across years is sig. for test group

but not control– Wave 2 exhibits similar trend, but interaction not sig.

12

Dependent Interview Nonresponse Follow-up Techniques

If “Don’t know” “Refused” or “Same as last time”….

• Earnings: Things may have changed since then, but I have

recorded from last time that [NAME] earned about $X,XXX a

month from this job with [EMPLOYER NAME]. Does that still

sound about right?

• Unearned Income: It says here that [NAME] received [$XXX] in

food stamp benefits last [MONTH]. Does that still sound about

right?

• Asset Income: Things may have changed a lot but I have

recorded from last time your income from these rental properties

was about $[X,XXX] annually. Does that still sound about right?

13

Assessing Dependent Interview Nonresponse Follow-up Techniques

• Item nonresponse for income amounts (don’t know, refused)

• Computed aggregate nonresponse ratio: # D.K. + Refused

# questions asked

• Average of ratios across all adults in universe

14

% Nonresponse to Earnings Items - 2002 Wave 2 (All Jobs/Businesses Combined)

20.0%

11.0%

0%

10%

20%

30%

Non

-Res

pons

e R

ate

Control

Test

15

% Nonresponse to Asset Income Items - 2002 Wave 2 (All Asset Types Combined)

20.2%

27.8%

0%

10%

20%

30%

Non

-Res

pons

e R

ate

Control

Test

16

% Nonresponse to Unearned Income Items - 2002 Wave 2 (All Programs Combined)

25.9%

17.3%

0%

10%

20%

30%

Non

-Res

pons

e R

ate

Control

Test

17

SIPP interview months and their associated reference period months (example):

OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG

M1 M2 M3 M4 M5 Wave 1 Interview

M1 M2 M3 M4 M5 Wave 2

M1 M2 [etc.]

18

Hypothetical Panel Survey Month-to-Month Transitions illustrating "seam" bias (Burkhead and Coder, 1985)

050

100150200250300350

Num

ber o

f Rep

orte

d Ch

ange

s

Food Stamps

Social Security

First Interview Second Interview

19

Dependent Interview Seam-Bias Reduction Techniques

• Unearned Income (and School Enrollment):

– If reported in M5 previous: “Last time I recorded that you received

[food stamps] in [month]. Is that correct?”

– If reported in M4 but not M5: “Last time I recorded that you received

[food stamps] in [month 4]. Did you continue to receive any benefits

from [food stamps] after [month 5] 1st?

• Health Insurance:

– “Last time I recorded [NAME] was covered by Medicaid. Is

[NAME] still covered by that type of insurance plan?”

20

Assessing Dependent Interview Seam-Bias Reduction Techniques

• Month-to-month transitions in recipiency/enrollment/coverage (yes/no)

• Transitions at the “seams” (between M4-M5) where fewer transitions = improvement

• Calculated % of month-to-month transitions for: unearned income, school enrollment, and health insurance coverage

21

% of Month-to-Month Transitions in Unearned Income Observed at the "Seam" - 2002 Wave 2

74.7%

62.2%

0%

20%

40%

60%

80%

% tr

ansi

tions

at s

eam

Control

Test

Expected % of transitions if no seam bias

22

% of Month-to-Month Transitions in School Enrollment Observed at the "Seam" - 2001 Production SIPP vs. 2004

40.4%

30.7%

0%

20%

40%

60%

80%

% tr

ansi

tions

at s

eam

2001 SIPP

2004 SIPP

Expected % of transitions if no seam bias

23

% of Month-to-Month Transitions (Yes-No and No-Yes) in School Enrollment Observed at the "Seam" -

2001 Production SIPP vs. 2004

13.1%

1.0%2.0% 1.4%

0%

20%

40%

Yes-No No-Yes

% tr

ansi

tions

at s

eam

2001 SIPP

2004 SIPP

24

% of Month-to-Month Transitions in Health Insurance Observed at the "Seam" (MPSIPP 2002 Control vs. Test)

73.8% 73.4%

0%

20%

40%

60%

80%

% tr

ansi

tions

at s

eam

Control

Test

Expected % of transitions if no seam bias

25

% of Month-to-Month Transitions (Yes-No and No-Yes) in Health Insurance Observed at the "Seam" -

MPSIPP 2002 Control vs. Test

4.7%

21.4%

5.2%

26.6%

0%

20%

40%

Yes-No No-Yes

% tr

ansi

tions

at s

eam

Control

Test

26

Conclusions

• D.I. In 2004 SIPP represents major departure from past practice

• Thorough testing program suggest:– Substantially improved interview experience from

Interviewer perspective– Respondents had few concerns with D.I.

(qualitative tests)– R.I.P didn’t have big impact on D.I. (3-5%)

27

Conclusions• D.I. beginning to impact attrition?• D.I. Nonresponse follow-up techniques

successful • Decreased item nonresponse for earning,

asset, and unearned income amounts• D.I. techniques helped to reduce seam bias in

income recipiency & enrollment• D.I. helped reduce “on/off” seam transitions?• Still much room for improvement• SIPP D.I. techniques first fielded in June 2004