11 How Much of Interviewer Variance is Really Nonresponse Error Variance? Brady T. West Michigan...
-
Upload
hilary-owens -
Category
Documents
-
view
224 -
download
0
Transcript of 11 How Much of Interviewer Variance is Really Nonresponse Error Variance? Brady T. West Michigan...
11
How Much of Interviewer Variance
is Really Nonresponse Error Variance?
Brady T. WestMichigan Program in Survey Methodology
University of Michigan-Ann Arbor
Kristen OlsonSurvey Research and Methodology Program
University of Nebraska-Lincoln
June 14, 2010International Total Survey Error Workshop 2010
22
Interviewer Variance: The Problem
An undesirable product of the data collection process, given interpenetrated sample designs
Responses for same interviewer are more similar than responses for different interviewers
Leads to inflation of variance in survey estimates due to intra-interviewer correlation, ρint
ρint = 0.01, 30 cases per interviewer 13.6% increase in SE of estimates
ρint usually less than 0.02, but can be larger
33
Research Question
Does ρint arise from complex interviewer-respondent interactions / probing for hard items?
High estimates of ρint (0.03–0.12) for factual (easy) and self-completion items in literature…
There is also consistent empirical evidence of interviewer variance in response rates
One estimates ρint with respondent data only, ignoring contributions of NR error variance
How much of interviewer variance can be attributed to nonresponse error variance?
44
The Wisconsin Divorce Study (WDS)
SRS of divorce records from four Wisconsin counties in 1989 and 1993
Sampled divorce records included official information also collected in a survey
The present study focuses on data collected using CATI: interviewer effects are likely attenuated relative to CAPI
n = 733 cases randomly sampled, and 355 CATI interviews performed by 31 trained interviewers
55
WDS Data
Six Survey Variables of InterestLength of Marriage in MonthsTime since Divorce in MonthsTime since Marriage in MonthsNumber of Marriages including the DivorceAge at MarriageAge at Divorce
Date of Divorce was recorded by an official body; other frame measures were reported by one member of the couple (possible errors)
66
Assigning Nonrespondents
Ideally, cases would not be worked by multiple interviewers (e.g., Singer and Frankel, 1982)
WDS used refusal conversion, and there were frequent changes in interviewers working non-finalized cases
This complicates the process of assigning nonrespondents to interviewers
77
Assigning Nonrespondents
Assumption: Interviewers working a particular shift work a random subsample of cases
The focus of this study is on interviewer variance within a shift, rather than across shifts
Persons with different characteristics are likely to be contacted at different times of the day
Account for shift: avoids possible confounding of nonresponse error with differences across shifts
88
Assigning Nonrespondents
Definitions of shifts:Weekday, 9-5pm (Shift 1: 26.8% of calls) Weekday, after 5pm (Shift 2: 44.1% of calls)Weekend, any time (Shift 3: 29.1% of calls)
Similar to work of Stokes and Yeh (1988)Interviewers worked multiple shiftsAlternative shifts were also considered,
and the study results did not change
99
Assigning Nonrespondents
Respondents were assigned to the interviewer completing the interview
Contacted refusals were assigned to: 1) the first interviewer receiving a refusal, or
2) the last interviewer to make contact
Non-contacts were assigned to the last interviewer making a call to the case
Random assignment of non-contacts was also considered; no change in results
1010
Assigning Nonrespondents
Limited power: 19-24 interviewers worked each shift, based on assignments
Large variability in assigned workloads across interviewers within a shift
Between 8 and 15 cases per interviewer within a shift, on average
Response rates lowest during the week, and especially before 5pm (41.1%)
1111
Analytic Approach
Examine interpenetration assumptions Estimate ρint for each survey variable within each
shift, based on respondent dataTest interviewer variance for significance
Estimate all variance components of the MSE of the respondent mean (possible with WDS data)
Estimate interviewer effects on (and interviewer contributions to) the variance components
Compute the proportion of interviewer-contributed variance due to NR error variance
1212
Example Derivation (Groves and Magilavy, 1984)MSE of respondent mean (u = # of refusals):
Example: Refusal error component
2 222
Noncontact
[ ( )]
Sampling
[
Measur( ) + + Refusae l ment
[ ]
+
[ )]
( ]
r
r r cr f rr ne
MSE x
Eu
E yE yc
En
Y yy yx yn
2
22
2 2
2 Refusal B[ ( )] + Refusal error variance
( ) ( )
ias
( )
r ref
r efe r rr f
u uE Y
uE y yn
uE Var y E Va Y
nr y
n n
13
Example Derivation, cont’d
Estimate variance components using linearized variance estimatorsAccounts for clustering due to interviewers and
unequal workloads
Estimate interviewer effects on variances based on estimates of intra-interviewer correlations in true values (Census Bureau, 1985)
13
1414
Example Derivation, cont’d
Estimated total contribution of interviewers to refusal error variance:
The estimated contribution is a function of intra-interviewer correlations in true values, for respondents and refusals
Similar derivations for other components
2 2
ˆ ˆˆ ˆ( ) ( 1) ( ) ( 1)r noint r r ref noint ref ref
u uVar y m Var y m
n n
1515
Results: Interviewer Variance Based on Respondent Data
Interpenetration evident in each shiftFour variable / shift pairs were found to
have unusually large estimates of ρint:Age at Divorce, Shift 2 (ρint = 0.08, p = 0.05)
Age at Divorce, Shift 3 (ρint = 0.10, p = 0.11)
Age at Marriage, Shift 2 (ρint = 0.11, p = 0.01)
Mths. since Marr., Shift 2 (ρint = 0.05, p = 0.13)
1616
Results: Sources of Interviewer Variance in Age at Divorce (Shift 2)Estimated intra-interviewer correlations in
Response errors: -0.003True values for respondents: 0.092True values for refusals: 0.008True values for noncontacts: -0.055
Total estimated variance of R mean: 0.423Total estimated variance contributed by
interviewers: 0.033Additional variance contributed by
interviewers is due to intra-interviewer correlations in true values for respondents!
17
Additional Results
Similar findings for age at divorce in shift 3Response error variance was main
contributor for age at marriage in shift 2 and months since marriage in shift 2
Response error variance may arise from outliers, as shown in the following graph
18
-4-2
02
4E
rro
r
Nonresponse Errors Measurement Errors
Mean Age at Divorce - Shift 2
-10
-50
5E
rro
r
Nonresponse Errors Measurement Errors
Mean Age at Divorce - Shift 3
-10
-50
5E
rro
r
Nonresponse Errors Measurement Errors
Mean Age at Marriage - Shift 2
-100
010
020
030
040
0
Err
or
Nonresponse Errors Measurement Errors
Mean Months Since Marriage - Shift 2
Illustration of Variance Sources
1919
Conclusions
Interviewer variance on key survey variables may arise from nonresponse error variance among interviewers
Interviewers may successfully obtain cooperation from different pools of respondents (e.g., older vs. younger)
Liking theory could be one explanation: variance in interviewer ages, voices variance in respondent ages (F. Conrad)
2020
Implications for Practice / Future Work
Monitoring Strategies: managers can continuously compare available features of R and NR for each interviewer, and intervene when large differences arise
Findings need to be replicated in a face-to-face setting with interpenetrated subsamples assigned to interviewers
Access to interviewer features would also enable use of multilevel modeling
2121
Thank You!
Bob Groves, Mick Couper, Frauke Kreuter and Paul Biemer have provided very helpful feedback and comments
Thanks to Vaughn Call for providing access to the WDS data
Please email [email protected] for these slides or a draft of the paper