Post on 18-Jan-2018
description
www.bls.gov
Assessing Nonresponse Bias and Measurement Error Using Statistical
MatchingJohn Dixon
U.S. Bureau of Labor StatisticsJune 15, 2010
The opinions expressed are those of the author, not the BLS
Nonresponse Bias and Measurement Error
Nonresponse bias and measurement error both refer to difficult to measure errors in surveys. Nonresponse bias refers to unmeasured persons, and measurement error refers to an unmeasured construct (which is mis-measured by the survey).
Contact history has the potential to describe the concerns of potential respondents as well as contactability. Those concerns have been found to relate to nonresponse, but little relationship to measurement error.
2
Consumer Expenditure Quarterly Survey
Household survey provided part of the “market basket” for CPI and other indices.
5 quarterly interviews for each household. Typically 6160 households a month. Nonresponse rate: 15% Refusal rate: 13% More probing than the similar Diary survey
National Health Interview Survey
Household survey provides health information.
Household, Family, Child, Adult modules.
Typically 35,000 households a year. Nonresponse rate: 12% Refusal rate: 6%
Contact History Instrument (CHI) Data
Mean Rates of CHI concerns
0
0.05
0.1
0.15
0.2
0.25
Mea
n Co
ncer
ns__
_
7
Predicting Nonresponse Logistic model Coefficients
-2-1.5
-1-0.5
00.5
11.5
22.5
3
Bivariate Models
NOTINT HUNGUP HOSTILE VOLUNTAR PRIVACY ANTIGOV QUESTION NOTAPP OTHHH SAMEINF BUSY NOSHOW SCHEDULE TIME MEMBER ISSUES SAMEFR TOOPERS TOOMANY TOOLONG QUIT
8
Predicting Nonresponse Logistic model Coefficients
-2-1.5
-1-0.5
00.5
11.5
2
Multiple Predictors
NOTINT HUNGUP HOSTILE VOLUNTAR PRIVACY ANTIGOV QUESTION NOTAPP OTHHH SAMEINF BUSY NOSHOW SCHEDULE TIME MEMBER ISSUES SAMEFR TOOPERS TOOMANY TOOLONG QUIT
Factor Pattern for Contact History Concerns
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Hostile Time Busy Privacy
HostileNotIntHangupSamefrTimeSameInfTooPersTooManyTooLongQuitBusyNoShowSchedulePrivacyAntiGovQuestionNotAppOthHH
Statistical Matching
10
Refusal effect for matched data.
00.5
11.5
22.5
33.5
4
1 2 3 4 5NHIS CE
Noncontact effect for matched data.
0
0.5
1
1.5
2
2.5
3
1 2 3 4NHIS CE
Summary
The Chi data was useful in modeling the relationship between concerns expressed by respondents and refusal/noncontact.
The CHI data showed factor patterns which could describe broad areas of concern. They related well in predicting nonresponse.
The nonresponse bias looked similar for the CE and NHIS for both refusal and noncontact
The NHIS had higher estimates of health expenditures, but the reason for those differences need investigation.
The matching method was sensitive to binning method, but a regression approach may have more robustness and better diagnostics.
Limitations and Future Research
The CHI data is limited in that it only reflects the concerns expressed by respondents. Some of the most common concerns may mask the real reasons, for example, “busy” may hide concerns about privacy, which weren’t expressed to the interviewer.
Statistical matching may be useful for suggesting further research, but the accuracy of the estimated differences needs other data sources to separate out method/survey differences.
Contact Information
www.bls.gov
Assessing Nonresponse Bias and Measurement Error
Using Statistical MatchingJohn Dixon
202-691-7516dixon.john@bls.gov