Survey Nonresponse
description
Transcript of Survey Nonresponse
1
Survey NonresponseSurvey Nonresponse
Survey Research LaboratoryUniversity of Illinois at Chicago
www.srl.uic.eduMarch 16, 2010
2
Survey Research Laboratory
OutlineOutlineI. Unit nonresponse
(UN)A. DefinitionB. CausesC. Impact on dataD. Strategies for
minimizingE. Correcting for UNF. Reporting issues &
UN
II. Item nonresponse (IN)A. DefinitionB. CausesC. Impact on dataD. Strategies for
minimizing E. Correcting for INF. Reporting issues &
IN
3
Survey Research Laboratory
I. Unit nonresponseI. Unit nonresponse
A.Definition: “Nonresponse occurs when a sampled unit does not respond to the request to be surveyed…” (Groves et al., 2002) increasing concern, especially in
telephone surveysB. Causes of unit nonresponse
noncontact noncooperation
4
Survey Research Laboratory
Failure to contactFailure to contact
• Non-contact patterns vary by survey mode & across different subgroups
Restricted-access buildings & communities
Telephone numbers connected to answering machines, caller ID. . .
5
Survey Research Laboratory
Percent of Eligible Sample Households by Calls to Percent of Eligible Sample Households by Calls to First Contact, for Five Surveys (Groves, 2001)First Contact, for Five Surveys (Groves, 2001)
0
10
20
30
40
50
60
1 2 3 4 5 6 7-8 9 ormore
NC
Calls to First Contact
Perc
ent o
f the
Hou
seho
lds
Current Events Mental Health Election
Welfare Reform Child Health
6
Survey Research Laboratory
Failure to gain cooperation (refusals)Failure to gain cooperation (refusals)
• Survey requests commonly misclassifed
• Repeated callbacks distinguish surveys
7
Survey Research Laboratory
Evidence that survey requests are Evidence that survey requests are misclassifiedmisclassified
• Refusals made quickly (<30 seconds)
• Many refusers reconsider later• Persons apparently focus on a small
number of factors when making the decision.
8
Survey Research Laboratory
Factors typically considered by Factors typically considered by respondentsrespondents
• Burden• Saliency (topic
importance)• Incentive• Authority of sponsor
9
Survey Research Laboratory
ImplicationsImplications
• Many different reasons for refusals• No single introduction may be
suitable for all• Interviewers must tailor
introductions information from initial refusal or
contact attempt
10
Survey Research Laboratory
C. Impact of UN on dataC. Impact of UN on data
• UN will results in nonresponse bias to the extent that respondents & nonrespondents differ on variables of interest types of nonresponse bias
• representativeness• data quality• distribution of responses
strategies for estimating nonresponse bias• number of calls or refusal conversions & data• data from studies with different response rates• using census data to compare respondents &
nonrespondents
11
Survey Research Laboratory
Nonresponse bias as function of response rate (Nonresponse bias as function of response rate (ttRR))
12
Survey Research Laboratory
Empirical research in general Empirical research in general finds no response rate effectfinds no response rate effect
• Keeter et al. (2000)• Curtin, Presser and Singer (2000)• Merkle and Edelman (2002)
13
Survey Research Laboratory
Example of no relationship between Example of no relationship between nonresponse & measurement errornonresponse & measurement error
From Merkle, D.M., & Edelman, M. (2002). Nonresponse in exit polls: A comprehensive analysis. In Groves, R.M., Dillman, D.A., Eltinge, J.L., & Little, R.J.A. (Eds.), Survey nonresponse (pp. 243-257). New York: Wiley.
14
Survey Research Laboratory
D. Strategies for reducing UND. Strategies for reducing UN
• Mode• Increase benefits
Incentives Convince potential
respondents of importance• introduction• publicity• trusted source or sponsor
• Reduce perceived burden
• Initial contact before interview (advance letter)
• Persistence: refusal conversions follow-ups change societal
perceptions mode changes interviewer changes
15
Survey Research Laboratory
E. Correcting for UNE. Correcting for UN
• Post data-collection weighting• Assumptions
respondents & nonrespondents in a particular category (e.g,. African Americans) are interchangeable
known population parameters of relevant characteristics
16
Survey Research Laboratory
F. Reporting issues and UNF. Reporting issues and UN
• Standards can be found at www.aapor.org• Disposition codes
Known eligible• Completes• Refusals
Known ineligible• Businesses• Disconnected numbers
Unknown eligibility• Ring, no answers• Always busy
17
Survey Research Laboratory
AAPOR AAPOR responseresponse rates rates
DenominatorNumerator
Known Eligible
Known Eligible + Unknown Eligibility
Known Eligible + e(Unknown Eligibility)
Completes LowestCompletes + Partials
Highest
18
Survey Research Laboratory
Reporting of response ratesReporting of response rates
• AAPOR standards• Examination of reporting in recent
journal articles done by Johnson and Owens (2003) AAPOR paper
19
Survey Research Laboratory
Journal article auditJournal article audit
• Sample of 95 papers selected from same set of 18 journals
• Published January 2000 or more recent
• Unit of analysis = individual• Noninstitutional population• Reporting primary survey data (no
secondary analyses)
20
Survey Research Laboratory
JournalsJournalsHealth science
journals• Amer J Psychiatry• Amer Jour Preventive
Medicine• Amer J Public Health• Annals of Epidemiology• JAMA• J Studies on Alcohol• Medical Care• Preventive Medicine• Substance Use & Misuse
Social science journals
• Amer J Community Psychology
• Amer J Political Science• Amer Political Science Review • International J Public Opinion
Research• J Community Psychology• J Health & Soc Behavior• Public Administration Review• Public Opinion Quarterly• Social Forces
21
Survey Research Laboratory
Disclosure typologyDisclosure typology
1. Report no information (5.3%)2. Report other information only (23.2%)3. Report non-AAPOR response rate (26.3%)4. Report AAPOR-consistent response rate
(24.2%)5. Report summary of sample dispositions
(21.1%)
22
Survey Research Laboratory
Disclosure typology IIDisclosure typology II
• Mail surveys are most likely to provide AAPOR response rate and/or full sample disposition information (88%).
• Telephone, face-to-face surveys are most likely to provide response rate information (44% and 43%).
• p < .001
23
Survey Research Laboratory
Other observationsOther observations
• Some argue that “no definitive formula for response rate estimates exists”
• No papers underestimate response rates
• Many appear to be overestimating response rates
24
Survey Research Laboratory
II. Item nonresponseII. Item nonresponse
A. Definition: When a given unit does not provide a substantive response to a particular question. Sometimes called “missing data”
B. Causes of IN ability to provide information willingness to provide information privacy and anonymity concerns
25
Survey Research Laboratory
C. Impact of IN on dataC. Impact of IN on data
• IN introduces nonresponse bias if it is associated with key variables
• Examine associations between IN & key variables representativeness (demographics) other substantive variables
26
Survey Research Laboratory
D. Strategies for minimizing IND. Strategies for minimizing IN
• reassurance of anonymity• follow-up question(s)• sometimes ask for less specific
information (e.g., income)
27
Survey Research Laboratory
E. Correcting for INE. Correcting for IN
• dropping variables• dropping observations• pairwise deletion• imputation of values for missing data• person mean substitution (scales)• item mean substitution• impact on central tendency
28
Survey Research Laboratory
F. Reporting issues and INF. Reporting issues and IN
• Steps to deal with IN should be reported
• If a correction strategy is used, the impact of that strategy should be reported (comparison of analysis with & without correction strategy)
29
Survey Research Laboratory
SummarySummary
• Ideally, reduce nonresponse• Correct for nonresponse• Distinction between nonresponse &
nonresponse bias• Test for nonresponse bias• Report nonresponse issues &
correction strategies