Survey Response Rates: Trends and Standards Karen Donelan, ScD Senior Scientist in Health Policy...
-
Upload
michelle-kent -
Category
Documents
-
view
215 -
download
2
Transcript of Survey Response Rates: Trends and Standards Karen Donelan, ScD Senior Scientist in Health Policy...
Survey Response Rates: Trends and Standards
Karen Donelan, ScDSenior Scientist in Health Policy
Massachusetts General Hospital/Harvard Medical School
AcademyHealth - June 27th, 2006
Overview
What is a response rate? Why do we care about response rates? Trends Does nonresponse = biased response? How should rates be calculated and
reported? How can we improve?
What is a Response Rate?
The number of complete interviews with reporting units divided by the number of eligible reporting units in the sample
Other Rates Cooperation rates - The proportion of all cases
interviewed of all eligible units ever contacted.
Refusal rates - The proportion of all cases in which a sampled unit or the respondent refuses to be interviewed, or breaks-off an interview, of all potentially eligible cases.
Contact rates - The proportion of all cases in which some responsible sampled unit member was reached.
Why Do We Care About Response Rates?
Sources of Error in Surveys Coverage, Measurement, Nonresponse, Processing,
Sampling Practical implications
Generalizability Publishability (often cutoff or major test of quality) Credibility Fundability
What Do We Know About Response Rate Trends?
http://www.census.gov/prod/2000pubs/tp63.pdf
Current Population Survey (CPS)
Behavioral Risk Factors Survey (BRFSS) : Telephone
Survey
Source: Groves et al. 2004
U Michigan: Survey of Consumers
Source: Groves et al. 2004
Why Does Nonresponse Happen?
Who is in the sample? (demographics, SES, lifestyle) # Contacts Schedule of contacts Mode of contact Respondent selection Respondent cooperation (refusal) Incentives ($, benefit) Respondent burden (time, boredom, frustration) Salience Respondent ability to respond to questions Sponsorship Privacy concerns
How Should Rates Be Calculated and Reported? Disclosure
Units drawn and attempted Addresses mailed Phone numbers called Households approached Patients invited
Sample Disposition: what happened? Eligible and ineligible respondents Bad contact information How many attempts, at what intervals # Never reached, # actually contacted # completed, refused, never got a
decision
www.aapor.org
Does Nonresponse = Biased Response? Low response rates may not be problematic Representativeness doesn’t necessarily increase with
response rate Sample representativeness is a function of the
difference between respondents and nonrespondents on the statistic(s) of interest
High response rates can can yield an unrepresentative sample (high nonresponse bias)
Low response rates can still yield a representative sample
Keeter et al. (2000), Curtin, Presser and Singer (2000), Merkle and Edelman (2002)
Measuring Nonresponse Bias
With response rates falling, understanding the impact on study findings is essential
Assess quantitatively if possible, but at least consider qualitatively
Methods for assessing nonresponse bias: Response rate comparisons across groups Follow-up interviews with non-respondents Comparing early vs. late respondents Comparisons to similar estimates from other
sources Post survey adjustment for nonresponse
Ways Not to Approach Nonresponse
Ignore it Obscure it Omit it Fail to measure/comment on
potential associated bias Give up on it
Improving Response Rates
Multiple contacts Multiple modes of contact Interviewer training Advance Notification/Endorsement Incentives Reduce respondent burden Increase relevance to respondent Time and money to get quality
Reporting Response Rates
Know and apply standards (AAPOR, CASRO)
Engage and manage with vendors Structure sample disposition Response and nonresponse analysis Publish enough information to allow others
to do the calculations Peer review: build awareness of standards