Mixed Mode solutions to coverage and nonresponse error ... · Coverage errors in telephone samples...
19
1 17.7.2013 Mixed Mode solutions to coverage and nonresponse error: Evaluation of the cost-error trade-off Presentation at the ESRA 2013, July 17 th “Is it worth mixing mode? New evidences on costs and survey error on mixed-mode surveys” Session Michèle Ernst Stähli, Caroline Roberts and Dominique Joye FORS and NCCR LIVES, University of Lausanne
Transcript of Mixed Mode solutions to coverage and nonresponse error ... · Coverage errors in telephone samples...
117.7.2013
Mixed Mode solutions to coverage and nonresponse error:
Evaluation of the cost-error trade-off
Presentation at the ESRA 2013, July 17th
“Is it worth mixing mode? New evidences on costs and survey error on mixed-mode surveys” Session
Michèle Ernst Stähli, Caroline Roberts and Dominique JoyeFORS and NCCR LIVES, University of Lausanne
217.7.2013
Overview
1. General design of the Swiss mixed mode experiment
2. Coverage errors in telephone samples
3. How to survey population not having an available telephone number (best mode or combination of modes)
4. Non-response errors on socio-demographics
5. Cost-error trade-off
317.7.2013
The challenges
Under-coverage in the telephone sampling Declining response rates and potential non-response
bias Rising costs for high quality surveys (especially FtF)
No single mode offers adequate solution
Mixing modes, but effects of different designs have to be empirically explored in national context
417.7.2013
The questionnaire
Concentrated on the topic of well-being
Mainly based on the ESS 2012 (without the democracy module)
Integration of central NCCR LIVES questions Strong socio-demographic module
Duration: 25-30 minutes
517.7.2013
Principles of the mixed mode design
Pure probability sample of 3600 residents of French speaking region of Switzerland (drawn from population register)
Cost-efficiency => sequential design (web-paper-cati/capi) Parallel modes for comparability of pure mode effects ESS 2012 CAPI as benchmark Specific sample of units without available telephone number
Supplementary principle: Respect of Dillman’s rules of the “tailored design method” (2009)
in the contact procedures
617.7.2013
The overall designgross sample size (N)
principal mode
N 1st contact (day 1)
2nd contact (day 4)
3rd contact (day 11)
4th contact (day 26)
5th contact (day 57)
6th contact (day 82)
TELEPHONE 600 prenotif ication letter w ith incentive
PAPI NRS by registered mail
end
PAPER 500 prenotif ication letter
PAPI letter w ith questionnaire and incentive
reminder by postcard
PAPI NRS by registered mail
end
WEB 1000 prenotif ication letter
CAWI letter w ith URL, code and incentive
reminder by postcard w ith URL
URL + PAPI CATI URL + NRS by registered mail
PAPER 500 prenotif ication letter
PAPI letter w ith questionnaire and incentive
reminder by postcard
PAPI NRS by registered mail
end
WEB 1000 prenotif ication letter
CAWI letter w ith URL, code and incentive
reminder by postcard w ith URL
URL + PAPI CAPI URL + NRS by registered mail
Incentive: 10.- CHF cashCATI: 25 minutes call on fixed line telephonePAPI: Printed paper questionnaireCAWI: Web survey (personalized access w ith URL and access code)CAPI: Face to Face contacts w ith previous contact letter and interview (25-30 minutes)NRS: Paper Non-Response Survey (1 page 2 sided)
ESS 2012N=2900 (58% have an available tel. number)
CAPI 60 min. FtF interv
2900 NRS
WITH available telephone number (N= 2100)
WITHOUT available telephone number (N= 1500)
contact letter, 5 FtF contacts, card for non-contacts, CATI contact, FtF refusal conversion, incentive 30.-
CATI calls, unlimited number of call attempts
717.7.2013
Fieldwork progressions
817.7.2013
Fieldwork progression (2)
917.7.2013
Completion rates by mode and initial sample
Answers by modePaper 1106CAWI 890CATI 406CAPI 81Total 2483
1017.7.2013
Coverage errors in telephone samplesno
t sig
.
1117.7.2013
Coverage errors in telephone samples
=> underrepresentation of: - young- unmarried- single- urban- foreign populations
Additional nonresponse errors on same characteristics!
How to address this coverage problem:- search for additional numbers, RDD, CATI on mobile phones- offering other mode(s) or mix of modes
1217.7.2013
How to survey population without telnumber
1317.7.2013
Nonresponse errors in samples without telephone number
1417.7.2013
Cost structures of data collection
1517.7.2013
Cost structures of data collection
1617.7.2013
Conclusions
Real problem of under-coverage in the telephone sampling This problem is reinforced by nonresponse errors
Need to integrate population without telephone number All modes and sequences of modes show up a similar NR-bias
pattern, sometimes with different strength Paper seems to be less biased than web only, but more is
expensive (even more with larger samples)
Offering additional modes to the web decreases some NR-bias But disproportionate additional cost of adding CATI/CAPI follow-
ups for the Web group (overall, not more expensive than paper only)
1717.7.2013
Conclusions
Mixed mode solutions appear to offer an effective way of correcting for under-coverage and reducing non-response bias (on a selection of socio-demographic variables)
However, in this cost-error trade-off, we should also include the extent of additional and confounded measurement errors associated with combining telephone with self-administered data collection modes
1817.7.2013
THANK YOU!
For more information:
Caroline Roberts, UNIL: [email protected]
Michèle Ernst Stähli, FORS: [email protected]
Dominique Joye, UNIL: [email protected]
1917.7.2013
Interest in politics by sample and mode
