117.7.2013

Mixed Mode solutions to coverage and nonresponse error:

Evaluation of the cost-error trade-off

Presentation at the ESRA 2013, July 17th

“Is it worth mixing mode? New evidences on costs and survey error on mixed-mode surveys” Session

Michèle Ernst Stähli, Caroline Roberts and Dominique JoyeFORS and NCCR LIVES, University of Lausanne

217.7.2013

Overview

1. General design of the Swiss mixed mode experiment

2. Coverage errors in telephone samples

3. How to survey population not having an available telephone number (best mode or combination of modes)

4. Non-response errors on socio-demographics

5. Cost-error trade-off

317.7.2013

The challenges

Under-coverage in the telephone sampling Declining response rates and potential non-response

bias Rising costs for high quality surveys (especially FtF)

No single mode offers adequate solution

Mixing modes, but effects of different designs have to be empirically explored in national context

417.7.2013

The questionnaire

Concentrated on the topic of well-being

Mainly based on the ESS 2012 (without the democracy module)

Integration of central NCCR LIVES questions Strong socio-demographic module

Duration: 25-30 minutes

517.7.2013

Principles of the mixed mode design

Pure probability sample of 3600 residents of French speaking region of Switzerland (drawn from population register)

Cost-efficiency => sequential design (web-paper-cati/capi) Parallel modes for comparability of pure mode effects ESS 2012 CAPI as benchmark Specific sample of units without available telephone number

Supplementary principle: Respect of Dillman’s rules of the “tailored design method” (2009)

in the contact procedures

617.7.2013

The overall designgross sample size (N)

principal mode

N 1st contact (day 1)

2nd contact (day 4)

3rd contact (day 11)

4th contact (day 26)

5th contact (day 57)

6th contact (day 82)

TELEPHONE 600 prenotif ication letter w ith incentive

PAPI NRS by registered mail

end

PAPER 500 prenotif ication letter

PAPI letter w ith questionnaire and incentive

reminder by postcard

PAPI NRS by registered mail

end

WEB 1000 prenotif ication letter

CAWI letter w ith URL, code and incentive

reminder by postcard w ith URL

URL + PAPI CATI URL + NRS by registered mail

PAPER 500 prenotif ication letter

PAPI letter w ith questionnaire and incentive

reminder by postcard

PAPI NRS by registered mail

end

WEB 1000 prenotif ication letter

CAWI letter w ith URL, code and incentive

reminder by postcard w ith URL

URL + PAPI CAPI URL + NRS by registered mail

Incentive: 10.- CHF cashCATI: 25 minutes call on fixed line telephonePAPI: Printed paper questionnaireCAWI: Web survey (personalized access w ith URL and access code)CAPI: Face to Face contacts w ith previous contact letter and interview (25-30 minutes)NRS: Paper Non-Response Survey (1 page 2 sided)

ESS 2012N=2900 (58% have an available tel. number)

CAPI 60 min. FtF interv

2900 NRS

WITH available telephone number (N= 2100)

WITHOUT available telephone number (N= 1500)

contact letter, 5 FtF contacts, card for non-contacts, CATI contact, FtF refusal conversion, incentive 30.-

CATI calls, unlimited number of call attempts

717.7.2013

Fieldwork progressions

817.7.2013

Fieldwork progression (2)

917.7.2013

Completion rates by mode and initial sample

Answers by modePaper 1106CAWI 890CATI 406CAPI 81Total 2483

1017.7.2013

Coverage errors in telephone samplesno

t sig

.

1117.7.2013

Coverage errors in telephone samples

=> underrepresentation of: - young- unmarried- single- urban- foreign populations

Additional nonresponse errors on same characteristics!

How to address this coverage problem:- search for additional numbers, RDD, CATI on mobile phones- offering other mode(s) or mix of modes

1217.7.2013

How to survey population without telnumber

1317.7.2013

Nonresponse errors in samples without telephone number

1417.7.2013

Cost structures of data collection

1517.7.2013

Cost structures of data collection

1617.7.2013

Conclusions

Real problem of under-coverage in the telephone sampling This problem is reinforced by nonresponse errors

Need to integrate population without telephone number All modes and sequences of modes show up a similar NR-bias

pattern, sometimes with different strength Paper seems to be less biased than web only, but more is

expensive (even more with larger samples)

Offering additional modes to the web decreases some NR-bias But disproportionate additional cost of adding CATI/CAPI follow-

ups for the Web group (overall, not more expensive than paper only)

1717.7.2013

Conclusions

Mixed mode solutions appear to offer an effective way of correcting for under-coverage and reducing non-response bias (on a selection of socio-demographic variables)

However, in this cost-error trade-off, we should also include the extent of additional and confounded measurement errors associated with combining telephone with self-administered data collection modes

1817.7.2013

THANK YOU!

For more information:

Caroline Roberts, UNIL: Caroline.Roberts@unil.ch

Michèle Ernst Stähli, FORS: Michele.ErnstStaehli@fors.unil.ch

Dominique Joye, UNIL: Dominique.Joye@unil.ch

1917.7.2013

Interest in politics by sample and mode