Centraal Bureau voor de Statistiek

Challenges of redesigning household surveys and maintaining output quality

Menno CuppenPaul van der LaanWim van Nunspeet

Statistics Netherlands

Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

May 6, 2010

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Outline

• Driving forces

• Approach

• Maintaining quality

• Results and perspectives

• Conclusions

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Driving forces

• Efficiency

• Quality

• Flexibility

Constraints:

• (European) quality / user satisfaction

• Solution for discontinuities

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Approach

Means:

• More use of registrative data

• More use of CAWI and mixed mode designs

• Integration of surveys in one survey system

• “Re-use” of respondents

• Implementation of advanced estimation

techniques

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Survey Model

Optimal use of these means in integrated design for household surveys:

• Core module and topic modules

• Well-balanced use of mixed mode (CAWI - PAPI - CATI - CAPI)

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Core Questionnaire

• Household composition• Labour status• Health• Religion• Voting behaviour• Occupation in employment• Economic sector in employment• Highest level of education completed• Screening, telephone numbers, e-mail adresses

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Standard mixed mode strategy

CAWI (+ PAPI)

CATI CAPI

If tel.nr. available If not tel.nr. available

25 – 35 %

± 45% ± 35%

55 - 65% 35 - 45%

Refusal No contact

± 5%

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Mode effects

Design principle: Unimode questionnaire design

• Selection effects

• Context effects

• Satisficing

• Interviewer effects

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Maintaining output quality

• Explain discontinuities for core

variables

• Quality indicators (framework)

• Pilots and experiments

• Parallel designs

• Time series modelling

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Quality indicators

• Relevance

• Accuracy

• Timeliness

• Accessibility

• Comparability

• Coherence

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Results

2009:

• Core questionnaire in Health Survey

• LFS sample size reduction by using data from unemployment offices

• Structural time series estimators for monthly unemployment

• Small area estimators for sub-regional unemployment statistics

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Results

2010:

• Core questionnaire in LFS

• Mixed mode in LFS (CAPI – CATI)

• Mixed mode in Health Survey (CAWI – CATI – CAPI)

• Topic modules after Health Survey

2011/2012:

• CAWI in LFS

• More topic modules after core questionnaire

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Parallel designs

LFS [1] CATI

LFS [2] CATI

LFS [3] CATI

LFS [4] CATI

LFS [1] CAPI

2010

Q1 Q2 Q3 Q4

New design

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Parallel designs

LFS [3] CATI

LFS [4] CATI

LFS [5] CATI

LFS [1] CATI

LFS [2] CATI

LFS [3] CATI

LFS [4] CATI

LFS [1] CAPI

2010

Q1 Q2 Q3 Q4

LFS [2] CATINew design

Old design

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Parallel designs

LFS [1] CAPI

LFS [2] CATI

LFS [3] CATI

LFS [4] CATI

LFS [5] CATI

LFS [1] CATI

LFS [2] CATI

LFS [3] CATI

LFS [4] CATI

LFS [1] CAPI

2010

Q1 Q2 Q3 Q4

LFS [2] CATINew design

Old design, parallel run

Old design

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Issues

• Unpredictibility CAWI results and impact on fieldwork planning in a mixed mode design

• Changing geographical distribution CAPI-samples

• Mode effects (methodology & best practices)

• Survey integration

• Creating uni-mode questionnaires

• Households/LFS and CAWI

• Flexibility in operation (methodology – survey case management system)

Centraal Bureau voor de Statistiek Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands

Conclusions

• Quality can be maintained

• Efficiency targets met

• Flexible in meeting new national and

European needs

Centraal Bureau voor de Statistiek

Challenges of redesigning household surveys and maintaining output quality

Menno Cuppen May 6, 2010

Statistics Netherlands

Challenges of redesigning household surveys and maintaining output qualityStatistics Netherlands