Page 1 Measuring Survey Quality through Representativity Indicators using Sample and Population...

Measuring Survey Quality through Representativity Indicators using

Sample and Population based Information

Chris Skinner, Natalie Shlomo,

Barry Schouten, Jelke Bethlehem

Li-Chun Zhang,

Representativity Indicators for Survey Qualitycollaboration between:

national statistical institutes of Netherlands, Norway and Slovenia

& universities of Leuven & Southampton

Representativity Indicators

• quality indicators for survey non-response

• to supplement response rate

• to measure how well respondents represent population

• tools for use at different stages of survey process (data collection+)

Aim of paper

• to consider estimation of representativity indicators using either:

• sample-based information (microdata for both respondents and nonrespondents), or

• population-based information (microdata for respondents and aggregate data for population)

How to define representativity indicator?

two approaches

• both based on idea of response propensity

Response propensity

Idea: R-indicator measures homogeneity of response propensities:

= probability of response for unit i

given values of auxiliary variables

i

R-indicator

where

if constant if

Schouten, Cobben & Bethlehem (2009, Survey Methodology)

( ) 1R ρ

21( ) ( )

1 i UUS

N

ρ

( ) 1 2 ( )R S ρ ρ

( ) 0R ρi( ) 0.5S ρ

Alternative Indicator

where

proposed by Särndal and Lundström (2008, JOS) in the context of selecting auxiliary variables for weighting adjustment

2 1 2( ) [ ] [ ( ) ]i i iU UQ ρ

1i i

Estimated R-indicator

Sample-based – auxiliary variables recorded for whole sample

(1) estimate response propensities using e.g. logistic regression

(2) 21ˆ ˆˆ( ) 1 2 ( )1 i i Us

R dN

ρ


Population-based – auxiliary variables only measured on respondents and in aggregated form for population

(1) Estimate response propensities using ordinary least squares

if population covariance matrix known. Estimate from respondents if only population

mean vector known

xxS

1'[ ( ')]OLSi i xx i ir

x N S xx d x

xxSx


Population based (continued)

(2) 1 21ˆ ˆˆ ˆ( ) 1 2 ( )1r i i i rr

R dN

ρ

Simulation Study

• Samples from Israel census data on 753000 individuals

• ‘realistic’ sampling with fractions 1:50, 1:100, 1:200

• ‘realistic’ non-response based on type of locality, household size, children in household – overall response rate 82%

Simulation Means of for Sample (S), Population with Known Covariance matrix (PC) and Population with Unknown Covariance Matrix (PUC) of Auxiliary Variables

ˆ(ρ)R

0.8

0.81

0.82

0.83

0.84

0.85

0.86

0.87

0.88

0.89

0.5% 1.0% 2.0%

True Model S based True Model PC based

True Model PUC based Less Complex S based

Less Complex PC based Less Complex PUC based

. 5% S . 5% PC . 5% PUC 1% S 1% PC 1% PUC 2% S 2% PC 2% PUC

0. 70

0. 75

0. 80

0. 85

0. 90

0. 95

R_indicator

gr oup

for Sample (S), Population with Known Covariance Matrix (PC) and Population with Unknown Covariance Matrix (PUC) of Auxiliary Variables – ‘True’ Model

)ρ(R̂


0. 70

0. 75

0. 80

0. 85

0. 90

0. 95

R_indicator

gr oup

for Sample (S), Population with Known Covariance Matrix (PC) and Population with Unknown Covariance Matrix (PUC) of Auxiliary Variables – Less Complex Model

)ρ(R̂


0. 70

0. 75

0. 80

0. 85

0. 90

0. 95

1. 00

R_indicator

gr oup

Simulation Means of for Sample (S) and Population (P) Based Auxiliary Variables

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

0.5% 1.0% 2.0%

True Model S Based True Model P Based

Less Complex S Based Less Complex P Based

2q

for Sample (S) and Population (P) Based Auxiliary Variables

0. 5% S 0. 5% P 1% S 1% P 2% S 2% P

0

0. 01

0. 02

0. 03

0. 04

q_squared

gr oup

2q

Conclusions

• representativity indicators defined in terms of response propensities

• can estimate accurately given either sample-based or population-based information

• have also applied to surveys from RISQ countries

• and have examined bias-corrected estimators and estimators of standard errors of estimators

Page 1 Measuring Survey Quality through Representativity Indicators using Sample and Population...

Documents

Transcript of Page 1 Measuring Survey Quality through Representativity Indicators using Sample and Population...