How survey design affects analysis Susan Purdon Head of Survey Methods Unit National Centre for...
-
Upload
roberta-martin -
Category
Documents
-
view
212 -
download
0
Transcript of How survey design affects analysis Susan Purdon Head of Survey Methods Unit National Centre for...
How survey design affects analysis
Susan Purdon
Head of
Survey Methods Unit
National Centre for Social Research
Two general principles
• Weighting affects both survey estimates and standard errors
• Complex sample design (clustering, stratification) affects standard errors but not survey estimates (as long as survey isn’t weighted).
Why weights are used
Two main reasons:• Because sample units are selected with non-equal
probabilities of selection• To reduce non-response bias
Weighting for non-equal probabilities
Non-equal probabilities of selection can be used:
• For good statistical reasons: stratification with disproportionate allocation (e.g over-sampling by region)
• For convenience: because sampling frame is of larger units than survey units, and need to sub-sample within units (e.g selection of one person per household).
• Weights are calculated as the inverse of the probability of selection.
Example 1: WERS98
Population Sample Samplingfraction (1
in ..)
Weight
10-24 197358 362 545 545
25-49 76087 603 126 126
50-99 36004 566 64 64
100-199 18701 562 33 33
200-499 9832 626 16 16
500+ 3249 473 7 7
Effect of selecting one adult per household
H’hldsize
H’hlds (per 100) Adults Adults selected Weight
1 38 38 38 1
2 51 102 51 2
3 9 27 9 3
4+ 2 8 2 4
Total 100 175 100
Weighting for non-response
• Many surveys use weights to reduce non-response bias.
• Up-weight low responding groups; down-weight high responding groups.
Weighting for non-response
Age Sample Population(pro-rata on
fullresponsesample)
Response rate Weight
16-24 510 920 55.4 1.8
25-34 650 1080 60.2 1.7
35-44 340 500 68.0 1.5
1500 2500
Effect of weights on estimates
• Weighting changes almost all survey estimates (means, percentages, odds ratios, correlation coefficients, regression coefficients etc.)
Effect of weights on estimates
Unweighted estimate=
(25%x38 + 17%x102+15%x27+9%x8)
/(38+102+27+8)=20%
Weighted estimate=
(25%x38x1 + 17%x102x2+15%x27x3+9%x8x4)
/(38x1+102x2+27x3+8x4)=18%
H’hldsize
Adults Weight % in poorhealth
1 38 1 25
2 102 2 17
3 27 3 15
4+ 8 4 9
Total 175 100
Effect of weighting on standard errors
• Standard errors for weighted and unweighted estimates are not the same.
• Weighting because of non-equal probabilities of selection tends to increase standard errors.
• Weighting for non-response sometimes increases/sometimes decreases ses. Impact tends to be smaller.
Impact of other design features - clustering
• Most face-to-face surveys are clustered• Clustering doesn’t change estimates, but it does
increase standard errors• Degree of increase depends on cluster size and
cluster homogeneity• To account for clustering need to identify the
primary sampling unit (psu) on dataset.
Impact of other design features - proportionate stratification
• Most surveys use proportionate stratification (either overall or within regions)
• Does not affect estimates. Tends to reduce standard errors. Degree depends on choice of stratifiers.
In summary:
• To get unbiased estimates need to use survey weights.
• To get correct standard errors need to take into account survey design, in particular weighting, clustering and stratification.