How representative are How representative are the samples?the samples?
Sabine Häder and Peter LynnSabine Häder and Peter Lynn
European Social Survey (ESS) –European Social Survey (ESS) –
Launch ConferenceLaunch Conference
Brussels, 25/26 November 2003Brussels, 25/26 November 2003
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 22
Sampling: Important part of a survey!Sampling: Important part of a survey! Precondition for comparability of countriesPrecondition for comparability of countries What do you need for achieving high-quality What do you need for achieving high-quality
comparable samples?comparable samples? In an In an ideal worldideal world you have you have
complete complete framesframes, from which you can select, from which you can selectrandomlyrandomly (with known probabilities) (with known probabilities)manymany individuals individualswho all like to who all like to respondrespond..
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 33
..... Sounds easy.......... Sounds easy.....
But: Could be found only in a minority ofBut: Could be found only in a minority of
ESS countriesESS countries
In reality:In reality:
Considerable variation between countriesConsiderable variation between countries
in constraints (e.g. availability of samplingin constraints (e.g. availability of sampling
frames) and survey practiceframes) and survey practice
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 44
Task: Minimising the impact of this variationTask: Minimising the impact of this variation
Requirements for sample designs:Requirements for sample designs:The same population (residents 15 years and The same population (residents 15 years and older)older)Same precision of results (Effective sample Same precision of results (Effective sample size of 1.500)size of 1.500)Highest achievable response ratesHighest achievable response ratesRecording sample design characteristics (e.g. Recording sample design characteristics (e.g. inclusion probabilities)inclusion probabilities)
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 55
Organisation of the work packageOrganisation of the work packageExpert PanelExpert Panel
Sabine Häder Sabine Häder (Centre for Survey Research and Methodology, Germany)(Centre for Survey Research and Methodology, Germany)
Siegfried GablerSiegfried Gabler (Centre for Survey Research and Methodology, Germany)(Centre for Survey Research and Methodology, Germany)
Seppo Laaksonen Seppo Laaksonen (Statistics Finland)(Statistics Finland)
Peter Lynn Peter Lynn (University of Essex, U.K.)(University of Essex, U.K.)
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 66
Guidance:Guidance:Identifying suitable sampling frames (e.g. Identifying suitable sampling frames (e.g. Austria)Austria)Developing new sample designs (e.g.Greece)Developing new sample designs (e.g.Greece)Maximising response rates (e.g. Switzerland)Maximising response rates (e.g. Switzerland)Implementing concept of design effects (e.g. Implementing concept of design effects (e.g. France)France)
Finally: ‚Signing off‘ the sample designsFinally: ‚Signing off‘ the sample designsResult: Best probability samples in all countriesResult: Best probability samples in all countries
—› —› As close to the ideal world as possibleAs close to the ideal world as possible
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 77
Variation in procedures to achieve Variation in procedures to achieve equivalence of outcomes: 2 examplesequivalence of outcomes: 2 examples
Advances in survey design and practice: 2 Advances in survey design and practice: 2 examplesexamples
Use of design weights in data analysisUse of design weights in data analysis
Some lessons for the futureSome lessons for the future
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 88
Variation in procedures to achieve Variation in procedures to achieve equivalence of outcomes:equivalence of outcomes:
Example 1: Age range 15+ (no upper cut-off)Example 1: Age range 15+ (no upper cut-off)
Address-based samples (e.g.UK, Greece)Address-based samples (e.g.UK, Greece)
Population register samples (e.g. Finland, Population register samples (e.g. Finland, Denmark)Denmark)
Electoral register sample (Italy)Electoral register sample (Italy)
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 99
Variation in procedures to achieve equivalence Variation in procedures to achieve equivalence of outcomes:of outcomes:
Example 2: Effective sample size 1,500+Example 2: Effective sample size 1,500+
Equal probability unclustered samples (e.g. Finland, Equal probability unclustered samples (e.g. Finland, Denmark)Denmark)
Variable probabilities due to address-based Variable probabilities due to address-based sampling (e.g. UK, Greece, Portugal)sampling (e.g. UK, Greece, Portugal)
Variable probabilities due to local aims (e.g. Variable probabilities due to local aims (e.g. Luxembourg)Luxembourg)
Clustered samples (most, but to different extents)Clustered samples (most, but to different extents)
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 1010
Example 2: Effective sample size, continuedExample 2: Effective sample size, continued
Design effectsDesign effects
ClustClust AddAdd OtherOther TotTot Equiv. Equiv. precisionprecision
SloveniaSlovenia 1.401.40 1.001.00 1.001.00 1.401.40 2,1002,100
NetherlandsNetherlands 1.001.00 1.191.19 1.001.00 1.191.19 1,7851,785
GermanyGermany 1.381.38 1.001.00 1.101.10 1.521.52 2,2802,280
IrelandIreland 1.201.20 1.331.33 1.001.00 1.601.60 2,4002,400
IsraelIsrael 1.221.22 1.291.29 1.011.01 1.591.59 2,3852,385
SwedenSweden 1.001.00 1.001.00 1.001.00 1.001.00 1,5001,500
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 1111
Advances in survey design and practice:Advances in survey design and practice:
Example 1: National probability sample of Example 1: National probability sample of households in Greecehouseholds in Greece
Probability samples previously used only by NSIProbability samples previously used only by NSI
No list of persons or addresses available to other No list of persons or addresses available to other organisationsorganisations
Quota sampling commonQuota sampling common
Exclusion of rural areas and most islands Exclusion of rural areas and most islands commoncommon
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 1212
Advances in survey design and practice:Advances in survey design and practice:
Example 1: Greece (continued)Example 1: Greece (continued)
Area-based probability sampling approachArea-based probability sampling approach
Census EAs as PSUsCensus EAs as PSUs
Field listing of addressesField listing of addresses
In-office selection of addresses from the listIn-office selection of addresses from the list
Carried out successfully - and good response Carried out successfully - and good response rate!rate!
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 1313
Advances in survey design and practice:Advances in survey design and practice:
Example 2: Inclusion of 15-17 year-olds in Example 2: Inclusion of 15-17 year-olds in electoral register based sample in Italyelectoral register based sample in Italy
Equal probability sampleof electors (18+)Equal probability sampleof electors (18+)
Interviewer lists all residents 15+ (r) at address of Interviewer lists all residents 15+ (r) at address of selected elector and randomly selects oneselected elector and randomly selects one
Interviewer also asks number of electors at address Interviewer also asks number of electors at address (e)(e)
Selection probability is Ke/r: all persons 15+ have Selection probability is Ke/r: all persons 15+ have known selection probabilityknown selection probability
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 1414
Design weightsDesign weights
In most countries, selection probabilities were In most countries, selection probabilities were unequalunequal
We ensured that the probabilities are knownWe ensured that the probabilities are known
We have converted these into a „design weight“ We have converted these into a „design weight“ that should be used in all analysesthat should be used in all analyses
e.g. address-based samples (Greece, UK, e.g. address-based samples (Greece, UK, Portugal ...): probability is inversely proportional Portugal ...): probability is inversely proportional to number of persons in householdto number of persons in household
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 1515
Design weights: ExampleDesign weights: Example
CZCZ ESES GRGR FIFI SESE
V1, unwV1, unw 25.725.7 14.914.9 18.718.7 22.922.9 21.121.1
V1, wtdV1, wtd 11.911.9 6.16.1 8.08.0 22.922.9 21.121.1
V2, unwV2, unw 26.326.3 24.224.2 38.038.0 13.813.8 11.611.6
V2, wtdV2, wtd 23.423.4 22.722.7 35.835.8 13.813.8 11.611.6
V1 = Single-person household (%)V1 = Single-person household (%)
V2 = 3+ hrs/week watching TV (%)V2 = 3+ hrs/week watching TV (%)
Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 1616
Lessons for the futureLessons for the future
Central co-ordination of sample design and close Central co-ordination of sample design and close liaison with national teams necessaryliaison with national teams necessary
Necessary not only to „agree“ the design, but also to Necessary not only to „agree“ the design, but also to monitor and discuss implementationmonitor and discuss implementation
Central team must be seen as „helping,“ not Central team must be seen as „helping,“ not „controlling“„controlling“
Specification of effective sample size was successfulSpecification of effective sample size was successful
Methodological research into sample designMethodological research into sample design
Top Related