How representative are the samples? Sabine Häder and Peter Lynn European Social Survey (ESS) –...

How representative are How representative are the samples?the samples?

Sabine Häder and Peter LynnSabine Häder and Peter Lynn

European Social Survey (ESS) –European Social Survey (ESS) –

Launch ConferenceLaunch Conference

Brussels, 25/26 November 2003Brussels, 25/26 November 2003

Häder/Lynn Häder/Lynn Representativeness of SamplesRepresentativeness of Samples 22

Sampling: Important part of a survey!Sampling: Important part of a survey! Precondition for comparability of countriesPrecondition for comparability of countries What do you need for achieving high-quality What do you need for achieving high-quality

comparable samples?comparable samples? In an In an ideal worldideal world you have you have

complete complete framesframes, from which you can select, from which you can selectrandomlyrandomly (with known probabilities) (with known probabilities)manymany individuals individualswho all like to who all like to respondrespond..


..... Sounds easy.......... Sounds easy.....

But: Could be found only in a minority ofBut: Could be found only in a minority of

ESS countriesESS countries

In reality:In reality:

Considerable variation between countriesConsiderable variation between countries

in constraints (e.g. availability of samplingin constraints (e.g. availability of sampling

frames) and survey practiceframes) and survey practice


Task: Minimising the impact of this variationTask: Minimising the impact of this variation

Requirements for sample designs:Requirements for sample designs:The same population (residents 15 years and The same population (residents 15 years and older)older)Same precision of results (Effective sample Same precision of results (Effective sample size of 1.500)size of 1.500)Highest achievable response ratesHighest achievable response ratesRecording sample design characteristics (e.g. Recording sample design characteristics (e.g. inclusion probabilities)inclusion probabilities)


Organisation of the work packageOrganisation of the work packageExpert PanelExpert Panel

Sabine Häder Sabine Häder (Centre for Survey Research and Methodology, Germany)(Centre for Survey Research and Methodology, Germany)

Siegfried GablerSiegfried Gabler (Centre for Survey Research and Methodology, Germany)(Centre for Survey Research and Methodology, Germany)

Seppo Laaksonen Seppo Laaksonen (Statistics Finland)(Statistics Finland)

Peter Lynn Peter Lynn (University of Essex, U.K.)(University of Essex, U.K.)


Guidance:Guidance:Identifying suitable sampling frames (e.g. Identifying suitable sampling frames (e.g. Austria)Austria)Developing new sample designs (e.g.Greece)Developing new sample designs (e.g.Greece)Maximising response rates (e.g. Switzerland)Maximising response rates (e.g. Switzerland)Implementing concept of design effects (e.g. Implementing concept of design effects (e.g. France)France)

Finally: ‚Signing off‘ the sample designsFinally: ‚Signing off‘ the sample designsResult: Best probability samples in all countriesResult: Best probability samples in all countries

—› —› As close to the ideal world as possibleAs close to the ideal world as possible


Variation in procedures to achieve Variation in procedures to achieve equivalence of outcomes: 2 examplesequivalence of outcomes: 2 examples

Advances in survey design and practice: 2 Advances in survey design and practice: 2 examplesexamples

Use of design weights in data analysisUse of design weights in data analysis

Some lessons for the futureSome lessons for the future


Variation in procedures to achieve Variation in procedures to achieve equivalence of outcomes:equivalence of outcomes:

Example 1: Age range 15+ (no upper cut-off)Example 1: Age range 15+ (no upper cut-off)

Address-based samples (e.g.UK, Greece)Address-based samples (e.g.UK, Greece)

Population register samples (e.g. Finland, Population register samples (e.g. Finland, Denmark)Denmark)

Electoral register sample (Italy)Electoral register sample (Italy)


Variation in procedures to achieve equivalence Variation in procedures to achieve equivalence of outcomes:of outcomes:

Example 2: Effective sample size 1,500+Example 2: Effective sample size 1,500+

Equal probability unclustered samples (e.g. Finland, Equal probability unclustered samples (e.g. Finland, Denmark)Denmark)

Variable probabilities due to address-based Variable probabilities due to address-based sampling (e.g. UK, Greece, Portugal)sampling (e.g. UK, Greece, Portugal)

Variable probabilities due to local aims (e.g. Variable probabilities due to local aims (e.g. Luxembourg)Luxembourg)

Clustered samples (most, but to different extents)Clustered samples (most, but to different extents)


Example 2: Effective sample size, continuedExample 2: Effective sample size, continued

Design effectsDesign effects

ClustClust AddAdd OtherOther TotTot Equiv. Equiv. precisionprecision

SloveniaSlovenia 1.401.40 1.001.00 1.001.00 1.401.40 2,1002,100

NetherlandsNetherlands 1.001.00 1.191.19 1.001.00 1.191.19 1,7851,785

GermanyGermany 1.381.38 1.001.00 1.101.10 1.521.52 2,2802,280

IrelandIreland 1.201.20 1.331.33 1.001.00 1.601.60 2,4002,400

IsraelIsrael 1.221.22 1.291.29 1.011.01 1.591.59 2,3852,385

SwedenSweden 1.001.00 1.001.00 1.001.00 1.001.00 1,5001,500


Advances in survey design and practice:Advances in survey design and practice:

Example 1: National probability sample of Example 1: National probability sample of households in Greecehouseholds in Greece

Probability samples previously used only by NSIProbability samples previously used only by NSI

No list of persons or addresses available to other No list of persons or addresses available to other organisationsorganisations

Quota sampling commonQuota sampling common

Exclusion of rural areas and most islands Exclusion of rural areas and most islands commoncommon



Example 1: Greece (continued)Example 1: Greece (continued)

Area-based probability sampling approachArea-based probability sampling approach

Census EAs as PSUsCensus EAs as PSUs

Field listing of addressesField listing of addresses

In-office selection of addresses from the listIn-office selection of addresses from the list

Carried out successfully - and good response Carried out successfully - and good response rate!rate!



Example 2: Inclusion of 15-17 year-olds in Example 2: Inclusion of 15-17 year-olds in electoral register based sample in Italyelectoral register based sample in Italy

Equal probability sampleof electors (18+)Equal probability sampleof electors (18+)

Interviewer lists all residents 15+ (r) at address of Interviewer lists all residents 15+ (r) at address of selected elector and randomly selects oneselected elector and randomly selects one

Interviewer also asks number of electors at address Interviewer also asks number of electors at address (e)(e)

Selection probability is Ke/r: all persons 15+ have Selection probability is Ke/r: all persons 15+ have known selection probabilityknown selection probability


Design weightsDesign weights

In most countries, selection probabilities were In most countries, selection probabilities were unequalunequal

We ensured that the probabilities are knownWe ensured that the probabilities are known

We have converted these into a „design weight“ We have converted these into a „design weight“ that should be used in all analysesthat should be used in all analyses

e.g. address-based samples (Greece, UK, e.g. address-based samples (Greece, UK, Portugal ...): probability is inversely proportional Portugal ...): probability is inversely proportional to number of persons in householdto number of persons in household


Design weights: ExampleDesign weights: Example

CZCZ ESES GRGR FIFI SESE

V1, unwV1, unw 25.725.7 14.914.9 18.718.7 22.922.9 21.121.1

V1, wtdV1, wtd 11.911.9 6.16.1 8.08.0 22.922.9 21.121.1

V2, unwV2, unw 26.326.3 24.224.2 38.038.0 13.813.8 11.611.6

V2, wtdV2, wtd 23.423.4 22.722.7 35.835.8 13.813.8 11.611.6

V1 = Single-person household (%)V1 = Single-person household (%)

V2 = 3+ hrs/week watching TV (%)V2 = 3+ hrs/week watching TV (%)


Lessons for the futureLessons for the future

Central co-ordination of sample design and close Central co-ordination of sample design and close liaison with national teams necessaryliaison with national teams necessary

Necessary not only to „agree“ the design, but also to Necessary not only to „agree“ the design, but also to monitor and discuss implementationmonitor and discuss implementation

Central team must be seen as „helping,“ not Central team must be seen as „helping,“ not „controlling“„controlling“

Specification of effective sample size was successfulSpecification of effective sample size was successful

Methodological research into sample designMethodological research into sample design

How representative are the samples? Sabine Häder and Peter Lynn European Social Survey (ESS) –...

Documents

Transcript of How representative are the samples? Sabine Häder and Peter Lynn European Social Survey (ESS) –...