Whom are we talking about? Exclusion, noncoverage and nonresponse in social s tatistics

Whom are we talking about?

Exclusion, noncoverage and nonresponse in

social statistics (and social surveys)

Ineke Stoop

Vienna, Q2014, June 2014




People living in The Netherlands




Non-citizens




Non-citizens

Age limits




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds

Abroad longer period




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds


Lang

uage

prob

lems




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds


Lang

uage

prob

lems

No landline phone




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds

Health problems

Lang

uage

prob

lems

No landline phone

No inte

rnet




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds


Lang

uage

prob

lems

No landline phone

No inte

rnet

Not in population register




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds


Lang

uage

prob

lems

No landline phone

No inte

rnet


No-go a

rea




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds


Lang

uage

prob

lems

No landline phone

No inte

rnet


No-go a

rea Noncontact




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds


Lang

uage

prob

lems

No landline phone

No inte

rnet


No-go a

rea Noncontact

Mentally or physically unable




Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds


Lang

uage

prob

lems

No landline phone

No inte

rnet


No-go a

rea Noncontact


Refu

sal



Non-citizens

Age limits

Non-re

siden

tial h

ouseh

olds


Lang

uage

prob

lems

No landline phone

No inte

rnet


Noncontact


Refu

sal


Conduct an interview


ContentWhat do we want?• Allow all members of the target population to participate in

our surveyDo we know what is happening?• Exclusion mechanisms• Availability of information on survey data collection process

What can we do?• Better documention• Become more inclusive

www.europeansocialsurvey.orgVienna, Q2014, June 2014

Aims ESS

• To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions

• To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond


ESS top data user countries (>100) (March 2014) (N=68,204)

Luxembourg

Korea, Republic of

Croatia

Cyprus

Australia

Philippines

Slovakia

Canada

Romania

Israel

Greece

Russian

Ireland

Finland

Austria

Switzerland

Italy

Spain

Norway

Netherlands

Belgium

0 1000 2000 3000 4000 5000 6000 7000


Strategy European Social Survey

• Face-to-face interviewing• CAPI/PAPI

• Random sampling• Different sampling frames

• High coverage• Translation +5% language minorities• But

– Exclusion non-residential population– Exclusion far away areas (Caribbean islands)

• Nonresponse• High target response rates: 70%• Specification of efforts, guidelines

CAPI

PAPI


Response rates ESS (selection all rounds)

Switzerl

and

Spain UK

German

y

Belgium

Norway

Denmark

Netherl

ands

Portug

al

Sweden

Sloven

ia

Finlan

d

Poland

0

10

20

30

40

50

60

70

80

2002 2004 2006 2008 2010 2012


Exclusion, noncoverage, nonresponse, cultural bias Different sampling frames

• Homeless• Illegal aliens• Opt-out register

Small language minority groups• No on-the-fly translation

Non-residential population• No upper age limit, but exclusion senior housing facilities• Cross-national differences

Not able due to mental or physical incapability• Survey mode

Cultural background• How important is it that immigrants have a Christian background?


Response outcomes European Social Survey, R5

BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IL LT NL NO PL PT RU SE SI SK UA0

10

20

30

40

50

60

70

80

90

100interview noncontact refusal not able/other

Report on quality assessment of contact data files in Round 5: Final report 27 countriesHideko Matsuo and Geert LoosveldtWorking paper Centre for Sociological Research (CeSO) Survey Methodology CeSO/SM/2013-3


40.00 45.00 50.00 55.00 60.00 65.00 70.00 75.00 80.00 85.000.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

BE

BG

HR

CYCZ

DK EEFI

FR

DE

GR

HU

IE

IL

NLNO

PLPTRU

SK

SI

ES

SECH

UA

UK

Response R4 %

R5%


0.00 1.00 2.00 3.00 4.00 5.00 6.000.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

BE

BGHR

CY

CZ

DK

EE

FI

FR

DEGR

HU

IEIL

NLNO

PL PTRUSK

SI

ES

SE

CH

UA

UK

Language R4 %

R5%


0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.000.00

1.00

2.00

3.00

4.00

5.00

6.00

BE

BG

HR

CY CZ

DK

EE

FI

FR

DE

GR

HU

IE

IL

NL

NO

PL

PT

RUSK

SIES

SE

CH

UA

UK

Not able R4 %

R5%


0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.000.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

BE

BG

HR

CY

CZ DK

EE

FI

FR

DE

GR

HU

IE

IL

NL

NO

PL

PTRU SK

SI

ES

SE

CH

UA

UK

Unavailable R4 %

R5%


Comparison official statisticsHealth surveys, LFS, EU-SILCMethodological information hard to findComparable response rates, effective sample size?• (EU-SILC 2010): “The effective sample size is not included

in the following table due to a lack of comparability among the methodologies used by countries for computing the design effect”.


Response outcomes EU SILC 2010

AT BE BGCH CY CZ DE DK EE EL ES FI FR HRHU IE IS IT LT LU LV MT NL NO PL PT RO SE SI SK UK0

10

20

30

40

50

60

70

80

90

100interview Interview rejected Refusal to cooperate AwayUnable to respond Other reasons Noncontacted

2010 COMPARATIVE EU INTERMEDIATE QUALITY REPORT Version 3 – October 2012


Undercoverage LFS

Eurostat, Quality report of the LFS 2012

People living in new dwellings Recent immigrants People without a fixed phone (LU, CH) Students in student halls People in non-private households Homeless people and other people without registered residence (e.g. people living in

huts, caravans) (DE). Hard-to-access groups are characterized either by extremely bad traffic conditions to

get to their place or by collective reluctance – usually within a small community – towards being interviewed (HU).

Population living in collective households or in dwellings outside the borders of built areas is not covered (EL).

New dwellings underrepresented in the sample, also population living in collective (institutional) households staying/or planning to stay in this places for over a year, homeless people and other people without registered residence (e.g. people living in huts, caravans) are out of the frame, emigrants staying abroad for more than one year (PL).

Substitution (IT) In many cases no or little information provided


Exclusion mechanismsOutdated or incomplete sampling framesExclusion particular groups• Non-private households• Far away, no-go areas• Difficult

Nonparticipation• Not able for mental or physical reasons• Language problems

Survey mode• Functional illiteracy


Do we know what is happeningNot reallyNon-standardised informationUnavailable informationMany groups drop out for different reasonEffect on statistics• The very elderly in the LFS• The illiterate in the LFS• The very elderly in health surveys


General population Subgroups Minority group

Precision and bias

Comparing groups Specific survey


1. General populationComplete population• Unemployment• Health• Political efficacy• Social participation

Survey problem• No identification necessary, as long as they take part• Precision: dependent on group size• Bias: Survey participation related to output variable

(through subgroup membership)

General population


2. General population: subgroupsDistinctions• Socio-demographic groups, deviant groups, analysis

Survey problems• In surveys, registrations• Have to be identified• Comparing groups

Standardised design

Subgroups


3. Specific groupTarget groupsSpecific topicsAdapted questionnaire

Survey problems• Sampling frame• Cooperation• Language• Culture• Measurement issues

Minority group


Paying more efforts to make it possible that everybody can

participate


Cons Pros• Efforts to include groups that

are not going to respond• Questionnaire not relevant• Costs• Small effect on final results• Responsibility respondent

• Completeness• Data quality• Cross-national comparability• Reporting on small subgroups• Fairness• Responsibility statistician,

researcher


Thank you for your [email protected]

Whom are we talking about? Exclusion, noncoverage and nonresponse in social s tatistics

Documents

Transcript of Whom are we talking about? Exclusion, noncoverage and nonresponse in social s tatistics