Comparing different weighting procedures for

volunteer online panels

Stephanie Steinmetz and Kea Tijdens

AIAS Lunch Seminar, 1. October 2009

erasmus studio

Outline

BackgroundSources of errors in ((volunteer) web) surveysWeighing - a solution?Example for the German and Dutch WageIndicator dataResultsConclusion and Outlook

Background

Increasing importance of web surveysGermany: between 2000 und 2007 from 3% to 27% (ADM, 2007)

Advantagestime and cost reduction, interactivity, flexibility, ‘worldwide’ coverage, no interviewer influence

DisadvantagesRepresentativeness? To what degree are(volunteer) web survey results representative of the general public?

Types of web surveys (see Couper, 2000)

Sample selection isprobability based = representative- intercept surveys,- online access panels,- mixed-mode surveys

not probability based = representative? - entertainment surveys - self-selected web surveys - volunteer online panels

Sources of error

Combination of causes1 (Non)Coverage: number of people having internet access +

differences between the persons with and without internet access.

2 Sampling/Self-selection: no comprehensive list of Internet users to draw probability-based sample + people with specific characteristics participate in a volunteer online panel.

3 Non-response: Not all persons finish the questionnaire, people with specific characteristics might have a higher non-response.

+ measurement, processing and adjustment errors

Weighting - a possible solution?

Weighting is a mean to correct subsequently for systematic survey errors and to adjust the sample to the target population.

Expectation disappearance of significant differences between web survey & random reference survey.☺ = web survey data can be adjusted to be representative

of general public.= persistence of differences due to other error sources,

like measurement and processing errors

Solution: Post-stratification weighting

Aim: Adjustment for demographic under- and over-representations between sample and target population

Method: %population (reference data) / %sample (web) = weighting coefficient

Findings: Necessary but has a rather limited impact (Vehovar et al. 1999, Loosvelt and Sonck 2008)

corrects for proportionality but not necessarily for representativeness of substantive answers

...but

Previous research comparing web and traditional methods

Significant differences can be observed for web respondents. They...– are more intensive users of the Internet, more

technically-oriented (Bandilla et al. 2003; Vehovar et al. 1999)

– have a larger social trust & a greater subjective control over their lives (Lenhart et al. 2003)

– are more politically and socially active (Duffy et al. 2005)

Solution: Propensity Score Adjustment (PSA)

Origin: experimental studies (Rosenbaum & Rubin, 1983)

Aim: to correct for differences due to the varying inclination to participate in web surveys (Harrison Interactive).

Findings: Mixed (Taylor 2005; Bethlehem & Stoop 2007)

- some differences disappeared by demographicweighting,

- some only after additional PSA, and

- others continued to exist or become even larger

PSA - method (see Schonlau et al. 2009)

Web and probability-based reference survey are combined in one data fileLogistic regression of people’s probability to participate in the web survey given demographic and/or attitudinal variables estimation of PSMake distribution of these propensity scores similar for web survey and random sample = calculation of weight wpsi (1/ psi if W = 1 (in the web survey), and 1/(1-psi) if W = 0 (in the reference survey))

web survey and random sample do not differ significantly for selected variables included in the PS

Example - the WageIndicator data

Web surveys: German and Dutch WageIndicatordata, year 2006, employees, age 16-75, cross monthly income 400€-10000€ (Dutch net hourly income)

NGerman= 21914NDutch = 8015

Reference surveys: Same restrictions Germany (GSOEP, 2006) N= 7993Netherlands (OSA, 2006) N= 2019

Selection bias - socio-demographics

02040

6080

100

men

wom

en

low

med

ium

high

16-3

4

35-4

4

45-7

5

sex education cohort

LS SOEP

02040

6080

100

men

wom

en

low

med

ium

high

16-3

4

35-4

4

45-7

5

sex education cohort

LW OSA

Germany Netherlands

Selection bias - Labour markert

0

20

40

60

80

100

manual non manual full part below above

occupation working time unemployment

LS SOEP

020406080

100

manual nonmanual

full part below above

occupation working time unemployment

LW OSA

Germany Netherlands

Selection bias - satisfaction

0

20

40

60

80

100

not satisfied satisfied not satisfied satisfied

health satisfaction job satisfaction

LS SOEP

0

20

40

60

80

100

not satisfied satisfied not satisfied satisfied

health satisfaction job satisfaction

LW OSA

Germany Netherlands

Summary

Similarities: underrepresentation of women, people between 45 und 75, part-timers,persons from regions with high unemployment,unsatisfied people

Differences: underrepresentation ofDE: highly educated, manual workersNL: low and medium educated, non-manual workers

Two possible solutionsa) Post-stratification weightingb) PSA

Weights

A) 6 post-stratification weights: W1= gender (2), education (2) and cohort (2) W2= gender (2), education (2), cohort (2) and part time (2)W3= gender (2), education (2), cohort (2) and nonmanual (2)W4= gender (2), education (2), cohort (2), part time (2) and jobsatW5= gender (2), education (2), cohort (2), nonmanual (2) and jobsatW6= part(2) and jobsat(2)

B) 4 PSA weightsPS1 = treat women edu2 coh2 nonman part perm nojob logwagemoPS2 = treat women edu2 coh2 nonman part perm nojob logwagemo +

healthsatPS3 = treat women edu2 coh2 nonman part perm nojob logwagemo +

jobsatPS4 = treat women edu2 coh2 nonman part perm nojob logwagemo +

healthsat jobsat

Results: Germany – Mean income

0 € 50 € 100 € 150 € 200 € 250 € 300 € 350 €

Diff

Diff1

Diff2

Diff3

Diff4

Diff5

Diff6

PS1

PS2

PS3

PS4

Diff Diff1 Diff2 Diff3 Diff4 Diff5 Diff6 PS1 PS2 PS3 PS4

Results: Germany – distributions

-30

-20

-10

0

10

20

30

men women 16-34 35-65+ lowmed medhigh manual nonman full part dissat sat

Gender Cohort Education Nonmanual Part-time Jobsat

Diff DiffW2 DiffW6 DiffPS1 DiffPS2

Results: Germany - Income Regression

Results: NL – mean income

-0,6 € -0,5 € -0,4 € -0,3 € -0,2 € -0,1 € 0,0 € 0,1 € 0,2 €

Diff

Diff1

Diff2

Diff3

Diff4

Diff5

Diff6

PS1

PS2

PS3

PS4

Diff Diff1 Diff2 Diff3 Diff4 Diff5 Diff6 PS1 PS2 PS3 PS4

Results: NL - distributions

-40

-30

-20

-10

0

10

20

30

40

men women 16-34 35-65+ lowmed medhigh manual nonman full part dissat sat

Gender Cohort Education Nonmanual Part-time Jobsat

Diff DiffW1 DiffW2 DiffW3 DiffPS1 DiffPS3

Results: Netherlands - Income Regression

Conclusion

Impact weightingBoth weighting methods show no substantial impact

Moreover- no consistency within weights- for some weights differences become larger (?!)- effect of weights differ between countries

Weighting cannot improve representativeness of (volunteer) web surveys

Problems- Reference surveys (also biased?), mode effects,

unobservables (not measured)

Discussion

Possible solutions for representativeness:Improving weights through inclusion of morevariables or advanced/mixed weighting proceduresOnly mixed-mode surveys (time and cost-reduction disappears)Non-representative use of web survey data

(only for experiments or exploratory analysis) OR

questioning the definition of representativeness (content vs. methodological)survey quality ≠ absolute

Representativness of surveys

0%

2%

4%

6%

8%

10%

12%

14%

Parttim

e Male

15-24

yr hi

gher

Fullti

me Male

15-24

yr hi

gher

Parttim

e Fem

ale 15

-24 yr

high

er

Fullti

me Fem

ale 15

-24 yr

high

er

Fulltim

e Fem

ale 15

-24 yr

lower

Parttim

e Male

25-44

yr lo

wer

Parttim

e Male

45-64

yr lo

wer

Fulltim

e Fem

ale 45

-64 yr

lower

Parttim

e Male

15-24

yr m

iddle

Parttim

e Male

15-24

yr lo

wer

Fulltim

e Fem

ale 25

-44 yr

lower

Parttim

e Fem

ale 15

-24 yr

lower

Fullti

me Fem

ale 15

-24 yr

midd

le

Parttim

e Male

45-64

yr m

iddle

Parttim

e Male

25-44

yr hi

gher

Fullti

me Fem

ale 45

-64 yr

high

er

Fulltim

e Fem

ale 45

-64 yr

midd

le

Parttim

e Male

25-44

yr m

iddle

Parttim

e Male

45-64

yr hi

gher

Fullti

me Male

15-24

yr lo

wer

Parttim

e Fem

ale 15

-24 yr

midd

le

Fulltim

e Male

15-24

yr m

iddle

Parttim

e Fem

ale 25

-44 yr

lower

Parttim

e Fem

ale 45

-64 yr

high

er

Parttim

e Fem

ale 45

-64 yr

lower

Fullti

me Fem

ale 25

-44 yr

midd

le

Fullti

me Fem

ale 25

-44 yr

high

er

Parttim

e Fem

ale 45

-64 yr

midd

le

Fullti

me Male

45-64

yr lo

wer

Parttim

e Fem

ale 25

-44 yr

high

er

Fullti

me Male

25-44

yr lo

wer

Fullti

me Male

45-64

yr hi

gher

Fulltim

e Male

45-64

yr m

iddle

Parttim

e Fem

ale 25

-44 yr

midd

le

Fullti

me Male

25-44

yr hi

gher

Fulltim

e Male

25-44

yr m

iddle

World Value Survey_NL_% _1999

Labour Force Survey_NL_% _2005

WageIndicator_NL_% _2005

Telepanel_NL_% _2002

Outlook

Comparison: more countries

Methods: - Combination of different weighting

techniques (see Lee & Valliant, 2009)

- Weighting with ‚better‘ reference survey and more webograhic variables (LISS panel= parallel survey, identical questionnaire + same mode)

The end

Thanks' for listening......questions ?...comments and suggestions?

contact: steinmetz@fsw.eur.nl

erasmus studio