© GfK 2016

Towards a New Math for Non-probability Sample Surveys

(Next Steps)

1

Mansour Fahimi, Frances Barlas, Nicole Buttermore, and Randall Thomas

© GfK 2016

Overview

Probability-based sample surveys (Neyman paradigm) require:

Complete (representative) sampling frames

Complete (high) response rates

Reasons for growing interest/acceptance for non-probability samples:

Virtually all sampling frames are incomplete

Many surveys struggle with single digits response rates

Budget/time

Where do we go from here:

Ignore the problem

Develop a “new math” for nonprobability survey sampling

Consider more robust weighting/calibration alternatives

2

© GfK 2016

Survey Sampling in Theory

3

Target Universe

(N) Sample

(n)

Random

Selection Inferred Universe Inferred Universe Inferred Universe

Weighting

© GfK 2016

Survey Sampling in Practice (20th Century)

4

Target Universe

(N*)

Sample

(n)

Respondents

(n*)

Qu

asi-R

an

do

m

No

nresp

on

se

Inferred Universe

)ˆ(N

*

*

* N

N

n

N

n

n

© GfK 2016

Survey Sampling in Practice (21st Century)

5

Sample

(n)

Respondents

(n*)

No

n-ra

nd

om

No

nresp

on

se

Inferred Universe Some Universe

© GfK 2016

Inadequacy of Geodemographic Weighting

6

-12%

-12%

-10%

-9%

10%

11%

11%

11%

12%

12%

13%

13%

14%

15%

15%

15%

15%

16%

17%

18%

20%

21%

24%

25%

Item donation

Cyber insecurity (businesses)

Cyber insecurity (others)

Feeling happy

Price sensitivity

Citizens influence politics

Feeling insecure

Like shopping in stores

Having meals with others

Government spends money effetely

Financial insecurity

Feeling lonely

Opinions hard to change

TV with others

Importance of my opinions

Enjoy surveys

Surveys are important

Technophilia

Risk Taking

Interested in news

Like shopping - online

Coupon usage

Importance of sharing Opinions

Email more than once a day

Significant Differences Between Weighted Estimates from Nonprobability Samples and Benchmarks

© GfK 2016

Calibration 1.0

Correcting for higher agreement propensity with:

I usually try new products before other people do

I often try new brands because I like variety

When I shop I look for what is new

I like to be the first among my friends to try something new

I like to tell others about new brands or technology

Concerns:

Univariate

Sometime insufficient

7

© GfK 2016

Calibration 2.0 (Why)

Social Engagement:

Take vacation with others

Exercise/play sports with others

Have a meal in someone’s home

Self-Importance:

Importance of sharing opinions

My opinions are hard to change

Feel confident in social settings

Shopping Habits:

Use of coupons when shopping

Enjoying shopping online

Brand compared to price

8

Happiness and Security:

Happiness with life

Feeling insecure and lonely

Concerned about cyber security

Politics:

Having influence on national politics

Views on government effectiveness

Closely following the news

Community:

Feeling part of a larger community

Number of moves in past 5 years

Religiosity

Altruism:

Donating blood

Donating items

Volunteering without pay

Survey Participations:

Experience with online surveys

Important of taking opinion surveys

Frequency of online surveys

Internet &Social Media:

Frequency of personal email

Frequency of accessing Internet

Time spent watching TV per day

© GfK 2016

Calibration 2.0 (How)

Number of online surveys taken in a month

Hours spent on the internet in a week for personal needs

Trying new products before other people do

Time spent watching television in a day

Using coupons when shopping

Number of relocations in the past 5 years

9

© GfK 2016

Calibration 2.0 (Assessment) Elections in IL and GA:

Percent Registered Voters

Percent Republican

Percent Conservative

Senate Race

Governor’s Race

BRFSS: Smoked 100 cigarettes in lifetime

Physical check-up past year

History of depressive disorder

One or more days per month physical health not good

Seven or more hours sleep per night

NSDUH: Always wear seatbelt as front passenger

Great risk when smoking one or more packs a day

Great risk when trying heroin once or twice

CPS / ACS: Householders receiving social security

Marital status

Homeownership status

Household income

Number of bedrooms in house

Number of automobiles

10

Nonprobability

Sample Weighted

Nonprobability

Sample Calibrated

Nonprobability +

Probability Samples

Calibrated

Probability Sample

Weighted

© GfK 2016

Calibration 3.0 (Robustness & Parsimony)

Hours spent on the internet for personal needs

Time spent watching television in a day

Trying new products before other people do

Frequency of expressing opinion online

Number of online surveys taken in a month

11

Nonprobability Sample

Weighted

Nonprobability Sample

Calibrated

Nonprobability +

Probability Samples

Calibrated

Probability Sample

Weighted

Probability Sample

Calibrated

© GfK 2016 12

Interpretation

Total Survey

Error

Errors of

Non-observation

Errors of

Observations

Errors of

Processing

Errors of

Dissemination

Coverage

Response

Sampling

Instrument

Interviewer

Mode

Coding/Editing

Imputation

Weighting

Data Compilation

Analysis

Takeaways

© GfK 2016

Takeaways

Taming nonprobability samples:

Geodemographic weighting is not enough

Without calibration external validity is poor and inconsistent

Evading calibration to keep design effects low is wishful thinking and misleading

Combining probability & nonprobability samples:

Can select probability-looking nonprobability samples

Larger samples accommodate more detailed weighting/calibration

Calibrated samples can perform better than individual samples

Improving probability samples:

Many surveys struggle with frame issues and single digit response rates

Traditional geodemographic weighting may not be adequate anymore

Even probability samples can benefit from calibration

13

© GfK 2016

Thank you!

14

Mansour Fahimi, Ph.D.

Senior Vice President, Chief Statistician

mansour.fahimi@gfk.com

T: 240-477-4570 | C: 240-565-8711