ESRC Research Methods Festival, Oxford, 17-20 July 2006 The Hunt for the Last Respondent How to...

ESRC Research Methods Festival, Oxford, 17-20 July 2006

The Hunt for the Last Respondent

How to decrease response rates and increase bias

Ineke Stoop


Enhance responserates

Minimize nonrespons

e bias

Enhance nonrespons

e rates

Maximize nonrespons

e bias


Design study

• Field experiences How to decrease response rates?

• Keeping the customer satisfied How to spruce up response rates?

• The sophisticated question How to maximize nonresponse bias?

• Sources International nonresponse literature European Social Survey Dutch experimental study

• Mainly face-to-face surveys


How to decrease response rates?


Preparation

• Short fieldwork period Timely results

• Fieldwork contracting Don’t specify: Survey organization knows best

• Stick to national and organisational traditions


Interviewers

• Foot soldiers of survey research• Economize on interviewer’ training and

instruction Experienced interviewers know best how to act Turnover rate is high and CAPI questionnaires

provide instructions anyway

• Economize on interviewer remuneration Not a real job

• No close monitoring fieldwork Interviewers prefer organizing their own schedules Too much rules and control will lessen motivation


Information and incentives

• No advance letters and brochures Nobody reads them Advance letters just scare off target respondents

• No toll-free telephone number In order to prevent respondent’ refusals

• No incentive required A survey is expensive enough as it is


Contacting sample persons

• Limit the number of contact attempts• Face-to-face calls during office hours only

Interviewers have a family life too Evening visits too dangerous

• Don’t send interviewers to problematic inner city areas

Dangerous and low response rate anyway

• Follow-up on appointments only when you need more respondents

• Stop contact attempts when you have enough respondents


Contactability ESS R1

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

PL IL IT LU AT SI GR BE FI HU IE DE ES GB NL CH PT

weekday morning/afternoon evening weekend


At home, interviewer calls, contact rate

0

10

20

30

40

50

60

70

80

90

100

6 12 18 24

morning afternoon evening

0

10

20

30

40

50

60

70


Obtaining cooperation• Use a questionnaire efficiently and include as

many questions as you can (better safe than sorry)

Matrix questions Small print (seems shorter)

• Do not spend too much time and effort on difficult respondents

Elderly people, language problems, rarely at home

• No bargaining about timing interviews• No refusal conversion (a refusal is a refusal)• Stick to a single interview mode


ESS: share of re-approached refusals that were converted

•

Greece

Poland

Finland

HU

Switzerland

Israel

Slovenia

I

Belgium

Austria

Netherlands

Spain

Great Britain

LUX

Italy

0 10 20 30 40 50 60 70 80 90 100

GR

GB

CH


Bad practices

What happens if you do not take

the previous recommendations seriously


Response rates ESS R1 and R2

0

10

20

30

40

50

60

70

80

GR PL PT FI SI NO HU SE NL DK AT BE IE ES GB DE LU CH

Round 1 Round 2


Dutch experiment

• F2F survey + long drop-offs every household member

67% response rate

• Follow-up survey lite among sample of persistent refusals

+70% responding refusals


Why did refusers cooperate?

• High quality interviewers (and telling them they are the best), motivation, remuneration

• Extensive briefing• Trust (money for incentives) and support

(newsletter) interviewers• Perceived importance (newspaper article,

involvement management)• Wide range of incentives• F2F, PAPI, CATI and internet• Commitment sponsor and fieldwork

organisation


Compare ‘easy’, hard to get, initial refusals and final refusals

• Differences small• Initial refusals more mr. and mrs Average than

easy respondents• Hard to reach respondents different• Final refusals: less participation, less religious,

fewer PCs, more popular culture


What if you can’t get away with low response rates


Do not spend an afwul lot of money


Focus on promising groups

• Rural areas• Families with small children• No apartment dwellers• People who can be reached on the phone


Better: redefine the denominator

• Lower upper age limit (older people take too long)• Raise lower age limit (parental permission or never at

home)• Exclude non-native speakers• Exclude mobile phone only• Exclude ex-directory telephone numbers• Exclude non-internet users• Exclude unused sample units from the net sample• Include only those who have agreed on the telephone to

participate in a f2f survey• Members of access panels (pre-recruirment panels)• Only those panel members that (almost always)

cooperate


Select conscientious access panel members: 70% or 80% guaranteed

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100

Response percentiles access panel

Res

pons

e ra

te


Still better

• Make nonresponse impossible Substitute nonrespondents by family members or

neighbours Quota sampling Volunteer panels


Added benefit

maximize bias

if topic of the survey is related to response mechanism or

selection mechanism


Different types of success

• Do a poor job Low response rates

• Focus on easy respondents and exclude different groups

Adequate response rates, large nonresponse bias

• Pre-select willing panel members Vey high response rates, low nonresponse bias, poor

survey quality


Are high response rates better?

• Maximum bias smaller• Difficult respondents similar to final

nonrespondents?• Hard to reach respondents similar to

noncontacts?• Converted refusals similar to final refusals?


Assumptions valid?

• High response surveys not necessarily better than low response surveys

Possibly because efforts are directed at ‘easy’ nonrespondents (situational refusals)

And survey-related or topic related refusal is left alone

• Low response rate especially harmful if nonparticipation is caused by single factor related to survey

Political interest (Groves, Presser, Dipko, 2004)


Increased fieldwork efforts

• Groves (1989)Devoting limited resources to increasing response rates with little or no impact on survey error is not money well spent

• Krosnick (1999)Prevailing wisdom that higher response rates are necessary for sample representiveness is being challenged

• Curtin et al. (2000)Large differences in response rates have only minor effects

• Keeter et al. (2000)Similar results Standard and Rigorous survey, significant differences between demographics

• Merkle and Edelman (2002)Response rates in exit pools not related to error

• Teitler et al. (2003)Diminishing returns at higher effort levels

• Loosveldt et al. (2003)ESS: converted refusals do differ in some countries (with high refusal conversion rate)


What to do?

• If obtaining high response rates is difficult• If high response rates not necessarily imply

high quality• If interviewing difficult respondents does not

always give better insight in final nonrespondents

• If enhancing response rates means getting more of the same

• If you want to minimize bias


Aim for high response rates

• Close monitoring• High contact rate in all areas among all groups• Obtain cooperation through multiple means

Respect respondents Provide information Internal and external incentives Motivated interviewers Organisational support Mixed mode interviewing


But maybe do not bother about the final few %

• Collect and code Sample information Observational data Fieldwork data (contact and cooperation) Information on respondent and nonrespondents

related to the topic of the survey Other sources Central Question procedure Survey among nonrespondents


Go for bias

• Surveys are very expensive: Not enhance response rates indiscriminately Not high response rates combined with non-

probability sampling and high coverage errors Not high response rates within specific groups Collect and use auxiliary information to assess bias

and to adjust for bias


Treat data collection

with a scientific, controlled approach

as used in sampling and data analysis

ESRC Research Methods Festival, Oxford, 17-20 July 2006 The Hunt for the Last Respondent How to...

Documents

Transcript of ESRC Research Methods Festival, Oxford, 17-20 July 2006 The Hunt for the Last Respondent How to...