Examination of Ipek Bilgen, Cheryl J. Recruitment Mailing ... 2.2 - Ipek Bilgen.pdf · HS grad or...

Examination of Recruitment Mailing Design Decisions for a Probability-based Research Panel

Ipek Bilgen, Cheryl J. Wiese, Lindsay Liebert, Erlina Hendarwan, Roopam Singh, Kanru Xia, Vicki J. Pineau, and J. Michael DennisCIPHER 2020, Washington DC,

February 26-27, 2020

2

Nonresponse during the recruitment stage is a source of risk to accuracy in estimates and panel representation in probability-based web/web-mostly panels

Mail recruitment is a large component of ABS-based web/web-mostly panels – Following visual design guidelines during mailing design is crucial – But… One size does not fit all (Dillman et al., 2014)

In this study, we examine the impact of key visual design decisions in recruitment mail materials and assess which design(s) work better for which part of the population

Introduction

3

AmeriSpeak Recruitment Methodology

Initial Recruitment Pre-notification postcard +5 days, 9 x 12 recruitment packet

w/ pre-incentive, letter, study brochure w/ privacy policy

+11 days, reminder postcard +18 days, reminder postcard In-bound CATI +25 days, Out-bound CATI

(call-outs to matched telephones)

NRFU Recruitment Federal Express study brochure,

letter, and enhanced pre-incentive In-person recruitment

4

Participants did not see uniqueness of AmeriSpeak in the materials. They saw it as another survey platform

The photos used to illustrate the points did not reinforce the instructions and may therefore distract from the instructional text

There was significant “visual noise” that may take away from the key information

Accordingly, the 2019 experiment designed to test how to effectively reduce the visual noise and simplify the content of the recruitment materials

Initial Qualitative Research Takeaways

5

Research Questions

What is the impact of each mailing design on the panel recruitment yield rate?

Do different mailing designs improve demographic representation of the panel sample?

How do different mailing designs impact the panel sample composition? One size fits all vs. tailor materials for households w/ different demographics.

What is the impact of recruitment mailing material design on AmeriSpeak Recruitment Rates and Panel Composition?

6

Overarching Hypotheses

Elimination of photos on mail recruitment materials will improve initial yield rates

Simplifying the mail recruitment material text will improve initial yield rates 8th grade reading level

Using formatting and layout to improve readability

Clear action items

The addition of a magnet as a non-monetary incentive will improve recruitment rates across all demographic groups

7

Pre-notification PostcardControl

Control Text

Treatment

8

What is AmeriSpeak?

Control: Photo + Full Text Envelope & Brochure Cover No Photo + Full Text Envelope & Brochure Cover

Photo + Simplified Text Envelope & Brochure Cover No Photo + Simplified Text Envelope & Brochure Cover

9

Initial Packet Cover Letter

Control Text

Magnet

Control Treatment

10

Reminder PostcardControl Treatment

11

0.37%(537)

2.53%(3,633)

0.65%(910)

2.99%(4,158)

6.41%Total Yield

Total Yielded9,238

(started survey)

Total Sampled144,198

Overall Recruitment Yield Rates – at a Glance

Reminder Postcard

Initial Packet Phone Follow-upPre-notification Postcard

12

Overall = 6.4%

Magnet = 5.6%No Graphics = 6.2%No Graphics + Magnet = 6.3%Simple Text + Magnet = 6.5%Simple Text = 6.8%Simple Text + No Graphics = 7.0%Simple Text + No Graphics + Magnet = 7.5%

Control = 5.8%

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10%

Recruitment Yield Rate

Rec

ruitm

ent M

ailin

g D

esig

n

1.7 pp increase = 29% more than the control grp yield rate(p<.0001)

Recruitment Yield Rates by Treatment Group

NOTE: CATI outbound phone yield rates were very similar among the treatment groups (ranged between 2.8 to 3.1%)

2019 Panel Recruitment: Composition Analysis

14

2019 AmeriSpeak Panel Recruits

Unweighted Panel Distribution by Mailing Design and Gender

Control versus ST + NG: ** p=0.0044Control versus ST+NG+M: * p=0.0333

0%

20%

40%

60%

80%

100%

Male Female

Rec

ruitm

ent Y

ield

Rat

e Benchmark(CPS 2019)

Control

Simple Text +No Graphics

Simple Text +No Graphics +Magnet

New designs bring in more male

panelists

15


Unweighted Panel Distribution by Mailing Design and Age

0%

10%

20%

30%

40%

50%

18-24 25-34 35-44 45-54 55-64 65-74 75+

Rec

ruitm

ent Y

ield

Rat


Control

No Graphics +Magnet


Control versus NG+M: + p=0.0946Control versus ST+NG+M: * p=0.0179

Design w/ no graphics and

magnet brings in younger panelists (18-24)

Design w/ simple text, no graphics, and

magnet brings in older panelists

(75+)

Overall, different designs appeal to

different age groups

(Tailored design)

16


Unweighted Panel Distribution by Mailing Design and Race

Control versus NG: p=0.1059Control versus ST+NG+M: p=0.1444

0%

10%

20%

30%

40%

50%

NH Black NH Asian NH Other Hispanic NH, 2+

Rec

ruitm

ent Y

ield

Rat


Control

No Graphics


We still need NRFU to bring in

moreHispanic

Minorities

New designs bring in slightly

but not significantly more Asians

17


Unweighted Panel Distribution by Mailing Design and Education

Control versus M: p=0.2152Control versus ST+NG+M: p=0.1353

0%

10%

20%

30%

40%

50%

No HS Diploma HS grad orequivalent

Some College BA or Above

Rec

ruitm

ent Y

ield

Rat


Control

Magnet


No significant impact of new design options on panelists w/ different education levels

18


Unweighted Panel Distribution by Mailing Design and Income

Control versus NG: p=0.5319Control versus ST+NG+M: p=0.7862

0%

10%

20%

30%

40%

50%

Less than$25,000

$25,000 to$49,999

$50,000 to$74,999

$75,000 to$99,999

$100,000 ormore

Rec

ruitm

ent Y

ield

Rat


Control

No Graphics


No significant impact of new design options on panelists w/

different income levels

19

• Overall, the mailing design interventions significantly increase the yield rates• Different mailing design options appeal to different age groups

• Consideration of tailored design model during future recruitment

• All interventions resulted in a significantly larger proportion of male respondents• Removing graphics and using magnet resulted in a larger proportion of younger

respondents (ages 18-24)• We still need NRFU to represent some H2R groups (e.g. Hispanic minorities,

low educated, and very high income level groups) • Future direction – Investigating the use of auxiliary data during recruitment to

predict HH demographics to determine how we can best tailor mailing materials based on the results of these experiments

Summary and Discussion

Thank You!

[email protected]

21

AAPOR RR3 (2014-2018 AmeriSpeak Panel Recruits)

Description Response Rate (AAPOR RR3, Weighted)

Household ResponseRate due to Initial Recruitment 5.6%

Household Response Rate due to NRFU 28.6%

Household Response Rate 34.2%

2019 Recruitment RR (Initial Recruitment Rate since we did not have NRFU last year) = 5.7%

2014 – 2019 Overall Household Response Rate = 24.1%

Examination of Ipek Bilgen, Cheryl J. Recruitment Mailing ... 2.2 - Ipek Bilgen.pdf · HS grad or...

Documents

Transcript of Examination of Ipek Bilgen, Cheryl J. Recruitment Mailing ... 2.2 - Ipek Bilgen.pdf · HS grad or...