Cell Phones for Data Collection: Costs and Challenges Michael Link 1, Michael Battaglia 2, Martin...

Cell Phones for Data Collection: Costs and Challenges

Michael Link1, Michael Battaglia2, Martin Frankel3,

Larry Osborn4, and Ali Mokdad5

1 Nielsen Media Research2 Abt Associates Inc.3 Baruch College, City University of New York

4 Knowledge Networks

5 Centers for Disease Control and Prevention

The Plague of Cell Phones!!!

Cell phones and telephone surveys Reliance on cell phones increasing (July-

December NHIS): 57.1% of households have a working cell phone 11.6% of households (11.8% of adults) are cell

phone only 25.4% for age 18-24, 29.1% for age 25-29, 54.0%

of unrelated adult households w/o children, 26.4% for renters

Result: increased potential for noncoverage bias

Cell Phone Summit 2005, TSM II 2006, AAPOR 2007 – special issue of POQ

Cell phones and telephone surveys

Conducting surveys via cell phones can be operationally challenging Cell phone frame may not be that efficient Geographic specificity is a problem Cannot use autodialers/predictive dialers Charges for incoming calls/minutes used Safety concerns Potential mode effects / measurement errors Level of cognitive engagement

Behavioral Risk Factor Surveillance System (BRFSS)

Monthly state-based landline RDD survey of health issues and related risk factors

50 states, District of Columbia, Puerto Rico, Guam, and Virgin Islands

350,000+ adult interviews conducted in 2006

Significant declines in participation overall, particularly among younger adults and males

2007 BRFSS cell phone pilot

Conducted in Georgia, New Mexico, & Pennsylvania

Target: 200 cell & landline / 200 cell-only adults (per state) 1,200 total interviews

Abbreviated BRFSS core interview: 66 questions 15-17 minutes (on average)

Incentives: $10 post-paid incentive for completing the

detailed interview $1 for completing the screener

Sample design Marketing Systems Group (MSG):

All designated cellular 1,000 banks Implicit stratification by area code and exchange Equal probability sample of telephone numbers

Survey Sampling Inc. (SSI): All 100 series banks designated as cellular “Mixed use” (landline / cell phone) banks

containing zero residential directory-listed numbers

Implicit stratification by FIPS, carrier, & 100-block Systematic random sample of 100 blocks Randomly generate last 2 digits of number

Screening questions Introduction

Confirmed telephone number

Is this a cellular telephone?

Are you 18 years of age or older?

Are you a resident of (state)?

“Do you also have a landline telephone that is used to make and receive calls?” Yes – took subsample of respondents No – took all respondents

Survey Participation Rates

Calculation of rates

Used detailed disposition codes modeled after Callegaro et al (2007) with some modifications/additions

Included “ring, no answer” and “voice mail” as working residential numbers Only cases confirmed by company message as

being not in service were excluded

Used AAPOR response rate guidelines

Calculated separate rates for: Screening for eligible respondent Completion of interview

Participation rates

GA NM PA

Starting sample size: 9,000 4,400 9,997

Completed interviews: 405 413 346

% Working cell numbers: 64.7 63.2 72.5

Screener rate: 40.2 47.5 34.3

Interview rate: 60.8 65.8 67.6

Response rate: 24.4 31.3 23.2

Interview rate by landline access

0102030405060708090

100

Total GA NM PA

Cell only Cell & landline

64.364.9

57.766.1

66.764.1 70.2

64.8

Percent working cell numberby sample vendor

0102030405060708090

100

GA NM PA

MSG SSI

63.366.1

61.265.3

70.574.5

Comparison of respondent demographics

Landline and Cell phonepopulations and frames

CELL PHONELANDLINE A B C

0

10

20

30

40

50

60

70

80

90

100

Landline (only) Cell & landline Cell & landline Cell (only)

46.651.1

38.237.9

Landline survey Cell phone survey

Percent male

State equalized design weight applied

0

10

20

30

40

50

60

70

80

90

100


24.0

51.4

19.614.5


Percent 18-34 years


0

10

20

30

40

50

60

70

80

90

100


15.2 21.412.216.8


Percent Hispanic


0

10

20

30

40

50

60

70

80

90

100


15.0 15.87.59.3Landline survey Cell phone survey

Percent black


0

10

20

30

40

50

60

70

80

90

100


39.848.5

33.6

60.3


Percent high school or less education


0

10

20

30

40

50

60

70

80

90

100


62.0

32.0

69.8

49.5


Percent married


Summary of significant differencesacross demographic subgroups

Cell only v.s. cell & landline adults (from cell phone survey): Significant differences for 12 of 24 subgroups

examined Particularly age, employment status &

marital status

Cell & landline adults (cell phone survey v.s. landline survey): Significant differences for 11 of 24 subgroups

examined Particularly sex, race, marital status, and

children in household

Comparison of key survey estimates

0

10

20

30

40

50

60

70

80

90

100


86.0

70.1

89.0

78.7


Percent any kind of health care coverage


0

10

20

30

40

50

60

70

80

90

100


16.324.9

10.220.4


Percent not received care due to cost barrier


0

10

20

30

40

50

60

70

80

90

100


19.731.1

17.324.8


Percent currently smoke cigarettes


0

10

20

30

40

50

60

70

80

90

100


43.6

54.2

36.637.5


Percent ever tested for HIV


0

10

20

30

40

50

60

70

80

90

100


13.023.521.1

11.0Landline survey Cell phone survey

Percent binge drink past 30 days


Comparison of Survey Costs

Cost per Interview

Data collection costs only

Level of effort: RDD = 7.4 calls/case Cell = 3.2 calls / case

Response rate: RDD = 38% Cell = 26%

Interview length: RDD = 25 minutes Cell = 12 minutes

$0

$50

$100

$150

$200

Landline survey

Cell phone survey

Cell only (via screening)

$60 $74

$196

What have we learned?

Group with both landline & cell phone differ across landline and cell phone surveys Mode effect? Response/nonresponse

effect? Frame effect? This is an important issue when we try to

combine landline & cell phone surveys

Cell phone only group differs significantly from landline group on some health variables, but not others Risk behaviors seem most problematic

What have we learned? Cell phone & landline usage varies

significantly across states Makes use of national estimates from the NHIS

for post survey adjustment problematic

Compared to landline surveys, cell phone surveys: Have lower rates of response at the screener

stage But similar rates at the interview stage

Working residential rates lower, but not as bad as expected

Are considerably more expensive, especially if we decide to screen for cell-only adults

Future directions Problem is not going away, but will continue to

worsen

Focus on combining estimates from cell and landline frames (Frankel and Battaglia are working on this for the BRFSS) Cell only households not the only problem Need more focus on primary cell users

Best to reach by landline or cell frame?

Mode effects – what measurement issues are raised by interviewing via cell phones?

Cell Phones for Data Collection: Costs and Challenges Michael Link 1, Michael Battaglia 2, Martin...

Documents

Transcript of Cell Phones for Data Collection: Costs and Challenges Michael Link 1, Michael Battaglia 2, Martin...