Mobile Optimization Considerations for

Survey Design –

A Case Study using the U.S. Census

Elizabeth NicholsCenter for Survey Measurement

U.S. Census Bureau

7th International Workshop on Survey MethodologyDaejeon, South KoreaSeptember 28, 2016

This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.

1

Korea has highest smartphone

ownership at 88%

2

By Lee Hana Korea.net http://www.korea.net/NewsFocus/Sci-Tech/view?articleId=133014

U.S. smartphone ownership at 72%

0

10

20

30

40

50

60

70

80

90

100

2011 2012 2013 2014 2015

Percent

Computer - PC

Smartphone

Tablet

Source: U.S. Census Bureau Internet and Computer Usage Supplement and Pew Research Center

3

Overview of Talk

Four considerations for offering a responsive design – that is, when the design displays optimally on a Personal Computer (PC) and on a mobile device

U.S. Census survey instrumentsMobile-optimized and un-optimized 2011 and 2014 American Community Survey – nationwide -

un-optimized 2014 Census Test – site test in and near Washington D.C. -

un-optimized 2015 Census Test – site test in Savannah, GA area and Maricopa

County, AZ – mobile-optimized 2016 Census Test – site test in Houston, TX and Los Angeles, CA

– mobile-optimized Offered in English, Spanish, Chinese and Korean

4

Consideration 1

Do you need to design your

survey for smaller devices?

5

Consider

Mobile device penetration rates

Whether there will be a bias in your estimates if you do not offer the mobile-optimized version in addition to a PC-optimized version

Respondent expectations

IT resources

6

U.S. Statistics

Device usage when answering the online

census questionnaire

0

10

20

30

40

50

60

70

80

90

100

2011 2014 2015

Percent Mobile device

Smartphone

Tablet

Computer - PC

Source: U.S. Census Bureau 2011 American Community Survey Internet Test, 2014 Census Test, 2015 Census Test

7

Public feedback from the

2016 Census Test

“We would prefer paper-form option but if we get assistance with online option it’s ok.” ~ an older adult, 64 years old or older

“I like the option of being able to get on my phone to fill out the survey.” ~ a college student, between the ages of 18-25

Source: Community Partnership Engagement Program in Harris County, Texas and Los Angeles, California

8

Consideration 2

What should you do before you

field a mobile-optimized design?

9

Consider

Your current un-optimized survey on a mobile phone What features are difficult to see and answer

The conventions of your population In the U.S. we read left to right, up to down

Take a look at popular sites

U.S. mobile standards – draft https://standards.usa.gov/

Draft a design, test with real users, repeat

10

An un-optimized online form

U.S. American Community Survey

11

Optimized for desktop/laptop Not optimized for smartphone

Design considerations

12

U.S. sources of inspiration

13

Usability testing

14

Consideration 3

What are some design features to

evaluate?

15

Preparing for a

mobile-optimized design

Respondents need to:

read question without zooming or horizontal scrolling

record answers without zooming

feel like the entire screen is the survey and not just a portion of the screen

see the question immediately (reduce the size of the banner, navigation, and menu)

16

U.S. convention

Important task needs to go up

Optimized for desktop/laptopOptimized for smartphone

17

No U.S. conventions: hamburger menu,

language icon, menu display

18

U.S. convention:

U.S. names on one line, left to right

19

2015 Census Test 2016 Census Test

U.S. convention:

Respondents are not used to scrolling

horizontally on a smartphone

20

U.S. convention:

Name, telephone number and email

vertically designed for mobile

21

2016 Census Test - English 2016 Census Test - Korean

U.S. Convention: Vertical

scrolling in mobile

22

Smartphone

No U.S. convention for label placement

Label above and near the field

23

Device conventionsExample is a drop down response option

24

iPhone Android

Spacing and symbolism

25

Final design for 2015 Census Test after usability testingInitial design for 2015 Census Test

SpacingNot all vertical design works well

26

Initial design for 2015 Census TestFinal design for 2015 Census Test after usability testing

Consideration 4

How can I evaluate the design

once it is in the field?

27

Consider

Response rates

Paradata analysis after fielding survey

Data associated with the survey completion task

Device used

Time per screen or session

Edit or Alert failures

Changed answers

Break-offs

28

Challenges with paradata results

Mobile takes more time than PC

Mobile increases break-offs compared to PC

Confounds might be self-selection in the U.S.

29

Time-on-task: Not optimized

1.2 tablet - 2014 Census Test

1.4 smartphone - 2014 Census Test

30

From Couper, Antoun, Mavletova Total Survey Error Conference Presentation

Time-on-task: Optimized

1.2 tablet - 2015 Census Test

1.3 smartphone - 2015 Census Test

31

Task completion: Not optimized

32

1.6 smartphone - 2014 Census Test

0.84 tablets - 2014 Census Test

Task completion: Optimized

33

1.14 tablets - 2015 Census Test

1.7 smartphone - 2015 Census Test

Things to consider

We know from usability testing that mobile optimization is more usable than not having it optimized

Usability metrics of time-on-task and task-completion are not necessarily the best measures of usability improvement Why longer?

Scrolling? Connectivity / Load time? Different device displays? Self-selection? Multi-tasking?

Uptick of online forms completed on mobile when optimized Evaluate the demographics of the respondents

34

Summary

Why optimize?

Is there bias and measurement error if you do not

Respondent expectations

How to optimize?

Design, test, redesign

How to measure?

Challenges

35

References-Anderson, M. “Technology Device Ownership: 2015.” Pew Research Center, October, 2015, Available at: http://www.pewinternet.org/2015/10/29/technology-device-ownership-2015

-Baker, R., (2014). Some thoughts on the 2015 Census for mobile. Internal correspondence.

-Couper, M. P., and Peterson, G. (2015). Exploring why mobile Web surveys take longer. Presentation at the GOR 2015 Conference. http://www.gor.de/category/gor-15

-Couper, M. P., Antoun, C. and Mavetova, A (2015). Mobile Web Surveys: A Total Survey Error Perspective. Presentation given at the 2015 International Total Survey Error Conference. Baltimore, Maryland.

-de Bruijne, M. and A. Wijnant. 2013. Comparing survey results obtained via mobile devices and computers: an experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey. Social Science Computer Review 31(4): 482–504.

-File, Thom. 2013. “Computer and Internet Use in the United States.” Current Population Survery Reports, P20-568. U.S. Census Bureau,Washington, DC.

-Geisen, E. and Olmsted, M. (2011). 2020 Census Coverage Study: Survey on Mobile Devices Literature Review. RTI International. RTI Project number 0212349.

-Gummer, T., & Rossmann, J. (2015) Explaining Interview Duration in Web Surveys: A Multilevel Approach. Social Science Computer Review 33(2) 217-234

-Horwitz, R., (2015). Usability of the American Community Survey Internet Instrument on Mobile Devices. U.S. Census Bureau. #ACS15-RER-04 Last accessed: https://www.census.gov/content/dam/Census/library/working-papers/2015/acs/2015_Horwitz_01.pdf

-Horwitz, R., (2013). Use of Paradata to Assess the Quality and Functionality of the American Community Survey Internet Instrument U.S. Census Bureau. #ACS12-RER-26-R1 Last accessed: http://www.census.gov/content/dam/Census/library/working-papers/2013/acs/2013_Horwitz_01.pdf

-Jue, A., and Luck, K. (2014). Update: Participation of mobile users in online surveys. Decipher White Paper. Retrieved from https://www.decipherinc.com/n/uploads/images/pages/Decipher_Mobile_Research_White_Paper_Update.pdf.

-Smith, A. (2013). Smartphone Ownership 2013 - Update. Pew Research Center’s Internet & American Life Project. http://pewinternet.org/Reports/2013/Smartphone-Ownership-2013.aspx

-Penzo, M. , 2006 Label Placement in Forms from, http://www.uxmatters.com/mt/archives/2006/07/label-placement-in-forms.php (accessed 8/18/2016)

Pew Internet project (2014). Mobile Technology Fact Sheet as of October 2014. http://www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/

-Peytchev, A. and Hill, C. A. (2010). Experiments in Mobile Web Survey Design: Similarities to Other Modes and Unique Considerations. Social Science Computer Review. 28(3) 319-335. DOI: 10.1177/0894439309353037.

-Rempel, H. G. and Bridges, L., (2013). That was then, this is now: Replacing the mobile-optimized site with responsive design. Information Technology and Libraries 32(4), 8-24.

-Seckler, M. Heinz, S., Bargas-Avila, J. A., Opwis, K., and Tuch, A. N. (2014). Designing Usable Web Forms – Empirical Evaluation of Web Form Improvement Guidelines DOI: 10.1145/2556288.2557265 Conference: In Proceedings of the 2014 annual conference on Human factors in computing systems (CHI’14). ACM, New York, NY, USA.

36

Contact

Elizabeth Nichols

elizabeth.may.nichols@census.gov

37