Making the Leap from Web to Mobile

Best Practices in Mobile User Experience Research

Amy Buckner, AnswerLab Kris Mihalic, Yahoo!

UPA 2009, June 12, Portland

3

CONFIDENTIAL

Agenda

Topic Minutes

1. Introduction 5 minutes

2. State of Mobile 10 minutes

3. Group Exercise: Mobile Device Task & Discussion 5 minutes

4. Mobile User Research Methods 20 minutes

5. Team Exercise: Develop Research Solution 15 minutes

6. Case Study 15 minutes

7. Mobile Usability Research Challenges & Tips 15 minutes

8. Q&A 5 minutes

4

CONFIDENTIAL

Introductions

Amy Buckner Managing Partner & Co-Founder of AnswerLab   Over 10 years experience leading user research programs for Fortune 500 companies

  Expertise in quantitative and qualitative methods, across web and mobile devices

  Previous West Coast Director of Professional Services at Vividence (now Keynote) and Marketing Manager at SmartPlanet

  Believes a glass of wine and a beef jerky stick count as dinner

Kris Mihalic, PhD Head of Mobile User Experience and Design Research, Yahoo!   Seasoned user experience and design researcher for mobile and cross-channel products

and services

  Expertise in defining and executing UX and design research strategy in cross-functional teams

  Experience teaching interface design methods, usability engineering, and telecommunication technologies

  Lives on coffee

5

CONFIDENTIAL

State of Mobile

6

CONFIDENTIAL

There are mobile phone subscribers worldwide

Only of US mobile phones are iPhones (~5 Million)

Social networking on mobile phones is growing Y/Y

There are more mobile IM users than mobile business email users in the US

4.1 Billion

2%

196%

1.4x

7

CONFIDENTIAL

Mobile Market Summary

Source: Experian 2008, eMarketer 2008

Globally, there are 800 million cars, 850 million PCs, 1.3 billion fixed landline phones, 1.4 billion credit cards, 1.5 billion TV sets – and 2.7 billion mobile phones – in use. (Experian 2008)

eMarketer projects that worldwide spending on mobile advertising will reach a total of $19.1 billion in 2012 – up from 4.6 billion in 2008.

8

CONFIDENTIAL

U.S. Mobile Web Users Growing Rapidly

Source: m:metrics/comScore, 2009

9

CONFIDENTIAL

Mobile Audience Size (# of users)

52%

31% 30%

10%

SMS Users Mobile Media Users 3G Smartphone 0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

In the U.S., the size of the Mobile Audience is 250 Million

Source: m:metrics/comScore, 2009

10

CONFIDENTIAL

U.S. Mobile Internet Usage

Mobile Internet users are heavy users

  60% access the Internet from their phone at least once/day

  48% access the Internet more than once per day

Source: TSM|TargetProfile 2007

11

CONFIDENTIAL

Device Traffic

Mobile phones dominate mobile Web traffic with 66% market share (US).

BlackBerry and Smartphones (Windows Mobile, Palm, etc.) are heavy users of mobile Internet – combined 26% of traffic

iPhone/iPod Touch has shown tremendous growth, capturing 8% of the mobile Internet market in the US

Source: m:metrics/comScore, 2009

12

CONFIDENTIAL

Widgets Improve Mobile Internet Experience

Source: ‘Widgets Improve the Convenience of the Mobile Internet,’ Forrester, 2008

13

CONFIDENTIAL

Group Exercise

14

CONFIDENTIAL

Take out your cell phone and search for times that ‘Star Trek’ is

playing in your neighborhood.

Raise your hand when you have finished the task.

15

CONFIDENTIAL

Discussion Points

  How long did it take?

 How did you do it? (web, app, SMS?)

 What problems, if any, did you have?

16

CONFIDENTIAL

Mobile User Research Methods

17

CONFIDENTIAL

Overview of Research Methods

  Quick and Dirty feedback

  Lab Studies

  Field Studies

  Online Surveys   Basic Survey   Survey with Behavioral Tracking (Keynote on iPhone only)

  Quantitative Behavioral Analysis

18

CONFIDENTIAL

Quick and Dirty – What are obvious problems?

Advantages

  Informal setting with Lo-Fi prototypes

  Fast iteration cycles

  Brain-damage check

  Finds critical usability issues in shortest time

Challenges

  Insider/stakeholder view

  Anecdotal, unstructured results

  Small sample size

  Difficult to convey real usage scenarios

Examples

  Cross-functional team feedback

  Internal alpha/beta

19

CONFIDENTIAL

Lab Studies – Can they use it?

Advantages

  Allows for probing by moderator

  Can record interactions

  Best for prototype testing

  Allows for real-time viewing of interaction – team engagement

Challenges

  Artificial environment

  Moderator bias; participant ‘pleasing’

  Small sample size

  Difficult to test all hardware / network scenarios

Examples

  Usability study

  Participatory design sessions

20

CONFIDENTIAL

Field Studies – What is the context of usage?

Advantages

  Assess usage under real conditions

  Candid user feedback

  Delivers unknown use-cases

Challenges

  Small sample size

  Difficult to observe and probe

  Participant engagement difficult

  Requires robust product

Examples

  Diary study

  Community study

21

CONFIDENTIAL

Online Surveys – What do they think of it?

Advantages

  Can deliver statistically valid results

  Provides qualitative and/or quantitative data

  Geographical diversity (remote)

  Broad representation of devices

Challenges

  Difficult to observe and probe

  Out-of-context interaction, esp. with desktop survey

  Technical constraints

Examples

  Desktop surveys

  On-device survey

  Survey with behavioral tracking

22

CONFIDENTIAL

Behavioral Analysis – How do they use it?

Advantages

  Reflects what people do, not say

  Statistically valid results

  Shows all behaviors, rather than those confined to a single task

Challenges

  Probing / deep-dive difficult

  Context and intent unknown

  Attitudes and perceptions unknown

Examples

  Log data mining

23

CONFIDENTIAL

Types of Prototypes

  Paper prototypes   Mocks

  Paper and pencil

  Interactive prototypes   Apps

  Flash

  Native, e.g. iPhone

  Browser-based   Simple HTML prototype

  High-fidelity prototype (with session variables)

24

CONFIDENTIAL

Team Exercise & Case Study

25

CONFIDENTIAL

Yahoo! Go

 Mobile Application  Search  News  Sports  Weather  Other Topics

26

CONFIDENTIAL

Y! oneSearch integrated into Y! Go

27

CONFIDENTIAL

Business Issue

  Drive mobile search uptake – Improved user experience can help accelerate growth of the service

  Improve relevancy – Understanding search intent can contribute to delivering more relevant results

  Product differentiation – New approach to search results page can deliver higher value to customers

28

CONFIDENTIAL

Research Objectives

  Understand mobile search behavior in users’ daily lives

  Identify content users seek when conducting mobile searches

  Assess the context surrounding mobile searches

  Evaluate effectiveness of new product

  Identify opportunities to improve the user experience

29

CONFIDENTIAL

1.  Form into teams of 4.

2.  Develop a research plan for Yahoo! Go.

3.  You have 10 minutes.

30

CONFIDENTIAL

  Understand mobile search behavior in users’ daily lives

  Identify content users seek when conducting mobile searches

  Assess the context surrounding mobile searches

  Evaluate effectiveness of new product

  Identify opportunities to improve the user experience

Research Objectives

  Develop a research plan for Yahoo! Go

  Include: •  Recommended method(s) •  Number of participants •  Profile of participants

Team Activity

31

CONFIDENTIAL

Discussion Points

  What methods do you recommend & why?

  What are your anticipated concerns?

  Any potential limitations?

32

CONFIDENTIAL

Our Solution: One-Month Field Study (1)

Voicemails & Pocket Card

  9 users

  Daily voicemails

  Laminated pocket card with key questions

Daily Mobile Survey

  Daily SMS mini-survey

  Linked to phone number for voicemail

Digital Photos

  Users sent photos of themselves or surroundings in the context of using oneSearch

33

CONFIDENTIAL

‘Pocket Card’ Questionnaire

34

CONFIDENTIAL

Sample Voicemail Diary Messages -- Winchelle

Friday 7:39pm I used oneSearch today, was looking for a hospital – directions and address to it. I was getting off work and sitting in my car, trying to find directions. It was about 4:30. I liked that it came up with the website and with information about that. And the directions on how to go. I didn’t like that it could not find my work address – which is saved on my Yahoo! directions (recently gone) saved places. I think using the actual internet through the PC will help me better because it knows where the address is. I’m not sure why the address is not coming up in my oneSearch and that sucked.

Saturday 8:06pm I didn’t use oneSearch today. I looked for info, looked for a store and I was at home. I wanted to know if the store was open STORE HOURS (9:45am). I used the internet with my phone and used Google to search. It was not very helpful because I wanted to see what time they opened and I couldn’t find it. I’ve used oneSearch in the past and I guess the results were the same but I guess it’d be nice to go to the site for that particular store.

Monday 10:48pm I used oneSearch to find Expedia, was at work on my lunch break. It was about 12:20. I liked that I can find it quickly. I guess the website didn’t support my mobile phone though, so I didn’t like that. Would rather book a flight/hotel at home because its supported.

35

CONFIDENTIAL

Our Solution: One-Month Field Study (2)

Pre-Interview

  An initial 45-minute interview

  Verified technical capabilities of phone

  Explained research program

Mid-Check Interview

  20-minute check-in phone interview

  Initial feedback

  Q&A

Wrap-up Interview

  Final in-person interview of 75 minutes

  Clarified voicemail reports

  Overall impressions of the product from the month-long usage

36

CONFIDENTIAL

Screening Criteria: 9 Experienced Mobile Users

  Have an unlimited data plan for WAP services through their mobile carrier

  Use their mobile for text messages (SMS) daily and use mobile more than three times per week for activities that require the transmission of online data

  Have Cingular, Sprint, T-Mobile, or Verizon as their mobile carrier

  Have a primary mobile device capable of downloading and using Y! Go

  Live or work within San Francisco

  Mix of gender, age (23-46+), and typical method of commuting to work (4 users primarily drive, 5 primarily take mass transit)

37

CONFIDENTIAL

Sliding Scale for Participant Incentives

  $100 for the initial in-person interview

  $50 for the mid-way phone interview

  $150 for the final in-person interview

  $3 each day a report is given

38

CONFIDENTIAL

  Natural environment key to context

  All research activities on the mobile device

  Mix of methods   Engage users in program   Breadth of insights – context, usability, impressions

Why did we choose this method?

39

CONFIDENTIAL

What did we learn? (1)

  Perceived value increased with usage

  Usage typically driven by lack of computer availability

  However, certain scenarios drove phone choice over computer:

  Social setting   Privacy   Convenience

40

CONFIDENTIAL

What did we learn? (2)

  Mental model based on computer usage

  Speed and relevance of search results highest area of frustration

  Lack of understanding that results were customized by widget

41

CONFIDENTIAL

Daily SMS Survey Results

*Website not adapted for mobile device

42

CONFIDENTIAL

What did we learn? (3)

Use Case Primary Content Wanted

Shopping Business’ number, location, directions, and hours of operation

Entertaining Schedules/Venues of concerts, sport games, movies, and TV shows

Researching Content from news/articles

Dining Restaurants’ number, location, directions, and hours of operation

Traveling Schedules and numbers of transportation (public transit, taxis, etc)

Media Seeking Photos/Ring tones

43

CONFIDENTIAL

Summary of Use Case Frequency

Note: The 166 searches were conducted by 9 users

44

CONFIDENTIAL

  Business’ number, location, directions, and hours of operation

  “While driving home from work, I did a search for Fry’s electronics on the local search and it came up with all the different Fry’s in the area, their numbers, as well as directions and all that. The results were all helpful and fast in helping me find the one closest.”

  “Since we were already out shopping, I wanted to find where the closest Citibank was to where we were.”

  Pricing/Product comparisons   “I was at Costco and comparing TV prices. I

wanted to know how much these same TV’s would be at the Circuit City down the street.”

Shopping

45

CONFIDENTIAL

  A restaurant’s number, location, directions, and hours of operation   “I just woke up and didn’t want to boot my

computer up but I wanted to find out what time this breakfast place opened so I could know how soon I could get there.”

  “I was in my car and wanted their number so I could call and place an order to pick it up on my way home from work.”

  Ideas of places to eat   “I was with my wife in the Haight and we

wanted to find nearby restaurants so we could decide what we wanted to eat that was within walking distance.”

  “I was in the South Bay with some friends and we wanted to find a good sushi place nearby.”

Dining

  Reservations & Reviews   “I was hoping to make reservations straight

from the phone, or at least call them to place them.”

  “I was out with some friends and curious about whether the place nearby was any good.”

46

CONFIDENTIAL

  Transit schedules and numbers   “I wanted to find out when the next MUNI but

the MUNI’s site wouldn’t work on my phone.”   “I wanted the number to call a taxi. I was

outside and tired of waiting for the bus.”

  Tourist activities   “I wanted to see what kind of ferry tours we

could do before looking for another museum that would be fun for my daughter.”

  “I was looking for fun things to do while at Hermosa beach.”

  Maps and directions   “I was trying to get a map of the Mt. Shasta area

and I wanted to plan which route to take. It would be nice if the driving directions had live updates on traffic and road conditions.”

  Flight status / check-in   “I was in the taxi to the airport and wanted to

check my flight’s status.”

Traveling

48

CONFIDENTIAL

Business Implications

  Drive mobile search uptake – Distribution through partnerships, marketing campaigns, opening to third party services and developers

  Improve relevancy – Focused on improving results for specific use cases (e.g. local businesses, flights, etc.)

  Product differentiation – Federated search results, integrated user experience across multiple services (e.g. search results + maps)

49

CONFIDENTIAL

Mobile Usability Research Challenges & Tips

50

CONFIDENTIAL

Variety of Mobile Devices

  Difficult to know and understand user experience across all devices

  Devices vary by model, browser type, carrier, and input type

  Moderator may be unable to help users through usability tasks – resetting prototypes, navigating – if interface is not well-known

  Design team should have a variety of devices simulators / devices on hand for testing

  Prototype designs should be tested across as many devices as possible

  Limit recruiting to only devices that have been tested

  Have a back-up plan with either a basic device or paper prototypes

Challenges  Tips 

51

CONFIDENTIAL

Challenges  Tips 

Mobile Devices are Small and . . . Well, Very Mobile

  Users move the device around while interacting and explaining, often moving video display out of focus

  Reflectors on screens and smudges can make video all reflection

  Small device screens make it difficult to see what users are doing

  Large fingers and long fingernails can cause unintended device responses

  Create a hot zone on the table, encouraging users to keep the device within a narrow frame

  Consider light source and potential reflection on the phone; continually adjust mid testing; turn-off lights

  Use remote-controlled video camera (with technician in back room) or have a second technician available in interview room

  Project the mobile screen on a larger monitor in interview room and back room

  Creatively screen out participants whose finger size may distract from findings (if non-touch screen)

52

CONFIDENTIAL

Challenges  Tips 

Varied Locations and Use Cases

  Mobile device usage occurs in a variety of places   59% of Americans check email while in

the bathroom*

  Difficult to recreate the true experience in a lab setting

  Create opportunities for feedback in context of usage

  Conduct field studies on beta products

MOBILE DEVICE ACCESS

Source: Nielsen Mobile 2008

55.2%

35.9% 15.8%

67.4%

6.4%

53

CONFIDENTIAL

Challenges  Tips 

Varied Carrier Network Coverage

  Network coverage is inconsistent; difficult to predict accessibility during lab testing

  Difficult to validate that out-of-town lab truly has adequate coverage

  Visit facility with colleagues who have various carriers to confirm network availability in lab

  From out-of-town labs, request:   Reference clients from last mobile

studies   List of carriers confirmed to have

consistent coverage   At a minimum, check carrier

coverage in various cities online

54

CONFIDENTIAL

Challenges  Tips 

Calling / Data Plans Vary Widely

  Participants may be charged for mobile web access or sample texts sent during testing

  Participants may change data plan between recruiting interview and date of study

  Screen during recruiting for:   Unlimited data plans   Willingness to send / receive text

messages during study

  Rescreen again upon arrival for study

55

CONFIDENTIAL

Challenges  Tips 

Varied User Experience

  Most mobile device owners only utilize a small percentage of total device capabilities   43% of mobile subscribers do not use

text messaging on a regular basis*

  Users may have no experience typing in a web address to access a site or with sending SMS messages (which may be critical to your study)

  If using a prototype phone for downloadable app, users may not be comfortable with it

  Pre-test participants to ensure appropriate device experience   Mobile address typing   Send SMS to recruiter

  Spend 2-5 minutes explaining how prototype phone works

* Source: Nielsen 2008 ‘The Short Code Marketing Opportunity’

56

CONFIDENTIAL

Other Helpful Tips

  Tell participants to bring recharger

  Simplify language (e.g., SMS vs. text message)

  Have a prototype strategy   Back-end SMS text simulation   Easy URL for access   Back-up paper prototypes   Index page for easy access to alternative flows   Have plan for interruption from phone call or text (resetting session variables)

  Plan for getting screenshots for your report   iPhone: Press hold button and home button at same time (saves into photo gallery)   Use documented comps

57

CONFIDENTIAL

Q&A

58

CONFIDENTIAL

Thank You.

Amy Buckner Managing Partner & Co-Founder, AnswerLab amy@answerlab.com

Kris Mihalic Head of Mobile User Experience and Design Research, Yahoo! kmihalic@yahoo-inc.com Twitter: suikris