Ilana Kaplan UX NZ 2013

Multi-national user testing: Tools, tricks and pitfalls

Multi country testing becoming more common place Understanding international customers is more

critical than ever as emerging market economies continue their rapid expansion

On the flip side, companies that don’t understand their international customers are at a strategic disadvantage and at greater risk of making missteps that could undermine their success in diverse markets

http://uxmag.com/articles/6-key-questions-to-guide-international-ux-research

Multi country testing becoming more common place International UX research is important even in

countries or regions with customers who speak the same language, as regional differences in nomenclature , cultural norms, and customer needs can dramatically impact the success of a product, and….

http://uxmag.com/articles/6-key-questions-to-guide-international-ux-research

Multi country testing becoming more common place International UX research is important even in

countries or regions with customers who speak the same language, as regional differences in nomenclature , cultural norms, and customer needs can dramatically impact the success of a product, and….

Legislation (medical devices) One size fits all – or does it? Soft launch in one county before we launch in

others Benchmarking against international competitors

Testing options

Moderated Unmoderated

In person Remote

Unmoderated Testing optionso First clicko Card sortingo Surveyso Task basedo Diaries

Cost

Data

Qu

al

Qu

ant

Less More

Testing options

Moderated Unmoderated

In person Remote

o Traditional user testingo Eye tracking

Testing options

Moderated Unmoderated

In person Remote

o Observer shares monitor and participant uses system

Testing options

Moderated Unmoderated

In person Remote

Cost

Data

Qu

al

Qu

ant

Less More

Medical deviceCoordinated out of USA

Face to face moderated usability testing• Observer travelled to each

country • Testing coordinated by

local UX partner

Global consumer site Coordinated out of Australia

Phase 4Face to face moderated usability testing in Australia and USA

• Study designed in Australia

• Skype briefing and debriefing

• Testing conducted by local partners in each country

Phase 2Unmoderated Treejack testing in USA, Australia and Germany

Phase 3Unmoderated ChalkMark testing in USA, Australia and Germany

• All recruit and analysis out of Australia, with help from German speaking staff

Phase 1Face to face moderated research on visual concepts

• Study designed in Australia

• Skype briefing and debriefing

• Testing conducted by local partners in each country

International websiteNZ company operating

internationally

1. User Survey on site in USA, Australia and NZ (around 1000 participants)

2. UX partners in Canada, the UK, and Hong Kong identified best practice examples in their country

3. Multi-national usability testing in Australia and NZ (34 particpants)

4. Eye-tracking in NZ5. Expert reviews

Smart Blood Pressure Monitor Study 2013

Qualitative insights into the purchase of a smart blood pressure monitor, the out-of-the-box experience and using the device over 10 days

Digital health and wellness

• More and more, patients are becoming consumers, spending more time monitoring their own health and well being and taking increased responsibility for themselves

• There is a whole new genre of Apps and devices that support and encourage this movement

• Electronic blood pressure monitors fit into this

• UX alliance partners set out to understand some of the short term factors which could impinge on the usefulness of these device

20 participants from 17 countries

BrazilCanadaJapanNew ZealandItaly

IndiaPolandDenmarkSwitzerland AustraliaUSA

FranceFinlandRussiaIrelandUnited KingdomGermany

The study – 3 stages

Purchasing the device

Out-of-the-box

experience

Using the

device

UX professionals purchase the device and record the experience

Observe and record the participant’s out-of-the-box experience

Participants measure their blood pressure and complete a survey daily, for 10 days

Smart Blood Pressure Monitor Study 2013: ‘Out-of-the-Box’ Experience

Unpacking device Assembling App First attempt

Purchase accessory

“It was hard to insert the batteries and there was no explanation about this in the manual.”

“It looks like a apple product. White package… it looks like an expensive product.”

“Reading on the package I realised that it wasn't iPhone 5 compatible. I didn't have [an] adapter though.”

“It was not clear which application you have to use with the device, and how to use it.”

Insights• It was not clear if an adapter was needed • Users struggled to read instructions in a foreign language• Most users had difficulties assembling the device

• Many users found that batteries were needed to charge the device after attempting to charge via connection to a computer

• Different apps were available for download which confused users• Most users were not successful reading their blood pressure the first time

“I spent something like 10 minutes trying to force the box open. I missed a transparent sticker that prevented the box to slide.”

“It worked fine from the very first time.”

“Not quite as slick as it could be.”

“The participant received a high reading. No information was given to prompt him to consider the cause.”

“There was no Russian version of the manual [or] list of languages on the cover of the manual so [the] participant had to look through the entire brochure.”

Multi-national user testing 21

Testing options – how to choose?

Moderated Unmoderated

Web/software UI More quantitative based Large, distributed sample or low

incidence Low(er) budget High penetration of Internet

access

Physical artifact Need rich qualitative feedback Need to have the human

connection Ensure high-level of consistency Uncertain of quality or

environment

Cultural differences In multi country moderated

testing

Planning

Running sessions

Planning

Check out the local calendar –

make sure you find out what really happens

In Japan, at the start of May there is

a week-long public holiday called

“Golden Week”; in August, there is

another week-long public holiday

called “Bon Festival”. Most

companies shut down so difficult to

recruit and test.

In Canada, hockey is a

religion! It’s difficult to recruit

evenings during the hockey

season.

In Brazil, don’t even think of recruiting

during Carnival (February or March, it

varies).

In France, avoid

running test sessions

in the first half of

May, around July

14th (Bastille Day)

In Australia and

many other

countries recruiting

participants is

difficult from mid

December to January

because of

Christmas and New

Year holidays.

Planning

Pick the time for sessions carefully

In Japan, evening and weekend sessions are common

because office workers are usually unable to attend daytime

sessions during weekdays.

In the US, it is difficult to schedule usability studies on the

weekend. Most US participants place a high premium on

their weekend time and reserve it for family, friends, or

weekend errands and chores.

In Moscow, many people have flexible working schedules;

some of them work late in the evening, so session times

can be varied. Free time is valuable (especially in large

cities), so it’s difficult to convince people to participate on

weekends.

Brazilians often run late – traffic and other occurrences that

prevent them from being in control contribute to laid-back

attitude to punctuality.

Planning

Consider local demographics

What does elderly mean?

Which languages do you consider?

Participants born abroad?

Planning

Plan ahead for recruitment.

Get help from someone local How many? Incentives?

• Italy over recruits by 20-30%

• UK 1 standby for each 6 participants

• Poland may one extra

• China, no shows rare

Planning

• Develop detailed

testing docs for

recruitment, running,

data collection and

reporting

• Translation

Allow enough time to plan

Protocol

• Recordings

• Sharing

files/streaming

• Cameras

• Test equipment

Technolog

y • Jetlag

• Where’s the lab?

• Time for a dry run

Travel

Running sessions

Does it matter who moderates?

English isn’t always English. In Australia, it’s

not unusual to use a local facilitator, even

when the client is English speaking and has

travelled to Australia to observe the sessions.In South Africa, it is most important to have

the right cultural fit between moderators and

participants. As such, moderators are not

typically experienced usability professionals.

What’s polite in one

culture can be

offensive in another

Running sessions‘Thinking aloud’ is very difficult in some cultures and too easy in others • The Brazilians and Irish love to talk, so thinking

aloud is no problem!

• Chinese users are often not willing to talk while

doing the tasks.

• Indians typically think aloud.

• For Germans and the British, small talk at the

front of sessions is very important to get

participants warmed up.

• In the UK, some participants are direct, while

others may be more circumspect. In some cases,

participants have struggled with tasks and yet

are still politely positive about it.

• In Japan, there is a cultural bias for men not to

say “I don’t know” or “I don’t understand,”

• Italians like to express their opinion on everything

- they are experts in all matters and when you ask

to think aloud during a task, you always get

opinion on the layout!

In conclusion

• There are valuable UX research options across a range of budgets

• Local culture, beyond language influences design, but also how you do

your research

• Allow enough time to plan, and

• Consider working with local partners

Ilana Kaplan UX NZ 2013

ilana.kaplan@optimalexperience.com

Dank U

Thank

You

Thank

you !

Merci

Kiitos

Grazie

mahal

o

DankeAny

question

s?