# USABILITY TESTING

Usability

Definition of Usability in ISO/IEC 9241-11 Standard

The extent to which a product can be used by specified

users to achieve specified goals with effectiveness,

efficiency and satisfaction in a specified context of use.

Objective

Device testing

Performance testing

Security testing

• Ease of use

• Ease of learning with the system

• Satisfaction with the entire experience

Documents• Ethics documents

• Consent form

• Post-test questionnaire

Participants

• Be representative of the target users

• Own a target device + OS

• Be available at the time and place

• Be ready to sign a consent form

Devices

Procedure

Step Time

Welcome / Signing of the consent form 5 min

Pre-test interview 5 min

Carrying out the test tasks 35 min

Post-test questionnaire / interview 10 min

Debriefing 5 min

What is debriefing?

• Going through and analyzing the actions performed by the

participant (conducted with the participant)

• Test session identifies problems, debriefing session provides

insight on why those problems occurred

Testing Tasks

• Register an account

• Sign into your account

• Log your diary data

• Retrieve Fitbit data

• Visualize data

Data CollectionQuantitative data

Qualitative data

• Screen recorder records the foot stage of clicks

• Timer recorders how long it took participants to complete each task

• Bio-sensors record physiological metrics

• Can you talk out loud …?

• Can you walk me through…?

• What do you think of …?

• Could you suggest 3 changes ?

Performance metrics

ISO/IEC 9126-4 Metrics

Effectiveness The accuracy and completeness with which users

achieve specified goals

Efficiency The resources expended in relation to the accuracy and

completeness with which users achieve goals.

Satisfaction The comfort and acceptability of use.

Effectiveness

Completion Rate

𝑒𝑓𝑓𝑒𝑐𝑡𝑖𝑣𝑒𝑛𝑒𝑠𝑠 =𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑎𝑠𝑘𝑠 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑑

𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑎𝑠𝑘𝑠× 100%

Number of Errors

Efficiency

Time-based Efficiency

𝑇𝑖𝑚𝑒 𝐵𝑎𝑠𝑒𝑑 𝐸𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑐𝑦 =

σ𝑗=1𝑅 σ𝑖=1

𝑁 𝑛𝑖𝑗𝑡𝑖𝑗

𝑁𝑅

N: total number of tasks

R: total number of users

𝑛𝑖𝑗: the result of task i by user j; 𝑛𝑖𝑗=0 if a user fails to complete the task; 𝑛𝑖𝑗

=1 if a user complete the task

𝑡𝑖𝑗: the time spent by user j to complete task i

Efficiency

Overall Relative Efficiency

𝑂𝑣𝑒𝑟𝑎𝑙𝑙 𝑅𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝐸𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑐𝑦 =σ𝑗=1𝑅 σ𝑖=1

𝑁 𝑛𝑖𝑗 ∙ 𝑡𝑖𝑗

σ𝑗=1𝑅 σ𝑖=1

𝑁 𝑡𝑖𝑗× 100%

N: total number of tasks

R: total number of users

𝑛𝑖𝑗: the result of task i by user j; 𝑛𝑖𝑗=0 if a user fails to complete the task; 𝑛𝑖𝑗 =1

if a user complete the task

𝑡𝑖𝑗: the time spent by user j to complete task i

Satisfaction

Measured through standard satisfaction questionnaires

Task Level Satisfaction

• Questionnaires should be given to participants immediately

upon they attempt a task (irrespective of whether they

manage to achieve its goal)

• Popular questionnaires

SEQ: Single Ease Question (1 question)

UME: Usability Magnitude Estimation (1)

SMEQ: Subjective Mental Effort Questionnaire (1)

ASQ: After Scenario Questionnaire (3)

NASA-TLX: NASA’s Task Load Index (5)

Satisfaction

Test Level Satisfaction

• Questionnaires should be given to participants at the end of the

test session

• Popular questionnaires:

SUS: System Usability Scale (10 questions)

SUPR-Q: Standardized User Experience Percentile Rank Questionnaire (13)

CSUQ: Computer System Usability Questionnaire (19)

QUIS: Questionnaire for User Interaction Satisfaction (24)

SUMI: Software Usability Measurement Inventory (50)

Results

• Background summary: what, where, who was tested,

who was the testing team, what equipment was used

• Methodology: demographic data, task scenario,

metrics collected

• Test results: quantitative data (completion rates, error,

task times, satisfaction ratings) and qualitative data

(problems experienced, good/bad points, suggestions).

• Findings and recommendations: good and bad

usability, what and how to improve.

To Do

• Prepare documents (consent forms, questionnaires)

• Create task list

• Conduct usability testing with 3 participants

(If your app is not ready for user test today, you can continue working

on your app and do a user test later after the class)

• Write usability testing report

Submit Usability Testing Report to test.u-tokyo@hotmail.com

by 9 Jan Tuesday 12:00 pm.