Soft PerformanceMeasuring

me

Dimiter Simov - Jimmy usability practitioner

product experience for SAP HANA Cloud Platform @ SAP

goal make you think about metrics when designing and testing

Performance

a task or operation seen in terms of how successfully it is performed

pay increases are now being linked more closely to performance

the capabilities of a machine, product, or vehiclethe hardware is put through tests which assess the performance of the processor

Source: Google define

recall ISTA 2013: performance has a soft sidetext and formattingFitts’ lawlayout and structureuser success and engagementpresentation of progressaesthetics 1 x 1.618 – the golden ratio

1. avoid messages, especially modal onesDON’T WRITE MESSAGES, DESIGN INTERACTIONS

2. if you have to give a message, make sure it is obvious who shows itwhat happenedwhywhat users can do about it

3. be practical

recall ISTA 2014: about messages

about measuring soft performance

look at how soft performance can be measured

success rates, satisfaction scores, orientation

higher scores predict better performance; not always

big data – a glimpse

trinity

effectiveness

efficiency

satisfaction

effectiveness | efficiency | satisfaction

effectiveness

effectiveness = result = success

shows whether users accomplish their tasks

the basic metric

1 or 0: users either succeed or not (could as well be a fraction)

measure success rate break down user journey into tasks and subtasks

assign 1 = success, 0 = failed

count and sum for each user and task

average across tasks, users, and user groups

report success rate aim for 100% but 70-80% is a great score

4 users out of 5 succeeded ≠ 80% success

confidence interval calculator[1]

[1] calculator: http://www.measuringu.com/wald.htm

sample

report

effectiveness | efficiency | satisfaction

efficiencyorientation

time on task

effort

errors

learnability

efficiency = productivity

ratio of what you get to what you invest

goal = maximum result with minimum effort and time

time

on

task

time on task = speed

shows how quickly usersperform tasks

measure by the clock relative: compare to another system or version

time on task

subjective time matters

example 10 websites: users performed their own personal tasks no two users performed the same tasks on any site measured perceived speed and actual speed per task

the faster website

about.comactual speed: 8 secondsperceived speed: slowest

amazon.comactual speed: 36 secondsperceived speed: fastest

perceived speed related to success

when a site is slow, if users succeed, they feel it fast

when a site is fast, if users fail, they feel it slow

estimate task times

rule of thumb

measure your timemultiply by 3 to 5

effort

effort

shows the consumption of resources needed to perform the tasks

involves counting: clicks,steps, calories, etc. effort

clicks and performance

fewer clicks usually mean better performance:nobody likes “clicky” UIs

clicks predict about 25% of task time [1]

[1] http://www.measuringu.com/blog/click-clock.php

KLM = Keystroke-Level Model

“predicts how long it will take an expert user to accomplish a routine task without errors” [1]

covers the straight case

[1] https://en.wikipedia.org/wiki/Keystroke-level_model

KLM example: dropdown versus radio buttons

m thinkp point the dropdown arrowb click the arrowr dropdown expands = 0m look at the listp point the item neededb click the itemr dropdown collapses = 0m look at the value to see it is selectedp move to the Submit buttonb click

m look at the listp point the item neededb click the itemp move to the Submit buttonb click

m p b r m p b r m p b = 7.2 sec m p b p b = 3.6 sec

m thinkp pointb click in the boxm look at the list againk type (between 8 and 24 characters)m look at the value to see it is correctp move to the Submit buttonb click

m p b m kkkkkkkk m p b = 8.24 – 12.72 sec

KLM calculator: http://courses.csail.mit.edu/6.831/2009/handouts/ac18-predictive-evaluation/klm.shtml

orientation

orientation = ability to navigate

shows how much users deviate from an optimal path

0 to 100 (could be percent)

orientation

measure coefficient of orientation define optimal path of steps for each task = 100% (O)

count the actual steps performed by users (A)

CoR = (1 – A/O)%

CoR = (1 - 3/6) = 50%optimal path actual path

1

2

3

1

3

2

1’

2’ 2’’

learnability

learnability

shows how fast users get to use the product

fast to slow (to never)

learnability

errors

errors = wrong UI

show why soft performance suffers

two types of errors to count slips = I know the correct way mistakes = I am guessing the correct way

errors

slip

I want to save the changesbut click Don’t Save instead of Save

my friend who has never before driven an automatic car is pressing the break pedal with his left foot

mistake

I wanted to change the background of a survey to be the same color as the orange color used on the ISTA website

R:G:B 220:110:15

the color I need is not in the palette

so I prepare an image with the desired color

and click the only option that looks different

…I dragand dropthe image file…

The photo you uploaded is too small! It must be at least 800 pixels wide and 200 pixels tall.

I edited the fileto 800 by 200 pixelsuploaded it and got this header

not the same as color I uploaded

effectiveness | efficiency | satisfaction

task 1

task 2

task 3

overall satisfaction

satisfaction = sense of accomplishment

shows how much users like the product from “not at all” to “completely”

caveat – users might be evaluating: look and feel self

task

satisfaction

task-level satisfaction

one question after each task

the single ease question (SEQ)

SEQ: http://www.measuringu.com/blog/single-question.php

topic-level satisfaction

overall

satisfaction

overall satisfaction defines the overall experience variety of methods and metrics

SUS = System Usability Scale NPS = Net Promoter Score SUPR-Q = Standradized Universal Percentile Rank Questionaire

…

SUS = System Usability Scale1. I think that I would like to use the system frequently2. I find the system unnecessarily complex3. I think the system is easy to use4. I think that I would need the support of a technical person to be able to use

the system5. I find the various functions in the system are well integrated6. I think there is too much inconsistency in the system7. I would imagine that most people would learn to use the system very quickly8. I find the system very cumbersome to use9. I feel very confident using the system10. I needed to learn a lot of things before I could get going with the systemSUS: https://en.wikipedia.org/wiki/System_usability_scale

scoring SUS

1. do this in Excel2. scale all values from 0 to 4 (four is the most positive)

odd items: user response - 1even items: 5 - user response

3. convert to range 0 to 100 sum the scaled responses for each user and multiply the total by 2.5

SUS shows three things [1]

satisfaction – the full set of items

learnability – items 4 and 10

usability – the other 8 items

[1] http://www.measuringu.com/sus.php

SUS explains likelihood to recommend [1]

NPS score ≈ 0.1* SUS score

SUS score of 80 ≈ NPS of 8

[1] http://www.measuringu.com/blog/nps-sus.php

average SUS

68

0 100

> 82promoters

[1] http://www.measuringu.com/products/SUSpack

overallbusiness apps

websites 75consumer apps

65cellphones

58average

41lowest

79highest

my studies in the last 2-3 years

ISTA website SUS

0 100

66satisfaction

60.9usability 86.2

learnability

as of 11:00, 19 Nov 2015, 68 participants

about the ISTA website, we can be: 95.0% confidentthe population SUS Scoreis between 61.78 and 70.14

97.5% confidentthe population mean SUS Scoreis above 61.78

mean = 66

margin of error = 4.18

promoters: 5.88%

passives: 32.35%

detractors: 61.76%NPS = -55.88%

trinity?

analytics

big data

internet of things

quality out of quantity data is out there (or in there) data is neither good, nor bad what matters is how we use data

can we use data to promote wellbeing and happiness?

things you might want to check quantified self PERMA happiness index – GNH = gross national happiness Sonja Lyubomirsky’s 12 steps to happiness

soft performance metrics to take home

effectivenesssuccess rate

satisfactiontask-level: SEQoverall: SUS

efficiencytime on taskeffort: Keystroke-Level Modelorientation: CoRerror rate

that’s

folks

@dsimovabout.me/

dsimov

all

all illustrations in this presentation are by Iliya Beshkov from the book „Тошко Африкански“ (Toshko Afrikanski)