Measuring Usability of BiometricsReview of Experiences at NPL

Linda Johnstone Sorensen

Linda.Sorensen@npl.co.uk

2

Outline

10 Years ago– BIOTEST project views on Usability Testing for Biometrics

1997–2008– Observations on usability measurement during …

• NPL Performance & Usability Testing 1999

• NPL Performance Evaluations 2000 – 2005

• UKPS Biometrics Enrolment Trial

2008 →– How would we update proposals from BIOTEST?

• What works well ? What doesn’t?

• What else should be included?

3

BIOTEST: 1996-1997

Collaborative EU Project

Objective: Develop methodologies to measure biometrics systems performance

Focussed on – Accuracy (I.e. error rates such as FMR, FNMR)– Security (I.e. robustness to spoofing, etc.)– Usability (Measuring ease of use etc.)

10+ years later …– Methodologies & metrics for assessing usability remain the

least well established

4

BIOTEST (1997)Usability measure proposals

What to measure?– Ease of enrolment & use– Acceptability of enrolment & use– Invasiveness of enrolment & use– Levels of Supervision– Enrolment risks– Exceptional enrolees

Not defined as usability– Physical characteristics

• Dimensions of device • Interfaces • Environmental conditions• Context of use

How to measure?– Quantitative measures

• Effectiveness– E.g. successful enrolment

• Efficiency– E.g. proportion of

unproductive time

– Time taken vs that of experienced user, …

– Qualitative measures• Expert assessment• Subject/Operator feedback

5

Usability testing of biometric systems for ATMs

Evaluation conducted in 1998/99– Assessment of verification

accuracy and usability – to guide a consideration of

implementing biometrics in an ATM system

– 2 fingerprint & 2 face recognition systems

– Opportunity to apply some of the methodology developed in BIOTEST

6

Usability testing of biometric systems for ATMs

– Verification accuracy assessed by Scenario Tests • Emulating sample of typical bank customers: 200+ test subjects

from staff on Teddington site and some relatives

• Emulating enrolling bank clerk: NPL staff on project team

– In-depth usability assessment• 20 subjects (demographic balance)

• Enrolment, training, & verification

• Separate observer

• Videotaping of the interactions

• Subjects interviewed before and after trial– Open questions

• Short questionnaire for all 200 test subjects– Closed questions

7

Change in opinions during the trial

Opinions Before using devices After the trial

Not comfortable using biometrics

Face: 4 / 20

Fingerprint: 4 / 20

Face: 6 / 20

Fingerprint: 3 / 20

Preferred biometric Face: 4 / 20

Fingerprint: 4 / 20

No preference: 12 / 20

Face: 5 / 20

Fingerprint: 9 / 20

No preference: 6 / 20

Other perceptions Fingerprints perceived as easy to forge

Fingerprints viewed as more stable than faces

Confidence in system improved by rejections (when doing something wrong)

8

Comparison with other methods of verification

Users asked how biometric devices compared to using a PIN

– 8/20 reported that biometric devices felt safer than PIN – 9/20 positive to biometrics not requiring memorisation – 10/20 said they would be willing to use the biometric

system to take out cash at an ATM – The reservations expressed loss of confidence due to

problems experienced during the trials and hence

9

Enrolment and verification problems observed

Face recognition– Height – Problems with eyes (e.g.

infections) – Wearing items such as glasses,

hats, sunglasses– Variations in hairstyle– Time taken to enrol/verify

Fingerprints– Poor quality fingerprints (e.g.

due to manual labour or accidents)

– Finger placement (e.g. just the tip of finger on sensor)

– Removing finger before image capture is complete

10

NPL 2005 evaluation

Impact of usability for operators

NPL Biometric Evaluation 2005– Pier 2-3 Handheld Iris camera– Holding the camera steady

• Expected to be difficult • Found easy after a practicing

– Intrusiveness• Expected that subject would

find the experience intrusive (camera held close to face).

• Findings – Operators also feel uncomfortable holding the camera so close

– Performance differences between operators not significant

11

NPL 2005 evaluation

Impact of usability for operators Correct

Thicker reference

lines on forehead and chin

NPL Biometric Evaluation 2005: 3D-face enrolment– Of the errors incurred, most

attributable to poor enrolment– Operator

• Instructs subject throughout• Raises/lowers camera• Checks subject

– Positioning– Pose – No smiling/talking– Fringe …

– Problems not always clear on on-screen display

Incorrect

Too far Too close (d (due to height)

12

NPL 2005 evaluation

Impact of usability for operators

NPL Biometric Evaluation 2005:2D-face enrolment– Warnings & operator advice for

non-optimal images– Resulting in …

• Perfect matching performance– No false matches and

– No false non-matches

• Longer enrolment times Shows multiple quality measures& whether these are adequate

Recommendation toaccept or retake

Shows whether algorithmcorrectly locates eyes

13

UKPS biometrics enrolment trial 20052003 Feasibility study on biometric ID cards– Main performance unknowns are around usability by all

sections of the population

2004/5 Biometrics enrolment trial– Focus on

• Enrolment & verification durations• Customer perceptions and reactions• Exception cases• Demographic differences

– Included a significant proportion of disabled usersdemographically balanced “quota” groupvolunteering members of the public

14

Error rate by demographic group

1st attempt enrolment errors

Face Iris Fingerprint

Age: 18-24 2% 21% 27%

Age: 35-34 3% 18% 29%

Age: 35-44 4% 17% 27%

Age: 45-54 4% 21% 29%

Age: 55-59 5% 26% 30%

Age: 60-64 5% 30% 31%

Age: 65+ 4% 41% 34%

Hearing impairment 10% 51% 35%

Learning disability 12% 56% 63%

Physical impairment 12% 52% 50%

Visual impairment 12% 65% 36%

Outlier groups showmore usability problems

17

UKPS Biometrics Enrolment Trial

Actual time taken vs. User feedback on time taken

– User response not particularly correlated with actual time taken

Don’t know A lot slower than expected A little slower than expected About the same as expected A little quicker than expected A lot quicker than expected

0 2 4 6 8 10 12 14 16 18 20 22 24 Fingerprint Enrolment: Time taken (minutes)

Average

How did the time it took to have your fingerprint image

recorded compare with what you

expected?

18

Findings on quantitative usability measures

Effectiveness– Measured by error rates

• Failure to Enrol, False Non-match rate, False Match Rate– These are and mainly determined by exception cases

• Exception cases (oldest / youngest / tallest / shortest / disabilities) often reveal more usability issues than typical users

Efficiency– Appropriate to measure use with

• Habituated subjects (familiar with using the systems) as well as

• Unhabituated subjects, with & without operator assistance

• Many tests use mainly unhabituated subjects with assistance

19

Findings on qualitative usability measures

Expert Assessment – Useful part of many evaluations: revealing errors due to usability– Assessor independent of the operator/subject interaction– Some (though not all) issues can be assessed by experts without

observing real use • Possibility of checklist?

Subject/Operator feedback– Limited usefulness as indicator of usability

• User’s goals not always the same as the system’s goals– Feedback is influenced by factors other than operational effectiveness

E.g.• Pre-conceptions• Novelty of the experience

20

Outstanding issues - 1

Acceptability of a biometric system

Classes of users – ‘People problems’; some groups of people have problems

with due to physical appearances – Technology may be challenging for certain user groups – Ergonomic concerns for specific user groups

User behaviour – Lack of behaviour compliance; when people don’t do what

you want them to do (ideal system do not require explanations).

Operator concerns with biometric systems

21

Outstanding issues - 2

How do we combine measures of usability in biometric systems? – We cannot always see what is most ‘usable’ from measures of

performance alone – We cannot see what is most usable from user feedback alone– Most errors observed in trials with biometric systems are not errors of the

system, but errors in its use!

Trust– How do we include trust?

• Trust in the technology– E.g. people like to see a ‘failure’ once in a while, to confirm that the system

works

• Trust in the system– E.g. security of the system

– Storage of biometric data – remotely or in personal chip-card

22

Thank you!