1

Topic 6:Usability Evaluation of IA

Applications and Mechanisms

Azene Zenebe, Ph.D.

Bin Mai, Ph.D.

Presentation Outline Introduction Usability of IA applications and mechanisms -

Reviewed Usability Evaluation: What, When and Why Usability Specification for Evaluation Usability Evaluation Methods

ˉ Analytical methods

ˉ Empirical methods Case Study Summary

2

3

Learning Objectives and Outcomes After completing this module, you should

be able to:

ˉ Describe the factors that affect usability of security systems

ˉ Describe the importance of evaluation of usability security systems

ˉ Prepare usability specification for evaluation

Learning Objectives and Outcomes (Continued)

ˉ perform usability evaluation or testing of a security system using an analytical method such as expert inspection

ˉ Perform usability evaluation or testing of a security system using an empirical method such as a field study or lab testing

ˉ Report results of usability evaluation as well as describe how the results can be used to make improvement

4

5

Introduction

Usability of IA application and mechanism - Reviewed ˉ Usability refers to the extent to which a product

can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user - ISO 9241-11

6

Multi-dimensionality of Usability

Ease of learning Efficiency of use Memorability Effectiveness Error frequency and severity Subjective satisfaction

7

“Usable” Security Systemscan easily and quickly learn a security system that they have never seen to accomplish basic tasks can remember enough to use them later without major cost are able to effectively perform and successfully complete security tasks supported by them cannot make sever and frequent errors are satisfied with the interface and functions of the systems

8

Framework for studying usability of security systems

four principal components in a human-machine system ˉ TOOL ˉ USER ˉ TASK ˉ ENVIRONMENT

9

Definers provide the policies, guidelines, and standards

Builders are the real techies, who create and install security solutions

Administrators operate and administer the security tools

End-users include home users and employees who are novice to CISS

Four groups of people involved in Security systems

Usability Evaluation: What, When and Why

Usability evaluation: whether a security system is usable for the users

Goal of usability evaluation: identify and correct flaws associated with ease of use of a security system

Performed during design and testing (or post-implementation) phases

Evaluation is iterative – an ongoing process

10

11

Usability Specification for Evaluation

Usability specifications are statements of required usability characteristics that are precise and testable

Task analysis provides a more precise specification of what users are expected to do in order to accomplish a task successfully

A sample usability specification - authenticity of a website

12

Subtasks Usability Outcomes Expected

Displaying the digital certificate of the website

A user with at least …previous usage experience should be able to display the certificate in a 40 seconds or less, with no errors, and should rate ‘easy of finding the menu item/icon for displaying the certificate’ no less than 6 on a 7-point rating scale.

Determining if the website is authenticate or not

A user with at least …previous usage experience should be able to read and comprehend the certificate information in a 80 seconds or less, with no errors, and should rate ‘easy of use the menu item/icon for navigating the certificate’ no less than 6 on a 7-point rating scale.

13

Usability Evaluation Methods

Analytical Methods - conduct analysis of a system’s features with the respect to their impacts for use

Empirical Methods – collect and use data from a system’s users. It is also referred as user-based testing

14

Analytical Methods Expert’s knowledge stated as a

heuristic rulesˉ Ten Usability Heuristics by Jakob

Nielsenˉ Shneiderman’s 8 Golden Rules of

Interface Design

15

Empirical Methods

What usability evaluators want to know is what happens when users use the systemDifferent techniques are ˉ Field studies ˉ Usability Testing in a laboratory ˉ Controlled Experiments

16

Quick Quiz What are the main advantages and

disadvantages for analytical methods and empirical methods?

Come up with two sample scenarios in IA field where you think analytical methods should be preferred, and two other scenarios where you think empirical methods should be preferred

17

Steps for usability testing Identify and profile the representative users Select the setting Decide what tasks users should perform Decide how and what types of data to collect Perform necessary activities before test session Perform necessary activities during test session Perform necessary activities after test session

18

Usability Testing in a Laboratory

Validity concerns are associated with the following questions for lab based testingˉ Is the prototype system used in the testing

missing any important features ˉ Are test participants really the kind of users

who will use the system ˉ Will actual users do tasks like these

participants ˉ Will actual users be more distracted in their

offices

19

Using the Results of Usability Testing

Results need to be looked at and actionable information regarding usability problems and issues should be made for design teams

Provide recommendations to address the identified problems

20

Automated Usability Testing Tools

A List of 24 Web Site Usability Testing Toolsˉ http://www.usefulusability.com/24-usabi

lity-testing-tools/

UMD list of usability testing tools ˉ http://otal.umd.edu/guse/testing.html#se

ct3a

Jay Forbes’ presentation about usability testing tools ˉ http://www.gslis.utexas.edu/~l385t6rb/a

uto_tools.pdf

Quick Quiz

Suppose you are testing the usability of an IDS your company decided to implement. ˉ What will be the setting of the testing? ˉ Who will be the representative users?ˉ What type of data should you collect?

Justify your answers.

21

Quick Quiz

Among IT managers, business managers, usability specialists, or general public, who do you think are the main users for automated usability testing tools? Why?

 What aspects of a usability study do you believe can never be automated? Why?

22

23

Case Perspectives: Usability Evaluation

Perspectives is a new approach to help clients securely identify Internet servers in order to avoid "man-in-the-middle" attacks

works with Firefox 3 extension Demo

Mission of the Perspectives detect whether a self-signed certificate is valid detect the fake security certificate attack and will

warn you

24

Usability Evaluation Design User Population

Potential Users: Novice, Intermediate and Expert in Security and IT

Targeted Users: Subset of the Potential Users Context of Uses

Using the Internet Home, free WiFi sites, and/or work Quite or Not Quite environment

Tasks: Banking, Shopping, etc.

25

Usability Evaluation DesignPerspectives: evaluating the authenticity of a public key based on accompanying signatures and making use of a Browser’s built-in mechanisms for such evaluation Requirements gathering

ˉDevelop usability specificationUsability Evaluation

ˉUsing InspectionˉUsing Empirical

26

Summary From this module, reader should take

away the following:ˉ Usability is a combination of factorsˉ Usability requires that users understands

the organization policy and rulesˉ There exist frameworks that guide the

usability evaluationˉ For different stakeholders, the goals of

usability differ

27

Summary (continued)From this module, reader should take away the following:

ˉUsability specification is required for usability evaluationˉThere are two categories of usability evaluation methodsˉThere existing some tools that automate usability testing

28

Discussion Topics

What are the advantages and disadvantages of Inspection method?

What are the advantages and disadvantages of Empirical method?

Compare and contrast the different methods of data collection. Describe the advantages and disadvantages of these methods.

29

Discussion Topics

How useful are these Heuristics for security systems? Which of the two is more relevant to security systems? Are these methods security systems dependent?

Is there a heuristics for security system interface design? Is there a methodology?

30

Discussion Topics

Describe and discuss scenarios where a system’s usability is important to one type of users, while not so important to another type

What are your opinions regarding the ideas that, as described by Jay Forbe, “automated usability testing is too good to be true”?

31

Project Ideas

Suppose your friend Joe opened an E-bay store online to sell his comic book collections, what data do you collect to evaluate his website’s usability?

Suppose a university Registrar Office hires you to evaluate the usability of its online registration system. What data would you collect?

32

Project Ideas

Prepare a sample usability specification built to track usability of a scenario for setting a firewall in Windows XP.

Develop a usability evaluation desing to track usability of an IDS (Intrusion Detection System)

Design a usability evaluation study for the latest release of PGP.

33

References 1. Braz, C. and Robert, J.-M. Security and usability: the case of the user authentication

methods. In Proceedings of the 18th International Conferenceof the Association Francophone d'Interaction Homme-Machine ACM, Montreal, Canada 2006 199-203

2. Garfinkel, S.L. Design Principles and Patterns for Computer Systems That Are Simultaneously Secure and Usable Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY, Boston, 2005, 470.

3. Hoonakker, P., Bornoe, N. and Carayon, P., Password Authentication from a Human Factors Perspective: Results of a Survey among End-Users. In 3rd Annual Meeting of the Human Factors and Ergonomics Society, (San Antonio, TX, 2009).

4. Josang, A., Alfayyadh, B., Grandison, T., Alzomai, M. and Mcnamara, J., Security usability principles for vulnerability analysis and risk assessment. in Twenty-Third Annual In Computer Security Applications Conference, (Miami Beach, Florida, 2007), 269-278.

5. Lazar, J. Web Usability: A User-Centered Design Approach. Pearson, Addison Wesley, Boston, 2006.

6. Nielsen, J. Usability Engineering. Morgan Kaufmann, San Francisco, 1994.

34

References

7. Rosson, M.B. and Carroll, J.M. Usability Engineering: Scenario-based development of human-computer interaction. Morgan Kaufmann, San Francisco, 2002.

8. Shackel, B. Usability - Context, Framework, Definition, Design and Evaluation. in Richardson, S. ed. Human Factors for Informatics Usability, Cambridge University Press, Cambridge, 1991.

9. Shneiderman, B. and Plaisant, C. Designing the User Interface. Addison-Wesley, Boston, 2005.

10.Weir, C.S., Douglasa, G., Carruthers, M. and Jacka, M. User perceptions of security, convenience and usability for ebanking authentication tokens. Computer & Security, 28 (1-2). 47-62.

11.Whitman, M.E. and Mattord, H.J. Management of Information Security. Course Technology, Thomson Learning, Inc., Canada, 2004.

12.Whitten, A. and Tygar, D., Why Johnny can't encrypt? In USENIX, (1999).