1

Qualitative Evaluation Approach – Usability Engineering

•  Objectives –  Identify user problems and issues with systems –  Provide specific recommendations for improving system –  Evaluate effectiveness of IT –  Provide input into system redesign and improvement –  Detect and prevent technology-induced errors

•  Level –  Individual/multiple users carrying out work tasks

Dr. Andre Kushniruk, University of Victoria, andrek@uvic.ca

Background

•  Usability – measure of “ease of use” and usage of a system

1. Learning 2. Effectiveness 3. Efficiency 4. Enjoyability 5. Safety

Usability engineering - scientific methods to improve system usability

Brief Description of Approach

•  Usability Testing •  Representative users and tasks •  “Think Aloud” Protocol Analysis •  Video Recording

•  Cognitive Task Analysis •  Process centered •  Includes focus on mental operations

4

Low Cost Rapid Usability Testing

Usability Testing in Health IT

•  Video and audio recording of representative users (e.g. physicians, nurses etc.) as they carry out representative tasks using systems

•  Qualitative and quantitative analysis of resultant data

•  Feedback to make systems better •  Can and should be used to predict (and fix)

errors before systems are released 5

History of Approach

6

•  Publication of “Usability Engineering” by Nielsen 1993 •  “Think aloud” protocol analysis was first introduced in

the early 1990s with the usability testing of users of electronic patient record systems (EPR)

•  Used in conjunction with video and audio recording of subjects (e.g. doctors and nurses) interacting with systems

•  Publication of work in applying usability to improving health information systems (MedInfo 1995)

•  Emergence of labs in both healthcare software industry and hospital settings – 2000 onwards

Applications in Healthcare

7

•  Been used in evaluation of : – Electronic health record systems – Decision support systems – Educational systems (medical and nursing) – Best practice guidelines – Patient information systems – Web health applications – Mobile health applications

Example #1

n  How effective are Clinical Prediction Rules embedded within an institutional Electronic Health Record (EHR) system….

n  Application of low-cost usability testing approach to assess user problems (usability problems) as well as impact on cognition and workflow

9

n  Video record of user interactions n  Screen recordings (using

Hypercam ® screen recording software)

n  “think-aloud” data of subject (audio data)

Data Collection

10

•  Content

•  too much information •  not enough information •  innapropriate information •  incorrect (out-of-date) information •  relevance of information

•  Comprehension •  graphics •  text •  audio

Data Analysis – Use of Coding Schemes to Analyze Video and Audio Data

11

•  Navigation •  ability to go back or forward •  ability to select/find wanted screen

•  User Control/Pace •  ability to pause •  pace of material

•  Understandability •  Understandability of information presented •  consistency of operations

•  ETC.

Coding Categories (Continued)

12

3:40 Clinical Prediction Rule triggered (within EHR) 3:42 Subject scrolls through text in the rule “I’m reading from the top, clinical prediction rule, telling me that the diagnosis is strep throat, which score of 4 would have me think ” 3:50 Two antibiotics appear checked off in computer generated Smart Set “And those are antibiotics I would use, not necessarily both of them that are already checked , I would choose one or the other.”

COMMENT – CONTENT - ANTIBIOTICS APPROPRIATE INDIVIDUALLY (NOT TOGETHER)

“And pending no allergies click and give them penicillin and go down to the next portion of the group for supportive therapies. Not sure what the normal corresponds to”

UNDERSTANDABILITY -- not sure what “normal” means

Video Analysis (Log File) of a User’s Interaction with the System

Example #2 – Analysis of Impact of a System on Healthcare Work Using Medication

Administration Tasks (Scenarios)

13

14

FIRST ORDER: 00:14.3 SEARCH FOR PATIENT ON COMPUTER 00: 45.7 VIEW ORDER LIST 00:51:9 SELECT ORDER 00:55:.3 VERIFICATION SCREEN APPEARS MOVES OVER TO PATIENT 00:59:6 TALKING TO PATIENT S. “Nice to meet you. Is your name Toridai, right? I will now give you

an IV drip” 01:09.5 SCAN PATIENT ID (FROM WRIST) MOVES BACK TO COMPUTER 01:25:2 VIEW EXECUTION INFORMATION MOVES OVER TO PATIENT AND SETS BAG SECOND ORDER: 02:24.6 SEARCH FOR PATIENT ON COMPUTER 02:40.2 VIEW ORDER LIST

Video Log File of User (Nurse) Administering Medications

Post Task Interview

15

E: Do you find any difficulty with handling the barcode reader?

S: In today’s operation there were no problems. But in the real situation, sometimes the scanner doesn’t respond to the barcode. Also sometimes the cord of the scanner is too short to reach the patient.

E: Do you find any difficulty with entering the data?

S: Sometimes in the emergency we have to skip this procedure due to its time-taking process and someone might need urgent help, but with this system I don’t think I’d be able to do that …

Results

•  New medication system changed process from parallel to serial

•  Can be safer in some situations •  However, in other situations, e.g.

emergencies, could lead to dangerous situation (i.e. if workflow becomes too rigidly “locked in”)

•  Implications: development of an “emergency override” capability 16

Lessons Learned – What works and what doesn’t

•  Applying the approach in real/realistic environments can be

•  Low cost and rapid (does not require huge expense) •  Lead to realistic predictions

•  The earlier the testing in development life cycle the better

•  Qualitative approaches to data analysis needed –  Think-aloud protocol analysis –  Video analysis –  Combining the above two leads to best data

18

Conclusions

•  Approach can be used to assess degree of system-user fit

•  Approach can be used to provide specific input into system redesign and improvement – Can also be used in system selection process

•  Approach can be applied at multiple levels – From individual users to study of work tasks

involving groups of participants (e.g. physicians and patients, and other healthcare workers)

19

Future

•  What remains to be done: – Approach needs to be more widely

disseminated and deployed –  Centralized usability testing (by vendor or

government agency) is not sufficient, also need for local in-situ testing

•  Expected impact of approach: – Will have major impact but needs to be applied

to much greater degree (and level of depth) before systems are released