Development and Usability Testing of CDS Alternatives

January 8th, 2008Jason J. Saleem, PhD & Laura Militello, MA

LEADERS SYMPOSIUM“Strategic Planning to Inform a Funded Project onhow to Achieve Workflow Integration in Developing and Implementing CDS for CRC Screening”

2

Study Aims

Specific Aim 1: Identify key approaches to CDS development for colorectal cancer screening at two VAMC sites and two nationally recognized non-VA sites, for effective CDS integration into clinical workflow.

Specific Aim 2: Develop and test CDS design alternatives for improved integration into clinical workflow through a controlled simulation study and subsequent implementation.

33

Key Informant Interviews of site-specific best-practices for integration of colorectal cancer screening CDS into workflow

Direct Observation of colorectal cancer screening CDS for barriers and facilitators to workflow integration

Implementationin primary care clinic after simulation study

Rapid Prototyping of CDS design alternatives based on Phase 1 findings

Simulation Study to test impact of CDS design alternatives on efficiency, usability, and workload

Evaluation in primary care clinic after simulation study

Figure. Project Overview

Phase 1 Phase 2 Phase 3

44

Human-Computer InteractionInformation Technology

Laboratory(HCI / IT LAB)

D-5014

VA Laptop•Morae•Snag-it

VA Desktops•Visual Basic

IU Laptop•Morae•Snag-it

IU Desktops•Visual Basic•Azure X 1PC

47 inch monitor

17” monitor & player forresearcher’s office

- VA LAN OUTLET

- IU LAN OUTLET

Portable DigiCam

w/WA adapter

5

Rapid Prototyping

Prototype design alternatives to address barriers to effective CDS workflow integration identified during the field observations and key informant interviews.

Rapid – demonstration of designs with no back end programming; low-fidelity mock-ups with prototyping software.

Screen captures of the current design will be used as a visual base and then redesigned.

Links and buttons will be made interactive to mimic the actual function of how the electronic medical record and CDS would work if fully programmed.

6

Simulation Study Usability experiment with simulated outpatient

clinic exam room Comparison study of the current VHA colorectal

cancer CDS and a redesigned prototype. Demonstrate improved integration into workflow

through relevant surrogate measures of performance, usability, and workload Improved efficiency using the CDS, or time to complete

tasks, suggests the CDS is better aligned with the providers’ clinical workflow.

Through improved workflow integration we also expect to see an increase in user satisfaction (usability) and decrease in workload.

77

Simulation Study At least 20 resident physicians Within-subject comparison Morae usability software to drive video and audio data

collection, and provide direct link between the participant’s clinical workstation and the experimenter’s station

Record audio and two video sources, one of the computer screen interaction and one of each of the participants’ faces

Opportunity to record qualitative data on usability critical incidents

Representative scenarios to construct patient encounters

Research team member playing the part of the patients;

Or – use a Standardized Patient (Linder et al., JBI, 2006) Debrief session

88

Dependent Measures

Attribute Measure or Tool

Efficiency Time to complete scenarios

Perceived workload NASA Task Load Index (TLX)

User satisfaction Usability questionnaire

99

NASA Task Load Index (TLX)

10

Usability Questionnaire

Likert-type scale (1=strongly disagree; 7=strongly agree)

Sample questions: The organization of the information on the

systems screens is clear. The display layouts simplify tasks. The sequence of displays is confusing.

11

Findings Validate and refine prototype through pre-

implementation usability testing Improved colorectal cancer screening CDS full

development and implementation in live clinic for pilot test evaluation (Phase 3)

Phase 1 (naturalistic observation and key informant interviews) and Phase 2 (rapid prototyping and simulation study) methods based on successful previous work (Saleem et al., JAMIA, 2005; Saleem et al., JAMIA, 2007). But fell short of direct impact to current VA system Phase 3: Implementation & Dissemination