ONC HIT Certification Program Test Results Summary for 2014 … · Test Results Summary for 2014...

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Apr-2015

ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification

Part 1: Product and Developer Information

1.1 Certified Product Information

Product Name: HealthAxis EHR Product Version: 2.0 Domain: Ambulatory Test Type: Complete EHR

1.2 Developer/Vendor Information

Developer/Vendor Name: HealthAxis Group Address: 5415 Mariner Street Suite 222 Tampa FL 33609 Website: http://www.healthaxis.com/ Email: [email protected] Phone: 813-830-8494 Developer/Vendor Contact: Dhaval Patel

of 11

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Apr-2015 Part 2: ONC-Authorized Certification Body Information

2.1 ONC-Authorized Certification Body Information

ONC-ACB Name: Drummond Group

Address: 13359 North Hwy 183, Ste B-406-238, Austin, TX 78750

Website: www.drummondgroup.com

Email: [email protected]

Phone: 817-294-7339

ONC-ACB Contact: Bill Smith

This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:

Bill Smith

Certification Body Manager ONC-ACB Authorized Representative Function/Title

4/5/2015

Signature and Date

2.2 Gap Certification The following identifies criterion or criteria certified via gap certification

§170.314

(a)(1) (a)(17) (d)(5) (d)(9)

(a)(6) (b)(5)* (d)(6) (f)(1)

(a)(7) (d)(1) (d)(8)

*Gap certification allowed for Inpatient setting only

x No gap certification

of 11

http://www.drummondgroup.com/

mailto:[email protected]

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Apr-2015 2.3 Inherited Certification

The following identifies criterion or criteria certified via inherited certification

§170.314

(a)(1) (a)(14) (c)(3) (f)(1)

(a)(2) (a)(15) (d)(1) (f)(2)

(a)(3) (a)(16) Inpt. only (d)(2) (f)(3)

(a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only

(a)(5) (b)(1) (d)(4)

(f)(5) Optional & Amb. only (a)(6) (b)(2) (d)(5)

(a)(7) (b)(3) (d)(6)

(f)(6) Optional & Amb. only (a)(8) (b)(4) (d)(7)

(a)(9) (b)(5) (d)(8) (g)(1)

(a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(2)

(a)(11) (b)(7) (e)(1) (g)(3)

(a)(12) (c)(1) (e)(2) Amb. only (g)(4)

(a)(13) (c)(2) (e)(3) Amb. only

x No inherited certification

of 11

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Apr-2015 Part 3: NVLAP-Accredited Testing Laboratory Information

Report Number: KAM-032615-1971

Test Date(s): 08/08/13, 05/12/14, 06/30/14, 09/30/14, 12/04/14, 03/26/15

3.1 NVLAP-Accredited Testing Laboratory Information

ATL Name: Drummond Group EHR Test Lab

Accreditation Number: NVLAP Lab Code 200979-0

Address: 13359 North Hwy 183, Ste B-406-238, Austin, TX 78750

Website: www.drummondgroup.com

Email: [email protected]

Phone: 512-335-5606

ATL Contact: Beth Morrow

For more information on scope of accreditation, please reference NVLAP Lab Code 200979-0.

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative:

Kyle Meadors

Test Proctor ATL Authorized Representative Function/Title

4/5/2015

Nashville, TN (Remote) Signature and Date Location Where Test Conducted

3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

Additional Software Applicable Criteria Functionality provided by Additional Software

DrFirst Rcopia a.1, a.2, a.10, b.3 e-Prescribing Surescripts Network for Clinical Interoperability

b.1, b.2, e.1

Direct HISP

No additional software required

of 11

http://ts.nist.gov/standards/scopes/2009790.htm

http://www.drummondgroup.com/


http://ts.nist.gov/standards/scopes/2009790.htm


3.2.2 Test Tools

Test Tool Version

x Cypress 2.4.1 x ePrescribing Validation Tool 1.0.4

HL7 CDA Cancer Registry Reporting Validation Tool 1.0.3 HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool 1.8

x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool 1.8

x HL7 v2 Laboratory Results Interface (LRI) Validation Tool 1.7 x HL7 v2 Syndromic Surveillance Reporting Validation Tool 1.7 x Transport Testing Tool 179 x Direct Certificate Discovery Tool 3.0.2

No test tools required

3.2.3 Test Data

Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter]

No alteration (customization) to the test data was necessary

3.2.4 Standards

3.2.4.1 Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted

Criterion # Standard Successfully Tested

(a)(8)(ii)(A)(2)

§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(a)(13)

x §170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release

§170.207(j) HL7 Version 3 Standard: Clinical Genomics; Pedigree

of 11


Criterion # Standard Successfully Tested

(a)(15)(i)

x §170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(a)(16)(ii) §170.210(g)

Network Time Protocol Version 3 (RFC 1305)

§170. 210(g) Network Time Protocol Version 4 (RFC 5905)

(b)(2)(i)(A)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(b)(7)(i)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(e)(1)(i)

Annex A of the FIPS Publication 140-2 [list encryption and hashing algorithms] AES SHA-1

(e)(1)(ii)(A)(2) x §170.210(g)

Network Time Protocol Version 3 (RFC 1305)

§170. 210(g) Network Time Protocol Version 4 (RFC 5905)

(e)(3)(ii)

Annex A of the FIPS Publication 140-2 [list encryption and hashing algorithms] AES SHA-1

Common MU Data Set (15)


§170.207(b)(2) The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)

None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards

of 11


The following identifies the newer version of a minimum standard(s) that has been successfully tested

Newer Version Applicable Criteria

No newer version of a minimum standard was tested

3.2.5 Optional Functionality

Criterion # Optional Functionality Successfully Tested

x (a)(4)(iii) Plot and display growth charts

(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(f)(3) Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)


Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)


Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)

No optional functionality tested

of 11


3.2.6 2014 Edition Certification Criteria* Successfully Tested

Criteria # Version

Criteria # Version

TP** TD*** TP TD

x (a)(1) 1.2 1.5 x (c)(3) 1.6 1.6 x (a)(2) 1.2 x (d)(1) 1.2

x (a)(3) 1.2 1.4 x (d)(2) 1.5 x (a)(4) 1.4 1.3 x (d)(3) 1.3 x (a)(5) 1.4 1.3 x (d)(4) 1.3 x (a)(6) 1.3 1.4 x (d)(5) 1.2 x (a)(7) 1.3 1.3 x (d)(6) 1.2 x (a)(8) 1.2 x (d)(7) 1.2 x (a)(9) 1.3 1.3 x (d)(8) 1.2 x (a)(10) 1.2 1.4 (d)(9) Optional 1.2 x (a)(11) 1.3

x (e)(1) 1.8 1.5 x (a)(12) 1.3 x (e)(2) Amb. only 1.2 1.6 x (a)(13) 1.2 x (e)(3) Amb. only 1.3

x (a)(14) 1.2 x (f)(1) 1.2 1.2 x (a)(15) 1.5 x (f)(2) 1.3 1.7.1

(a)(16) Inpt. only 1.3 1.2 x (f)(3) 1.3 1.7

(a)(17) Inpt. only 1.2 (f)(4) Inpt. only 1.3 1.7

x (b)(1) 1.7 1.4

(f)(5) Optional & Amb. only 1.2 1.2

x (b)(2) 1.4 1.6

x (b)(3) 1.4 1.2

(f)(6) Optional & Amb. only 1.3 1.0.3

x (b)(4) 1.3 1.4

x (b)(5) 1.4 1.7 (g)(1) 1.7 1.9 (b)(6) Inpt. only 1.3 1.7 x (g)(2) 1.7 1.9

x (b)(7) 1.4 1.6 x (g)(3) 1.3 x (c)(1) 1.6 1.6 x (g)(4) 1.2

x (c)(2) 1.6 1.6

No criteria tested *For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD)

of 11

http://www.healthit.gov/certification


3.2.7 2014 Clinical Quality Measures*

Type of Clinical Quality Measures Successfully Tested:

x Ambulatory

Inpatient

No CQMs tested

*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)

Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version

x 2 v3 90 136 155

22 117 x 137 v2 156

x 50 v2 122 x 138 v2 157

52 123 139 x 158 v2

56 124 140 159

61 125 141 160

x 62 v2 x 126 v2 142 161

64 127 143 163

65 128 144 164

66 129 145 x 165 v2

68 130 146 166

x 69 v2 131 x 147 v2 167

74 132 148 169

75 133 149 177

x 77 v2 134 x 153 v2 179

82 135 154 182

Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version

9 71 107 172

26 72 108 178

30 73 109 185

31 91 110 188

32 100 111 190

53 102 113 55 104 114

60 105 171

of 11

http://www.cms.gov/


3.2.8 Automated Numerator Recording and Measure Calculation

3.2.8.1 Automated Numerator Recording

Automated Numerator Recording Successfully Tested

(a)(1) (a)(9) (a)(16) (b)(6)

(a)(3) (a)(11) (a)(17) (e)(1)

(a)(4) (a)(12) (b)(2) (e)(2)

(a)(5) (a)(13) (b)(3) (e)(3)

(a)(6) (a)(14) (b)(4)

(a)(7) (a)(15) (b)(5)

x Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Measure Calculation Successfully Tested

x (a)(1) x (a)(9) (a)(16) (b)(6)

x (a)(3) x (a)(11) (a)(17) x (e)(1)

x (a)(4) x (a)(12) x (b)(2) x (e)(2)

x (a)(5) x (a)(13) x (b)(3) x (e)(3)

x (a)(6) x (a)(14) x (b)(4)

x (a)(7) x (a)(15) x (b)(5)

Automated Measure Calculation was not tested

3.2.9 Attestation

Attestation Forms (as applicable) Appendix

x Safety-Enhanced Design* A

x Quality Management System** B

x Privacy and Security C

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product

3.3 Appendices

Attached below.

of 11

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Apr-2015 Test Results Summary Change History

Test Report ID Description of Change Date

2014 Edition Test Report Summary

of 11

170.314.g.3 UCD HealthAxis Group, LLC. 2014 Page 1 of 90

EHR Usability Test Report of HealthAxis EHR

This report is based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports

Product Name & Version: HealthAxis EHR Version 2.0

Date of Usability Test: Wednesday, November 12, 2014

Date of Report: November 18, 2014

Report Prepared By: Sabiha Sharma, Training & Implementation (310) 895-7768-108 [email protected]

Palak Shah, Vice President, Customer Service (813) 365-3260 [email protected]

Company: HealthAxis Group, LLC. 5415 Mariner St, Ste. 222 Tampa, Florida 33609

Prepared for: Drummond Group, Inc. 13359 North Hwy 183, Ste. B-406-238 Austin, TX 78750 (817) 294-7339




Table of Contents 1 ATTESTATION ......................................................................................................... 4

2 SUMMARY ............................................................................................................... 5

3 INTRODUCTION ...................................................................................................... 9

4 METHOD .................................................................................................................. 9

4.1 PARTICIPANTS ................................................................................................. 9

4.2 STUDY DESIGN .............................................................................................. 11

4.3 TASKS ............................................................................................................. 11

4.4 PROCEDURES ................................................................................................ 12

4.5 TEST LOCATION ............................................................................................. 13

4.6 TEST ENVIRONMENT .................................................................................... 13

4.7 TEST FORMS AND TOOLS ............................................................................ 14

4.8 PARTICIPANT INSTRUCTIONS ...................................................................... 14

4.9 USABILITY METRICS ...................................................................................... 15

4.10 DATA SCORING .............................................................................................. 16

5 RESULTS ............................................................................................................... 17

5.1 DATA ANALYSIS AND REPORTING .............................................................. 17

5.1.1 USABILIY TESTING RESULTS ................................................................ 18

5.1.2 SYSTEM USABILITY SCALE .................................................................... 19

5.2 DISCUSSION OF THE FINDINGS ................................................................... 19

5.2.1 MAJOR FINDINGS .................................................................................... 20

5.2.2 AREAS FOR IMPROVEMENT .................................................................. 20

6 APPENDICES......................................................................................................... 23

6.1 RECRUITING PARTICIPANTS ........................................................................ 24

6.2 PARTICIPANT DEMOGRAPHICS (SUMMARY) ............................................. 25

6.3 PARTICIPANT DEMOGRAPHICS (DETAILED) .............................................. 26

6.4 NON-DISCLOSURE AGREEMENT FORM...................................................... 27

6.5 PARTICIPANT CONSENT FORM ................................................................... 28

6.6 COMPENSATION RECEIPT FORM ................................................................ 29

6.7 MODERATOR’S GUIDE .................................................................................. 30

6.7.1 MODERATOR FORMS ............................................................................. 38

6.8 TEST CASES, PARTICIPANT ASSESSMENT, NOTE-TAKER LOGS ............ 42

6.9 SYSTEM USABILITY SCALE (SUS) QUESTIONNAIRE ................................. 87

6.10 QUICK-REFERENCE GUIDE .......................................................................... 88


6.10.1 DRUG-ALLERGY INTERACTION .......................................................... 88


1 ATTESTATION

HealthAxis Group attests to the use and implementation of a hybrid User Centered Design process in the ongoing development of its EHR software. Our design process adheres – in part – to processes outline in the NISTI 7741 standard.1

The following principles are followed. 1) The design is based upon an explicit understanding of users, tasks, and environments. 2) Users are involved at the end of the design process before development begins to sign off on the proposed designs. 3) Users are involved with the “release to candidate” version using our test environment once development has passed the UAT phase. 4) Our design process is iterative. 5) The designs encompass the whole user experience. 6) The design team includes multidisciplinary skills and perspectives, including project managers, business analysts, UI designers, and developers.

The details of the HealthAxis EHR implementation are described in the remainder of this document. Our non-industry standard design approach has been applied in the development and measurement of the seven (7) ambulatory EHR criteria outlined in 170.314(g)(3) Safety-enhanced design document.

HealthAxis Group, LLC. Senior Management Approval:

John Carradine Printed Name Signature

Chief Financial Officer November 19, 2014

Title Date Signed

1 (Reference: National Institute of Standards and Technology. (2010). NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records (NISTIR 7741). Gaithersburg, MD. www.nist.gov/manuscript-publication-search.cfm?pub_id=907313)

http://www.nist.gov/manuscript-publication-search.cfm?pub_id=907313


2 SUMMARY

Our usability test of the HealthAxis EHR version 2.0 – ambulatory setting –application was conducted by the HealthAxis Group Usability team on Wednesday, November 12, 2014, in Torrance, California.

Purpose: The purpose of this event was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, seven (7) healthcare professionals matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative, tasks.

This study collected performance data on the following tasks typically conducted in an EHR:

1. Medication Allergy List

2. Drug-Allergy Intervention

3. Medication List

4. Drug-Drug Intervention

5. CPOE – Radiology

6. CPOE – Laboratory

7. CPOE – Medications / ePrescribing

8. Clinical Decision Support

9. Clinical Information Reconciliation

10. DD-DA Severity Management

11. Clinical Decision Support - Setup

Participant Arrival: Each participant was greeted by the administrator and asked to verify his/her personal identification and sign the Participant Sign-in Sheet. Participants were then asked to sign the Non-disclosure Agreement and Informed Consent document before the session began (these forms are found in Appendices 6.4 and 6.5 respectively).

Orientation: Participants were escorted to the testing facility conference room and seated. Participants were asked to fill out the About Me questionnaire, which was collected before the training began. The administrator explained the purpose of the test and then introduced the participants to the EHRUT. After the introduction, participants were asked to fill out a First Impressions survey, which was collected immediately upon completion.

Participants were divided into three groups and assigned to a group leader. The administrator introduced the testing procedures to all groups and explained that participants would be completing a series of tasks (given one at a time) using the EHRUT.

Testing: During the testing, group leaders timed each test and, along with the data loggers, recorded user performance data on paper. At the end of each task, participants


filled out the Participant Task Assessment Form, which they used to rate the efficiency effectiveness, and satisfaction with the EHRUT. Neither the administrator nor group leaders gave participants assistance in how to complete any task. Each participant was given a quick-reference guide designed to assist him/her in navigating the EHR to access specific tasks to be performed.

Data Collection: Neither participant screens nor audio were recorded for subsequent analysis. The following types of data were collected for each participant:

Number of tasks successfully completed without assistance

Time to complete the tasks

Number and types of errors

Path deviations

Participant’s verbalizations

Participant’s satisfaction ratings of the system

Participant Identification: All participant data was de-identified, that is, no corresponding reference could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire and Final Impressions survey.

Compensation: Before departing, participants were compensated with $200 corporate check for their time.

Summary Performance/Data Rating: Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records 7741, were used to evaluate the usability of the EHRUT. Table 1 (found on page 7) is a summary of the performance and rating data collected on the EHRUT.

THIS SPACE LEFT BLANK INTENTIONALLY.


Table 1 Summary Performance and Data Rating

Measure

N Task

Success Path

Deviation Task Time Errors

Task Ratings [1]

Task # Mean (SD) Deviations Observed/

Optimal Mean (SD)

Deviations Observed/

Optimal Mean Mean (SD)

1. Medication Allergy List (access, modify, record) 7 5.5 1.58 79% 4.58 1.51 0.96 0 2.29 1.22

2. Drug-Allergy Interactions 7 7 0.00 100% 3.54 1.28 0.88 0 2.29 1.38

3. Medication List (access, modify, record) 7 6 1.36 86% 5.28 1.94 0.96 0 2.29 1.38

4. Drug-Drug Interactions 7 4 4.23 57% 3.67 2.10 0.81 0 2.43 1.13

5. CPOE Radiology (access, modify, record) 7 5.5 1.84 79% 5.62 2.33 1.25 0 2.57 1.51

6. CPOE Laboratory (access, modify, record) 7 2.5 2.75 36% 5.68 2.81 1.26 0 2.43 0.93

7. CPOE Medication (access, modify, record) / e-Prescribing 7 6.5 1.40 93% 4.26 1.59 0.95 0 2.21 1.63

8. Clinical Decision Support Patient Interventions 7 6.5 0.70 93% 2.05 0.23 0.84 0 2.29 1.89

9. Clinical Information Reconciliation 7 4.5 3.57 64% 3.70 1.80 0.93 0 2.13 1.51

10. DD-DA Severity Level Management [2] / [3] 2 2 0.00 100% 2.88 0.31 1.64 0 1.50 0.71

11. Clinical Decision Support Setup 3 2 2 0.00 100% 2.71 0.14 0.77 0 1.50 0.71

[1] Task rating 5= Very Difficult, 4= Difficult, 3= Neither Difficult Nor Hard, 2= Easy, 1= Very Difficult [2] To validate this task DD-DA severity levels were reset and participants repeated CPOE medication/ePrescribing task to trigger interventions only. [3] Only two participants – administrative level – participated in the test task


SUS Summary: The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be 54. Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.

Qualitative Observations: Based on the average Ease of use satisfaction rating of each task, participants found the system intuitive overall but difficult to use.

We observed at the beginning of task testing participants had difficulties focusing on completing tasks due to being unfamiliar with the system. However, as participants completed the first few tasks, their time efficiency improved, navigation became easier, and confidence in system use increased, allowing greater focus on completing remaining tasks.


3 INTRODUCTION

The EHRUT tested for this study was HealthAxis EHR, version 2.0, ambulatory. Designed to present medical information to healthcare professionals in private practices and community clinics across a variety of specialties, the EHRUT consists of patient registration and billing, medical history and current health status, vitals tracking, CPOE and electronic prescribing, visit documentation, visit summaries, referral tracking, clinical decision support, patient-specific education, CQM reporting, patient portal access to health records, and health information exchange. The usability testing attempted to represent realistic exercises and conditions for the more commonly used workflows.

The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHRUT. To this end, measures of effectiveness, efficiency, and user satisfaction such as task success, task deviation, time on task, errors, ease of use were captured during the usability testing.

4 METHOD

4.1 PARTICIPANTS

Recruiting: A total of seven (7) participants were selected to test the EHRUT. Participant occupations/roles comprised two (2) Clinicians – RNPs; three (3) Clinical Support – 2 Medical Assistants and 1 Scribe; and two (2) Administrative Personnel – 1 Receptionist and 1 Office Manager.

For testing purposes, end-user characteristics were identified and an email explaining the testing event and a request for participation was sent to potential participants based on our desired characteristics. (Section 6.1 contains these characteristics and associated correspondence.)

Recruited participants had a mix of backgrounds and demographic characteristics conforming to our recruitment criteria. Table 2, found on page 10, lists participants by characteristics, including demographics, occupation, role, years’ experience in current role, computing experience, and assistive technology needs. Participant names were replaced with participant IDs so that data could not be tied back to these individuals. Participants were recruited by members of the HealthAxis Group Implementation Team.

Orientation and Training: Participants were oriented to and trained on the EHURT according to a modified training plan. Due to the abbreviated training, they were given a quick-reference guide designed to assist them in navigating to the areas in the system were tasks were performed. (Appendix 6.10 contains a copy of the quick-reference guides given to each participant.) The administrator and group leaders was allowed to give immaterial guidance and clarification on tasks, but not instructions on use.

Compensation: Each participant was compensated with a $200 corporate check for their time.


Prior Experience with and Affiliation to EHRUT: Six (6) of seven (7) participants were former users of our Practice Management application but not the EHRUT. The seventh participant was recruited through referral and had no prior experience with EHRUT. No participants had any direct connection to the development of the EHRUT.

Table 2 Participant Demographics

#

Par

tici

pan

t ID

Gen

der

Age

Occ

up

atio

n

Ro

le

Year

s'

exp

erie

nce

in

cu

rren

t ro

le

Co

mp

uti

ng

Ski

ll Le

vel

Ass

isti

ve

Tech

no

logy

N

eed

ed?

Pri

or

exp

erie

nce

usi

ng

an E

HR

1 1 F 24 Clinical Support

Scribe 4 Adv. No Yes

2 2 F 29 Admin. Receptionist 6 Basic No Yes


MA 4 Basic No Yes

4 4 F 36 Admin. Office

Manager 8 Basic No Yes

5 5 F 48 Clinician NP 10 Basic No Yes

6 6 F 38 Clinician NP 14 Adv. No Yes


MA 12 Inter. NO Yes

Actual Participation: Seven (7) participants (matching the demographics in section 4.1) were recruited and all participated in the usability test.

Participant and Testing: After orientation and training (which lasted approximately 60 minutes), participants were divided into three groups each being led by a test observer and data collector. Participants from each group were seated in the same cubicle area to perform test tasks. The test tasks were divided into four sessions: Session 1 comprised tasks 1 – 4 with a 5-minute break; session 2 comprised tasks 5 – 7 with a 5-minute break; session 3 comprised tasks 8 and 9 with a 5-minute break; and session 4 comprised tasks 10 and 11. Note: Only the administrative participants completed session 4. The post-test session System Usability Scale questionnaire and the Final Impressions survey were completed by each participant and then collected by each group leader thereafter.

To manage this testing sample, participants were divided into three groups. Group leaders recorded his/her observations on paper during the testing process. Post-event, this data was collated and transferred to spreadsheets.

Except for the two administrative tasks (D-D/D-A severity checking and Clinical Decision Support Intervention creation), all participants were given the same tasks to complete.


4.2 STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well – effectively, efficiently, and satisfactorily – and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability but also to identify areas where improvements must be made.

During the usability test, participants interacted with one (1) EHR application. Each participant used the hardware and software system in the same location and was provided with the same instructions. The system was evaluated for effectiveness, efficiency, and satisfaction as defined by the following measures collected and analyzed for each participant:

Number of tasks successfully completed without assistance

Time to complete the tasks

Number and types of errors

Path deviations

Participant’s verbalizations (comments)

Participant’s satisfaction ratings of the system

Additional information about the various measures can be found in section 4.9 starting on page 15.

4.3 TASKS

A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including:

1. Medication Allergy List (Access, Modify, Record)

2. Drug-Allergy Intervention (Identify and apply D-A intervention)

3. Medication List (Access, Modify, Record)

4. Drug-Drug Intervention (Identify and apply D-A intervention)

5. CPOE Radiology order (access, modify, and record new item)

6. CPOE Lab order (access, modify, and record new item)

7. CPOE Medication (access, modify, and record new item)/ ePrescribing (Create prescription)

8. Clinical Decision Support patient-specific intervention

9. Clinical Information Reconciliation

10. Administrative Task: Adjust DD – DA severity alerts

11. Administrative Task: Create a CDS intervention


Tasks were selected based on their frequency of use, criticality of function, and difficulty of use.

4.4 PROCEDURES

Arrival and Consents: Upon arrival, participants were greeted; their identity was verified and matched with the participant register. Participants were given their participant ID and login information. Each participant reviewed and signed a Non-disclosure Agreement and an Informed Consent Form. A representative from the Usability Test Team witnessed each participant’s signature. (These forms are found in appendices 6.4 and 6.5 respectively, starting on page 27.)

Test Team Qualifications: To ensure that the test ran smoothly, the testing administrator, two data loggers, and two observers participated in this test. The Usability Test Team conducting the test were inexperienced usability practitioners. However, the test administrator has a Bachelor’s Degree in Finance with four (4) years’ experience as an EHR implementation specialist and business analyst. One data logger has a Bachelor’s Degree in Biochemistry and Kinesiology, as well as a Master’s Degree in Public Health with five (5) years’ experience as an ER technician and scribe and four (4) years’ experience as an EHR implementation specialist. The second data logger has a Bachelor’s Degree in Computer Science with ten (10) years’ experience in EHR implementation, training, customer support, software requirements, and meaningful use consulting.

Test Team Responsibilities: The administrator moderated the session, introduced the EHRUT, oriented participants to the application, provided participant test instructions, and administered the post-session System Usability Scale Questionnaire (SUS). The administrator and data loggers (assigned as group leaders) provided application training, provided understanding of tasks to be tested 2, administered each test, monitored task times, obtained post-task participant comments and rating data, and noted task success, path deviations, the number of and types of errors, and participant comments.

Participant Resposibilties: Each participant was instructed to perform all tasks as follows:

Complete tasks quickly, making as few errors and deviations as possible.

Complete tasks without using a “think aloud” technique.

Complete tasks without assistance.

Exception: The administrator was allowed to give immaterial guidance and clarification on tasks, but not instructions on use.

Audibly state “Done” once task was complete where applicable.

Testing Process: Before each task was tested, group leaders trained their participants on the task. Participants were asked to read the task test to be performed and then indicate when he/she was finished reading. Note: Participants received a copy of the task tests in their participant packet that was distributed at check-in. Participants were given time to ask

2 Task Test Instructions: As each test was administered, participants were asked to read each task on their own. They were allowed to ask for clarification before the testing began.


for test clarification before each test began. Task timing began once participants read the task test and asked questions to clarify understanding. Task time stopped once the participant indicated he/she had successfully completed the test. Immediately after each task, participants were asked to evaluate the difficulty level of the task using the Participant Task Assessment form. (Scoring is discussed in section 4.10, starting on page 16.)

Following the testing session, the administrator asked each participant to complete the System Usability Scale Questionnaire and the Final Impressions Survey (items found in their participant packet). Group leaders collected these items once participants completed them.

Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, post-task ratings, and post-test questionnaire were recorded on paper during the session but were collated and transferred onto a spreadsheet post-testing session.

Post-test Events: At the conclusion of the testing, participants were compensated. Participants signed a receipt indicating that they had received the promised compensation of $200 (sample receipt is found in Appendix 6.6).

Participants were thanked individually for their time and escorted out of the testing facility.

4.5 TEST LOCATION

The test facility (located at 3848 W. Carson, St, Ste 200, Torrance, CA 90503) included a waiting area with chairs and a couch. The testing area comprised cubicles from which participants would sit for the duration of the test session. To perform the test, participants each used a laptop and an off-board mouse for use as a pointing device. The administrator, participants, data loggers, and all observers where situated in the cubicle area. To ensure that the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a normal range. All of the safety instruction and evacuation procedures were valid, in place, and visible to the participants.

4.6 TEST ENVIRONMENT

The EHRUT being tested during this event would be the same EHR used in a healthcare office setting. In this instance, the testing was conducted at the Torrance facility.

The system used when interacting with the EHRUT (and providing a representative computer configuration in a typical healthcare office setting) comprised the following:

Equipment:

Laptop computer

Independent mouse (for use as a pointing device)

High-speed fiber optics internet connection 3

LAN connection

Computer System configuration:

Window 7 professional operating system

3 It is possible that some healthcare offices may configure their facilities using T1 or fractional T1 Internet connections.


15 1/2” monitor

Screen resolution set to 1366 x 768

32-bit color scheme of 32-bit

Internet Explorer version 11.0

EHRUT Application:

The HealthAxis EHR, version 2.0, ambulatory application was set up by the Torrance, California, IT technician according to the Implementation’s standard EHR system setup and configuration.

System performance (response time) was closely representative to what actual users would experience in a field implementation.

Additionally, participants were instructed not to change any of the default system settings such as browser font size of system screen resolution.

4.7 TEST FORMS AND TOOLS

During the usability test, various documents and instruments were used, including:

Sign-in Sheet

Recruiting Criteria and eMail Communication

Non-disclosure Form

Informed Consent Form

“About Me”

Initial Impressions Survey

Moderator’s Guide

Participant Assessment Form

Data Logger Task Observations

Post-test Questionnaire (SUS)

Final Impressions Survey

Incentive Receipt and Acknowledgment Form

Test Task Quick-Reference Guides

Examples of these documents can be found in the appendices starting on page 24.

Data collection was accomplished by modifying the Moderator’s Guide so as to be able to capture required data more easily. Data loggers used a similar form to capture path deviations; and test participants used a form for rating and commenting on system functionality.

4.8 PARTICIPANT INSTRUCTIONS

The following sample instructions were read aloud by the administrator to the participants (also see the full moderator’s guide in section 6.6, starting on page 29).


“Thank you for participating in this study. Your input is very important. Our session today will last about two (2) hours. During that time you will use an instance of an electronic health record. I will ask you to complete a few tasks using this system and answer some questions. You should complete the tasks as quickly as possible making as few errors as possible. Please try to complete the tasks on your own following the task instructions very closely. Please note that we are not testing you, we are testing the system. Therefore, if you have difficulty all this means is that something needs to be improved in the system. I will be here in case you need help understanding your tasks, but I am not able to instruct you on or provide help in how to use the application.

Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it, so please be honest with your opinions. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Should you feel it necessary you are able to withdraw at any time during the testing.”

System Orientation: Following the procedural instructions, participants were shown the EHR and as their first task, were given time (10 minutes) to explore the system and make comments. Once this task was complete, the administrator administered the First Impressions questionnaire. Once participants completed this questionnaire, the administrator explained the testing process. Application training was conducted by test task, that is, before each task test began, participants were trained on how to use programs associated with the task. Upon completing the training, the task testing began with the administrator providing the following instructions:

“For each task, you will read the details. When you are finished reading the task, please tell our group leader.” Once all group leaders have audible confirmation that each of you is ready to begin, I will announce begin. At that point, please perform the task and audibly say “done” once you believe you have successfully completed the task as indicated on your task sheets. I would like to request that you not talk aloud or verbalize while you are doing the tasks. Once you are done, you will be asked for your impressions about the task. Please complete the participant task assessment form, which can be found in participant packet. Once you complete the form, please give it to your group leader.”

4.9 USABILITY METRICS

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records 7741, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and satisfactorily. To this end, metrics for effectiveness, efficiency, and user satisfaction were captured during the usability testing. The goals of this usability testing on the EHRUT were to assess:

1. Effectiveness by measuring participant success rates and errors. 2. Efficiency by measuring the average task time and path deviations. 3. Satisfaction by measuring ease of use ratings.


4.10 DATA SCORING

Table 3, starting on page 16, details how tasks were scored, errors evaluated, and the timed data analyzed.

Table 3 Details of how observed data were scored

Measures Rationale and Scoring

Effectiveness –Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.

The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.

Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency.

Effectiveness –Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a “Failures.” No task times were taken for errors.

Efficiency – Task Deviations

The participant’s path through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path.

Efficiency – Task Time

Each task was timed from when the administrator said “Begin” until the participant said, “Done.” If he or she failed to say “Done,” time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis.

Satisfaction – Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-event questionnaire. After each task, participants was asked to rate “Overall, this task was… on a scale of 1 (Very Difficult) to 5 (Very Easy).”

To measure participants’ confidence in and likeability of the [EHRUT] overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. (See full System Usability Score questionnaire in section 6.8 found on page 42.)


5 RESULTS

5.1 DATA ANALYSIS AND REPORTING

The results of the usability test were calculated according to the methods specified in the Usability Metrics section 4.9 above.

Participants who failed to follow task instructions had their data excluded from the analyses. To this end, no data was excluded from the analyses nor did we have any testing irregularities or issues that affected data collection or interpretation of the results.


5.1.1 USABILIY TESTING RESULTS

The usability testing results for the EHRUT are detailed in Table 4. The results should be seen in light of the objectives and goals outlined in section 4.2 (Study Design) found on page 11. The data should yield actionable results that, if corrected, yield material, positive impact on user performance.

Table 4 Usability Testing Results

Measure

N Task

Success Path

Deviation Task Time Errors

Task Ratings [1]

Task # Mean (SD) Deviations Observed/

Optimal Mean (SD)

Deviations Observed/

Optimal Mean Mean (SD)

1. Medication Allergy List (access, modify, record) 7 5.5 1.58 79% 4.58 1.51 0.96 0 2.29 1.22

2. Drug-Allergy Interactions 7 7 0.00 100% 3.54 1.28 0.88 0 2.29 1.38

3. Medication List (access, modify, record) 7 6 1.36 86% 5.28 1.94 0.96 0 2.29 1.38

4. Drug-Drug Interactions 7 4 4.23 57% 3.67 2.10 0.81 0 2.43 1.13

5. CPOE Radiology (access, modify, record) 7 5.5 1.84 79% 5.62 2.33 1.25 0 2.57 1.51

6. CPOE Laboratory (access, modify, record) 7 2.5 2.75 36% 5.68 2.81 1.26 0 2.43 0.93

7. CPOE Medication (access, modify, record) / e-Prescribing 7 6.5 1.40 93% 4.26 1.59 0.95 0 2.21 1.63

8. Clinical Decision Support Patient Interventions 7 6.5 0.70 93% 2.05 0.23 0.84 0 2.29 1.89

9. Clinical Information Reconciliation 7 4.5 3.57 64% 3.70 1.80 0.93 0 2.13 1.51

10. DD-DA Severity Level Management [2] / [3] 2 2 0.00 100% 2.88 0.31 1.64 0 1.50 0.71

11. Clinical Decision Support Setup 3 2 2 0.00 100% 2.71 0.14 0.77 0 1.50 0.71

[1] Task rating 5= Very Difficult, 4= Difficult, 3= Neither Difficult Nor Hard, 2= Easy, 1= Very Difficult [2] To validate this task DD-DA severity levels were reset and participants repeated CPOE medication/ePrescribing task to trigger interventions only. [3] Only two participants – administrative level – participated in the test task


5.1.2 SYSTEM USABILITY SCALE

The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on performance with these tasks to be: 54. Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.

5.2 DISCUSSION OF THE FINDINGS

Effectiveness: measures whether tasks were completed on time. Based on observations made and analysis of the data one outlier skewed the mean for tasks, especially CPOE. Other areas that took longer than expected were adding medication allergies and revealing D-D/D-A interactions. Administrative setup was completed in less time than expected. No errors that would have led to failure of completion of tasks were observed. All participants were able to complete tasks assigned to them. The high task success rate and low occurrence of errors indicates the straightforwardness and intuitiveness of the system.

Efficiency: measures time on task. Optimal time was calculated by adding two (2) minutes to the expert time for the completion of each task. Out of eleven (11) tasks performed, CPOE (total of 3 tasks) and finding D-D interactions were completed outside the target completion time. With CPOE, participants were able to view and order in allocated time but the modification of orders took longer. Path deviations were relatively low, most of the users experienced very little to no deviations at all. Out of all the tasks, the highest performing areas were triggering clinical decision support and reconciling clinical information, as well as accessing patient’s medication list, medication allergy list and existing order. Most of the patient information is easily accessible for all participants. Overall, the results indicate that the EHRUT is effective in being straightforward and minimizing path deviations.

Errors: Any path deviation that led to eventual failure to complete a task was considered an error. If an error were to occur, it would have been recorded by the data loggers and included in the Areas of Improvement.

Satisfaction: In order to calculate the user satisfaction, each participant was asked to complete a Participant Task Assessment form after conclusion of each task. In this form the participants rated each task on a scale of 1 to 5, where 1 indicated that the task was perceived as very easy to complete and 5 indicated the task was perceived as very difficult to complete. The overall rating of the entire system was derived by calculating the mean of all perceived ease of use ratings across the participants for all the tested tasks. The system was rated 1.97 for the ease of use on all tasks except the administrative setup (CDS and D-D/D-A severity level setup). For the two administrative setup tasks the ease of use was rated at 1.50. Both the means were lower than expected; our target goal was 3, which indicated the system was neither easy nor difficult. Out of a total of 11 tasks, D-D interaction, Lab and radiology CPOE received higher than 1.97 rating, with the highest being 2.57 for radiology CPOE. Overall, the ease of use measured in the current study falls under the target range, qualifying the system as user-friendly.


5.2.1 MAJOR FINDINGS

The major goals of this study were to calculate the usability of the various criteria’s tested as well as to identify the weaknesses of the system and use the feedback provided by the participants in the continuity of user experience improvisations. Several actionable items have been reported based on the Final Impression of Our EHR form filled out by the participants at the end of the testing session.

Overall the system was rated intuitive and easy to use by all participants except for CPOE order modifications and D-D interactions. Although the satisfaction ratings were not drastically low, the data loggers noticed that the number of questions asked significantly increased for those tasks compared to all other tasks they had to complete. The proof can be seen in the increased target time for completion of these tasks. All recommendations made by the participants in the Final Impression of Our EHR form were taken into consideration by the administrator but unfortunately, the D-D interactions and medication orders functionality is via a third party user interface and hence, the recommendations cannot be included to improve the usability of HealthAxis EHR version 2.0.

Clinical Information Reconciliation received a lot of positive feedback on the ease of use in importing the files and merging the desired items for each record. We were expecting a lot of questions for this test case but surprisingly received no questions at all.

Even though the ease of use for Clinical Decision Support setup was reported relatively easy, the users did mention that it required too many clicks. The data loggers also noted the verbal reactions of the participants indicated difficulty in identifying what information is required to complete the CDS setup as they felt there were excessive required fields on that screen which caused confusion. Furthermore, the data loggers recorded that the patient’s medical record page had too much information which for a new user can be overwhelming. The participants did express that they appreciated the intuitive tabs in the system, which made navigation very easy even with minimum user experience. This can be seen in the low mean calculated for path deviation. The data loggers also observed that for majority of the tasks, most participants did not ask a lot of “how to” questions and were able to figure out the path with little to no help.

5.2.2 AREAS FOR IMPROVEMENT

The findings of the usability test indicate that even though there are areas of improvement, overall the system meets the usability standards based on adequate Ease of Use and Satisfaction ratings. 5 out of 7 participants reported that they would like to use this system and recommend it to others. Although some of the areas cannot be improved due to third party dependencies, all other areas directly designed by HealthAxis Group will undergo design improvements in future iterations based on current feedbacks. In addition to the data collected during the current study, usability improvement suggestions provided by participants in the Participant Task Assessment and Final Impression of Our EHR will also be reviewed and incorporated in the future releases of the software.


Although the results of the current report reflect that the current system is usable, as well as delivers a high task completion rate with minimal risk associated with end user error, the areas of improvement listed below will be considered for future enhancements to increase time efficiency and usability:

CPOE: Modifying existing lab, radiology and medication orders were found to be less intuitive according to the Participant Assessment Forms and from data loggers’ observations. Participants were able to create new orders and view them easily but somehow got lost when they had to modify the existing order. Longer task times and increased steps in addition to the optimal path, indicates that the participants had a difficult figuring out how to complete these two tasks.

o The interactions are generated in the third party user interface screen; hence, HealthAxis cannot make any changes. Future roadmap for HealthAxis Group does include discontinuing the use of third party interface for medications CPOE and interactions but release date has not been scheduled.

o Once the lab order is created and if any modifications are required, the users are required to manually save and close the screen and re-open the lab order screen to make any changes. One of the highest priorities for HealthAxis will be to eliminate this extra step, hence, eliminating extra clicks required to complete this task.

o Radiology order can only be created one at a time. Which means that if a user needs to create multiple radiology orders for a patient, they have to repeat the same steps multiple time which can cause frustrations on the users’ end. In order to increase the time efficiency and satisfaction for this task, HealthAxis will incorporate the lab order workflow into the radiology order screen to have consistency and ease of use in the CPOE module.

User feedback indicated that the D-D/D-A interactions were found to be less intuitive according to the Participant Assessment Form. Even though these tasks were completed within the target time, users expressed confusion on how to get to the screen where the interactions pop-up.

o The interactions are generated in the third party user interface screen; hence, HealthAxis cannot make any changes. Future roadmap for HealthAxis Group does include discontinuing the use of third party interface for medications CPOE and interactions but release date has not been scheduled.

Participants were able to access Clinical Decision Support Interventions easily but recommended displaying the interventions in order of urgency. We would like


to consider this enhancement request in our future iterations to help improve user experience.

Clinical Decision Support interventions setup was completed in less than the target time with a score of 1.5 in the ease of use rating. However, the data loggers noticed that the users were not able to recognize the required fields to create the interventions quickly. It was only after the user selected “save” that they got a notification that the CDS setup could not be completed due to the missing data in required fields.

o To improve the usability experience for the CDS setup, marking the required fields with red asterisk will be considered in the future iterations to increase user satisfaction and reduce errors.


6 APPENDICES

The following appendices include supplemental data for this usability test report:

1. Recruitment criteria and email template (p. 24)

2. Participant demographics (summary) (p. 25)

3. Participant demographics (detailed) (p. 26)

4. Non-Disclosure Agreement (NDA) form (p. 27)

5. Informed Consent form (p. 28)

6. Moderator’s Guide (p. 29).

7. System Usability Scale (SUS) Questionnaire (p. 42)

8. Compensation form (p. 29)

9. Quick-Reference Guides (p. 88)


6.1 RECRUITING PARTICIPANTS

End-User Characteristics: For our testing event, the following characteristics were the basis for our reaching out to potential participants:

Clinician (MD, NP, PA)

Administrative (Office Manager)

Clinical Support (MA, Scribe)

Minimum 3 years’ EHR experience

Basic computing skills

Participant Request: The following email was sent to potential participants:

“Subject line: Health Axis invites you to participate in an EHR usability test and earn $200

Hello,

My Name is Rosie Soria, and I’m helping to run a usability test for eHealthCare systems, a Division of Health Axis Group. In an effort to improve our EHR, we are looking for people who may be interested in trying out our EHR and giving us feedback after using it. You will receive a $200 corporate check to participate.

What you will be doing in a usability test?

You will be asked to do several short tasks using our EHR. You will also be asked questions about your experience and perception of our EHR.

How long is the session?

3 hours

When and where?

The usability test will be held November 12, 2014 at 5:30 pm. you will be asked to participate in person at our office locates at 3848 W. Carson St Suite 200.

Interested in participating?

Please reply to this email with your contact information

If you have any questions, please feel free to call me or email me. My contact information is below. You may also contact Sabiha Sharma at 424-271-2940 and [email protected].

Thank you for your interest,

Rosie Soria”



6.2 PARTICIPANT DEMOGRAPHICS (SUMMARY)

Table 5 summarizes the demographics of participants used in this study. See section 6.3, found on page 26, for a detailed participant demographic breakdown.

Table 5 Summary Participant Demographics

Gender

Men ................................................................. 0

Women ......................................................... 7

Total (participants) .................................... 7

Occupation/Role

Physician ...................................................... 0

Physician’s Assistant .................................... 0

Registered Nurse Practitioner ...................... 2

APN (CNP, CNM, CNS, CRNA) ................... 0

Clinical Support (2 MA, 1 Scribe ) ................ 3

Administrative (1 OM, 1 Clerk) ..................... 2

Total (participants) .................................... 7

Years of Experience

Years’ experience (48 ÷ 7) ................... 6.857

Facility Use of EHR

All paper .................................................... NO

Mostly electronic ....................................YES

All electronic .............................................. NO


6.3 PARTICIPANT DEMOGRAPHICS (DETAILED)

# Participant

ID Gender Age Education Occupation Role

Years' experience in Current

Role

Computing Skill Level

Assistive Technology

Needed?

Prior Experience

using the eHCS EHR?

1 1 F 24 Bachelor's Clinical Support

Scribe 4 Adv. No No

2 2 F 29 HS Diploma Admin. Clerk 6 Basic No Yes

3 3 F 24 HS Diploma Clinical Support

MA 4 Basic No Yes

4 4 F 36 Some College Admin. Office Manager 8 Basic No Yes

5 5 F 48 Bachelor's Clinician NP 10 Basic No Yes

6 6 F 38 Bachelor's Clinician NP 14 Basic No Yes

7 7 F 31 Vocational Clinical Support

MA 12 Inter. NO YES


6.4 NON-DISCLOSURE AGREEMENT FORM

Non-Disclosure Agreement

THIS AGREEMENT is entered into as of __________________ , between ______________________________ (“the Participant”) and the testing organization HealthAxis Group, LLC., located at 3848 W. Carson St, Ste. 200, Torrance, CA, 90503.

The Participant acknowledges his or her voluntary participation in today’s usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the course of today’s study.

By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts.

Any information the Participant acquires relating to this product during this study is confidential and proprietary to Test Company and is being disclosed solely for the purposes of the Participant’s participation in today’s usability study. By signing this form the Participant acknowledges that s/he will receive monetary compensation for feedback and will not disclose this confidential information obtained today to anyone else or any other organizations.

Participant’s printed name: _______________________________________

Signature: _________________________________ Date: _________________


6.5 PARTICIPANT CONSENT FORM

Participant Consent Form

The purpose of this usability study is to evaluate the design of the HealthAxis EHR version 2.0. We are interested in determining if people can accomplish common tasks and easily find information using our EHR. The session will not ‘test' you or your ability, rather the session will test the HealthAxis EHR version 2.0 and provide information on areas that might be improved. Please be advised that there are no risks associated with participation in this session. During this session, you will be asked to complete some tasks using the EHR application and fill out a user satisfaction questionnaire. As you complete the tasks, members of HealthAxis Usability Experience Group will observe and take notes. The session will last no longer than three hours. Refreshments will be provided. I understand that participation in this usability study is voluntary, and I agree to immediately raise any concerns or areas of discomfort during the session with the Usability Experience Group. I understand and agree that the data collected from this study may be shared outside of HealthAxis Group. I understand and agree that data confidentiality is assured, because only de-identified data – i.e., identification numbers not names – will be used in analysis and reporting of the results. Date: _________ Please sign your name: __________________________________________________ Please print your name: __________________________________________________ Thank you! We appreciate your participation.


6.6 COMPENSATION RECEIPT FORM

Usability Test Compensation Receipt Form

Please sign below to indicate that you have received the promised compensation of $200 corporate check for your participation in testing today. Date: _________ Amount received: _________ Please print your name: __________________________________________________ Please sign your name: __________________________________________________ Thank you! We appreciate your participation


6.7 MODERATOR’S GUIDE

EHRUT Usability Test Moderator’s Guide

Administrator/Data Logger: Sabiha Sharma

Data Logger(s): Melissa Genove and Robin Walton

Observer(s): Palak Shah and Scott Woodcock

Date: November 12, 2014 Start Time: 06:30 PM End Time: 08:35 PM

Testing Location: 3848 W. Carson St, Ste. 200, Torrance, CA 90503

A. Review Checklist

1. Pre-Testing Final Preparations:

a. Confirm schedule with Participants (contact participants day before test to remind them of the test).

b. Ensure EHRUT test environment is running properly.

c. Ensure data is set up for each test case.

d. Ensure all questionnaires, forms, are copied and collated.

e. ID participant packets or forms.

f. Team Assignments: Be sure all Testing Team members know their responsibilities and are prepared to assist in this event.

i. Participant liaison:

1. Reminds participants of test.

2. Greets participants as they arrive.

3. Thanks participants as they depart.

ii. Sign-in:

1. Signs in participants

2. Checks IDs

3. Witnesses Participant Signature

iii. Check-out:

1. Signs out participant

2. Distributes incentive payment


iv. Escorts:

1. Escorts participants to the building exit and ensures safe departure

v. Moderator:

1. Administers test

2. Times Tasks

3. Debriefs participants between tasks

vi. Data Loggers:

1. Observes and records paths taken by each participant.

2. Evaluates and records completion, noting how a participant completes a task with/without help / correctly / or not at all.

3. Records participant comments

vii. Observer:

1. Watches participant as he/she takes test

2. Notes any difficulties and reports to data logger or administrator

viii. Data Gatherer (store for analysis event):

1. Collects data from all loggers and observers

2. Transfers data from paper to spreadsheet

ix. Testing Environment Manager:

1. Sets up testing environment, ensuring all software and hardware are ready for use

2. Takes down testing environment, uninstalling any one-time use software installed for the administration of this test

x. Data Analyzer:

1. Reviews and scores data

xi. Report Preparer:

1. Fills in usability results

2. Ensures all fields are populated.

3. Prepares draft for approval.

4. Prepares final report.

5. Submits report to EHR Project Manager.

xii. Reporter:

1. Submits report to Drummond Group


2. Check-in:

a. Greeter welcomes participants and directs them to the sign-in table.

b. During sign-in:

i. Participants sign the sign-in sheet with a witness observing the signature.

ii. Sign-in staff must match check IDs against the sign-in sheet

iii. Once confirmed, give each participant his/her login sheet and the About Me questionnaire, and assigned cubicle location.

c. Escort participants to their assigned cubicle.

d. Once all participants are signed in, orient them to restroom, break room facilities, and the evacuation sign.

e. Ask participants to fill out the document “About Me” as soon as possible. Tell them the document will be collected at the start of the session.

3. Session Start – Participant Instructions, orientation, and training:

a. Procedural instructions

b. System orientation

c. Application training

4. Test Administration:

a. Before each task:

i. Be sure participants know which task they are on.

ii. Ask participants to log in if the system has timed them out.

b. After each task, debrief participants:

i. Hand out the after-task questionnaire.

1. Score level of difficulty.

2. Write down comments.

ii. Collect questionnaires before starting the next task.

c. After Last Task:

i. Be sure to follow the steps in 4.b. Then,

ii. Handout SUS questionnaires tell participants to indicate when they have completed the questionnaire.

iii. Collect questionnaires.

5. Session End – Participant Instructions:

a. Thank participants.

b. Direct participants to go to the check-out table.


6. Check-out:

a. Participants sign acknowledgment form.

b. Distribute incentive payments.

c. Staff escorts each participant out of the testing facility.

7. Post-Event Wrap-up:

a. Collate all notes.

b. Uninstall any third-party software used in the test session administration or data collection.

B. Conduct Introduction

The moderator stated the following to participants:

“Thank you for participating in this study. Our session today will last approximately two (2) hours. During that time you will take a look at an electronic health record system.

I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system itself? Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. We will appreciate your honest and open with your opinions.

The product you will be using today is version 2.0 pre-production version. Some of the data may not make sense as it is placeholder data.

We are not recording any audio or screenshots of our session today. All of the information that you provide will be kept confidential, and your name will not be associated with your comments at any time. You many withdrawal from this event at any time.

Do you have any questions or concerns?”

There were no questions from any participants.


C. Preliminary Questions Participant’s “About me”

Allotted Time: There was no time limit to completing this inquiry. Participants completed prior to the participant orientation and EHRUT training.

Preparation:

1. Print a copy of these questions, found on page 39, for your reference.

Begin Response

1. Ask participants to locate this document in their participant packets.

2. Instruct the participant to fill in his/her participant ID before answering any questions.

3. Collect completed survey from participants.

Group Introduction

1. Review the information to stimulate a group discussion and awareness.

D. Conduct Participant Orientation:

Preparation:

1. Print a copy of the First Impressions survey.

2. Ask participants to locate this document in their participant packet.

3. Tell participants that you are going to ask them to answer these questions at a certain point in the presentation and that they should be thinking about these questions are you orient them to the system.


5. Instruct participants that you will let them now when they are able to begin filling in their responses.

Begin Orientation:

1. Tell participants “Do not click on anything just yet.”

2. Show test participants the EHRUT.


E. Obtain First Impressions

Allotted Time: There was no time limit to finishing this survey.

Preparation:

1. Print a copy of these questions.

2. Ask participants to locate this document in their participant packets.

3. Tell participants that you are going to ask them to answer these questions at a certain point in the presentation.

4. Ask participants to scan the questions, telling them that they should be thinking about these questions as you orient them to the system.

5. Instruct participants that you will let them now when they are able to begin filling in their responses.

Begin Responses:


2. Instruct participants “Start your responses.”

3. Collect this information once completed.

Data Logger only:

There was no time limit to completing this document.

36

F. Task Testing: For each test case, prepare or perform the following tasks. See test cases, participant assessment forms, and note-taker data logging worksheets starting on page 42.

1. Pre-event Preparation: Ensure test patient has an encounter and appropriate data prepared for each task test.

2. Test-event Administration: Follow these steps for each test case.

a. Ask participants to find the named task test case document (located in the participants’ participants packets.

b. Ask participants to read task to themselves and to signal their group leaders when they have finish reading.

c. Ask participants if any clarifications are required.

d. Train participants on using the program associated with the named test case.

e. Remind participants that once they feel they have completed the task, to state out loud to their group leader that they are “done.”

3. Administer Test:

a. When participants are ready, state aloud “begin.”

b. Record Start Time, end time, and total test time for each task.

4. Test Debrief:

a. Once participants finish each task test or time is stopped, begin the named test case debrief.

i. Ask participants to locate the Participant Task Assessment survey found in their participant packet.

ii. Explain how to complete this survey.

iii. Instruct participants to begin and to tell their group leaders once they finish.

iv. Collect the assessments once all participants finish.

5. Data Collection:

a. Each group leader should use the Note-taker Data Log document to collect record observations such as difficulty of performing test, participant verbalizations, task deviations, and errors.

G. Obtain Final Impressions

1. Print Final Questions document.

2. Give one copy to each participant.

3. Instruction participants to write in their ID #s in the space provided before starting.

4. Ask participants to answer these questions without using a think aloud process.

37

H. Administer Post-Tasks Testing Questionnaire

1. Administer the SUS questionnaire (see Appendix 6.8 found on page 42).

a. Print questionnaire.

b. Give one copy to each participant after task testing is completed.

c. Instruction participants to write in their ID #s in the space provided before starting.

d. Ask participants to complete survey without using a think aloud process.

I. Conclude Test Session

1. Thank participants.

2. Distribute Incentive Payments

a. Print acknowledgement form (see Appendix 6.6 found on page 29).

b. Instruction participants to fill in the information requested.

c. Ask participant if he/she wants a copy of this receipt.

d. Distribute Incentive Payments

3. Escort participants from testing facility

38

6.7.1 MODERATOR FORMS

Because all test participants were tested on the same day and at the same time, these documents were given to each participant to complete at the beginning, during, or after the testing session portion of this usability testing event. Documents begin on the next page.

39

Preliminary Questions (About Me)

Objective: To get to know you!

Instructions:

1. Please tells us about you by answering these questions. 2. You have XX minutes to complete it. 3. Please tell the moderator when you have finished.

Participant Name: __________

1. What is your job title / appointment?

_______________________________________________________________

_______________________________________________________________

2. How long have you been working in this role?

_______________________________________________________________

_______________________________________________________________

3. What are some of your main responsibilities?

_______________________________________________________________

_______________________________________________________________

4. Tell me about your experience with electronic health records.

_______________________________________________________________

_______________________________________________________________

40

Your First Impressions of our EHR

Objective: To get your impressions of our EHR after being oriented.

Instructions:

1. Please fill out this document. 2. You have XX minutes to complete it. 3. Please tell the moderator when you have finished.

Participant ID: _________________

1. “Have you heard of eHealthCare (HealthAxis Group)?” Yes No

“If so, tell me what you know about them.” _______________________________________________________________________________

_______________________________________________________________________________

2. Just from looking at this first page, what kinds of information do you think you could get from this application? Please be specific.

____________________________________________________________________

____________________________________________________________________

3. What do you think this site is designed for? Please be specific.

____________________________________________________________________

____________________________________________________________________

4. From looking at the EHR, who manages this application?

____________________________________________________________________

____________________________________________________________________

41

Your Final Impressions of our EHR

Objective: To get your impressions of our EHR after testing.

Instructions:

1. Please fill out this document. 2. You have XX minutes to complete it. 3. Please tell the moderator when you have finished.

Participant ID: __________________

1. What was your overall impression of this system?

_________________________________________________________________

2. Which aspects of the system did you like most?

_________________________________________________________________

3. Which aspects of the system did you like least?

_________________________________________________________________

4. Were there any features that you were surprised to see?

_________________________________________________________________

5. Which features did you expect to see but did note encounter?

_________________________________________________________________

6. Compare this system to other systems you have used.

_________________________________________________________________

7. Would you recommend this system to your colleagues? Why or why not?

_________________________________________________________________

42

Participant Sign-in Sheet

Participant Instructions:

Please print your first and last name

Please show your ID to the sign-in person

Participant First Name and Last Name ID

Confirmed Witness Initials

1.

2.

3.

4.

5.

6.

7.

6.8 TEST CASES, PARTICIPANT ASSESSMENT, NOTE-TAKER LOGS

This section comprises all test cases, participant assessment forms, and note-taker data log forms used for this usability testing.

43

TEST CASE: TASK 1 – CPOE MEDICATION ALLERGY LIST One of your established patients presented a week ago with what seemed to be an acute case of

urinary tract infection. You feel that you should prescribe an antibiotic. Before you prescribe these medications, your first task is to check her list of current medication allergies. On the medication allergy list, you see that she is allergic to penicillin class drugs. You ask her to confirm either medication allergy. She denies the penicillin allergy stating that she has never had a problem with it. You strongly believe penicillin inclusion was due to a clerical error. Your second task is to update her record to indicate that the penicillin allergy is not valid.

She also states that she has taken sulfa drugs in the past and was told she had watery eyes but took Septra DS when she had her last UTI with no issues. Your third task is to sulfa to her medication allergy list.

After entering this information into the order form, you get a warning stating that there is a potential drug-allergy interaction between the trimethoprim-sulfamethoxazole and her reported sulfa allergy. You feel that the trimethoprim-sulfamethoxazole is necessary, and given that she has tolerated trimethoprim-sulfamethoxazole in the past, you determine to move forward with the prescription anyway, but warn her to call you if there are any serious side effects. Your last task is to over-ride the warning with a note stating your follow-up intention and plan.

1. Navigate to the full allergy list and verbally state whether she has a sulfa allergy.

2. Navigate to the full allergy list to modify the penicillin allergy using the information in Table 1

to reflect that is no longer valid, and then verbally state when you believe that the record

modification has been completed.

3. Navigate to the full allergy list and add the following allergy to her drug allergy list using as

much of the information in Table 2 as you can. Verbally state when you believe you have

successfully completed this task.

4. Navigate to the medication order form, and prescribe trimethoprim-sulfamethoxazole using as

much as the information from Table 3 as is required. Verbally state when you believe you have

successfully completed this step.

5. Override the drug-allergy interaction with a comment stating that the patient has tolerated

trimethoprim-sulfamethoxazole in the past, finalize the prescription order using as much

information from Table 3 as required, and then verbally state when you believe that the order

has been completed.

Table 1. Allergy Information

Drug Name/Class Penicillin class Allergy Status Invalid/Clerical Error

Table 2. Drug Allergy Information

Allergen Type Drug Allergen Sulfa Severity Mild Reactions/Comments Watery eyes Onset Adulthood Onset Date Unknown

44

PARTICIPANT TASK ASSESSMENT

TASK 1A: MEDICATION ALLERGY - VIEW AND MODIFY

Start: Patient’s Medical Record

End: Successfully Delete the Penicillin allergy

Effectiveness

Easily completed

Completed with difficulty or help (describe)

Not completed

Notes:

Efficiency

Task Time: _______(mm:ss)

Notes:

Satisfaction

“On a scale from 1 to 5, 1 being very easy and 5 being very difficult, how would you rate this task?”

Notes:

Rating: _______________

45


TASK 1B: MEDICATION ALLERGY - RECORD

Start: Allergy List

End: Successfully add a new allergy- Sulfa

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________

46

NOTE-TAKER DATA LOG

TASK 1 A & B: MEDICATION ALLERGY LIST Pathway(s)

Access Records: Find Medical Record Select Patient Location Allergy Summary. Then, Review list.

Change Record(s): Select allergy(s) modify data save record return to summary list

Record Data: Locate option to add new allergy add data as indicated save record return to summary list

Success (Circle 1)

Correct

Minor deviations Describe _______________________________________________________

Major deviations Describe _______________________________________________________

Success (Circle 1)

0 Not completed 1 Completed with difficulty or help 2 Easily completed

If completed with difficulty or help, describe: ______________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:

Note why user was or was not successful.

Wrong pathways Confused by page layout Navigation issues Terminology

Other Describe: _________________________________________________________________________________________

Record Observations. ______________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________

47

TEST CASE: TASK 2 – DRUG ALLERGY INTERACTION

1. Navigate to the medication order form, and prescribe trimethoprim-sulfamethoxazole using as much

as the information from Table 3 as is required. Verbally state when you believe you have successfully

completed this step.

2. Override the drug-allergy interaction with a comment stating that the patient has tolerated

trimethoprim-sulfamethoxazole in the past, finalize the prescription order using as much information

from Table 1C as required, and then verbally state when you believe that the order has been

completed.

Table 3. Drug Information

Drug Name Bactrim Generic Name trimethoprim-sulfamethoxazole Strength 160mg-800mg Dose 1 tablet two times daily Duration 10 days Form Tablets Route Oral Dispense Amount 20 Brand Necessary No Refills No Reason Cellulitis in hand Override Reason Has tolerated previously; will monitor and follow-up

48


TASK 2: DRUG-ALLERGY INTERACTION

Start: Medication List

End: Successfully override Sulfa allergy and prescribing Bactrim

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________


NOTE-TAKER DATA LOG

TASK 2: DRUG-ALLERGY INTERVENTION

Pathway(s)

Access Records: Find Medical Record Select Patient Location Medications Summary. Then, Review list.

Change Record(s): Select medication modify data save record return to summary list

Record Data: Locate add new medication option add data as indicated save record return to summary list

Correct

Minor deviations Describe ______________________________________________________

Major deviations Describe ______________________________________________________

Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: ___________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TEST CASE: TASK 3 – MEDICATION LIST Two months ago, when you met with her, you diagnosed her with major depression and prescribed

Prozac (20 mg). A few weeks later, she reported anxiety due to her mother passing away. She presents today with severe anxiety. Your first task is to review the medications that she is currently taking to ensure that it is complete and correct.

After locating the medication list and reviewing it, you ask her to verify if she is still taking 20 mg Prozac. However, this time she has the bottle with her and she notices that it states 40 mg. Your second task is to correct this in her record.

After making the Prozac correction, you ask her if she is taking any other medications. She reports that she is taking ibuprofen for regular aches and pain. Your third task is to enter these drugs in her medication list.

After consideration, you decide to prescribe Xanax to alleviate her anxiety. You next task is to order this drug using the EHR system.

While entering the order for Xanax in the EHR, you get an alert from the EHR system about interactions between Prozac and Xanax. You feel that Xanax is the best choice, so your final task is to override the alert with the plan to monitor her and follow-up with her in a few days.

1. Navigate to her full medication list and verbally state the name and strength of the first medication on

her medication list.

2. Navigate to a place where you can update drug information and change the Prozac dosage from 20 mg

to 40 mg, using as much information as necessary from Table 1, and then verbally state when you

believe you have successfully completed this step.

3. Navigate to a place where you can add a medication to her medication list and add ibuprofen using as

much information as necessary from Table 2. Verbally state when you believe you have successfully

completed this task.


Drug Name Prozac

Strength 40 mg

Dose 1 capsule every night Duration 30 days

Form Capsules

Route Oral

Dispense Amount 30

Brand Necessary No

Refills 2 refills

Reason Major Depression Table 2. Drug Information

Drug Name Ibuprofen

Strength 200mg

Dose 2 tablets PRN

Duration ---

Form Tablets

Route Oral

Brand Necessary No


Refills 2 refills

Reason Aches and pains



TASK 3A: MEDICATION LIST – VIEW AND MODIFY


End: Successfully change Prozac 20mg to Prozac 40 mg

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________



TASK 3B: MEDICATION LIST -ADD


End: Add Ibuprofen to patient’s medication list

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________


NOTE-TAKER DATA LOG

TASK 3 A & B: MEDICATION LIST Pathway(s)




Correct



Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: ___________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TEST CASE: TASK 4 – DRUG-DRUG INTERACTION 1. Navigate to the medication order form and prescribe Xanax using as much as the information

from Table 3 as is required. Verbally state when you believe you have successfully completed

this step.

2. Override the drug-drug interaction, noting that you plan to follow-up, and then finalize the

Xanax prescription. Verbally state when you believe you have successfully completed this task.


Generic Name Xanax

Strength 0.5 mg

Dose 1 tablets twice a day as needed

Duration 30 days

Form Tablets

Route Oral

Dispense Amount 60

Brand Necessary No

Refills No

Reason Anxiety

Override Reason Will monitor and follow-up



TASK 4- DRUG TO DRUG INTERACTION

Start: Patient’s Medication list

End: Successfully prescribe Xanax and override drug-drug interaction with Prozac

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________


NOTE-TAKER DATA LOG

TASK 4: DRUG-DRUG INTERACTIONS

Pathway(s)




Correct



Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: ___________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TEST CASE: TASK 5 – CPOE RADIOLOGY

When she arrives for her yearly exam, she also presents with a dry cough that has bothered her

for the last two weeks. She denies postnasal drip, heartburn, shortness of breath, and fever, but you

note crackles in her left lower lobe. She also has had pain in her neck, which occurred after she

engaged in lifting some heavy furniture 3 months ago. The neck pain has persisted despite taking

ibuprofen and using heating pads. Her cardiovascular and lung exam are normal. She states that she

hasn’t had ibuprofen yet today, and her current neck pain is an 8/10. Your first task is to order a X-RAY

CHEST 2-VIW PA & LAT to evaluate the cough. Your second task is to order is to order a CT C-SPINE

W/O CONTRAST to evaluate the radiculopathy.

After ordering a CT C-SPINE W/O CONTRAST she now states that she remembers the neck pain

started after a trip and fall. Your third task is to change your CT C-SPINE W/O CONTRAST to a CT C-

SPINE W/ W/O CONTRAST. Your final task is to view all of the orders.

1. Navigate to where you can order imaging tests and order a X-RAY CHEST 2-VIW PA & LAT using

as much information as possible from Table 1 and then verbally state when you believe you

have successfully completed this step.

2. Navigate to where you can order imaging tests and order a CT C-SPINE W/O CONTRAST by using

as much information as possible from Table 1. Verbally state when you believe you have

successfully completed this task.

3. Navigate to where you can order imaging tests and change a routine MRI (without contrast) of

the lumbar spine to C-SPINE W/ W/O CONTRAST by using as much information as possible from

Table 2. Verbally state when you believe you have successfully completed this task.

4. Navigate to where you view the imaging tests orders you have created. Verbally state when you

believe you have successfully completed this task.

Table 1. Imaging Orders

Test Name Chest x-ray View/Location 2-VIW PA & LAT Urgency Routine Reason Persistent cough

Test Name CT C-SPINE W/O CONTRAST View/Location Cervical Spine Urgency Routine Reason Screen for radiculopathy

Table 2. Imaging Orders

Test Name C-SPINE W/ W/O CONTRAST View/Location Cervical Spine Urgency Routine Reason Screen for radiculopathy



TASK 5A: CPOE: RADIOLOGY - ORDER


End: Successfully order one X-Ray and one MRI

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes:

Rating: _______________



TASK 5B: CPOE: RADIOLOGY- MODIFY AND VIEW

Start: Patient’s Orders

End: Successfully modified the MRI ordered and view the current orders

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes:

Rating: _______________


NOTE-TAKER DATA LOG

TASK 5 A & B: CPOE RADIOLOGY ORDER

Pathway(s)

Access Records: Find Medical Record Select Patient Go to Today’s Date of Service User Chart Tabs Locate Radiology Order Review

Record Data: Create radiology order enter data save record

Success (Circle 1)

Correct



Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: _________________________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TEST CASE: TASK 6– CPOE LAB ORDERS One week ago, she came back for a check-up. She presented with diabetes symptoms and an

elevated blood glucose level. Your task is to order a glucose serum.

After talking to her more, she reports that she has an extensive family history of diabetes. You decide that a glucose tolerance would be a better test. Your second task is to change your glucose serum test to a glucose tolerance. Your final task is to view all of the orders.

1. Navigate to a place where you can order tests, order a glucose serum by using as much

information in Table 1 as possible. Verbally state when you believe you have successfully


2. Navigate to a place where you can order tests, change your glucose serum test order to a

glucose tolerance test order by using as much information in Table 2 as possible. Verbally state

when you believe you have successfully completed this task.

3. Navigate to where you view the lab orders you have created. Verbally state when you believe

you have successfully completed this task.

Table 1. Test Orders

Test Name Glucose Serum Urgency Routine Reason Type II Diabetes

Table 2. Test Orders

Test Name Glucose Tolerance Urgency Routine Reason Type II Diabetes



TASK 6A: CPOE: LAB - ORDER


End: Successfully order Glucose Serum for the patient

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________



TASK 6B: CPOE: LAB- MODIFY AND VIEW

Start: Patient’s Lab orders

End: Change the previously ordered Glucose Serum to Glucose Tolerance and view the existing orders

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________


NOTE-TAKER DATA LOG

TASK 6: CPOE LABORATORY ORDER Pathway(s)

Access Records: Find Medical Record Select Patient Go to Today’s Date of Service User Chart Tabs Locate Lab Review lab

Record Data: Create outside laboratory order enter data save record

Success (Circle 1)

Correct



Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: _________________________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TEST CASE: TASK 7 – CPOE MEDICATION ORDER / E-PRESCRIBING

She has presented with many symptoms related to exercise-induced asthma. After reviewing the history of

her symptoms and pulmonary function test results, you are strongly convinced that asthma is the correct

diagnosis and prescribe albuterol for her. Your goal is to enter the order for albuterol into the electronic

prescribing function of your EHR system.

After speaking to her more. She tells you that she is going on a trip to Europe for 2 months, and may need

more than 2 refills. Your next task is to modify her prescription order to 4 refills.

After modifying her order, you decide that you have made the right decision prescribing Albuterol with 4

refills to treat her asthma. Your final task is to send this order to the pharmacy.

1. Navigate to the medication order form and prescribe albuterol using as much as the information from

Table 1 as is required. Verbally state when you believe you have successfully completed this task.

2. Navigate to her full medication order list. Verbally state when you believe you have successfully


3. Navigate to modify the prescription for Albuterol that you have just created. Using as much information

from Table 2 modify her prescription to reflect 4 refills, and then verbally state when you believe that

the record modification has been completed.

4. Navigate to the medication order you have created and send this prescription to the pharmacy. Verbally

state when you believe you have successfully completed this task.


Generic Name Albuterol

Strength 90 mcg

Dose 2 puffs before exercise

Form Metered-dose inhaler

Route Inhaler

Dispense Amount 1 inhaler (200 puffs)

Brand Necessary No

Refills 2 refills

Reason Exercise induced asthma


Generic Name Albuterol

Strength 90 mcg

Dose 2 puffs before exercise

Form Metered-dose inhaler

Route Inhaler

Dispense Amount 1 inhaler (200 puffs)

Brand Necessary No

Refills 4 refills

Reason Exercise induced asthma



TASK 7A: CPOE: MEDICATION ORDER - VIEW AND ORDER


End: Successfully order Albuterol for the patient

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________



TASK 7B: CPOE: MEDICATION ORDER- MODIFY

Start: Patient’s Active Orders

End: Successfully change Albuterol order for the patient before sending it

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________



TASK 7C: CPOE/ E-PRESCRIBING

Start: Patient’s Medication List

End: Successfully send the electronic prescription for the patient

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________


NOTE-TAKER DATA LOG

TASK 7 A & B & C: CPOE MEDICATION ORDER / EPRESCRIBING Pathway(s)

Access Records: Find Medical Record Select Patient Go to Today’s Date of Service Orders Tab Locate ePrescribing Create ePrescription Return to EHR

Record Data: Create In-house Medication order enter data save record

Success (Circle 1)

Correct



Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: _________________________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TEST TASK: TASK 8 – CLINICAL DECISION SUPPORT

While meeting with her, you receive several notices from the EHR system. Navigate to the patient’s

visit record for today and locate the following interventions:

1. Problem: Type II Diabetes - Routine HbA1c

a) Navigate to the EHR module or screen with clinical reminders and locate the alert to

order an HbA1c test for diabetic patients every 6 months and verbally indicate when you

believe to have found it.

2. Medication: Aspirin

a) Navigate to the system notification recommending that she take aspirin or other

antiplatelet medication due to her history of coronary artery disease and verbally

indicate when you believe to have found it.

3. Medication Allergy: Sulfa Allergy

a) From the intervention list, locate the alert warning you regarding her sulfa allergy and

the Bactrim DS you have prescribed. Verbally indicate when you believe to have found

the correct alert.

4. Demographics: Overweight

a) Navigate to the system notification alerting you regarding her BMI indicating that she is

overweight. Verbally indicate when you believe to have found the correct alert.

b) Navigate to the assessment and plan. You will see that “Overweight” has already been

added to Today’s Assessment for you. Navigate to Education materials. Print out

education materials for body weight. Verbally indicate when you believe to have found

the correct alert.

5. Lab Test with Results:

a) From the intervention list, locate the alert notifying you regarding her abnormal glucose

serum. Verbally indicate when you believe to have found the correct alert.

6. Vital Signs: Blood Pressure

a) Navigate to the system notification alerting you regarding her blood pressure that is out

of range/high. Verbally indicate when you believe to have found the correct alert.

Table 1. Problem List

Problem Type II diabetes mellitus Alert Routine HbA1c every 6 months

Table 2. Medication

Medication Aspirin


Alert Take 81mg Aspirin due to history of coronary artery disease

Table 3. Allergy

Allergy Sulfa Alert Contraindication between sulfa allergy

and Bactrim DS prescription

Table 4. Demographics

Demographic Overweight Alert Patient is overweight Education materials Body weight

Table 5. Lab Tests With Results

Lab Test Glucose Serum Alert Abnormal/High

Table 6. Vital Signs

Vital Sign Blood Pressure Alert Out of Range/High



TASK 8 A - CLINICAL DECISION SUPPORT: VIEW ALL INTERVENTIONS

Start: EMR quick Access page

End: Successfully view all the interventions for the patient

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________



TASK 8B CLINICAL DECISION SUPPORT- IDENTIFY DIAGNOSTIC & THERAPEUTIC INFO.

Start: EMR Quick Access

End: Give appropriate patient education based on patient’s diagnosis

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________


NOTE-TAKER DATA LOG

TASK 8 A & B: CLINICAL DECISION SUPPORT Pathway(s)

Access Intervention: Find Medical Record Select Patient Locate today’s visit review CDS list

Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: _________________________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TASK 9 – CLINICAL INFORMATION RECONCILIATION

Your patient with type II diabetes, which you diagnosed a little more than a year ago, is here today for a

follow-up visit to his her recent hospitalization for abdominal pain. Your first task is to determine

whether any information from the hospital visit has been sent to your office for this patient. Your

second task is to review the report you have received. Your third task is to add a problem, medication,

and a medication allergy from the data you received from the into this patient’s medical record.

1. Navigate to today’s visit for this patient. Look for record received electronically for this

particular patient. Verbally state when you believe you have successfully completed this task.

2. After finding record for this patient, view the CCD report. Verbally state when you believe you

have successfully completed this task. Use the data found in Table 1 to perform the

reconciliation. Verbally state when you believe you have successfully completed this task.

Table 1. Lab Order

Problem Abdominal Pain

Medication Norco

Medication Allergy Aspirin



TASK 9A CLINICAL INFORMATION RECONCILIATION – ADD ACTIVE PROBLEM

Start: Home Page

End: Reconcile Active Problem List

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________



TASK 9B CLINICAL INFORMATION RECONCILIATION – ADD ACTIVE MEDICATION

Start: Home Page

End: Reconcile Active Medication List

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________



TASK 9C-CLINICAL INFORMATION RECONCILIATION – ADD ACTIVE MED ALLERGY

Start: Home Page

End: Reconcile Active Allergy List

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________


NOTE-TAKER DATA LOG

TASK 9: Clinical Information Reconciliation Pathway(s)

Access Records: Go to Medical Records go to CCDA records received Location patient on the list

View CCD: Select CCD document to review

Attach CCD: Select attach option choose patient’s record fill in attachment details Save attachment

Reconcile Data: Locate reconcile option select records to add approve records perform reconciliation

Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: _________________________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TEST CASE: TASK 10 – ADJUST SEVERITY LEVEL DD –DA INTERVENTIONS

The providers at your practice are complaining that while e-prescribing they are getting way too

many drug-drug interactions that they have to override. When you check the Severity Level interaction,

you notice that it is set to “severe and contraindicated”. To lower the number of interventions, as an

administrator you decide to change the intervention level to “contraindicated.”

1. Navigate to the e-prescribing Options Menu to access the administrative configuration settings.

Change the "When checking for drug-drug interactions"

Table 1. Severity Level Drug-Drug Intervention

Drug-Drug Severity Level Contraindicated



TASK 10: DRUG-DRUG INTERACTION: ADJUST SEVERITY LEVEL

Start: Home Page

End: Change the severity level of drug-drug interaction

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes: Rating: _______________


NOTE-TAKER DATA LOG

TASK 10: DD / DA SEVERITY MANAGEMENT Pathway(s)

Access ePrescribing Program: Medical Records Manage ePrescriptions

Open Patient’s ePrescriptions: Find patient go to Rcopia Options System Preferences Change Settings Look up Medication that will trigger Drug-Drug and Drug-Allergy Alert Return to EHR once interactions are is triggered.

Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: _________________________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


TEST CASE: TASK 11 – CLINICAL DECISION SUPPORT - MODIFY

As you open her electronic record, you notice a system alert stating that all diabetic patients need

HbA1c testing every 6 months. After speaking to your clinic director, he states that he would like for all

diabetic patients to have their HbA1c test done every 3 months. As an administrator is your job to

change these settings. Your goal is to change the clinical decision support intervention for diabetic

patients to have their HbA1c done every 3 months.

1. Navigate to the administrative system configuration settings for Clinical Decision Support.

Modify this alert to state that a diabetic patient will need to have their HbA1c test done every 3

months. Verbally state when you believe you have successfully completed this task.

Table 1. Intervention Information

Problem Type II diabetes mellitus

Alert Routine HbA1c every 3 months



TASK 11: CLINICAL DECISION SUPPORT- CONFIGURATION

Start: Home Page

End: Setup a Clinical Decision Support Intervention in the Admin Setup

Effectiveness

Easily completed


Not completed

Notes:

Efficiency


Notes:

Satisfaction


Notes:

Rating: ________________


NOTE-TAKER DATA LOG

TASK 11: CDS INTERVENTION SETUP Pathway(s)

Access CDS Setup Program: Home Page Administration EMR Clinical Decision Support

Create Intervention: Click New Ellipse button Configure rule by entering all required data Save record Close Program Return to Administration EMR Menu Options

Success (Circle 1)



_________________________________________________________________________________________

_________________________________________________________________________________________

Notes/Observations:



Other Describe: ___________________________________________________________________________


_________________________________________________________________________________________

_________________________________________________________________________________________

_________________________________________________________________________________________


6.9 SYSTEM USABILITY SCALE (SUS) QUESTIONNAIRE

System Usability Scale (SUS) Questionnaire

Objective: To get your feedback on this EHR system.

Instructions:

1. Please circle the answer that best describes your opinion. 2. You have XX minutes to complete it. 3. Please tell the moderator when you have finished.

Participant ID: _____________

Questions Strongly disagree

Disagree Neither

Agree nor Disagree

Agree Strongly

agree

1. I think that I would like to use this system frequently.

1 2 3 4 5

2. I found the system unnecessarily complex.

1 2 3 4 5

3. I thought the system was easy to use.

1 2 3 4 5

4. I think that I would need the support of a technical person to be able to use this system.

1 2 3 4 5

5. I found the various functions in this system were well integrated.

1 2 3 4 5

6. I thought there was too much inconsistency in this system.

1 2 3 4 5

7. I would imagine that most people would learn to use this system very quickly.

1 2 3 4 5

8. I found the system very cumbersome to use.

1 2 3 4 5

9. I felt very confident using the system.

1 2 3 4 5

10. I needed to learn a lot of things before I could get going with this system.

1 2 3 4 5


6.10 QUICK-REFERENCE GUIDE

The following documents were given to each test participant for use as summary guide to navigating the EHRUT.

MEDICATION ALERGY LIST

Navigating to the Rcopia Medication Allergy List

1. Go to EMR Quick Access.

2. Locate your patient.

3. Select go to MR.

4. From this point, begin your Medication Allergies List test task.

Note: You will be redirected to the Rcopia ePrescribing program 5. When you are finished with your patient’s allergy list return to the EMR home page.

6.10.1 DRUG-ALLERGY INTERACTION



3. Select go to MR.

4. From this point, begin your DAI test task.

5. When you are finished, return to the EMR home page.

MEDICATION LIST

Navigating to the Rcopia Medication List



3. Select go to MR.

4. From this point, begin your Medications List test task.

Note: You will be redirected to the Rcopia ePrescribing program. 5. When you are finished with your patient’s medication list return to the EMR home page.

DRUG-DRUG INTERACTION



3. Select go to MR.

4. From this point, begin your DDI test task.



CPOE RADIOLOGY ORDER

Navigating to Radiology Orders



3. Select go to MR.

4. From this point, begin your Radiology Order test task.

5. Once your order is complete return to the EMR home page.

CPOE LAB ORDER

Navigating to Lab Orders



3. Select go to MR.

4. From this point, begin your Lab Order test task.


CPOE MEDICATION ORDER / EPRESCRIBING

Navigating to Rcopia Medication Orders



3. Select go to MR.

4. From this point, begin your Medication / ePrescribing test task.


CDS – PATIENT INTERVENTIONS

Navigating to a patient’s CDS Interventions



3. Select go to MR.

4. From this point, begin your Patient CDS Management test task.


RECONCILE DATA

Navigating to the CDS Interventions Setup Program

1. Go to Medical Records.

2. Select Care Transition Records.

3. From this point, begin your Reconcile Data test task.

4. When you are finished, return the EMR home page.


ADMINISTRATIVE – CREATING CDS INTERVENTIONS

Navigating to the Clinical Decision Support Interventions Setup Program

1. Go to Administration.

2. Select System Configuration.

3. Select EMR.

4. Locate the Clinical Decision Support program.

5. From this point, begin your Create CDS interventions test task.

6. When you are finished, return the EMR home page.

ADMINISTRATIVE - ADJUSTING D-D/D-A SEVERITY LEVELS

Navigating to the Rcopia Medication List



3. Select go to MR.

4. From this point, begin your D-D/D-A severity level adjustments.

Note: You will be redirected to the Rcopia ePrescribing program. 5. When you are finished with your patient’s medication list return to the EMR home page.

ONC HIT Certification Program Test Results Summary for 2014 … · Test Results Summary for 2014...

Documents

Transcript of ONC HIT Certification Program Test Results Summary for 2014 … · Test Results Summary for 2014...