Test Results Summary for 2014 Edition EHR Certification 15 ......Test Results Summary for 2014...
Transcript of Test Results Summary for 2014 Edition EHR Certification 15 ......Test Results Summary for 2014...
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 1
Part 1: Product and Developer Information1.1 Certified Product Information
1.2 Developer/Vendor Information
Part 2: ONC-Authorized Certification Body Information2.1 ONC-Authorized Certification Body Information
Test Type: Complete EHR
Developer/Vendor Name: EndoSoft, LLCAddress: 135 Broadway
ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification
Product Name: EndoVault®Product Version: 3.0Domain: Ambulatory
Developer/Vendor Contact: Katie Tracy
ONC-ACB Name: InfoGard Laboratories, Inc.Address: 709 Fiero Lane Suite 25
Schenectady, NY 12305Website: www.endosoft.comEmail: [email protected]: (518) 831-8057
ONC-ACB Contact: Adam Hardcastle
This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:
Adam Hardcastle EHR Certification Body ManagerONC-ACB Authorized Representative Function/Title
San Luis Obispo, CA 93401Website: www.infogard.comEmail: [email protected]: (805) 783-0810
Signature and Date2/28/2016
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 2
2.2 Gap Certification
(a)(1) (a)(19) (d)(6) (h)(1)(a)(6) (a)(20) (d)(8) (h)(2)(a)(7) (b)(5)* (d)(9) (h)(3)(a)(17) (d)(1) (f)(1)(a)(18) (d)(5) (f)(7)**
*Gap certification allowed for Inpatient setting only**Gap certification allowed for Ambulatory setting only
2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification
(a)(1) (a)(16) Inpt. only (c)(2) (f)(2) (a)(2) (a)(17) Inpt. only (c)(3) (f)(3)
The following identifies criterion or criteria certified via gap certification§170.314
No gap certification
§170.314
(d)(3) (a)(6) (b)(1) (d)(4)
(f)(6) Amb. only (a)(7) (b)(2) (d)(5)
(a)(3) (a)(18) (d)(1) (f)(4) Inpt. only
(a)(4) (a)(19) (d)(2) (f)(5) Amb. only
(a)(5) (a)(20)
(a)(10) (b)(5) (d)(8) (g)(2) (a)(11) (b)(6) Inpt. only (d)(9) Optional (g)(3)
(a)(8) (b)(3) (d)(6) (f)(7) Amb. Only
(a)(9) (b)(4) (d)(7) (g)(1)
(a)(14) (b)(9) (e)(3) Amb. only (h)(2) (a)(15) (c)(1) (f)(1) (h)(3)
(a)(12) (b)(7) (e)(1) (g)(4) (a)(13) (b)(8) (e)(2) Amb. only (h)(1)
No inherited certification
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 3
Part 3: NVLAP-Accredited Testing Laboratory Information
3.1 NVLAP-Accredited Testing Laboratory Information
3.2 Test Information
3.2.1 Additional Software Relied Upon for Certification
No additional software required
Accreditation Number: NVLAP Lab Code 100432-0Address: 709 Fiero Lane Suite 25
San Luis Obispo, CA 93401Website: www.infogard.com
Report Number: 15-3332-R-0034 V1.0Test Date(s): N/A
ATL Name: InfoGard Laboratories, Inc.
For more information on scope of accreditation, please reference http://ts.nist.gov/Standards/scopes/1004320.htm
Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative:
Milton Padilla EHR Test Body Manager ONC-ACB Authorized Representative Function/Title
Email: [email protected]: (805) 783-0810ATL Contact: Milton Padilla
MaxMD Configuration Mgr b1, b2 Transmit/receive CCDA Pop-Health c1-3 Upload QRDA files
Signature and Date
Additional Software Applicable CriteriaFunctionality provided by
Additional Software
First Data Bank a1-2, a6-7, a8, a10, Drug Database
2/28/2016
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 4
3.2.2 Test Tools
Version
No test tools required
3.2.3 Test Data
3.2.4 Standards3.2.4.1 Multiple Standards Permitted
HL7 v2 Laboratory Restults Intervace (LRI) Validation ToolHL7 v2 Syndromic Surveillance Reporting Validation ToolTransport Testing ToolDirect Certificate Discovery Tool
Edge Testing Tool
Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter ]
Test ToolCypressePrescribing Validation ToolHL7 CDA Cancer Registry Reporting Validation ToolHL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting Valdiation Tool
(a)(13)
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
§170.207(j)HL7 Version 3 Standard: Clinical Genomics; Pedigree
No alteration (customization) to the test data was necessary
The following identifies the standard(s) that has been successfully tested where more than one standard is permitted
Criterion # Standard Successfully Tested
(a)(8)(ii)(A)(2)
§170.204(b)(1)HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 5
None of the criteria and corresponding standards listed above are applicable
3.2.4.2 Newer Versions of Standards
No newer version of a minimum standard was tested
(a)(15)(i)
§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide
(b)(7)(i)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(b)(8)(i)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(a)(16)(ii) §170.210(g) Network Time Protocol Version 3 (RFC 1305)
§170. 210(g)Network Time Protocol Version 4 (RFC 5905)
(b)(2)(i)(A)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(e)(3)(ii) Annex A of the FIPS Publication 140-2
Common MU Data Set (15)
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
§170.207(b)(2)The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)
(e)(1)(i) Annex A of the FIPS Publication 140-2
(e)(1)(ii)(A)(2) §170.210(g) Network Time Protocol Version 3 (RFC 1305)
§170. 210(g)Network Time Protocol Version 4 (RFC 5905)
The following identifies the newer version of a minimum standard(s) that has been successfully tested
Newer Version Applicable Criteria
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 6
3.2.5 Optional Functionality
No optional functionality tested
(a)(4)(iii) Plot and display growth charts
(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)
Criterion # Optional Functionality Successfully Tested
(f)(3)
Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)
(f)(7) Ambulatory setting only – transmission to public health agencies – syndromic surveillance - Create Data Elements
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)
(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(e)(1) View, download and transmit data to a third party using the standard specified at §170.202(d) (Edge Protocol IG version 1.1)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 7
3.2.6 2014 Edition Certification Criteria* Successfully Tested
TP** TD*** TP** TD***
(a)(1) (c)(3) (a)(2) (d)(1) (a)(3) (d)(2)
Criteria #Version
Criteria #Version
(a)(7) (d)(6) (a)(8) (d)(7) (a)(9) (d)(8)
(a)(4) (d)(3) (a)(5) (d)(4) (a)(6) (d)(5)
(a)(13) (e)(3) Amb. only
(a)(14) (f)(1) (a)(15) (f)(2)
(a)(10) (d)(9) Optional
(a)(11) (e)(1) (a)(12) (e)(2) Amb. only
(a)(19) (a)(20) (f)(6) Optional &
Amb. only (b)(1)
(a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only
(a)(18) (f)(5) Optional & Amb. only
(b)(5) (g)(3) (b)(6) Inpt. only (g)(4) (b)(7) (h)(1)
(b)(2) (f)(7) Amb. only
(b)(3) (g)(1) (b)(4) (g)(2)
*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)**Indicates the version number for the Test Procedure (TP)***Indicates the version number for the Test Data (TD)
(b)(8) (h)(2) (b)(9) (h)(3) (c)(1) (c)(2)
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 8
3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:
CMS ID Version CMS ID Version CMS ID Version CMS ID Version2 90 136 155
22 117 137 15650 122 138 15752 123 139 15856 124 140 15961 125 141 16062 126 142 16164 127 143 16365 128 144 16466 129 145 16568 130 146 16669 131 147 16774 132 148 16975 133 149 17777 134 153 17982 135 154 182
CMS ID Version CMS ID Version CMS ID Version CMS ID Version9 71 107 172
26 72 108 17830 73 109 18531 91 110 18832 100 111 19053 102 11355 104 11460 105 171
Ambulatory Inpatient No CQMs tested*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)
Ambulatory CQMs
Inpatient CQMs
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 9
3.2.8 Automated Numerator Recording and Measure Calculation3.2.8.1 Automated Numerator Recording
Automated Numerator Recording was not tested
3.2.8.2 Automated Measure Calculation
Automated Measure Calculation was not tested
3.2.9 Attestation
(a)(3) (a)(12) (a)(19) (b)(8)(a)(4) (a)(13) (a)(20) (b)(9)
Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)
(a)(7) (a)(16) (b)(4) (e)(3)(a)(9) (a)(17) (b)(5)
(a)(5) (a)(14) (b)(2) (e)(1)(a)(6) (a)(15) (b)(3) (e)(2)
(a)(4) (a)(13) (a)(20) (b)(9)(a)(5) (a)(14) (b)(2) (e)(1)
Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)(a)(3) (a)(12) (a)(19) (b)(8)
Attestation Forms (as applicable) Appendix
(a)(6) (a)(15) (b)(3) (e)(2)(a)(7) (a)(16) (b)(4) (e)(3)
*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9)**Required for every EHR product
Safety-Enhanced Design* A Quality Management System** B Privacy and Security C
(a)(9) (a)(17) (b)(5)
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 10
An inaccurate description of the summative usability testing measures used for Effectiveness, Efficiency, and Satisfaction was provided in the "Results" section of the report. The information provided in the table of results data did not match the results as described for measures of Effectiveness, Efficiency, and Satisfaction.
Appendix A: Safety Enhanced Design
User-Centered Design (UCD) process and standard
ISO/TR 16982
SO/TR 16982:2002 provides information on human-centered usability methods which can be used for
design and evaluation. It details the advantages, disadvantages and other factors relevant to using
each usability method.
It explains the implications of the stage of the life cycle and the individual project characteristics for
the selection of usability methods and provides examples of usability methods in context.
https://www.iso.org/obp/ui/#iso:std:31176:en
ISO 9241-210 (2010)
ISO 9241-210:2010 provides requirements and recommendations for human-centered design principles and activities throughout the life cycle of computer-based interactive systems. It is intended to be used by those managing design processes, and is concerned with ways in which both hardware and software components of interactive systems can enhance human–system interaction.
https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-1:v1:en
• §170.314(a)(1) Computerized provider order entry
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(a)(2) Drug-drug, drug-allergy interaction checks
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(a)(6) Medication list
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(a)(7) Medication allergy list
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(a)(8) Clinical decision support
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(a)(16) Inpatient setting only –electronic medication administration record
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(a)(18) Optional – computerized provider order entry –medications
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(a)(19) Optional – computerized provider order entry –laboratory
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(a)(20) Optional – computerized provider order entry –diagnostic imaging
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(b)(3) Electronic prescribing
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
• §170.314(b)(4) Clinical information reconciliation
- Standard used – ISO/TR 16982
- Standard used – ISO 9241-210 (2010)
Page 1 of 11
EHR Usability Test Report of EndoVault
EHR Version 3.0
EndoVault EHR Version 3.0
Standard Used: ISO/IEC 25062: Common Industry Format for Usability Test Reports
Date of Usability Test: Usability Tests Conducted in December 2014- January 2015
Date of Report: December 12th 2014 to January 2nd 2015
Report Prepared By:
Abhishek Bajaj
Clinical Director
518-831-8064
135 Broadway, Schenectady, NY 12305
Page 2 of 11
EXECUTIVE SUMMARY
A usability test of EndoVault EHR, version 3.0, Inpatient, Outpatient and Ambulatory
was conducted on December 12th, 2014 through January 2nd 2015 at Jewish General
Hospital. Dr. Grossman and Price, Dr. Gaston and Segal Cancer Center Clinical Staff
participated. The purpose of this test was to test and validate the usability of the current
user interface, and provide evidence of usability in the EHR Usability Test (EHRUT).
During the usability test, 12 healthcare providers and clinical and non-clinical staff
matching the target demographic criteria served as participants and used the EHRUT in
simulated, but representative tasks.
This study collected performance data on 12 tasks typically conducted in an
EHR:
Find information in Patient Summary screen
Use Pt. chart to find lab results
Review Consults
Review Procedure Notes
Drug-drug / drug-allergy checks
Medication List
Allergy List
Clinical Decision Support
Reconciliation
eRx
CPOE
eMAR
During the test, clinical and non-clinical staff members along with new
employees were requested to participate (but not required) in the Usability test.
They were instructed that they could withdraw at any time. During the Usability
testing at different sites, some participants (physicians) had experience with
EndoVault 3.0’s EHR. Others along with ancillary staff at certain sites did not
have prior experience with the EHR.
The administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) created by EndoSoft LLC using the
EHRUT guidelines. During the testing, the administrator used data logger(s)
Page 3 of 11
who recorded the participants’ performance data on paper. The administrator
did not give the participants assistance in how to complete the tasks.
NO training or help materials were provided.
Usability test scenarios for all eleven situations were designed based on real end-user needs to perform
the tasks needed to provide for the patient on a day to day basis.
The following types of data were collected for each participant:
• Number of tasks successfully completed.
• Number and types of errors
• Path deviations (Correct path, Minor deviations, Major Deviations)
• Participant’s satisfaction ratings of the system
• Ease of use. (Easily Completed, Completed with Difficulty or Help, Not
completed)
All participant data was de-identified – no correlation could be made from the identity of the
participant to the data collected. Following the conclusion of the testing, participants were
thanked for their help. No dollar amount was exchanged. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes Approach for
Improving the Usability of Electronic Health Records, were used to evaluate the usability of the
EHRUT. Following is a summary of the performance and rating data collected on the EHRUT.
4 of 11
Measure
Task
N Task Success Path Deviation Easily
Completed
Minor
Deviations
Completed
with difficulty
Mean(SD)
Non
Applicable
Task Ratings
Easy
# Mean Deviations
(Comp
w/difficulty
/Optimal/Minor
deviations)
Mean Mean Mean Mean
1.Find information in
Patient Summary
screen
12 24(12)/12
Mean=24
10 Optimal
2 Comp
w/difficulty
22(10)/12
=18.33
14(2)/12
=2.33
22(10)/12
=18.33
2.Use Pt. chart to find
lab results
12 24(12)/12
=24
11 Optimal
1 Minor deviation
23(11)/12
=21.08
13(1)/12
=1.08
24(12)/12
=24
3.Review Consults 12 24(12)/12
=24
9 Optimal
3 Comp
w/difficulty
21(9)/12
=15.75
15(3)/12
=3.75
21(9)/12
=15.75
4.Review Procedure
Notes
12 24(12)/12
=24
9 Optimal
3 Comp
w/difficulty
21(9)/12
=15.75
15(3)/12
=3.75
21(9)/12
=15.75
5.Drug-drug / drug-
allergy checks
12 24(12)/12
=24
7 Optimal
1 Minor Deviation
4 Non Applicable
16(7)/8
=14
16(7)/8
=14
16(4)/12
= 5.33
16(7)/8
=14
6.Medication List 12 24(12)/12
=24
7 Optimal
1 Minor Deviation
4 Non Applicable
16(7)/8
=14
16(7)/8
=14
16(4)/12
= 5.33
16(7)/8
=14
7.Allergy List 12 24(12)/12
=24
12 Optimal 24(12)/12
=24
24(12)/12
=24
5 of 11
8.Clinical Decision
Support
12 24(12)/12
=24
12 Optimal 24(12)/12
=24
24(12)/12
=24
9.Reconciliation 12 24(12)/12
=24
3 Optimal
2 Minor
Deviations
7 Non Applicable
10(3)/5
=6
10(2)/5
=4
19(7)/12
=11.08
10(3)/5
=6
10.eRx 12 24(12)/12
=24
2 Optimal
10 Non Applicable
4(2)/2
=4
22(10)/12
=18.33
4(2)/2
=4
11.CPOE 12 24(12)/12
=24
8 Optimal
1 Minor Deviation
3 Non Applicable
20(8)/12
= 13.33
13(1)/12
= 1.08
15(3)/12
= 3.75
21(9)/12
= 15.75
12.eMAR 12 24(12)/12
=24
2 Optimal
10 Non Applicable
4(2)/2
=4
22(10)/12
=18.33
4(2)/2
=4
The results from the System Usability Scale scored the subjective satisfaction with the
system based on performance with these tasks to be: Easily completed by the majority of the
clinical and non-clinical staff.
In addition to the performance data, the following qualitative observations were made:
o Major findings
The majority of the people who participated in the Usability test found the application to be quite
straight forward and easy to use. There were occasional users that had minor deviations from the
optimal path mostly due to the lack of experience with EHR’s. Some just didn’t notice certain
obvious and non-obvious choices and had reactions such as: “Oh yeah, missed that for sure” as
reported by the administrator of the usability test. The more inexperienced usability testers had
some difficulty navigating through certain sections as they were unaware of the clinical flow of a
patient and completed certain tasks with difficulty.
o Areas for improvement
According to the feedback received certain users mentioned that there were multiple clicks
required in certain sections.
Some users found certain sections to be too complex for a couple of reasons:
o They had used a different system in the past and had a different workflow in
mind.
o They never had experience with certain sections because it was out of the scope
of their practice.
o Reduction of clicks was a major suggestion and we will be working on it for
sections indicated by the users.
6 of 11
INTRODUCTION
The EHRUT(s) tested for this study was EndoVault, Version 3.0. Designed to present medical information to
healthcare providers in Inpatient/Outpatient and Ambulatory facility types and specialties such as Oncology (Medical,
Surgical, Hematology, Gynecology and Dermatology), Breast Surgery, Gastroenterology, Pulmonology, and
Administration (placing orders such as medication renewal). The EHRUT consists of realistic scenarios and we were
asked to demonstrate the uses of application as a trained user would use in real life patient care. The usability testing
attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability
of the current user interface, and provide evidence of usability in the EHR Usability Test (EHRUT). To this end,
measures of effectiveness, efficiency and user satisfaction, such as ease of use, flexibility of the application, most used
section being the highlight and were asked to complete the tasks necessary for any facility using the EHR. Tasks such
as: Reviewing patient summary, Checking Lab results, Reviewing Procedure notes, Reviewing Consults, CPOE, Drug-
Drug/ Drug –Allergy check, Medication List, Allergy List, Clinical Decision Support, Reconciliation, eRx,eMAR etc.
were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 12 participants were tested on the EHRUT(s). Participants in the test were Physicians, Nurses,
Medical Technicians, Clerical Staff and Non-Clinical staff. Participants were recruited by facility
administrators where EndoVault is available and installed currently. Participants were not influenced or
contacted by EndoVault other than providing a Usability Testing document. There was no compensation
provided to the users in efforts to reduce biased opinions. In addition, participants had no direct
connection to the development of or organization producing the EHRUT(s). Participants were not from
the testing or supplier organization. Participants were given the opportunity to have the same orientation
and level of training as the actual end users would have received. In certain User testing cases the users
were provided no training other than given a “User ID and password for login and a test patient on the
Schedule Screen already scheduled for testing purposes.” For the test purposes, end-user characteristics
were not requested by the developing company but were released by the administrators after the testing
was complete.
ID Age Gender Education Profession Occupational Role Computer
Experience
Yrs of
Experience
Product
Experience
101 32 Female Bachelors Practice Admin Administrator 15 + yrs 8 yrs 2 years
102 21 Female Bachelors Student Office Assistant 5 + yrs 2 yrs 0 yrs
103 25 Female Bachelors Medical Tech Medical Assistant 8 + yrs 7 yrs 6 months
104 67 Male Doctorate Physician Gastroenterologist Unknown 39 yrs 6 yrs
105 35 Male Doctorate Physician Breast Surgery Unknown 10 yrs 2 yrs
106 31 Female Doctorate Physician Medical Oncologist 15+ yrs 5 yrs 2yrs
107 36 Female Doctorate Physician GYN Oncologist 20+ yrs 8 yrs 2yrs
108 27 Male Unknown Medical Tech Technician 10 + yrs 6 yrs 0 yrs
109 55 Male Doctorate Physician Gastroenterologist Unknown 25 Yrs 2 Yrs
110 40-45 Female Unknown Admin Asst. Assistant Unknown 15 + yrs 0 yrs
111 25 Male Medical
Student
Student Student 10 + yrs 0 yrs 0 yrs
112 72 Male Doctorate Physician Breast Surgeon Unknown 47 + yrs 1 yrs
Page 7 of 11
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application performed well –
that is, effectively, efficiently, and with satisfaction – and areas where the application failed to
meet the needs of the participants. The data from this test may serve as a baseline for future
tests with an updated version of the same EHR and/or comparison with other EHRs provided
the same tasks are used. In short, this testing serves as both a means to record or benchmark
current usability, but also to identify areas where improvements must be made.
During the usability test, participants interacted with EndoVault 3.0 EHR. Each participant
used the EndoVault 3.0 under the supervision of the site administrator. The Usability test was
conducted in Medical University Teaching Hospitals, ASC’s, and Stand-alone physician
practices, and were all provided with the same instructions. The system was evaluated for
effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for
each participant:
Number of tasks successfully completed without assistance
Number and types of errors
Path deviations
Participants’ verbalizations (comments)
Participants’ satisfaction ratings of the system
TASKS
A number of tasks were constructed that would be realistic and representative of the kinds of
activities a user might do with this EHR, including:
1. Find patient’s information in the patient summary screen
2. Find results of recent blood work
3. Check and review Consult Notes.
4. Check and review Procedure Notes
5. Drug-drug / drug-allergy checks
6. Medication List
7. Allergy List
8. Clinical Decision Support
9. Reconciliation
10. eRx
11. Place Orders from the CPOE Screen
12. eMAR
Page 8 of 11
Tasks were selected based on their frequency of use, criticality of function, and those that may be most
troublesome for users. Tasks should always be constructed in light of the study objectives.
PROCEDURES
Participants were selected at random from the site administrators where the Usability tests
were conducted. Participants were asked to volunteer to take the Usability Test therefore
consenting verbally and then the time and date were confirmed by the administrators on site.
To ensure that the test ran smoothly, the site administrator did one on one testing therefore
data logging and recording any verbalizations from the Testers. The usability testing staff
conducting the test were experienced usability practitioners with medical experience, a
practice management background and a number of years of experience in the field of
medicine.
The administrator moderated the session including administering instructions and tasks. The administrator
also monitored, obtained post-task rating data, and took notes on participant comments. They also served as
the data logger and took notes on task success, path deviations, number and type of errors, and comments.
Participants were instructed to perform the tasks as quickly as possible making as few errors and deviations
as possible. (See specific instructions below):
All participant data has been de-identified and kept confidential.
• Without assistance; administrators were allowed to give immaterial guidance and clarification on
tasks, but not instructions on use.
• Without using a think aloud technique.
For each task, the participants were given verbal commands for each of the
tasks.
Tasks began once the administrator finished reading the question.
Following the session, the administrator gave the participant the post-test
verbal questionnaire (e.g., the System Usability Scale, suggestions), and
thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors,
deviations, verbal responses, and post-test questionnaire were recorded into the
testing script for the participants.
Page 9 of 11
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table,
computer for the participant, and recording computer for the administrator.
Only the participant and administrator were in the test room. All observers and
the data logger worked from a separate room where they could see the
participant’s screen and face, and listen to the audio of the session. To ensure
that the environment was comfortable for users, noise levels were kept to a
minimum with the ambient temperature within a normal range. All of the
safety instruction and evacuation procedures were valid, in place, and visible to
the participants.
TEST ENVIRONMENT
The EHRUT would typically be used in a healthcare office or facility.
In this instance, the testing was conducted in Medical University Teaching Hospitals, ASC’s,
and Stand-alone physician practices. For testing, participants used PC’s, Tablets (Surface Pros
or IPad’s) and laptops depending on the facility running Windows XP, Windows 7, and
Windows 8.
The participants used a mouse and keyboard, touch screen pens or finger touch
when interacting with the EHRUT.
The EHRUT used various screens and resolutions based on the availability of the facility.
The application was set up by the vendor in advance according to the vendor’s
documentation describing the system set-up and preparation. The application itself was
running on a windows platform using a test database on a LAN / WAN connection.
Technically, the system performance (i.e., response time) was representative to what actual
users would experience in a field implementation. Additionally, participants were instructed
not to change any of the default system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used, including:
1. Moderator’s Guide (testing script)
2. Post-test Satisfaction Verbal Questions by the administrator
Page 10 of 11
The participant’s interaction with the EHRUT was captured and recorded digitally with screen
capture software running on the test machine. Reaction and verbal comments were recorded
by administrators and conveyed to the developing company EndoSoft.
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant:
Thank you for participating in this study. Your input is very important. Our session today will
last about 15-20 minutes. During that time you will use an instance of an electronic health
record. I will ask you to complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few errors as possible. Please
try to complete the tasks on your own following the instructions very closely. Please note that
we are not testing you we are testing the system, therefore if you have difficulty all this means
is that something needs to be improved in the system. I will be here in case you need specific
help, but I am not able to instruct you or provide help in how to use the application.
Overall, we are interested in how easy (or how difficult) this system is to use, what in it would
be useful to you, and how we could improve it. I did not have any involvement in its creation,
so please be honest with your opinions.
Following the procedural instructions, participants were shown the EHR and as their first task, were given time- 5
minutes to explore the system and make comments.
Participants were then given 12 tasks to complete as listed above. An analysis following the
results was performed.
DATA SCORING
RESULTS
DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the
Usability Metrics section above. NONE of the participants who failed to follow session and task
instructions were excluded from the analyses. No Exclusions to report. There were no
irregularities during testing process or interpretation of the data provided by testers.
DISCUSSION OF THE FINDINGS
EFFECTIVENESS
In light of the testing performed, the application seemed to be quite effective in providing
the medical practices what they need and what the medical professionals require.
Page 11 of 11
EFFICIENCY
According to the data collected during the EHRUT it seems that the application was quite
efficient overall, although some users had minor deviations from the optimal path, most
testers navigated through the application with ease.
SATISFACTION
Based on the data collected the satisfaction rate was a range from 75-100% in each category.
Overall the satisfaction with the EHR seemed to be high.
MAJOR FINDINGS
Overall based on the findings provided by the administrators the application seemed to be
easy to use and simple for the users and non-clinical users. The majority of them were able
to navigate quite efficiently. There were some users who were not so tech savvy and had
difficulties navigating.
AREAS FOR IMPROVEMENT
Common suggestions from a few of the testers were to reduce the number of clicks
navigating to specific sections.
Usability Test Reports (Additional)
EndoVault EHR 3.0
Descriptions of the steps for each measures:
§ 170.314(a) (1) (CPOE);
Step 1: Find patient by using search routine.
Step 2: Find Item on patient summary Screen
Step 3: Go to order tab
Step 4: using CPOE routine to add orders.
Assistive Technology used: None
Optimal Time: 30 seconds
Average Actual Time: 25 seconds
§ 170.314(a)(2) (Drug- drug, drug-allergy interaction checks);
Step 1: Find Patient by using search routine.
Step 2: Find Item on patient summary screen
Step 3: Add Drug to the existing chart of the Patient
Step 4: find Drug-drug, drug-allergy interaction
Assistive Technology used: None
Optimal Time: 35 seconds
Average Actual Time: 23 seconds
§ 170.314(a)(6) (Medication list);
Step 1: Find patient by using search routine.
Step 2: Find Item on patient summary screen
Step 3: Click on Medication list Tab
Assistive Technology used: None
Optimal Time: 25 seconds
Average Actual Time: 22 seconds
§ 170.314(a)(7) (Medication allergy list);
Step 1: Find Patient by using search routine.
Step 2: Find Item on patient summary screen
Step 3: Click on Medication allergy Tab
Assistive Technology used: None
Optimal Time: 25 seconds
Average Actual Time: 22 seconds
§ 170.314(a)(8) (Clinical decision support);
Step 1: Find Patient by using search routine.
Step 2: Go To patient chart
Step 3: Clinical decision support will be triggered.
Step 4: Cancel or take action.
Assistive Technology used: None
Optimal Time: 40 seconds
Average Actual Time: 35 seconds
§ 170.314(a)(16) (Electronic medication administration record);
Step 1: Find patient by using search routine.
Step 2: Go To Medication list tab
Step 3: use scan routine to search meds from order list.
Step 4: add meds to the list.
Assistive Technology used: Bar code Scanner
Optimal Time: 55 seconds
Average Actual Time: 43 seconds
§ 170.314(b)(3) (Electronic prescribing);
Step 1: Find patient by using search routine.
Step 2: Find Item on patient summary screen
Step 3: go to prescription tab
Step 4: add meds to the prescription list.
Step 5: click send button to send out eprescription.
Assistive Technology used: None
Optimal Time: 45 seconds
Average Actual Time: 38 seconds
§ 170.314(b)(4) (Clinical information reconciliation).
Step 1: Find patient by using search routine.
Step 2: Go to patient chart
Step 3: click on clinical reconciliation button
Step 4: take action.
Assistive Technology used: None
Optimal Time: 45 seconds
Average Actual Time: 32 seconds
TEST ENVIRONMENT
Facility: Jewish General Hospital
Computers: Desktop/Laptop/Tablet with Windows 7 Professional/Windows 8 with 4GB
of RAM
Screen sizes: 19 inches
Screen resolutions: 1920/1200
Color settings: true color (32 bit)
Facility: Dr. Grossman and Price
Computers: Desktop/Laptop with Windows 7 Professional with 4GB of RAM
Screen sizes: 21 inches
Screen resolutions: 1600/1200
Color settings: 256 color
Facility: Dr. Gaston
Computers: Desktop/Laptop with Windows 7 Professional/Windows XP with 4GB of
RAM
Screen sizes: 19 inches
Screen resolutions: 1600/1200
Color settings: true color (32 bit)
USABILITY METRICS
The goals of the test were to assess:
1. Effectiveness of EHRUT by measuring participant
success rates and errors
2. Efficiency of EHRUT by measuring the average
task time and path deviations
3. Satisfaction with EHRUT by measuring ease of
use ratings
DATA SCORING
The following table details how tasks were scored, errors
evaluated, and the time data analyzed.
Measures Rationale and Scoring
Effectiveness:
Task Success
A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.
The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency.
Optimal task performance time, as benchmarked by expert
performance under realistic conditions, is recorded.
Effectiveness:
Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a “Failures.” No task times were taken for errors.
The total number of errors was calculated for each task and
then divided by the total number of times that task was
attempted.
Efficiency:
Task
Deviations
The participant’s path (i.e., steps) through the application was
recorded. Deviations occur if the participant, for example, went
to a wrong screen, clicked on an incorrect menu item, followed
an incorrect link, or interacted incorrectly with an on-screen
control. This path was compared to the optimal path. The
number of steps in the observed path is divided by the number
of optimal steps to provide a ratio of path deviation.
Efficiency:
Task Time
Each task was timed from when the administrator said “Begin”
until the participant said, “Done.” If he or she failed to say
“Done,” the time was stopped when the participant stopped
performing the task. Only task times for tasks that were
successfully completed were included in the average task time
analysis. Average time per task was calculated for each task.
Variance measures (standard deviation and standard error)
were also calculated.
Satisfaction:
Task Rating
Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire.
After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants.
Common convention is that average ratings for systems judged easy to use should be 3.3 or above.
To measure participants’ confidence in and likeability of the
EHRUT overall, the testing team administered the System
Usability Scale (SUS) post-test questionnaire. Questions
included, “I think I would like to use this system frequently,” “I
thought the system was easy to use,” and “I would imagine that
most people would learn to use this system very quickly.
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 11
Appendix B: Quality Management System
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 12
Appendix C: Privacy and Security
Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 13
Test Results Summary Document History Version
V1.0V1.1 Updated Safety-Enhanced Design report February 28, 2016
Initial release July 6, 2015
END OF DOCUMENT
Description of Change Date