LRI Validation Suite

LRI Prototype Tool DemonstrationRob Snelick—NISTJanuary 31st, 2011

Overview• Validation Tool for LRI (Prototype for trial use)

– http://lri.sipilotdevelopment.org/lri-phase-2/• Web Application

– Firefox; Chrome; IE9 (in compatibility view enabled)• Source code will be public domain• Joint effort of NIST and LMCO (ONC)• Wiki Page—Validation Suite Resources• Google Group

– Discussion forum for LRI IG, Pilots, Validation Tool, Tool and Resources Updates, Release Notes, etc

– http://groups.google.com/group/hl7v2_lri_testing

Timeline• Demonstration of “proof of concept” validation tool

– Tuesday December 20th, 2011– Not for use (review in validation suite meetings)– Based on Ballot 1+ IG version

• Prototype for trial use– Goal is early February (~2 weeks before HIMSS)—Week of Feb. 13th– Based on IG version 0.15 (locked for Pilots)– Not all functionality and IG requirements included

• On-going updates– development will continue with periodic releases– Will include errata to IG Version 0.15

• Version 1 (“Production Ready”)– 3 months after publication of IG (after ballot 1—i.e., stable IG)**– Not complete number of test cases (5-7 core messages with variation)

• Version for MU testing—if in ONC rule??– Fall 2012 (Dependent on final publication of IG)**– 3 Months after final publication

**assuming current level of resources

LRI Validation Suite Meetings• Agenda

– Tool updates and status– User Questions/Comments– Test case/message reviews (Users can request test cases to add)– Other topics as needed

• Current Schedule– February 28th

– March 27th

– April 24th

– May 22nd

• Tuesdays, 1:00 PM EST• Additional meetings to be added if necessary• HL7V2_LRI_Testing Google Group will be used for notification

Validation Suite Products (Drafts)• Test Case Management Spreadsheet• Test Messages• Test Plans

– LIS Test Plan, EHR Test Plan• Testing Methodologies (approach to testing)• Vocabulary Spreadsheets (value sets for the IG), XML, and GUI

– Including HL7, UCUM, in-scope LOINC, in-scope SNOMED to come– Categorized in-scope LOINC to support testing

• XML Conformance Profile– Message Template capturing most message constraints (requirements)– LRI IG MWBHL7 V2.5.1 XML + CP + CS + Len HL7 V2.7.1 LRI IG XML

• Validation Tool (Focus on LRI—i.e., not a general HL7 V2 tool)– LIS Mode

• Context-Free (test any LRI message)• Context-Based (Generated Data Sheets from Test Cases – Scenarios)

– Associated with test cases (i.e., test data)

– EHR Mode• Generated Data Sheets and Juror Documents from Test Cases – Scenarios

– Support for NG and GU Profiles (NG initially)• Provide feedback to implementation guide analysis group

LRI Validation Suite WG Charter OverviewAmbulatory EHR

ONC S&I Framework Test Data

LIS LRI Validation

Tool

EHR LRITest

Harness

NIST LRI Test Tool

NIST LRI Test Tool1. Suite scope limited to the requirements specified in the LRI IG2. Develop test data

1. provided & verified by the S&I Framework community3. LIS LRI Validation Tool

1. Used to validate vendor LIS systems & Test Harness2. Develop a test plan3. Capture and validate LIS LRI messages

4. EHR LRI Test Harness1. Used to validate EHR systems2. Develop a test plan3. Simulation of an LIS system4. Manage, Send, Receive LRI Messages5. Inherently provides CLIA Testing6. Targets

1. S & I Framework EHR Pilots2. Vendor EHR Products 3. MU Certification

LRI ORU R01

LRI ACK R01

S & I FrameworkEHR Pilots

Vendor EHR Products

Ambulatory EHR Systems1. EHR Pilots

1. Draft standard trial use (implementation)2. Verifies LRI IG can be implemented—provide feedback3. Use LRI test harness to perform conformance testing4. Function of the S&I Framework LRI Pilots WG

2. Vendor Product EHRs1. Can use LRI test harness to perform conformance

testing2. LRI Test Harness expected to be used for MU

certification

Part of validation suite deliverables Anticipated users of validation suite products

LRI Validation – High Level Overview• LIS Mode

– Context-Free– Context-Based (Generated Data Sheets from Test

Cases – Scenarios)• EHR Mode

– Generated Data Sheets and Juror Documents from Test Cases – Scenarios

• Conformance Profile Browser• Vocabulary Browser

LIS (or Proxy)

LIS Test Tool

Load Cut/paste Send

HL7 V2 Lab Results Message

ValidationReport

Context Free Testing:1. Context free testing will validate a LRI message

created by the LIS2. The context (e.g., the type and results of the

lab test) is unknown to the validation tool3. Therefore not all conformance requirements

of the LRI implementation guide can be assessed

4. However, the validation provides a simple and convenient method for testing message structure and most vocabulary

Process:1. An LRI message is created by the vendor’s LIS2. The message is sent, pasted, or loaded in the

test tool3. A validation is performed

LIS Context-Free Testing

Phase 1 Core Test Messages• Erythrocyte sedimentation rate

– 30341-2 OBR.4 LN code– Valid Minimally Populated Message – Final Results– Valid Typically Populated Message – Final Results– Valid Typically Populated Message – Corrected Results– Valid Maximally Populated Message – Final Results

• CBC W Auto Differential panel in Blood– 57021-8 OBR.4 LN code– Valid Typically Populated Message – Final Results– 27 Results (OBXs)

• Lipid 1996 panel in Serum or Plasma – 24331-1 OBR.4 LN code– Valid Typically Populated Message – Final Results– 4 Results (OBXs)

LIS

LIS Test Tool select test case

Test Data Sheet

Manual entry of test data

Load Cut/paste Send

HL7 V2 Lab Results Message

Use Case

Test Case

ValidationReport

EHR transmits Hemoglobin blood test order to the LIS for John Doe along with pertinent demographic and order request data

Lab Results Data SheetJohn DoeDOB: 05/23/1959Gender: MResult: 13.7 g/dL Range 13.2 – 16.2Status: Finaland more…

• A lab test is ordered for a patient• The specimen is collected, and is received and processed in the lab• The lab result is produced and stored in the LIS database• The lab result message is created•The lab result is transmitted to an ambulatory EHR•The lab result is viewed in the ambulatory EHR

Process:1. A technician enters lab results data into LIS

based off the data sheet provided2. The message is sent, pasted, or loaded in the

test tool3. A validation is performed

Context-based Testing:1. Context-based testing will validate a LRI message

created by the LIS2. The context (e.g., the type and results of the lab test)

is known to the validation tool3. Therefore all conformance requirements of the LRI

implementation guide can be assessed4. Supports various scenarios (FC, rejected specimen)

LIS Context-based Testing

EHRLab Message

ACK

Validation

LRI EHR Test HarnessInspection Testing can be performed by:

1. On-site inspection2. Over a webex like technology3. Screen-scraper or screen-capture (include clock)4. Printed Reports

Inspection Testing Techniques1. EHR display screens2. Database access3. Configuration files

Communication

Automated TestingAcknowledgement MessageLimited Utility

LRI Test Message

Validation Report

Test Data

Test Case

Use Case

JurorDocument

ACK

LRI EHR Testing – Test Harness

Conformance Profile• Machine processable version of the LRI implementation guide• HL7 V2.7.1 Standard Conformance Profile

– XML– Chapter 2B– Captures most of the LRI IG requirements

• Message Structure, usage, cardinality, condition predicates, conformance statements, lengths, etc.

– Key differences from V2.5.1 (and not handled by the MWB)• Min/max length• C(a/b) – configurable conditionals• Conformance Statements

– NIST added elements for handling condition predicates and conformance statements (not defined in 2.7.1—NIST will propose for V2.9)

• Support for min/max shortly

• LRI IG MWBHL7 V2.5.1 XML + CP + CS + Length HL7 V2.7.1 LRI IG XML +

Test Case Management Spreadsheet Processor

Message Generation

Engine

Tab Delimited Text File

LRI Test Message

Generation Context File

(XML)

Test Data

Test Cases

Use Case

JurorDocument

(XML)

Test Case Management Spreadsheet

Validation Context File

(XML)

EHR Data Sheet (XML)

LIS Data Sheet (XML)

Message Validation

Engine

Validation Report

LIS Test Plan and LRI Test

Tool

Message Profile (XML)

EHR Test Plan

EHR Test Plan and LRI Test

Tool

LRI Test Tool

Message Model

JurorDocument

(HTML Tables)

Modifies Message Model

Validation Context File

(XML)

Regenerate

Test Case Filter

LRI Test MessageUser

Configuration

Test Case Management Spreadsheet Processing – Priming the Validation Tool

LIS

LIS Test Tool select test case

Test Data Sheet

Manual entry of test data

Load Cut/paste Send

HL7 V2 Lab Results Message

Use Case

Test Case

ValidationReport

EHR transmits Hemoglobin blood test order to the LIS for John Doe along with pertinent demographic and order request data

Lab Results Data SheetJohn DoeDOB: 05/23/1959Gender: MResult: 13.7 g/dL Range 13.2 – 16.2Status: Final

• A lab test is ordered for a patient• The specimen is collected, and is received and processed in the lab• The lab result is produced and stored in the LIS database•The lab result is transmitted to an ambulatory EHR•The lab result is viewed in the ambulatory EHR

1. Local code2. No code—text3. LOINC code

No LOINC given for results

• Test tool inspects OBX.3 for a specific LOINC code• Inspect for 718-7 only or inspect for one of 718-7, 20509-6, 30313-1, 30352-9, 30351-1, and 30350-3?

For Discussion:1. No suggested method given (Should we?)2. With no method given we should expect one

from a set of valid LOINC codes for this order (Lab dependent?)

3. If a specific method is given (is this typical/possible?) then should we look for a specific LOINC code in the sent message?

For Discussion:1. Does the LAB expect a local code for the order in

a electronic order request?2. Use Cases to consider: agreement/no agreement

between LIS and EHR for local codes

It is assumed Lab has capabilities for manual entry

Dat

a E

ntry DB

Message

EHRLab Message

ACK

Validation

LRI EHR Test Harness

CommunicationLRI Test

Message

Validation Report

Test Data

Test Case

Use Case

JurorDocument

ACK

718-7 (LN)

718-7 (LN)

718-7 (LN)S Name Value REF Range

F Hemoglobin 13.7 13.2 – 16.2 g/dL

For Discussion:1. What are the requirements on the EHR with regards to LOINC—

given the LRI IG, CLIA, and meaningful use? 2. What are the EHR display (GUI) requirements for the received lab

results?3. What are the EHR storage requirements for the received lab

results?4. Given that the LOINC code is likely to be translated into the EHR

internal representation, what lab test names need to be displayed in exact form and what lab test names can be displayed in an equivalence text (same concept)?

5. What qualifies as “equivalence—same concept”? How is that determined? Can it be? Expert inspector only? Should displaying of the LOINC text (only) be the requirement—i.e., no mapping?

6. What is the impact of sending only the LOINC code (i.e., no local code) on testing?

Translated into internal

representation

Displayed as internal display text representation