STATEWIDE HIO - Weeblynataliesschwartzmd-eportfolio.weebly.com/uploads/8/8/4/7/8847566… ·...
Transcript of STATEWIDE HIO - Weeblynataliesschwartzmd-eportfolio.weebly.com/uploads/8/8/4/7/8847566… ·...
STATEWIDE HIO Vendor Selection Yolande Calhoun, Chris Davis, Kyle Johnson, Joanne Rimac & Natalie Schwartz 2/22/2011
Table of Contents
1. Introduction: ..................................................................................................................... 3
2. Project Objectives .............................................................................................................. 4
3. Political Environment ......................................................................................................... 4
4. Key Design Decisions and Assumptions .............................................................................. 6
5. Selection Criteria .............................................................................................................. 9
Methodology and Reasoning behind Selection Criteria- HIE Organizational landscape is too new,
No proven sustainable models ......................................................................................................................................... 9
Selection Criteria ................................................................................................................................................................... 12
6. Analysis of Vendor Responses .......................................................................................... 13
Customer Testimonials: ..................................................................................................................................................... 14
7. Scores ............................................................................................................................. 16
8. Cost Proposals ................................................................................................................. 16
9. Acceptance Testing ......................................................................................................... 18
Describe the role of acceptance testing .................................................................................................................... 19
The role of acceptance testing play in the decision ............................................................................................ 20
Acceptance testing performed on the product ...................................................................................................... 21
Subcommittee List of Objectives for the testing process ..................................................................................................... 21
Steering Committee List of Objectives for the testing process .......................................................................................... 21
When will acceptance testing be performed? ........................................................................................................ 22
10. Change Management ..................................................................................................... 22
Statewide HIE Change Request Form ........................................................................................................................ 24
11. Discussion ...................................................................................................................... 25
Attachments ....................................................................................................................... 26
References .......................................................................................................................... 26
3
1. Introduction:
Statewide HIO has been engaged in the process of collecting responses to our Request for Proposals
(RFPs) to assist in the vendor selection process to provide HIE services to state X. After approval by the
Statewide HIO Board of Directors, the RFP was released for responses on February 11, 2011. After the
RFP due date, the Statewide HIO RFP Selection Committee carefully reviewed and compared each
vendors responses. We respectfully submit this document to the Board of Directors with our final
vendor recommendation for approval.
Members of the HIO RFP Selection Committee include:
Evaluator HIO Subcommittee Membership Title/Organization
1.Thomas Brady Clinical Advisory M.D., State X University Hospital
2.Maria D’ Angelo Privacy & Security Privacy Officer, St. Vincent Medical Center
3.Jeffrey Fox Technical CIO, Southern General Hospital
4.Rajendra Kothari Technical Systems Analyst, Federated Bank
5.Carol Manning Clinical Advisory Clinical Nurse Specialist, County General
6.James Prince Clinical Advisory M.D., Erie Community Hospital
7.Sara Rowe Clinical Advisory M.D., State Department of Health
8.Randall Smith Finance J.D., C.P.A., The Smith Group
HIO Steering Committee Members Title
Yolande Calhoun Management Analyst
Chris Davis M.D.
Kyle Johnson CIO
Joanne Rimac R.N.
Natalie Schwartz M.D.
2. Project Objectives
The Statewide HIO Selection Committee considered the following core objectives in evaluating each
vendor’s RFP responses:
Exchange of clinical information in a standard format, such as Continuity of Care Document (CCD).
Exchange of clinical messaging and referrals between participants of the HIE.
The ability to connect to the following entities through virtual or HL7 gateways: o Hospitals o Physician offices o State Public Health agencies (for immunization, syndromic surveillance and
reportable lab registries) o ePrescribing entities o Payers
Provide security of clinical information in a HIPAA compliant fashion.
Conform to national standard vocabularies and code sets being established by the HIT Standards Committee Vocabulary Task Force, the HIT Standards Implementation Workgroup, and the ONC Standards and Interoperability Framework Contractors.
Provide technology which delivers high availability of the HIE services.
Provide the ability to connect to HIEs in other regions, states, and with a National Health Information Network (NHIN).
Support the information technology needs of the HIO (internet, email, web services).
Additional objectives needed to support Phase 2 and 3 of Meaningful Use which were considered in the
RFP process include:
Provide an electronic consent repository.
Provide directory services for providers, payers, and public health entities.
Provide Public Key Infrastructure (PKI) to allow for more sophisticated secondary authentication.
Provide the ability to electronically submit quality reporting to state and federal agencies.
3. Political Environment
Since the implementation of the SOW which includes this RFP, the following changes have occurred:
Acquisition of HIE Vendors by Payers
5
Prior to the release of this RFP, Axolotl and Medicity were acquired by Ingenix and Aetna respectively. These acquisitions are examples of a growing trend for larger established health insurers who are heavily invested in IT to acquire HIE vendors. It is predicted that within the next five years, there will no longer be small, independent HIE vendors. The independent HIE vendors that survive will be a division of a far larger company. These insurers who have the infrastructure to collect and analyze lots of patient data and to
manage risk exposure want to make their technology and data more easily available to
doctors. Procuring HIEs will allow the payers to be able to easily share and access patient
data and electronic health records. With these acquisitions, a closer relationship between
ACOs and payers will be forced. This change does not affect the RFP, but is a change in the
political environment worth noting.
Privacy concerns
The acquisition of HIE vendors will give health insurers the ability to better share administrative patient data with doctors. However, this may also require that doctors also share clinical patient data with insurers. The risk with this is that it could lead to insurers denying care to the riskiest patients. For the statewide HIO, supporting this infrastructure could put the entity at risk for involvement in lawsuits by the patient if such a scenario should occur. With this in mind, the negotiated contract with the selected vendor will have to be written in a language where the statewide HIO is protected from any lawsuits such as these.
The potential risk that HIEs may pose in complying with Health Insurance Portability and Accountability Act (HIPAA) regulations has been expressed by the provider community. Because of these concerns, there may be a reduction in the quantity of data the providers are willing to share which reduces the quantity of data available.
Changes in leadership – Governor, representatives, etc.
With new leadership, come new ideas and new changes. A new governor was elected for
this state in January and has appointed new leadership to sit on the steering committee.
Also, there have been changes in the leadership of the statewide agency with the
appointment of a new commissioner of the healthcare agency sponsoring the statewide
HIO. Whereas, this initiative is still supported, there may be different ideas or changes in
direction from the steering committee on what the statewide HIO may look like than was
originally agreed upon to support this RFP.
Also, with the most current election, a Republican majority U.S. House of Representatives
was elected. Currently, there is a battle over the budget and some of the dollars allotted for
healthcare reform are on the chopping block. If the budget is not passed by March, the
funding that was originally targeted to support this initiative may not be available and may
delay the project’s ability to award a vendor according to the schedule originally defined in
the statement of work. If the budget is passed and there is a reduction in the allotted
dollars for the ONC to work with for this fiscal year, there may be a delay in the trickling
down of money to the steering committee. With this possible reduction or delay in funding,
the state and steering committee will have to seek other sources of funding which will
change the schedule for implementing the statewide HIO. This lack of funding or trickling
affect may also affect the scope of the HIO and may require that a scaled down version of
the statewide HIO be initially implemented.
4. Key Design Decisions and Assumptions
Statewide HIO outlined several key design decisions and assumptions that would be required to meet
the needs of an HIE application. The HIE landscape is immature and in a high state of change and
growth as an industry. Industry experts predict that 50% of all HIEs today will fail and not become
sustainable. The most important consideration for most HIEs is cost, scalability, and technology. The
primary areas that are most important to the Statewide HIO are:
Data Exchange – Statewide HIO has determined the approach to data interoperability that will best meet its overall business needs. If you compare an HIE to music available through the Apple iTunes store, customers log in and download their music from a repository and take the music – or in the case of an HIE – the data away. In a federated model, the data is collected in edge servers where it is indexed or retained in its original repository and accessed by record locator pointers that contain minimal patient identifying data to locate data related to that patient. The Hybrid HIE model stores some data in a central repository within the HIE application for registry or population reporting. Most other data is stored at the edge server level and accessed via the record locator service. The Hybrid model does a better job of managing security and privacy concerns of the HIO and this model generally is lower cost than the central repository.1
Statewide HIO has made a decision to begin with the Federated approach in order to stand up
the HIE quickly. This will allow the HIO to meet basic data exchanged for providers. The long-
term vision of the HIO is to move to the Hybrid approach. The HIO believes the Hybrid approach
best matches their overall business and security needs. Starting with the Federated model will
allow the HIO to spend less dollars up front before moving to the Hybrid model. Both Axolotl and
Medicity offer both approaches, although this is part of the base offering of Axolotl and is an
add-on feature for Medicity.
7
Open Systems Architecture – Statewide HIO seeks to find a vendor who has adopted an Open Systems Architecture approach to their HIE application. Open Systems Architecture can take advantage of all of the healthcare message & transaction standards –
Both Axolotl and Medicity can accommodate the message and transaction standards required
by Statewide HIO.
HITECH Security Standards - Security policies to be considered revolve around access, authentication, audit, and authorization – the administrative safeguards of HITECH. Statewide HIO requires its HIE application to provide all of these appropriate safeguards. Security becomes an important issue for provider willingness to exchange their data.
Both Axolotl and Medicity, in either a Federated or Hybrid data exchange approach, will assist
Statewide HIO in meeting its security goals.
System Reliability, Availability, and Scalability - Consistent system reliability and guaranteed availability of the system are critical for seamless patient data exchange. The data exchange services will become as mission critical as any provider EHR. Statewide HIO is looking for a vendor who has a proven track record for stabile system operations and 99.999% uptime availability. HIE solutions also need to be scalable for future growth of the HIE. HIE’s begin exchanging a few data types – laboratory, radiology, and transcribed document results. The HIE will grow in other data types and services such as referral, consent, e-prescribing, registry, and population reporting services. As the HIE expands in its services, the HIE application will need to scale up to accommodate the growth.
Axolotl offers both reliability guarantees and 99.999% uptime availability and has a proven track
record for meeting HIE scalability based on the HIO needs. Medicity offers the reliable systems
Message Standards and Transport Protocols
CCD
CCR
DICOM
EDI X12 – Administrative transactions
HL7 V2.x
Hl7 V3
NCPDP
SOAP (Web Services)
SSL/TLS
XML
guarantee, and also offers a 99.99% uptime guarantee. Medicity has had some challenges with
scalability, and also requires more planned downtime during the year.
Remote Hosted vs. Self-hosted Technology Models – Statewide HIO decided to use the RFP HIE search and selection process to help inform the HIO about which hardware hosting approach that would be best for the business. Statewide HIO has determined that its core business competency is not hardware hosting and infrastructure and has decided to find a vendor that can provide these services for the HIO.
Both Axolotl and Medicity have cloud computing offerings which will allow the HIO to remote
host its hardware operation for the HIE application.
HIO Privacy Requirements – Statewide HIO’s position on patient consent is “opt-out”. The HIO requires an HIE application that will support an opt-out approach. It is important to the HIO that the HIE application can also support opt-in and no-opt approaches in the future in the event that Statewide HIO changes its consent policy.
Another concern from a privacy perspective is the issue of malpractice. The HIO could find itself
in a malpractice lawsuit if a provider either chose not or did not use information available to
him/her in the delivery of patient care. Statewide HIO will address this as part of the privacy
terms and conditions in the vendor contract.
Both Axolotl and Medicity can provide the selected option for patient consent to the HIO.
Vendor Stability and Reliability – In the HIE application RFP, Statewide HIO outlines stability and reliability its vendors must meet:
o The vendor must have experience within the last 2 years in HIE business planning and development, technical planning and design, implementation and operations support. The response must provide the details of each area of expertise and the exact years of experience.
o The vendor must be the prime vendor in the operation of at least three HIE installations.
Axolotl and Medicity both meet the requirements of vendor stability and reliability.
9
Cost – Cost is an important consideration in choosing an HIE, but not the only consideration. Other important considerations are listed in this section – features and functions; open architecture; security & privacy; system reliability, scalability, and availability; technology; and vendor reliability and stability. All of these aspects of the HIE application will be thoughtfully weighed and the vendor solution providing the closest match to the overall business needs will be chosen.
Medicity is the lowest cost option of our vendors, with Axolotl coming in as a close second.
5. Selection Criteria
Methodology and Reasoning behind Selection Criteria- HIE Organizational landscape is too new, No proven sustainable models
The selection criteria, which follows relied on the Delphi process to decide an appropriate scoring
technique for selecting the vendor. The criteria are a culmination of objective and subjective data
collection, tangible and intangible asset evaluation and evaluation of the vendor’s current customers.
The vendors completed the worksheets identified in the RFP and the responses were analyzed and
applied to the appropriate selection criteria. Due to the known lack of HIE sustainability, the categories
receiving higher scoring priority are as follows: Cost, Vendor Experience, Scalability, Ability to provide
infrastructure, design and host services and Customer site visits.
Cost Comments
A. Implementation costs Costs spread out over 5 years vs. Higher upfront
costs. Fear over how the ARRA money will trickle down
to the HIE level and what the future funding will be after
incentives disappear.
B. Software pricing
C. Professional Services
D. Hardware
Vendor Respondent Experience
A. Two years experience in HIE business
planning
Few vendors can boast significant live customer
experience, but looking for an experienced vendor with
solid, proven solution
B.Prime vendor for 3 HIE’s
Vendor Minimum Requirements
A. Provide basic HIE functions-clinical
messaging, referrals, inquiry, e-prescribing,
public health reporting, EMPI, record locator
services
Depositing patient data directly into the EHR so
that it becomes actionable should not be the immediate
goal. Most successful HIEs to date are those with the
least complex approaches. More focused on passing
packets of data around without taking ownership of the
data---some manual intervention will be required with
almost every transaction.
What will be critical in the future is analytics and
business intelligence and you cannot run reports in a
federated model---but, it is easy to deploy. Centralized
HIE models are fading due to logistical issues, security
liabilities and upfront expenses. Can we afford data
warehouses?
Hybrid HIE with a central server to hold onto data for a
finite period and for compiling reports will become the
industry standard. In the future, PaaS will allow the HIE
vendor to host the service in the cloud and health care
providers will be able to customize it.
B.Conform to national standards Even vendors that are CCHIT certified have a
rudimentary ability to send out results in a standardized
way. Public HIEs- direct and temporary government
backing. Demand strong standards-based technology
C.Comply with privacy & security requirements
for an HIE
Very important factor, since the potential risks that HIEs
may pose in complying with HIPAA may be a barrier to
stakeholder participation
Response to Request for Additional Information Comments
A. Timely Response<1 week
11
B. Answered as directed
Scalability Public HIEs- direct and temporary government backing.
Demand strong potential scalability
A. Consent Repository Vendors that can best manage patient consent
management are most successful
B.Quality Reporting Will require a centralized or hybrid solution
C.Vocabulary Services
D.Security tokens
Infrastructure
A. Software components Vendors with the most flexible, customizable
application programming interfaces and software
development kits will succeed in the PaaS model
B.Hardware requirements
C. Reliability of the System
Design Comments
A.Integration
B.Adapter Framework
C.Disaster Recovery
D.Programming Language
Host Services Advantage for SaaS- costs? Implementation? Scalability?
Maintenance?
A. Independently
B. Relationship with Host Provider
Usability
A. Workflow
B. Warranty
C. Training Tools
D. Support 24/7
Vendor References
A. Listed 3 HIE’s involved with
B. Provided 3 customers who verified project plan
involvement
Vendor Demonstration
Customer Site Visits
Selection Criteria Category Score
Price of the system 15
Vendor respondent experience 15
Vendor qualifications meet the Minimum Requirements stated in the RFP
10
Response to requests for additional information from vendor respondent
10
Scalability of the proposed system to meet 15
13
future growth of the HIE
Vendor respondent ability to provide all aspects of the RFP: Infrastructure, design and host services
15
Usability of the system for end-users and IT staff maintenance within the HIE
10
Vendor References 10
Vendor Demonstration 5
Customer Site Visits 15
Total Score 120
6. Analysis of Vendor Responses
There are very few experienced vendors with live HIE’s who have met their objectives. Four of the
leading HIE vendors within the industry responded to our Statewide HIO RFP by the deadline, Axolotl,
Medicity, RelayHealth, and Epic. We used the Delphi process to support a side-by-side vendor
evaluation of these contenders. We selected eight leading members from our panel of steering
committees, to serve as key evaluators of the respective products. The evaluators represented the
clinical, technical, privacy and security, finance, and legal constituencies of the HIO, as well as the broad
interests of the private and public health care sectors across State X. These evaluators were responsible
for scoring each of the vendors based upon:
Their responses to the Features Functions Worksheet of our RFP
The HIO Vendor Selection Criteria
Submitted Client References
Site Visits
Vendor Demonstrations
Client Testimonials
The eight key evaluators used a Vendor Evaluation Scorecard (Appendix A) to score each of the four
vendors on their individual responses to the Features Functions Worksheet of our RFP. The average
vendor scores were 7.61 for Axolotl, 7.31 for Medicity, and 5.01 for RelayHealth. Epic only connects Epic
software clients and was not fully scored.
Based upon the vendor evaluation scoring, we eliminated RelayHealth and Epic as viable partners in our
Statewide HIE initiative. We continued to vet the remaining vendors, Axolotl and Medicity, which are
industry leaders in HIE innovation and implementation. In fact, according to the 2010 KLAS report,
Medicity and Axolotl were considered in 23 percent and 22 percent, respectively, of the HIE buying
decisions.2
We requested three current HIE client references from both Axolotl and Medicity, and performed full
day site visits at two live HIE operations of each vendor. We also invited our Board of Directors, Steering
Committee members, Key Evaluators, and all interested stakeholders to a number of scheduled vendor
product demonstrations.
Customer Testimonials:
While cost, technology, scalability, integration, and privacy concerns are major determinants in our
selection process of an HIE vendor, there are few proven models of success in this evolving and
precarious industry, and finding the ‘perfect’ vendor has proven to be uniquely challenging. Aside from
the objective criteria, we need to align with a vendor that shares Statewide HIO’s vision for the future of
health information exchange, is committed to transparency and a spirit of partnership in all negotiations
and implementations, and fits the culture of our organization. Some of these softer criteria can be
assessed by focusing in on the marketing strategies and attitudes of the vendors’ representatives during
site visits, vendor demonstrations, and, even, informal conversations.
We have gained a deeper understanding of the organizations behind the RFP responses and marketing
tools of our final two competing vendors, by referencing KLAS, a company that evaluates health care
vendors, products, and services. KLAS spent one year investigating the solutions of the HIE industry
leaders. As part of their initiative, they elicited client satisfaction responses regarding vendor products
and services. 2
15
The KLAS report indicated that Axolotl was one of the “largest commercial vendors in the HIE space,
with a fast-growing client base evenly distributed between acute to acute and acute to ambulatory HIE
organizations. It has the most live acute to acute HIE clients". Clients spoke enthusiastically about a
“straightforward product…that was easy to use”, “well-done implementations”, and “customizable
workflows and flexible report distribution features”. Many customers cited Axolotl’s ‘lite EMR solution’
(for sites that do not have a full-blown EMR) as an impetus for widespread health information exchange.
Most customers claimed measurable improvements in their operational efficiency. Customers were
satisfied viewing data via Web portals and taking advantage of remote access. Interviewed live sites
were content with manual search models and receipt buckets that collected requested information, but
looked forward to Axolotl’s next phase in which information would populate the resident EMR without
interfacing, or, perhaps, have the EMR proactively search for needed information and display it for
physicians. Clients were positive about Axolotl’s account management and executive involvement, and
acknowledged that “the organization contributed to smooth contracting and implementations.” Other
customers commented that the “contracting process was fair with key executives involved throughout”.
One client stated, “Part of my frustration is that Axolotl knows a lot, but we don’t know what to ask. So
we trip over something and Axolotl will have the answer but will never have told us.” In addition, many
customers asked for “better training and faster development cycle for upgrades”. 2
Medicity customers applauded the company’s strength and efficiency in acute to ambulatory exchange
of healthcare information. They described the Novo Grid product as “simple and easy to use…very
successful for pushing information to doctors’ offices.” It was “outpacing the competition by a wide
margin as a facilitator for simple results distribution...funneling lab results from hospital labs and
reference labs to ambulatory sites.” Success of the ProAccess solution was more difficult to measure,
as it was only in use at a few acute to acute HIE organizations. Other positive customer comments
included, “Once properly configured, the operation is fluid and stable to use”, “Novo staff are talented
and intent on implementing client feedback in application upgrades”, “Medicity is really good at getting
disparate software systems to talk to each other”, and “Medicity was the only vendor that came with a
business proposal and not just a technology proposal”. In contrast, some other customers made note
that, “Initial phases of Novo Grid’s implementation can be labor intensive and time consuming while
configuring the HIE platform”, “System documentations for new releases have room for improvement”,
and “Since Medicity acquired Novo, there are longer turnaround times”. Customers also wanted “more
regular feedback on in-progress interfaces”. 2
7. Scores
Selection Criteria Category Score
Axolotl Medicity
Price of the system 15 11.16
12.16
Vendor respondent experience 15 15 15
Vendor qualifications meet the Minimum Requirements stated in the RFP
10 10 10
Response to requests for additional information from vendor respondent
10 10 10
Scalability of the proposed system to meet future growth of the HIE
15 14.50
14.50
Vendor respondent ability to provide all aspects of the RFP: Infrastructure, design and host services
15 12.72
12.74
Usability of the system for end-users and IT staff maintenance within the HIE
10 9.91
9.91
Vendor References 10 9.81
9.44
Vendor Demonstration 5 4.875
4.625
Customer Site Visits 15 12.875
11.125
Total Score 120 110.85 109.50
8. Cost Proposals
17
In our request for Proposal, Statewide HIO was seeking pricing information it could use to do side-by-
side comparisons amongst its vendor respondents. Statewide HIO covers a population of 10 million,
25,000 physicians, and 175 acute care hospitals.
Four vendors were reviewed – Axolotl, EPIC, Medicity, and Relay Health. As a result of our features and
functions review and scoring criteria, only Axolotl and Medicity remain appropriate HIE options for
Statewide HIO.
The table below outlines side-by-side cost comparison of IT costs over a 5-year period of time. Axolotl
took a more traditional approach to their cost proposal. They have included a one-time enterprise
license quote for the entire Statewide HIO service area, professional services to cover phase I and phase
II implementation activities, and annual ongoing costs for remote hosting services that include
hardware, software maintenance, storage, high availability & disaster recovery, 24x7x365 support,
network & telecom fees.
Medicity took a different approach for their proposal. Their quote includes one-time enterprise license
fees for software and implementation services spread over a 4-year period of time. Medicity is Software
as a Service (SaaS) provider. Their annual support costs begin in year 5 and those fees include mostly
software maintenance costs and a small fee for on going SaaS costs. Medicity is not a repository, so the
hardware, storage, and hosting requirements are a completely different model from the Axolotl
proposal.
Axolotl & Medicity Cost Proposals2
One- Time Costs: Year 1 Year 2 Year 3 Year 4 Year 5 Totals
Software License Fees:
-Axolotl $725,000 $0 $0 $0 $0 $725,000
-Medicity $400,000 $800,000 $1,200,000 $2,000,000 $0 $4,400,000
Implementation Fees:
-Axolotl3 $942,000 $942,000 $491,000 $2,375,000
-Medicity $0 $0 $0 $0 $0 $0
Total One-Time Costs:
-Axolotl $3,100,000
-Medicity $4,400,000
Annual Recurring Costs:
Hosting Fees: -Axolotl $400,000 $400,000 $400,000 $400,000 $400,000 $2,000,000
-hardware
-storage
-high availability & disaster recovery
-24x7x365 support
-network & telecom fees
-software maintenance
Subscription Fees: -Medicity $0 $0 $0 $0 $450,000 $450,000
includes SaaS fees
Total Annual Recurring Costs:
-Axolotl $2,000,000
-Medicity $450,000
5-Year Total Cost of Ownership:1
Axolotl $2,067,000 $1,342,000 $891,000 $400,000 $400,000 $5,100,000
Medicity $400,000 $800,000 $1,200,000 $2,000,000 $450,000 $4,850,000
1. TCO does not include HIO labor costs or other professional fees not provided by vendors.
2. These costs represent non-negotiated vendor quotes.
3. Axolotl will charge implementation fees as incurred by year.
Both vendor quotes are within original budget projection. Statewide HIO plans to negotiate aggressively with both vendors to reduce quotes by 20% Medicity has a more flexible payment proposal, less out-of –pocket in the early years. Axolotl and Medicity offered enterprise licensing (no additional licenses would need to be purchased in the future for
current state size projections. Axolotl & Medicity include software maintenance in its annual hosting fee. Overall, Axolotl matches our selection criteria most closely.
9. Acceptance Testing
19
The Statewide HIO Steering Committee Technical experts will evaluate vendor testing on the completed
system. The expected test plan will include verification/validation for the following:
System requirements System design Interoperability Integration and Adapter Data Standards Data Aggregation, De-Identification, Re-Identification and Anonymization Analytics and Reporting Privacy and Security authentication Disaster Recovery
A team of experts (identify the committee members) will develop and initiate user acceptance test case
scenarios.
Describe the role of acceptance testing
The Statewide HIO Steering Committee Technical experts will evaluate vendor testing on the completed
system. The expected test plan will include verification/validation for the following:
System requirements
System design
Functional requirements
Interoperability
Integration and Adapter
Data Standards
Data Aggregation, De-Identification, Re-Identification and Anonymization
Analytics and Reporting
Privacy and Security authentication
Disaster Recovery
A team of experts comprised of the Sub Committee members who are representative of stakeholders
throughout State X and the Steering Committee will have the responsibility of managing the user
acceptance testing. The subcommittee members include Thomas Brady, M.D., Maria D’Angelo, Privacy
Officer, Jeffrey Fox, CIO, Rajendra Kothari, Systems Analyst, Carol Manning, Clinical Nurse Specialist,
James Prince, M.D., Sara Rowe M.D., Randall Smith’s J.D., C.P.A. The team of experts will develop and
initiate user acceptance test case scenarios applicable for the technical requirements identified within
the RFP. The chosen panel of experts from the various disciplines ensures a balanced approach when
testing the system and verifies the vendor meets the agreement set forth by the Statewide HIO.
The role of acceptance testing play in the decision
The Statewide HIO is committed to providing a quality product for its users and the citizens of State X.
The Statewide HIO Steering Committee expects the vendor to follow a rigorous testing plan conforming
to ISO/IEC 29119 Software Testing Standard. The testing process provides a method to ensure
reliability, accountability, and quality assurance of the system.
http://softwaretestingstandard.org/part2.php
The vendor should demonstrate a test plan, which includes the elements listed below prior to any User
Acceptance Test Plan:
Unit Testing
21
Integration Testing
System Testing
System Integration Testing
Software Performance Testing
Software Load Testing
Stability Testing
Usability Testing
Security Testing
Destruction Testing (http://en.wikipedia.org/wiki/Software_testing)
Acceptance testing performed on the product
Subcommittee List of Objectives for the testing process
Provide well-defined, easily understood tests, which are applicable to the functionality of the
system
Enter relevant information in the system and guarantee the expected result of data entry
Validate that all test cases are documented
Document the problems identified and work with the vendor to resolve
Report necessary changes to the Steering Committee
Steering Committee List of Objectives for the testing process
Track and manage test problems
Communicate change recommendations to the vendor
When will acceptance testing be performed?
The user acceptance testing timeline is contingent upon the final contract agreement. The expert team
members have identified high-level objectives and the required tasks applicable to user acceptance
testing in the table below.
Objectives Tasks Associated with Objectives
Plan Acceptance Testing for State HIE Acceptance test schedule
Identify Testing Requirements Test matrix
Establish Test Environment Pilot site agreements
Conduct Acceptance Test Readiness
Review
Draft of test plan
Execute Tests Complete testing checklists
Complete Acceptance Testing Test summary report to Statewide HIO Steering Committee
Document Acceptance Testing Final acceptance test report to Statewide HIO Steering
Committee
Testing Case Scenarios – Please refer to Attachment (Excel Document)
Results Unacceptable during Acceptance Testing
There are several results, which would be unacceptable throughout the user acceptance-testing period,
and it is the expectation a fix occurs within five business days. It is imperative the user, role and patient
authentication is without incident due to federal and state government rules and regulations, which
address patient privacy and organizational policies pertaining to information security. The inability to
transmit what is considered a small amount of aggregated data from Hospital A to Hospital and data not
deidentified is not satisfactory. Lastly, prolonged periods of unplanned downtime is not suitable for the
timely exchange of patient information, which is a primary objective of the Statewide HIO.
10. Change Management
In the SOW, a change management process was established which was to submit a completed change
request form to the statewide HIO steering committee. The change request form contains a description
23
of the change, and a justification for the change. This form is reviewed by the steering committee which
will either approve or reject the change request. The same form presented in the SOW is used to
manage changes to the RFP.
During the review of the RFP, it became apparent that not all vendors were able to offer a complete
product as requested in the RFP. It is understood that the goal of the statewide HIO is to get an HIE up
and it is recommended that a step by step approach be taken so that the selection will allow for the
Statewide HIO to deal with the HIEs as they are. In the future, as analytics are developed, they can work
on developing centralized server at a later time.
The following change request form was submitted and approved by the statewide HIO steering
committee during the vendor evaluation process.
Statewide HIE Change Request Form
Sponsor Name Statewide RFP Vendor Change # RFP – 001
Project Name Statewide HIE Date Submitted 2/22/2011
Requested By Robert F. Proposal Vendor Date Reply Due 2/28/2011
Change Request Type: __New Requirement X Change Requirement __Remove Requirement
Description of Change:
None of the responding vendors were able to offer a complete product as requested in the RFP. To meet the objectives of getting the HIE
established in the desired timeline, this request is for approval to allow the adoption of a step by step approach in the implementation of the
Statewide HIO. The first step will be to begin with a Federated approach in order to stand up the HIE quickly. It will allow the HIO to meet basic
data exchanged for providers. This approach also allows the organization to select an HIE with their current offerings and to enhance their
systems over a defined time period of time to reach the ultimate goal of a hybrid model system.
Justification:
In reviewing the RFP responses, none of the HIE vendors offered a complete offering of a complete product as requested in the RFP. Also,
because of possible funding shortages, approval of this change will allow the HIE to stand up quickly, and meet basic data exchanges for
providers. It will also allow the HIO to spend less dollars upfront before moving to the long-term vision of a hybrid model.
Effect on Schedule, Deliverables, and Project Cost
No. Deliverable / Item Revised End Date Net Change, Hours
Increase or (Decrease)
Net Change, Cost
Increase or (Decrease)
This item affects the overall project
which cannot be outlined. Therefore,
this item is a TBD item to be revisited and completed after the final contract is
awarded and a workplan defined.
Totals
Revised Project End Date
Steering Committee Member Approval:
Signature : _________________________ Title: __________________________ Date: _________
Comments:
25
11. Discussion
While Axolotl and Medicity were very close finalists in this RFP process, Axolotl is recommended by the
HIO RFP Selection Committee for several reasons. First, the pricing structure of Axolotl whereby the
majority of costs are incurred during the beginning of the HIE implementation is more attractive for
Statewide HIO. Given that federal grants are only available to help offset startup costs of HIEs by means
of the HITECH Act, long term sustainability is a valid concern. The pricing model offered by Medicity
escalates costs each year during the 5 year total cost of ownership. If federal funds are no longer
available as health care reform evolves, the transition to HIE sustainability with Medicity could be more
challenging if the majority of the costs of the HIE implementation are still being paid.
Statewide HIO would like to rapidly implement basic functionality of the HIE utilizing a federated model.
As additional functionality is needed to meet stage 2 and 3 meaningful use requirements, transition to a
more hybrid model will be necessary to support features such as a consent repository. Axolotl offers the
tools needed for both a federated and a hybrid model with their base product. In contrast, adding
repository functionality with a hybrid model is not included in the base Medicity product, resulting in
additional costs in the future.
Comparing the experience of these two vendors, Axolotl has more customers live with their HIE
products. They began their clinical networking company in 1995. Axolotl also has experience with other
customers like Statewide HIOs powering both RHIOs and other statewide HIEs.
The references and site visits validated the comments documented in the latest KLAS report. The
customers we spoke with were very satisfied with Axolotl as a vendor. When problems did arise, Axolotl
promptly engaged with the customer in troubleshooting and resolution of the issue to the customer’s
satisfaction.
Two vendors were excluded during the RFP process. While Epic is a KLAS leader in the world of
electronic health records, it’s HIE product, Care Everywhere, is designed to connect Epic customers with
each other. There is a variation of Care Everywhere (formerly called Care Elsewhere) that allows a
similar connection between an Epic customer and other EHRs, but this product is contingent on use of
Epic’s EHR. Therefore, Epic was not considered as an option for the Statewide HIO due to the fact that it
is not a stand-alone HIE software solution and requires the use of Epic’s EHR.
Relay Health was also excluded from the list of HIE vendors during this RFP process. This HIE solution
currently only involves a unidirectional interface between hospitals and ambulatory physicians. While
their ability to connect physician offices with the patient’s personal health record makes them stand out
from a patient centric philosophy, their experience in acute to acute facility exchanges is lacking.
Additionally, Relay Health’s parent company, McKesson, recently reported layoffs of more than 400
employees, bringing their stability as an organization into question.
Attachments
Selection Criteria Final Evaluation.xls
User Acceptance Testing.xlsx
vendor selection form-2.xlsx
References
1. “Whats-on-tap-for-health-information-exchange-at-HIMSS-2011.htm”, SearchHealthIT.com, February 11, 2011.
2. KLAS. Health Information Exchanges. The Reality of HIE Adoption. February 2010. 3. http://chilmarkresearch.com/2010/06/30/where-is-the-consumer-in-hies/ 4. http://infosite.medicity.com/theHIEblog/
5. http://www.ehow.com/how_5224184_perform-user-acceptance-test-_uat_.html 6. http://www.healthcareitnews.com/news/axolotl-initiate-partnership-creates-market-
opportunities
7. www.medicity.com
8. www.relayhealth.com