Hold My Beer, I'm going to do DevOps with DICOM, HIPAA, and Hospitals, or at least try - Gil Bahat -...

Hold my beer!

I’m going to do DevOps with DICOM, HIPAA and Hospitals. Or at least try.

A little bit about me

Independent DevOps consultant

Like to solve hard problems (Hold my beer…)

Social links:

www.linkedin.com/in/gilbahat

www.twitter.com/GilBahat

http://www.linkedin.com/in/gilbahat

http://www.twitter.com/GilBahat

Disclaimer

This lecture bastardized DevOps terminology, unapologetically mixing ideals and

real world concerns and generally sidesteps the whole “What is DevOps” debate

by flipping a finger.

We’re here to work and make this beast function somehow.

Two part lecture

In part 1 We’re going to talk about DICOM, one of the two major healthcare

protocols.

In part 2 We’re going to talk about healthcare regulations and how they change our

workflow and thinking.

Even if you don’t care about DICOM, consider it an exercise in onboarding non-

standard protocols.

Even if you don’t care about healthcare regulations, consider it an exercise in

handling any regulatory environment

Motivation

Learn how to handle a non-HTTP protocol:

Get familiar with the protocol

Scope its usage in our system

Understand key features we care about (performance, security, …)

See how we can realize them.

Get to know DICOM

DICOM stands for “Digital Imaging and COmmunication In Medicine”. It’s both a

wire protocol and extensible serialization format.

Its designed by NEMA (electronics manufacturers) - makers of modalities (the

medical term for xray/CT/MRI/… machines)

Its design philosophy is “Be conservative with what you send and liberal with what

you accept”.

DICOM is a highly specialized protocol. Very few implementations, much less open

source.

Scoping DICOM for our system

Few Implementations means trouble.

With DevOps you really want to avoid re-inventing the wheel.

Some SDKs, much less integrated ‘plug and play’ packages.

Auxillary components are generally missing: load balancing, monitoring, utility

parsers etc. For the exact same reasons.

Scoping DICOM for our system

We expect to use DICOM to transfer data from the hospital to our processing

domain.

Internally we will avoid the wire protocol as soon as possible.

As for serialization, YMMV. Risks and benefits either way.

DICOM Technical features

The protocol is binary, overlays a full OSI model over TCP/IP.

Strictly regulated enumerators and hard-coded behaviours.

Let’s see how easy or hard it is to retrofit stuff we need for scalability and security

onto DICOM.

Analysis of DICOM features

Commands are sent from SCU (“client”) to SCP (“server”). Only SCU is allowed to

cleanly release a connection.

Commands are split to notification and operation. Async operations supported,

optional and negotiated.

Analysis of DICOM features

Commands are either A (association), C (composite entities) or N (normalized

entities). We’ll be looking at A and C lingo. We see the following vocabulary: A-

ASSOCIATE, A-RELEASE, A-ABORT, C-STORE, C-FIND, C-MOVE, C-GET and C-

ECHO.

P-DATA (payload contents basically) may be fragmented. Messages have unique

IDs and are all acknowledged.

Incoming DICOM

We can pipeline when multiplexing, if negotiated during the connection.

Since we’re dealing with studies, we’re looking at a lot of short transfers in same

bulk.

If we keep long-term established connections, we have to plan our load balancing a

lot more carefully.

Incoming DICOM

Observation: Connection establishment involves 3-way TCP and 3-way ACSE (TLS

not included - yet). We can keep connections alive with C-ECHO.

We cannot multiplex different ACSE names (either client or server) though, each will

result in a new connection.

Load balancing DICOM

What is it that we want from a load balancer, anyway?

Balancing load between instances, to allow for good resource utilization, good

responsiveness, avoid overloaded instances.

Detect instance failure, remove failing instances from duty to avoid data loss /

outage.

Reflect a safe error condition towards the caller if the system is unresponsive

Modify/tweak client behaviour to better suit the server needs (optionally)

Handle/offload encryption and authentication

Securing Incoming DICOM

DICOM Security: TLS (part of the standard). Some people do it with IPSEC VPNs,

we’ll dedicate a slide for that option.

DICOM authentication: User/Password, Kerberos, SAML

Digital signatures supported.


TLS is part of the DICOM standard, since mid-2000’s. Albit specified in a highly

braindamaged constraining fashion, to 3 profiles:

TLS 1.0 RSA with NULL encryption

TLS 1.0 RSA with 3DES encryption

TLS 1.0 RSA with AES encryption

Entity authentication is optional.

Load balancing DICOM

Load balancing:

Selected algorithm: leastconn

Detecting failed instances: C-ECHO pings, dummy C-STORE actions

Static 504 replacement: A-ASSOCIATE-Rejector

TLS termination + TLS certificates

Without a full-fledged DICOM implementation, this is pretty much what we can get.


Overall, things are looking great for us right now. It looks like we can achieve our

stated goals, without investing too much effort into it.

A bit of haproxy lua wizardry around ACSE primitives and presto!

https://github.com/gilbahat/haproxy-dicom-lua

A full-fledged DICOM load balancer is probably a major undertaking.

https://github.com/gilbahat/haproxy-dicom-lua

DICOM in the wild (west)

Strike #1:

No auth methods are supported by modalities. Today’s magic sentence is:

“This product does not conform to any defined DICOM Security Profiles. It is

assumed that the product is used within a secured environment.”


Strike #2:

No TLS transport supported by modalities. Today’s magic sentence is still:

“This product does not conform to any defined DICOM Security Profiles. It is

assumed that the product is used within a secured environment.”


Offside: why IPSEC-VPN is a bad idea.

IPSEC is a cumbersome protocol with lots and lots of rope to hang yourself with.

Expect long debugging sessions. Staff your support accordingly.

It’s also technologically inferior to TLS with regards to ease of tunnel setup, for

failover scenarios.


Strike #3:

Commonality of lost images. There is no strong guarantee that all images are being

sent, and a missing image makes the entire study untrustworthy.

Protocol-level guarantees are no good since they do not account for host/policy

override and/or are not supported.

Remember C-FIND? It’s now our job to C-FIND the missing parts of a study. What

joy!


Strike #4:

C-FIND means we have to initiate a connection from our side to fetch the missing

part.

This would have messed up our entire architecture, but this option is impossible

due to security considerations.

So now we have to design and maintain our own in-premise equipment.


Strike #5:

Our favorite Open-source DICOM router does not multiplex.

It does support multiple connections. But you have to tweak them really fine: too

many and you lose images. Too little and your performance suffers.


Strike #6:

We’re barred from using client side certificates. This would have been our preferred

way of authenticating, given the DICOM auth mechanism selection and the fact we

don’t want to rely on DICOM authentication for security, and aiming for a TLS

deployment

The reason we’re barred from doing so is because DLP gateways will MITM any

traffic which doesn’t pass a designated IPSEC tunnel.


HTTP would have done somewhat little to avoid our woes here: we’d have to have

on-site presence due to policy and access controls. We’d have to go with VPN

tunnels due to DLP policies.

We’ll have to suffice with trying to shift as much workload / smarts to where we can

control them.


Actually, why did we prefer a zero-footprint installation, intuitively?

Because of the release cycle and control. This is a weak spot when deploying in

medical environments and may very well still be so in the next few years.

This means that our design still needs to make our in-hospital presence as minimal

and well-tested as possible. Almost as if it’s an embedded/IoT medical device (in a

sense, it is).


Can we apply some “DevOps for embedded/IoT” best practices?

Probably, but DICOM and Hospitals are going to make life hard for us here as well.

These environments are not easily reproducible or simulated due to exotic

protocol, exotic equipment, complex integration environment, security filters...


This turns our DevOps transformation into a people’s problem, more than a

technical one.

The right way to do DevOps in regulated environments is to understand the on-site

requirements and matching the deployment.

This is extremely hard to genericize, so you have to assume the worst. Keep

baseline exceptions to the minimum.

This means a non-typical customer engagement profile. Do it. Full-Circle holistic

design.

Regulations (HIPAA)

Intro to HIPAA

HIPAA:

HIPAA is a regulatory framework governing privacy and accountability of entities

processing US healthcare data. (there are other frameworks governing EU, IL, etc).

Terminology:

PHI - Protected Health information.

BA/BAA - Business Associate / Business associate agreement. Governs sharing

your data and responsibilities with third parties.

Intro to HIPAA

Get familiar with the regulations

Scope its footprint in our system

Understand key requirements/constraints we care about (security, auditing…)

See how we can realize them.

Intro to HIPAA

From a DevOps perspective, we are looking at any of the following being assigned

/ coordinated / supported / designed:

Encryption, Vulnerability scans, Asset management, Disaster recovery,

Configuration management, Incident investigation, Auditing, Logging, Coding

standards, Operating procedures.

Intro to HIPAA

Learning our lesson from before, there’s regulations and there’s unwritten

regulations.

Before you architect your solution, be sure to collect “unwritten” regulations from

potential customers. Stick to product management / sales / marketing if needed. Do

not relent.

(also, lest they promise something which is impossible from this aspect)

Intro to HIPAA

Accountability is a key line of thought for regulations. Get ready to procedurize(™)

yourself to hell and back. This can align nicely with DevOps principles.

Accountability implies auditing framework. This encourages an RCA culture, if

actively pursued.

Give heavy weight to logging solutions and their ability to handle PHI.

Intro to HIPAA

Security updates can be fast-tracked. Don’t miss this opportunity.

Encryption is your friend: at rest and at transit.

But also in data store too: The encryption keys can serve as a basis for

authorization - having the key implies a transferrable authorization context.

Intro to HIPAA

DevSecRegOps(™)!

Keeping your environment compliant and provably compliant.

So, for all the topics we’ve mentioned before (encryption etc), we’re now looking at

a three-way assessment of their impact / requirements and desired processes /

culture.

It’s important that they all talk the same language, and definitely doable.

Regulated DevOps culture

Understand the boundary between patient-impacting and non-patient impacting, as

well as the PHI borderline.

It’s hard to keep two distinct cultures. YMMV if you want to do a cultural split

personality thing or not.

Embedded development with its release cycle and methodologies is already

different enough.


Healthcare open source kits are good enough for scripting tests you may need.

Keeping on the cultural aspects just discussed, embrace a test culture throughout.

You know you wanted to anyway, now regulations are pushing you to do the right

thing(™).

Then again, we may need to consider some facets of testing in regulated

environments...


Pro tip for testing in live - always roll back if patients start dying.

An open architectural problem - roll back not yet implemented for patients.

Until that feature is available (being worked on, no ETA unfortunately) - No testing

in live, particularly because it’s alive and we want to keep it this way.

But we can test in preprod with live data - but only after anonymization. Get a

design partner.


Replicating the hospital test environments is pretty much impossible - including

oddball devices, policies (some may even be hidden from you) and even datasets

and integrations.

However, in an online/cloud architecture we can treat it as a black box and just

emulate the feed coming off of it after anonymizing it.

So, split environments/cultures again...


Tenancy models: getting this stuff right.

Single-tenant - eases the regulatory requirements, at the cost of resource waste

Multi-tenant - more complex regulations, allows better resource sharing.

Hybrid - twice the development/testing

Shared compute, separate data - with enough data access layer sophistication.

Accounting for hotspots, etc. A fun challenge.

(geo) Data retention laws make this decision less straightforwards than other SaaS

deployments.


Tenancy models affect security and regulatory architecture:

Single tenant - emphasizes things like network ACLs, IAM permissions

Multi tenant - emphasizes things like application ACLs, multiple encryption keys,

token services.


Regulations will inevitably be a political driver for many company decisions around

the development processes (and hence DevOps)

Potentially it’s a double-edged sword, e.g. with external services not offering BAA

or refusing PHI.

DevOps can be an enabler for conceptual regulatory requirements (e.g.

contingency, software acquisition, backups).

Software acquisition example - mirroring/proxying sourced software


Shared responsibility model - allowing some form of (needed!) relief.

What you don’t have to do - don’t do it.

Physical security, disposal of physical media, all that stuff. Happily delegated.

3rd parties are your friends - only if they offer HIPAA whitepapers and design docs.

Questions?

Now, will you hold my beer already?

Hold My Beer, I'm going to do DevOps with DICOM, HIPAA, and Hospitals, or at least try - Gil Bahat -...

Technology

Transcript of Hold My Beer, I'm going to do DevOps with DICOM, HIPAA, and Hospitals, or at least try - Gil Bahat -...