Can you recover your data?

MEDITECH Best Practices for Backup and Disaster Recovery

A Little About Frank• Frank Tollefson

• Send me email: tolleff@inhs.org

• Network Services Manager at INHS

• Healthcare Infrastructure System Engineering.  Manage a team of 11 Engineers.

• Degree in Electronics Engineering

• Industry Certifications – VMware VCA-CLOUD, VCA-WM, VCA-DCV, VCP5-DCV and VCP41 - Microsoft MCSE and MCP

• I have been working in IT for over 20 years with over 17 years supporting healthcare and financial customers, with 11 at INHS.

• MEDITECH infrastructure hosting and consulting for hospitals running all versions of Meditech utilizing Meditech traditional backups or Bridgehead Software.

• MEDITECH SAN certification testing

• Die hard ocean and fresh water fisherman

A Little About Chris

• Chris Welch

• Follow me: @Chrisw767

• Send me email: Chris.Welch@bridgeheadsoftware.com

• I think a lot about Backup, Archiving, and Disaster Recovery Business Continuity

• Visited hundreds of MEDITECH hospitals throughout North America

• Worked with all of the recent versions of MEDITECH and implemented all of BridgeHead's MEDITECH solutions, including first FileStore deployments for MEDITECH SCA

• Private pilot

• Living in upstate New York

Goals for this Discussion

• Recent Events at Customer Sites

• MEDITECH Best Practices

• General Best Practices

• INHS Model for Uptime and Protection

5 BridgeHead Software / Healthcare Data Management

Is this your disaster recovery plan?

Where are YOUR tapes?

Real-World Scenario: Critical Access Hospital, Weather Event

• Hospital building was seriously affected by a weather event- sections of the hospital were uninhabitable

• The datacenter was flooded with water due to aftereffects

• Hospital’s tapes were in a cardboard box next to SAN

7 BridgeHead Software / Healthcare Data Management

Real-World Scenario: Regional Hospital, Datacenter Modification

• Hospital was adding new air handler to datacenter

• Contractor drilled a large hole in the concrete floor to run a power line for air handler – and left core and concrete dust below floor

• When air handler was turned on …

• Servers, storage units, and switches pull air from front to back, and the dust was sucked into every running piece of equipment in the datacenter

• Hospital had a disk based backup solution which was also affected by the dust cloud

9 BridgeHead Software / Healthcare Data Management

Real-World Scenario: Large Hospital Chain, Disk Corruption• MEDITECH EMR disk slowly corrupted over several months-

corruption was not significant enough to bring down the EMR

• Backups to tape were happening on a daily basis and were dutifully backing up the corruption

• As corruption progressed, server became slower and slower- finally staff rebooted the server and it would not come up.

• When the previous night’s backup was restored, it contained the corruption and would also not boot.

• Customer had replicated MEDITECH data using array replication technology, but the replicated copy was corrupted as well

• Customer finally found a good backup with no corruption from several months back

• In the meantime, consultants were trying to find a way to fix the corruption

• Customer was faced with a difficult question: do I restore from a known good backup from months ago and lose a lot of EMR data or wait for a solution to resolve the corruption?

11 BridgeHead Software / Healthcare Data Management

Real-World Scenario: Overwrite of MEDITECH SCA Data

• MEDITECH Scanning and Archiving (SCA) servers contain images of Point of Care scans (drivers licenses, insurance cards, etc) and Reports archived from other modules within MEDITECH. It can also contain data from other applications and sources.

• MEDITECH SCA disks contain millions of files and grow very large. They are difficult to protect using standard backup strategies because of the large volume of data and files.

• MEDITECH overwrites data in SCA disks on a regular basis.

• A change was made in this hospital’s MEDITECH environment which caused over a million reports in SCA to be overwritten

• Because the backup window for SCA was so lengthy, backups had been stopped months ago

• The customer was replicating the disk with the SCA data, which caused the data at the replicated site to be unusable as well.

Disasters You May Not Have Considered

• Train car full of chemicals overturns next to the hospital- staff is not allowed in the hospital

• Sprinkler system in datacenter activated by high heat floods servers and storage (you don’t have water pipes in your datacenter, do you?)

• SAN fails due to bad firmware upgrade

• Wind breaks the window of the datacenter and rain gets into the core switch (I’ve seen it happen)

• Your backup tapes are overwritten every day by staff trying to save money

• An upgrade breaks a system and can’t be backed out

• Hospital becomes triage center for latest SARS epidemic

• With a Category 5 hurricane bearing down, your IT staff evacuates with their families

Some Surprising Facts About MEDITECH Recovery

• The most common root causes to failures are: – Data corruption – User error

• The vast majority of day-to-day restores come from monthly tapes

• Often, IT is not the limiting factor to restore time

• Patient data is always important, but in some scenarios it’s not required as quickly as other data

• During recovery, almost all hospitals discover that there is some application or set of applications that they have not been protecting

• They almost always thought they had been adequately protecting all of their applications

DR Language: Planning Against RPO and RTO

• RPO = Recovery Point Objective

– Formal Definition: The time from before the disaster event occurs, from which you have a recoverable set of data

– Practical Definition: How much data can you afford to lose?

• RTO = Recovery Time Objective

– Formal Definition: The time that it takes after a disaster event occurs, for you to recover your healthcare applications and data

– Practical Definition: How long can you afford to be down?

Understanding RPO and RTO

Point in Time at which the Disasteroccurs

RPO:How much data can you lose?

RTO:How much time can you wait?

1 min4 hrs1 week 1 min 4 hrs 1 week

The “sweet spot” for most hospitals

What Does “Good” Look Like?MEDITECH Best Practices

• MEDITECH application consistent backups (ISB, IDR, MBF)• Establish well thought out, realistic RPOs and RTOs for all applications.

• Geographically dispersed protection- multiple sites and multiple formats

– Disk and tape provide both fast restore and offsite protection

– No onsite tapes (the next building doesn’t count!)

– Cloud is a good option if you don’t have a DR site

• Deep generational protection

– MEDITECH recommends:

• Daily Backups- Keep for 2 Weeks

• Weekly Backups- Keep for 1 Month

• Monthly Backups- Keep for 12 Months

• Yearly Backups- Keep for 7 Years

What Does “Good” Look Like?General Best Practices

• If multiple sites, be sure you can back up in your secondary datacenter!• Your hospital may be able to recover from one disaster, but what if another

hits while you are recovering?

• Have a well thought out and documented plan for disaster recovery- it may not be you doing the restore

• A well trained staff knowledgeable in your backup product- make sure your staff is correctly monitoring your backups!

• Know who in your organization can make critical decisions involving data loss and recovery in a disaster

What Does “Good” Look Like? Good Behavior

What Does “Good” Look Like?General Best Practices

• Encryption- make sure you’re doing it and doing it correctly

• Does your second site only have the data, but no servers? Can you successfully run at your secondary datacenter

• Can your clients connect to the secondary datacenter in a disaster? Have you tested it?

• Failback process and testing

• Restore testing!

– Document the process

– Test the Documentation

– How long does it take?

– Quarterly testing

INHS ModelRobust Backups and Recovery starts with a Robust

Production Environment

Tools for Production Uptime Success

• Deploy only MEDITECH Certified Hardware Solutions

• Utilize a MEDITECH certified backup and recovery solution

• Follow MEDITECH Configuration guidelines and best practices

• Utilize Virtualization for MEDITECH Servers for Maximum Uptime

• Maintain server instances to Microsoft best practices, e.g., patching and OS hardening

• Establish a Disk Defragmentation Policy for Client Server and 6.x Environments

INHS ModelRobust Backup and Recovery for Maximum Data

Protection

Tools for Backup/Restore and Disaster Recovery Success

• Establish and Follow a Retention Schedule for MEDITECH Backups

• Send Backups Offsite Daily- Whether Tape, Offsite DR site, or Cloud

• Utilize VM Image Backups for Virtualized Servers System Drives

• Perform Disk based Backups to Lower RTO

• Document your backup, recovery and disaster recovery plan and procedures

• Test your backup plan at least yearly and but quarterly is preferred

Thank-you!

Questions?