Do you have a DR plan in place: so, don't let a disaster defeat your business

You Have a Disaster Recovery Plan:

So, Don’t Let a Disaster Defeat Your

Business

MWLUG

August 22, 2012

No one is safe from

Mother Nature!

August 29, 2012 Velocity Proprietary and Confidential2

93% of businesses that suffer a significant loss of data are out of business within 4 years.

The Bureau of Labor

Sobering Statistic

Disaster Preparedness Checkpoint #1:Are Disasters Viewed as a Real Threat?

Disasters never really happen.

It’s just a vendor sales gimmick…


Why Is This Area “ Vital ” ?

� Expectations of the IT Services are demanding

� Technology is an enabler of business

� Penalties are becoming more severe

� Business is becoming more competitive

� Businesses rely more heavily on their enterprise applications

than any other time in history

5

What is a Disaster ?

ANYTHING !!!!

That stops your business from functioning & that cannot be corrected within an acceptable amount of time….

What is Disaster Recovery?

Reaction to a sudden, unplanned event that enables an organization to continue critical business functions until

normal business operations resume.

“…It is not enough to arrange for hardware replacement;…planning must address continuation of business operations, or

business continuation.”

8

Common Issues with Disaster Recovery Planning

We tend to let our guard down when times improve.

As planners, we must always be ready and prepared.

• Are the expectations of your DR plan realistic?

• Has your plan kept up to date with your IT integrations?

• Do you have a qualified staff to execute the plan?

• Is DR integrated into your Change Control process?


9

Protect Important Assets

Primary assets needed to operate Information Systems:

• Hardware and networks can be replaced

• Facilities can be rebuilt or relocated

• Your Data is Priceless !

• People are assets too!


Checkpoint # 2: Of course we have a DR plan… Somewhere

Plan it yourself or get someone to do it, but whatever you do.. Plan!


11

DR Plans Come in Many Flavors

� To be on holidays when the

big one hits

� Cross your Fingers

� RESUME

� Resume stored offsite

12

No / Incomplete Disaster Recovery Plan

Guarantees:

• Confusion

• Lack of direction

• Conflict

• Lost Customers or credibility

13

The Products of the Plan

Who will execute recovery actions

What is needed to continue, resume, recover or restore business functions

When business functions and operations must resume

Where to go to resume corporate, business & operational functions

How detailed procedures for continuity, resumption, recovery or restoration

CLASSIC: WHO-WHAT-WHERE-WHEN-HOW

Checkpoint # 3: Ensuring Your Plan Meets Your Business Needs

Does your business know what your IT department can deliver?


Recover data to last known good

point before outage

RECOVERY POINT

OBJECTIVE

The amount of time it takes to

recover production systems

and resume operations

RECOVERY TIME

OBJECTIVE

Days MinsHrsWks Secs

Recovery Point

Mins DaysHrsSecs Wks

Recovery Time

What Are Your Recovery Objectives?


RTO - Recovery Time Objective

48 hours RTO – Systems Availability

RPO - Recovery Point Objective

24 Hours RPO - Tape Recovery

Recovery Solution Examples


Categorize Your Technology Inventory as Critical vs. Necessary vs. Not at All

Typically look for a 60, 20, 20 breakdown.

• 60% of all your servers are critical for recovery• 20% are necessary but not required immediately• Remaining 20% are optional or not required at all.

This means you can focus on 60% not the 100%.


Email

It has become Mission Critical.

It represents a way you communicate with customers, partners, employees, and the world.

Imagine no Email !!

Should EMAIL be an H/A application?


Traditional Backup Process

Offsite Storage

Facility

Traditional Backup & Recovery Process

Offsite Storage

( Bonded Provider )

Backup

Software


How Much Data Can You Afford to Lose?

• Lost data is lost money!

• Most IT shops depend on tape backups to protect their data

• Data considerations for tape

• Minimum of 24 hrs. of lost data

• Application access may be more critical


• Successfully deliver RTO objectives

Configure hardware

Install OS

Configure OS

Install backup agent

Load each and Every application from CD / Tape, FTP etc.

RestoreVM

Poweron VM

True recovery process comparison

24 + hrs

< 2+ hrs

Recovery Timeline Sample



Does your Business Know What IT Can Deliver?

August 29, 2012 Velocity Proprietary and Confidential23 23

Management Awareness

� Management shocked that the IT folks lost all orders

from previous day.

� 48 Hour RTO - Who agreed to this ?

� Do you have a new Senior Management Team ?

Checkpoint #4: Backups

Build it and they will come.

Put into practice what you say you will do.


Your System Just Lost All of Its Data!

Are you worried ?

Tape Backups - Pros

� Easy to run

� Easy to manage – Tape Management Products

� Easy to Automate – Automation Software Solutions

� High Speed & High Capacity reduced backup windows

� Capacity of tapes reduces number of Tapes

� It’s the way we have always done things

� Virtual tape libraries

� Local Electronic Backups

• Media, interface, or drive errors

• Seldom validated – How do you know your backups are valid?

• Time to ship, travel and restore tapes at Hotsite in a disaster

• Loss of data because backup only runs once per day

• Backups omitted in lieu of system access

• Lost tapes – Misplaced!

Tape Backups - Cons

27August 29, 2012 Velocity Proprietary and Confidential27

August 29, 2012 Velocity Proprietary and Confidential28 28

Backup = Recovery

How many people backup their system ?

38% are in-complete

19 % iSeries ( AS400) are un-recoverable

32 % Windows are un-recoverable

30 % Oracle DB are un-recoverable

15 % Virtual Infrastructure un-recoverable

29

“Backups ran fine. I never heard of any issues.”

• With a sign off, the process implies correctness

• Many backup solutions are partially broken

• Backup Success Rate = 97%

93

94

95

96

97

98

99

1 2 3 4 5 6 7 8 9 10 11 12

Failure rate = 3 %

This equals 11 days per year

with NO Backups


30

Sample Log

• Starting SAVDLO of folder *ANY to devices TAP01.

• 2574 document library objects saved.

• Starting save of list *LINK to devices TAP01.

• 43917 objects saved. 342 not saved.

• Save of list *LINK completed with errors.

• Starting save of media information at level *OBJ to device

• 18 objects saved from library QUSRBRM.

• Save of BRM media information at level *OBJ complete.

• DAILY *BKU 0070 *EXIT CALL PGM(BBSYSTEM/ENDDAYBU).

• Control group DAILY type *BKU completed with errors.

Backup/Recovery Best Practices

• Examine your backup strategy and policies for data

• Review backup strategy and policies for server infrastructure

Document server related information with signoff

• Are your network configurations backed up?

• Review off-site tape and vital records storage strategies

• Have you audited your backups?

• Can you demonstrate recovery procedures for daily, weekly, and monthly backups?


Checkpoint #5: Building a Recovery Team

Getting the right people:

Do they have the Right Stuff?


Perspective

Too often, companies populate their DR teams with raw or inexperienced staffers

Volunteers to satisfy an auditor or the sacrificial lamb

Ideal Team Member Characteristics

• Considered an expert by his or her peers

• A go-to person for anything and/or everything

• Works well under pressure

• Sense of urgency, willing to fix problems created by others

• Confident

• Trusted by peers

• Dedicated – A company person


Recovery is only possible if your staff is available

to put IT back together again.

Disasters affect people in unpredictable ways:

• It can devastate people both physically and mentally

• Can effect others around them

• Make the individual unable to function

• Emotional breakdown

Respect the situation!

The Effects of an Unprepared Recovery Staff Are Real


The Role of Executives

Executives are not typically involved and should NOT be.

Why:

• Executive Pressure is hindering

• Intimidating

• Often Nasty

• Glorified Experts

The team must provide regular status reports & the

Executives should be accessible when required.

• Feed your staff - Have snacks handy

• Stay away from High Energy Drinks

• Enforce no shift duration to exceed 12 hours

• Encourage people to take breaks

• Provide distractions during breaks

Best Practices During a Disaster

Family Comes First

Until the basic personal needs are met

• Family comes first. Always !!

• Staff members will not focus on the Enterprise recovery

• Staff members may or will not be available

Irene was no lady!

People Evacuated

2, 465, 000

Checkpoint #6: Is Your Data Offsite?

It’s eleven o’clock.

Do you know where your data is?


Recipe for Disaster

• One average filing cabinet

• Your recovery tapes/CD’s

• One average business fire


Recipe for Disaster

Directions:

• Place media in filing cabinet.

• Bake in a fire for 20 minutesat 800°

• Let cool.

• Open filing cabinet.


Storing Backup Media on Site …Your Business Is Toast!


Offsite Strategy

DISASTER

STRIKES

10 AM

W T F S S M T W T F S S M T W T F S

FullBackup

S

OPERATIONS RECOVERY

POSSIBLEUNPROCESSED

DATA

PLANNED

OFFSITE

BACKUP

DAILY EXPOSURE

Incremental backup


Checkpoint # 7: Virtualization

Practice smart virtualization


Compute

Storage

46

– OS & applications have 1:1 dependencies on hardware configuration

– Complex to physically recover OS, applications & data

– Separate processes for system and application data

Slow and Unreliable Process,

Expensive Infrastructure

Physical Wintel Recovery Challenges Today

cd, tape or ghost image

Application

OS

OS files

localstorage

Storage

WAN

Application

OS

OS files

localstorage

Storage

Production Server “Boot & Pray”

47

DR Scenario – Without Virtualization

Hardware Gap - Your Production & DR must always be equal

Gap grows as new severs are deployed

Can you afford NOTto modernize your recovery solution?

49

� Break dependencies between OSand hardware

� Manage OS and application assingle unit by encapsulatingthem into VMs

� VM’s are hardware-independent: they can be provisioned anywhere

With Virtualization

50

Physical Servers Converted into a Virtual Solution

Production Site

Convert images to virtual machines

Image Repository

Import virtual machines to ESX Servers

DR Site

51

Physical to Virtual (P-V) then Recovery

How:

• Converter creates virtual machines matching physical machines

• Copy virtual machines to Denver

If Recovery is required:

• Boot virtual machines on any hardware

• Start data recovery of application data

imaging

imaging

imaging

conversion

conversion

conversion Replication

P2V

P2V

P2V

JAS

Deploy

Load-Balance

Checkpoint # 8: Testing your DR Plan

It worked last year.

Nothing has really changed…


It’s not all bad news if your plan fails during a test…

Frequent testing identifies gaps in your recovery process


Passive Testing

• Hands on plan review

• Paper walkthrough

– Assessment of current Recovery Plan

– Assessment of risk areas

– Involve every member of your recovery team. This includes the business and management!


Active Testing

Validates the Recovery Plan in terms of:

1. Recovery capabilities and abilities

2. Alternate site configuration

3. Network recovery

4. Offsite records

5. Identify weaknesses in the plan

6. Provides training

This all equals success during a disaster


Testing / Hotsite Best Practices

• Examine your hardware schedule in your DR contract

— Check for back level compatibility

— Do you need additional capacity?

— Performance Profile – Have database requirements changed?

— Travel and expenses

• Avoid testing from “special” backups

• Test your entire technology infrastructure

• Train your alternates

• Evaluate vendor performance

• Make sure your “hot box” is current for vital records


Disaster Recovery Planning Is an Ongoing Process

Plan

ImplementTest

Business

Requirements

…To be prepared, you must

regularly go through the cycle


Remember to Consider All the Key Checkpoints

1. Recognize that disasters do happen. Know how your business units

may be impacted and understand their expectations

2. Ensure you have an actionable and repeatable DR plan

3. Confirm your DR plan supports your business

4. Backup: Put into practice what you say you will do

5. Build a recovery team

6. Keep your data stored safely offsite and know how to access it.

7. Practice smart virtualization

8. Test your plan, and then update the plan accordingly


StaffingStaffing

TestingTesting

Disaster RecoveryDisaster Recovery

RiskRisk

StandardizationStandardization

CareerCareer

Aging InfrastructureAging Infrastructure

GovernanceGovernance

What’s on the Mind of CIOs . .

DR Preparedness for 21st Century

Understand and are Ready

– Top Gun

Understand and are not Ready

– Sleepless in Seattle

Understand but don’t want DR

- Dumb and Dumber

Don’t Understand. Why Bother !

– Clueless or Armageddon

www.mcpressonline.com

62

Richard Dolewski

VP Business Continuity

[email protected]

www.velocity.cc

Do you have a DR plan in place: so, don't let a disaster defeat your business

Technology

Transcript of Do you have a DR plan in place: so, don't let a disaster defeat your business