April 13, 2007 1

Operational Recovery Planning

Presented by the California State Information Security Office

April 13, 2007 2

Agenda

Introductions – name and agency CA State Information Security Office Definitions Four Types of Continuity Plans Review of BL 07-03 – ORP Changes ORP-COOP/COG Alignment Discuss Test Scenarios

April 13, 2007 3

April 13, 2007 4

State Information Security Office

o Vision• Leading the way to secure the State's information assets.

o Mission• To manage security and operational recovery risk for the State's information assets by providing statewide direction and leadership.

April 13, 2007 5

Definitions

o Emergency Response

o Business Continuity Planning (BCP)

o Operational Recovery Planning (ORP)

o Continuity of Operations (COOP)

o Continuity of Government (COG)

April 13, 2007 6

Emergency Response

o The immediate reaction and response to an emergency situation commonly focusing on ensuring life safety and reducing the severity of the incident.

• Definition from Disaster Recovery Journal (DRI) website at: http://www.drj.com/glossary/

April 13, 2007 7

Business Continuity Planning (BCP)

o Process of developing and documenting arrangements and procedures that enable an organization to respond to an event that lasts for an unacceptable period of time and return to performing its critical functions after an interruption.

Similar terms:  business resumption plan, continuity plan, contingency plan, disaster recovery plan, recovery plan.

• Definition from Disaster Recovery Journal (DRI) website at: http://www.drj.com/glossary/

April 13, 2007 8

Operational Recovery Planning (ORP)

o The management approved document that defines the resources, actions, tasks and data required to manage the technology recovery effort.  Usually refers to the technology recovery effort.  This is a component of the Business Continuity Management Program. 

DISASTER RECOVERY PLAN (also known as – Operational Recovery Plan)

• Definition from Disaster Recovery Journal (DRI) website at: http://www.drj.com/glossary/

April 13, 2007 9

Continuity of Operations (COOP)

o The activities of individual departments and agencies and their sub-components to ensure that their essential functions are continued under all circumstances. This includes plans and procedures that delineate essential functions; specify succession to office and the emergency delegation of authority; provide for the safekeeping of vital records and databases; identify alternate operating facilities; provide for interoperable communications; and validate the capability through tests, training, and exercises.

• Office of Emergency Services (OES)

April 13, 2007 10

Continuity of Government (COG)

o The preservation, maintenance, or reconstitution of the institution of government. It is the ability to carry out an organization’s constitutional responsibilities. This is accomplished through succession of leadership, the pre-delegation of emergency authority and active command and control.

• Office of Emergency Services (OES)

April 13, 2007 11

Relationship of Plans

Continuity of Operations Continuity of Government

OperationalRecovery

EmergencyResponse

Business Continuity

April 13, 2007 12

Inter-Dependencies

April 13, 2007 13

Three Phases of Continuity

Emergency Response - Life Safety

First 72 Hours

Damage Assessme

ntFirst 72 hours

Restoration

Business

back to

normal

IT Operational Recovery

up to 30 daysBusinessRecoveryup to 30

days

Planning, Documenting, Testing, and Training

Departments

Phase I Phase II Phase III

April 13, 2007 14

IMPLEMENTATION OF PLANS

o Disruption of business occurs and you are informed, next steps

1. Emergency Response – safety and security of staff.

2. Securing the site.

3. Activate COOP/COG Plan to ensure the continuation of essential functions.

4. Implementation of the communication plan.

5. After assessing incident, determine if implementation of BCP & ORP is required.

6. Contact SISO to report incident.

7. Implement BCP and ORP

April 13, 2007 15

Budget Letter 07-03

o SAM Section 4843 – Operational Recovery Planning• Use results from risk analysis and

business impact analysis to identify critical business functions.

• Include the operational recovery considerations and costs in FSRs.

• Develop ORP as part of a complete continuity program.

April 13, 2007 16

Budget Letter 07-03 – Continued

o SAM Section 4843.1 – Agency Operational Recovery Plan

• Rewritten to clarify and enhance operational recovery requirements.

• Removal of minimum components from policy.• SIMM 65A – ORP Documentation for Agencies

Preparation Instructions• Requires ten minimum components in ORP.• Additional three components for agencies without

a BCP or COOP/COG.

April 13, 2007 17

ORP Documentation Revised

o Components to be included in the ORP – updated in January 2007.

o The April and July quarterly filers must provide a cover sheet indicating where the information for each topic area in SIMM 65A is located in the agency’s Operational Recovery Plan.

o All components listed in SIMM 65A must be

addressed and included in agencies’ ORPs beginning in October 2007.

April 13, 2007 18

Changes for ORP Development

Overall Requires more details

New Components Backup and offsite storage Data Center Services Contact information

Removed from SAM and Policy Damage Recognition Preparation of cost-benefit analysis Selection of alternative SIMM Section 140A

April 13, 2007 19

New Requirements

o ORPs must describe:1. Agency Administrative Information2. Critical Business Functions/Applications3. Recovery Strategy4. Backup and Offsite Storage Procedures5. Operational Recovery Procedures6. Data Center Services7. Resource Requirements8. Assignment of Responsibility9. Contact Information10.Testing

April 13, 2007 20

Supplemental Requirements

Agencies that have not developed and implemented a full business continuity plan or COOP/COG must also address and include the following in their ORP:

1. Damage Recognition and Assessment2. Mobilization of Personnel3. Primary Site Restoration and Relocation

April 13, 2007 21

Agency Administrative Information

A communication plan should include strategy on: How information will flow (escalation) Decision making processes Interrelationship among agency

resources for response, recovery and resumption

April 13, 2007 22

Example - Escalation Process

Single site, minor impact. User calls into Help Desk with possible virus infection. Communication Plan strategy includes:

• Process to dispatch field support to check PC

o If infected, take steps to identify and quarantine

• notify ISO and IT Management• Eradicate virus• Verify virus has not spread

April 13, 2007 23

What would you do?

Multiple site, major impact. The virus outbreak has spread from your headquarters to your remote offices and is running rampant. The anti-virus software will not eradicate it and all the systems in your agency are being impacted.

What would your communication plan need to include?

April 13, 2007 24

Communication Plan

Document Who to contact and under what circumstances Lists name, phone #, cell #, home #, email

address Includes Chain of Command Management,

other pertinent staff (ISO, ORP Coordinator, etc), and contractors

Distribute to applicable staff Providing training to staff Collect when duties change or staff leaves

April 13, 2007 25

Sample Call Lists

Wallet size cards: Name, work #, cell #, home #, email

Call Tree: Manager calls supervisor Supervisor calls his/her staff

April 13, 2007 26

Critical Business Functions/Applications

This section includes a description of: Critical business functions and their

supporting applications Maximum Allowable Outage (MAO) for

each application Recovery priorities

April 13, 2007 27

Example - Critical Business Function

Single site, minor impact. Help Desk identifies that the services on the email server are not working. As a critical business function, recovery strategy includes: Process for IT staff to check services

If denial of service, follow internal procedures to identify and mitigate.

Notify ISO and IT Management

April 13, 2007 28

What would you do?

Multiple site, major impact. The email server has crashed, there are both hardware and software failures. Rebuilding the server will require replacement hardware, which will take several days to acquire and configure.

What would your Critical Business Functions / Applications need to include?

April 13, 2007 29

Procedures for Critical Functions

Document Critical Business Functions Recovery Procedures Responsible individuals or team for

recovery

Distribute procedures to applicable staff Provide training

April 13, 2007 30

Sample Procedure

Repair/replace hardware Restore database structure Restore post office Restore gateway connectivity Rebuild database Keep users/management informed

April 13, 2007 31

Recovery Strategy

Recovery strategy should include alternate recovery site/sites that include: Location of all sites Requirements of facilities/equipment Contact numbers

April 13, 2007 32

What would you do?

Single site, minor impact: Your department is located in several locations. A building adjacent to one location has a fire, the fire did not spread to your site. The Fire Dept and Law Enforcement block the street, so there is no access into your building.

What would your recovery strategy need to include?

April 13, 2007 33

Recovery Strategy

Communication plan for employees, management, and contractors.

List all office locations.

Identify the alternate location. If multiple locations are available, prioritize them.

Address what functions could be restored at each site.

Determine who would need to be called, include as the contact list.

April 13, 2007 34

Sample Recovery Strategy

Department has three locations: 1234 Headquarters St., Sacto, 95814 5678 Anywhere St., Sacto, 95825 9876 SomePlace St., LA 90210

Critical operations would be restored at an unaffected site (identify priority and equipment needed).

Contact: J Resto at (916) 555-1212 for Headquarters R Quick at (916) 444-1212 for Anywhere M Pia at (213) 555-1212 for SomePlace

April 13, 2007 35

Backup and Offsite Storage

The backup and offsite storage procedures should include: Retention schedule Procedures List of authorized staff Account information Contacts of offsite storage

April 13, 2007 36

What would you do?

The data on one of your critical applications was corrupted and its MAO is 4 hours. It is 5:30 pm on Friday and Monday is a holiday. The business area have staff scheduled to work Saturday on this system. Technical staff has gone home, and several are out of town for the weekend.

What would your backup and offsite storage procedures need to include?

April 13, 2007 37

Details – Backup and Offsite Storage

Document: Retention schedule Detailed procedures

Hardware and software (include version) Offsite storage details (location, acct #) Retrieval of backups (contacts (24x7) and

personnel authorized to retrieve) Process to identify data to be restored

April 13, 2007 38

Operational Recovery Procedures

These procedures systematically detail the operational procedures for recovery in a timely and orderly way, they should include: Detailed procedures that the backup or

other IT professional could follow High-level network diagram that includes

all critical applications

April 13, 2007 39

Data Center Services

This section should include a: Description of service to be provided. Interagency agreements,

memorandums of understanding, or contracts.

Specific coordination efforts with the data center critical to the recovery efforts.

April 13, 2007 40

Example – Minor Impact

Single site, minor impact. Your Web server providing access to one of your critical applications located at DTS has been compromised. You have contacted DTS and DTS is working to get the server back online within the hour.

What would your need to include?

April 13, 2007 41

What would you do?

Multiple site, major impact. There was a fire in a facility adjoining DTS facility where the servers are housed. The sprinkler system was activated and the servers had to be powered down. There is significant water damage. There is an estimate that it will take 14 to 21 days to reestablish services.

What would your plan need to include?

April 13, 2007 42

Details - Data Center Services

Expectations Meet with Data Center to identify

Hardware/Software requirements Services required Timeframe for services

Document Agreement – Before it’s needed Create a Service Level Agreement (SLA) or

Memorandum of Understanding (MOU) Develop Recovery Procedures

April 13, 2007 43

Resource Requirements

This is a comprehensive list of: Equipment Software Telecommunication needs Data Hard copy manuals Personnel essential for recovery

April 13, 2007 44

Assignment of Responsibility

Designation of responsibilities and assignments should be listed. Procedures should include job title, and not individual names, for the recovery process.

Individuals names can be placed in a single location for ease of maintenance.

April 13, 2007 45

Contact Information

There are two types of contact information to be collected: Employees, including management.

Resource List including contractors, Major Service providers, vendors, other government entities, and outside resources critical to the recovery process.

April 13, 2007 46

Contact List

Employee contact information should be designated as sensitive, and provided to authorized individuals.

Resource lists typically have business contact information. This information can be provided more widely.

April 13, 2007 47

Testing

Annual testing of the ORP is essential to: Ensure for training the management

and recovery teams. Validate that the procedures have the

appropriate level of detail. Verify Call Back lists are current. Confirm that Recovery strategies are

appropriate for your environment.

April 13, 2007 48

Governor’s Office of Emergency Services

Introduction

Mission and Goals of OES

SEMS/NIMS

Disaster Service Worker

April 13, 2007 49

Planning

Be Smart, Be Responsible. Be Prepared. Get Ready Campaign

Your Intranets and Emergency Preparedness

Executive Order S-04-06

State Emergency Plan / COOP-COG/ORP

April 13, 2007 50

Training and Testing

Emergency Management Training Requirements for Public Employees

The California Specialized Training Institute (CSTI)/OES Training Branch

How to develop a Table Top Exercise (TTex)

Definition of a TTex The 8 Step Process Used to Design a TTex

After Action/ Corrective Action Process

California Master Exercise Calendar (CMEX)

April 13, 2007 51

State IT Strategic Plan Action Item

To align the ORP and COOP/COG, a work group has been established to:

review processes

define terminology

evaluate reporting requirements

April 13, 2007 52

Resources

SISO web site: http://www.infosecurity.ca.gov/ORP/

Budget Letter 07-03 – ORP Policy Changeshttp://www.dof.ca.gov/OTROS/StatewideIT/IT_BdgtLttrs.asp

ORP Policy in the State Administrative Manual (SAM):Operational Recovery Planning: http://sam.dgs.ca.gov/TOC/4800/4843.htmOperational Recovery Plan http://sam.dgs.ca.gov/TOC/4800/4843.1.htm

ORP – SIMM 65A: http://www.infosecurity.ca.gov/Policy/

April 13, 2007 53

Contact Us

Rosa.Umbach@dof.ca.gov(916) 445-1777 ext. 3242

Colleen.Pedroza@dof.ca.gov(916) 445-1777 ext. 3224

SISO Office:email: security@dof.ca.govTelephone: (916) 445-5239www.infosecurity.ca.gov