A Disaster Planning A Disaster Planning Case StudyCase Study

For PALnet's joint Integrated Library System

Stephanie John, PALnet

Mark Evans, PALnet

Roy Teahen, Baker College

TODAY’S GOALSTODAY’S GOALS

• Something for every level• Examples of inter-agency

cooperation• Real-life examples to tickle your

brain• Concerns unique to e-libraries• Learning from each other

NOT INCLUDED: IT basics like backups, log analysis, antivirus

Communication disastersCommunication disasters

• 70% - by people cable cuts, work errors, vandalism &

sabotage

15% - equipment failure software, power loss, circuit board failures

• 10% - environment related loss of a/c, excessive humidity

• >5% - natural causes tornadoes, flood, fire & lightning

Top 10 causes of data lossTop 10 causes of data loss

Cause Weight

1 Mechanical hard drive failure 25

2 Data structure corruption 17

3 Accidental or intentional data deletion 13

4 RAID server failure 10

5 Backup tape malfunction 8

6 Physical tape damage 6

7 Accidental overwrites 6

8 Software corruption 5

9 Viruses 5

10 Natural disasters 4

The consequences of failureThe consequences of failure

Trust

Perceived competency and value

Goodwill – the ‘emotional bank account’

Revenues

Faculty time and effort

‘Votes’ at budget time

What is PALnet?What is PALnet?

Baker CollegeKettering University

Mott Community College

PALnet Governing Board

PALnet Director

PALnet Advisory Committee(library directors)

recommendations

Office documents server, private and DMZ backup server, and helpdesk server.

Web Reporter. Statistics Main PALnet server Runs all business functions of libraries, catalogs. Horizon Information Portal. Provides public catalogs – 32+ interfaces. Multifunction: performs DNS, email, proxying for online resources, the PALnet website, and other functions. Remote Patron Authentication. Access to databases, ebooks.

FLPOP Router

Merit

Legend Router

Location Router

Switch

PALNET Equipment

Firewall Server

Global Xubg bkup

12 Mbps

The Internet

DS3 link

KTU

MCC

Web Reporter Server

PALnet Switch

DMZ Switch

Firewall Server

Horizon Server

HIP Server

Netbox (DNS) Server

RPA Server

Baker Router

PALNET Office Server

Private Switch

Playing well with othersPlaying well with others

• Baker CIS

• Mott ET and IT

• Kettering IT

• Lapeer Public Library

InsuranceInsurance

• Read EXCLUSIONS carefully– Usual: $0 for data reconstruction

• Data insurance

• Business interruption insurance

• Business continuity coverage– Pay vendor for ASP hosting?

– Pay for move to backup site?

• Principle of mitigation

• Discounts available for good prep

Hardware maintenance contractsHardware maintenance contracts

• Single item based response time– 24/7/365 for critical– 4 or 8 hour for others

• Service person permanently assigned to your account

• System monitoring optional• Parts and spares kept locally• Site moves (post-disaster)

Start SIMPLEStart SIMPLE

• Paper and online phone trees– PALnet staff– Building staff– Agency directors– Baker CIS, Security– MCC and KTU IT Adv Comm members– Baker Clinton Twp director contact

information

• Service bureau• Major vendor (SirsiDynix)• OCLC, MELCat, other partners

Identify prioritiesIdentify priorities

Your needs AND your users’ needs• Firewall data• Horizon database• Horizon server & OS• Remote Patron Authentication (RPA)

data• Online Catalog (HIP) configuration

EvacuationEvacuation

Take: • Backup tapes• Laptops• Paper licensing file• Vendor contract(s)• Service Express agreement and

contact info• Network diagram• Contact info: personnel, insurance,

consortia

Mirrored data on remoteMirrored data on remote

• IP range data for all agencies• Password list for systems• Firewall info• Most recent images of servers• Photos & inventory of all equipment,

furnishings• Photos of offices, server room• Copies of contact lists• Copy of:

http://www.techsoup.org/katrina/ITRecoveryManual_vI.pdf

Where will you move?Where will you move?

1. Baker College Flint Library

2. Baker College Auburn Hills campus – Library

FireFire

• Where is your extinguisher?

• Where is your alarm?

• Is there a special phone #?

• PRACTICE

SuppliesSupplies

• Tool kit(s)• Wrist strap(s) for static• Plastic sheeting• Trash bags• Fans• USB drives – easy to share

common files before network restored• FLASHLIGHTS

PowerPower

• UPS protection• Backup generators & testing

A/C and temp monitoringA/C and temp monitoring

• A/C unit evaluation• Temp alarms• Temp monitoring• Auto shut-down for temp alerts• Campus security and alarm

monitoring

Backup Internet Backup Internet

• Backup Internet connections• Global Crossing• Backup fiber ring• Router reconfigurations

Baker WANBaker WAN

Cadillac

Clinton TwpFlint

Muskegon

Fremont

Jackson

Coldwater Auburn Hills

Port Huron

Cass City

Sandusky

Allen Park

Owosso

West Branch

InternetInternet

InternetInternet

InternetInternet

Unique to librariesUnique to libraries

• Licensing of e-books: Choose vendors carefully

• Alternate access to databases for when auto-authentication down RPA, Ezproxy, squid

• RPA can authenticate against a flat file, output from db

• Offline circulation software• Partnerships/collection sharing

ILS vendorsILS vendors

• ILS = Integrated Library System• Vendors or 3rd parties offer service

bureaus• Use service bureau as backup

facility• Establish basic profile, test backup

tapes• Run circulation from remote vendor

site until local server restored

Single site disasterSingle site disaster

• Closing a single library site– Stop fines– Keep materials out of shared

catalog– Suspend overdues & bills

• Verify items are not showing in public catalog

• Holds disallowed

Other ResourcesOther Resources

• EZProxy URL rewriterhttp://www.usefulutilities.com/

• Remote User Auth in Librarieshttp://library.smc.edu/rpa.htm

Nagios service and network monitoringhttp://www.nagios.org/

Other ResourcesOther Resources

• Disaster Recover Journal’s Sample Plans and Outlines

http://www.drj.com/new2dr/samples.htm• University and Library Disaster Planshttp://palimpsest.stanford.edu/bytopic/

disasters/plans/TechSoup:Restoring IT Infrastructurehttp://www.techsoup.org/katrina/

ITRecoveryManual_vI.pdf