Real Life Recovery:Real Life Recovery:Perspective, Preparation & Perspective, Preparation & PerformancePerformance

RMOUG Training Days ‘99RMOUG Training Days ‘99Daniel W. FinkDaniel W. Fink7S Consulting, Inc.7S Consulting, Inc.

IntroductionIntroduction

��Perspective Perspective Shifting the Paradigm

��PreparationPreparationDecisionDecision--making and Practicemaking and Practice

��PerformancePerformanceExecuting the recovery

PerspectivePerspective

TraditionalTraditional•• Focus is on Focus is on

backing up the backing up the databasedatabase

•• Technology Staff Technology Staff makes decisionsmakes decisions

•• If you care about If you care about data, use data, use archivingarchiving

NewNew•• Focus on properly Focus on properly

recovering a recovering a databasedatabase

•• Business staff Business staff makes decisionsmakes decisions

•• Backup & Backup & Recovery strategy Recovery strategy is a business is a business exerciseexercise

Cost Cost v.v. Lost Revenue Lost Revenue ModelModel

��Business processBusiness process��Balance Costs against RisksBalance Costs against Risks��Educated decisionEducated decision--makingmaking��Clear, documented understanding Clear, documented understanding

of what is possible, probable and of what is possible, probable and what is notwhat is not

Educate UsersEducate Users

�� Identify the appropriate decisionIdentify the appropriate decision--makersmakers

��Oracle 101Oracle 101•• Transactions and related structuresTransactions and related structures•• Types of backups, archiving and Types of backups, archiving and

recoveriesrecoveries

Set BoundariesSet Boundaries

��BudgetBudget•• TimeTime•• MoneyMoney

��Pick 2Pick 2•• GoodGood•• FastFast•• CheapCheap

��Allowable data loss/downtimeAllowable data loss/downtime

Service Level AgreementService Level Agreement

��Backup ProceduresBackup Procedures•• TypeType•• FrequencyFrequency•• Storage policiesStorage policies

��Data loss & downtimeData loss & downtime•• AllowableAllowable•• ExpectedExpected

��Anticipated ScenariosAnticipated Scenarios��Contact PointsContact Points

Backup PrinciplesBackup Principles

�� Never have 2 points of failureNever have 2 points of failure�� Never backup a file to the same physical Never backup a file to the same physical

device or controllerdevice or controller�� Always have at least 3 control filesAlways have at least 3 control files�� Redo Logs should be multiplexed at the Oracle Redo Logs should be multiplexed at the Oracle

levellevel�� Archived redo logs must be backed up to at Archived redo logs must be backed up to at

least 2 separate tapesleast 2 separate tapes�� OFA Installation eases backup managementOFA Installation eases backup management

BackupsBackups

��Offline (cold) Offline (cold) ��Online (hot)Online (hot)�� Logical (export)Logical (export)

Offline BackupOffline Backup

�� BenefitsBenefits•• Easy to implementEasy to implement•• Copy direct to Copy direct to

tapetape•• Can be performed Can be performed

as part of system as part of system backupbackup

•• Not all Not all tablespaces must tablespaces must be backed upbe backed up

�� Costs & RisksCosts & Risks•• Database is down Database is down

if process failsif process fails•• Cannot detect Cannot detect

datablock level datablock level corruptioncorruption

Online BackupOnline Backup

�� BenefitBenefit•• Required for 24x7 Required for 24x7

SystemsSystems•• Database is up if Database is up if

process failsprocess fails

�� Costs & RisksCosts & Risks•• Additional disk Additional disk

and processes and processes requiredrequired

•• More difficult to More difficult to implementimplement

•• Special care Special care required if required if database crashesdatabase crashes

•• Cannot detect Cannot detect datablock level datablock level corruptioncorruption

Logical BackupLogical Backup

�� BenefitsBenefits•• Easy extraction of Easy extraction of

individual objectsindividual objects•• Can expose data Can expose data

corruptioncorruption•• Useful for Useful for

upgrades, upgrades, database database copies/movescopies/moves

�� Costs & RisksCosts & Risks•• Cannot be used to Cannot be used to

recover recover transactionstransactions

•• Requires target Requires target database to insert database to insert objectsobjects

•• Cannot backup Cannot backup complete complete databasedatabase

No ArchivingNo Archiving

�� BenefitsBenefits•• Default state of Default state of

databasedatabase•• Minimal Minimal

management & management & performance performance impactimpact

�� Costs & RisksCosts & Risks•• All data since last All data since last

backup is lostbackup is lost•• Restoration Restoration

requires all requires all database filesdatabase files

ArchivingArchiving

�� BenefitsBenefits•• Transactions can Transactions can

be recoveredbe recovered•• Only affected files Only affected files

must be restoredmust be restored•• Required for Required for

Online Online backupsbackupsOptional for Optional for Offline backupsOffline backups

�� Costs & RisksCosts & Risks•• Increased Increased

managementmanagement•• Additional disk Additional disk

and processesand processes

RestoreRestore

��Copying files required for instance Copying files required for instance restart or database recoveryrestart or database recovery

�� If final state of recovery process, If final state of recovery process, committed transactions since last committed transactions since last backup are lostbackup are lost

RecoverRecover

��Apply committed transactions Apply committed transactions since last backupsince last backup•• Automatic Automatic -- using redo logsusing redo logs•• Manual Manual -- using keyboard or other using keyboard or other

source for datasource for data

��Database returned to state after Database returned to state after last backup/before failurelast backup/before failure

Rebuild & ReloadRebuild & Reload

��Using export or source data to Using export or source data to recreate the databaserecreate the database

��Appropriate for nonAppropriate for non--volatile volatile databasesdatabases

Practice, Practice, PracticePractice, Practice, Practice

��Only method of verifying backup Only method of verifying backup procedure effectiveness is to procedure effectiveness is to perform several different perform several different recoveriesrecoveries

��The first performance should not The first performance should not be on a live, down databasebe on a live, down database

��Titanic Syndrome Titanic Syndrome -- No backup is No backup is ‘unsinkable’‘unsinkable’

RecoveryRecovery

�� Follow established guidelinesFollow established guidelines�� Take time to do it right the first time, Take time to do it right the first time,

there may not be a second!there may not be a second!�� Have the Right people in the Right place Have the Right people in the Right place

at the Right timeat the Right time�� DoubleDouble--check and document each movecheck and document each move�� If time and circumstances permit, If time and circumstances permit,

backup database before attempting backup database before attempting recoveryrecovery

StopStop PanickingPanicking

��Calm is critical, don’t make a Calm is critical, don’t make a mistake by being mistake by being tootoo hastyhasty

��Accept the pressure of downtimeAccept the pressure of downtime��Refer to documented steps for Refer to documented steps for

each type of anticipated failureeach type of anticipated failure•• Loss of disk or other hardwareLoss of disk or other hardware•• Loss of datafile, redo log, archived log, Loss of datafile, redo log, archived log,

controlfilecontrolfile•• Loss of tableLoss of table

Identify Cause of FailureIdentify Cause of Failure

��Differentiate between symptoms Differentiate between symptoms and causesand causes

��Know how to determine if problem Know how to determine if problem is internal or external to Oracleis internal or external to Oracle

Correct Cause of FailureCorrect Cause of Failure

�� If the failed component can be If the failed component can be replaced, e.g. new disk, a spare replaced, e.g. new disk, a spare should be easily accessibleshould be easily accessible

��Be prepared to bypass the failed Be prepared to bypass the failed componentcomponent

��Restoring files to a bad disk will Restoring files to a bad disk will require another recoveryrequire another recovery

Restore affected filesRestore affected files

��Restore only those files to be Restore only those files to be recovered, this will minimize recovered, this will minimize downtimedowntime

�� If no archiving is being done, all If no archiving is being done, all files must be restoredfiles must be restored

Perform proper recoveryPerform proper recovery

��Cause of failure usually determines Cause of failure usually determines the type of recoverythe type of recovery

��Complete Complete -- up to the point of up to the point of failurefailure

�� Incomplete Incomplete -- prior to point of prior to point of failurefailure•• Required if hole exists in archive logsRequired if hole exists in archive logs•• Know when failure occurred for timeKnow when failure occurred for time--

basedbased

PostPost--mortemmortem

��Number 1 issue Number 1 issue -- What could have What could have been done to prevent failure and been done to prevent failure and minimize downtime and data loss?minimize downtime and data loss?

��Document recovery execution and Document recovery execution and critiquecritique

��Recovery caused by preventable Recovery caused by preventable scenario is a waste of valuable scenario is a waste of valuable timetime

ConclusionConclusion

��Shift the focusShift the focus��Practice, Practice, PracticePractice, Practice, Practice��Don’t PanicDon’t Panic

For more information,For more information,

www.orcldba.comwww.orcldba.com