Lab13 ASM and Rman Crash Restore and Recovery Scenarios

ASM HANDS-ON TRAININGLab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Alejandro Vargas | Principal Support ConsultantOracle Advanced Customer Services

INDEXSummary ....................................................................................................................................................................... 3 Setup Archivelog Mode ................................................................................................................................................. 3 Full Rman Backup ......................................................................................................................................................... 5 Enable change tracking file ........................................................................................................................................... 6 Make A Full Incremental Hot Backup ............................................................................................................................ 6 Checking the Backup .................................................................................................................................................. 11 Crash Restore and Recovery Scenarios ..................................................................................................................... 13 System tablespace loss ............................................................................................................................................... 13 Restore And Recover System Tablespace ................................................................................................................. 14 User datafile loss ......................................................................................................................................................... 18 .................................................................................................................................................................................... 21 Restore and Recover users tablespace ...................................................................................................................... 21 Online redologs loss .................................................................................................................................................... 25 .................................................................................................................................................................................... 27 Restore and Recover from Online Redolog loss ......................................................................................................... 27 Restore And Recover From Controlfile Loss .............................................................................................................. 38 Total Database Loss .................................................................................................................................................. 42 Restore And Recover From Total Loss ....................................................................................................................... 44 Scripts .......................................................................................................................................................................... 52 backup-incr-lev0-plus-recover.rmn .............................................................................................................................. 52 1-system-tablespace-loss ............................................................................................................................................ 54 1-restore-from-system-tablespace-loss ....................................................................................................................... 55 2-user-datafile-loss ...................................................................................................................................................... 56 2-restore-from-user-datafile-loss ................................................................................................................................. 58 3-online-redolog-loss ................................................................................................................................................... 59 3-restore-and-recover-from-online-redo-loss .............................................................................................................. 61 4-restore-and-recover-from-controlfile-loss ................................................................................................................. 62

1/67


5-total-database-loss ................................................................................................................................................... 63 5-restore-and-recover-from-total-database-loss ......................................................................................................... 65

2/67


Alejandro Vargas | Principal Support ConsultantOracle Advanced Customer Services

Summary

On this Lab we will review the following crash and recovery scenarios, and we will implement some of them:

• System tablespace loss • Recover system tablespace• User datafile loss • Recover users tablespace• Online redo loss • Recover from redo loss• Controlfile loss • Recover from controlfile loss• Database loss • Recover from total loss

Setup Archivelog ModeThe tests wil be done using archivelog mode and hot backups, then we will restart the database on archivelog mode.

[oracle@asmxpt app]$ ora-------------ORACLE_BASE=/u01/app/oracleORACLE_HOME=/u01/app/oracle/11g_dbORACLE_SID=sati11-------------oracle 6462 1 0 09:58 ? 00:00:06 ora_smon_sati11oracle 8904 1 0 Feb12 ? 00:00:02 asm_smon_+ASM

3/67


-------------oracle 8693 1 0 Feb12 ? 00:00:18 /u01/app/oracle/11g_db/bin/tnslsnr LISTENER_ASMXPT -inherit-------------[oracle@asmxpt app]$ sqlSQL*Plus: Release 11.1.0.7.0 - Production on Fri Feb 13 14:17:45 2009

Copyright (c) 1982, 2008, Oracle. All rights reserved.

Connected to:Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> shutdown immediate;Database closed.Database dismounted.ORACLE instance shut down.SQL> startup mount;ORACLE instance started.

Total System Global Area 351522816 bytesFixed Size 1313344 bytesVariable Size 268436928 bytesDatabase Buffers 75497472 bytesRedo Buffers 6275072 bytesDatabase mounted.SQL> alter database archivelog;Database altered.

4/67


SQL> alter database open;Database altered.

SQL> archive log listDatabase log mode Archive ModeAutomatic archival EnabledArchive destination USE_DB_RECOVERY_FILE_DESTOldest online log sequence 1Next log sequence to archive 1Current log sequence 1

SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options

Full Rman BackupHaving the database set on archivelog mode we will make a full backup to the flash recovery area

SQL> show parameters recovNAME TYPE VALUE------------------------------------ ----------- ------------------------------db_recovery_file_dest string +FRADGdb_recovery_file_dest_size big integer 6048Mrecovery_parallelism integer 0

5/67


Enable change tracking fileTo be able to make incremental backups that will quickly find only the changed blocks we will enable change tracking file.

SQL> ALTER DATABASE ENABLE BLOCK CHANGE TRACKING USING FILE '+fradg';Database altered.

SQL> select * from v$BLOCK_CHANGE_TRACKING ;STATUS FILENAME BYTES---------- -------------------------------------------------- ----------ENABLED +FRADG/sati11/changetracking/ctf.259.678750225 11599872

Make A Full Incremental Hot BackupThis backlup script is intended to make a backup that will be automatically updated with the last changes each time the script is invoked. The first time the script is called there is no level 0 backup to update so a full level 0 backup is created.The second time all changed blocks since the previous backup are backed upEvery other execution the last incremental is applied to the backup and a new incremental is created

RUN{ALLOCATE CHANNEL disk1 DEVICE TYPE DISK FORMAT '+FRADG/%U';ALLOCATE CHANNEL disk2 DEVICE TYPE DISK FORMAT '+FRADG/%U';RECOVER COPY OF DATABASEWITH TAG 'INCREMENTAL_DAILY_UPDATED' ;BACKUPINCREMENTAL LEVEL 1FOR RECOVER OF COPY WITH TAG 'INCREMENTAL_DAILY_UPDATED'

6/67


DATABASE FORMAT '+FRADG/%U'PLUS ARCHIVELOG FORMAT '+FRADG/%U';}

[oracle@asmxpt ~]$ rman target /Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 04:07:31 2009


connected to target database: SATI11 (DBID=124143874)

RMAN> @backup-incr-lev0-plus-recover.rmnRMAN> RUN2> {3> ALLOCATE CHANNEL disk1 DEVICE TYPE DISK ;4> ALLOCATE CHANNEL disk2 DEVICE TYPE DISK ;5> ALLOCATE CHANNEL disk3 DEVICE TYPE DISK ;6> ALLOCATE CHANNEL disk4 DEVICE TYPE DISK ;7> RECOVER COPY OF DATABASE8> WITH TAG 'INCREMENTAL_DAILY_UPDATED' ;9> BACKUP INCREMENTAL LEVEL 110> FOR RECOVER OF COPY WITH TAG 'INCREMENTAL_DAILY_UPDATED'11> DATABASE PLUS ARCHIVELOG ;12> }using target database control file instead of recovery catalogallocated channel: disk1channel disk1: SID=130 device type=DISK

7/67


allocated channel: disk2channel disk2: SID=131 device type=DISK



Starting recover at 14-FEB-09no copy of datafile 1 found to recoverno copy of datafile 2 found to recoverno copy of datafile 3 found to recoverno copy of datafile 4 found to recoverFinished recover at 14-FEB-09

Starting backup at 14-FEB-09current log archivedchannel disk1: starting archived log backup setchannel disk1: specifying archived log(s) in backup setinput archived log thread=1 sequence=2 RECID=2 STAMP=678773278channel disk1: starting piece 1 at 14-FEB-09channel disk2: starting archived log backup setchannel disk2: specifying archived log(s) in backup setinput archived log thread=1 sequence=1 RECID=1 STAMP=678751046channel disk2: starting piece 1 at 14-FEB-09channel disk1: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t040758_0.260.678773279 tag=TAG20090214T040758 comment=NONEchannel disk1: backup set complete, elapsed time: 00:00:16

8/67


channel disk2: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t040758_0.261.678773281 tag=TAG20090214T040758 comment=NONEchannel disk2: backup set complete, elapsed time: 00:00:16Finished backup at 14-FEB-09

Starting backup at 14-FEB-09no parent backup or copy of datafile 1 foundno parent backup or copy of datafile 2 foundno parent backup or copy of datafile 3 foundno parent backup or copy of datafile 4 foundchannel disk1: starting datafile copyinput datafile file number=00001 name=+DATADGNR/sati11/datafile/system.259.678706167channel disk2: starting datafile copyinput datafile file number=00002 name=+DATADGNR/sati11/datafile/sysaux.261.678706169channel disk3: starting datafile copyinput datafile file number=00003 name=+DATADGNR/sati11/datafile/undotbs1.260.678706169channel disk4: starting datafile copyinput datafile file number=00004 name=+DATADGNR/sati11/datafile/users.268.678706171output file name=+FRADG/sati11/datafile/users.265.678773315 tag=INCREMENTAL_DAILY_UPDATED RECID=1 STAMP=678773321channel disk4: datafile copy complete, elapsed time: 00:00:26output file name=+FRADG/sati11/datafile/undotbs1.264.678773305 tag=INCREMENTAL_DAILY_UPDATED RECID=2 STAMP=678773528channel disk3: datafile copy complete, elapsed time: 00:03:53output file name=+FRADG/sati11/datafile/system.262.678773299 tag=INCREMENTAL_DAILY_UPDATED RECID=4 STAMP=678773597channel disk1: datafile copy complete, elapsed time: 00:05:05output file name=+FRADG/sati11/datafile/sysaux.263.678773303 tag=INCREMENTAL_DAILY_UPDATED RECID=3 STAMP=678773597channel disk2: datafile copy complete, elapsed time: 00:05:05

9/67


Finished backup at 14-FEB-09

Starting backup at 14-FEB-09current log archivedchannel disk1: starting archived log backup setchannel disk1: specifying archived log(s) in backup setinput archived log thread=1 sequence=3 RECID=3 STAMP=678773603channel disk1: starting piece 1 at 14-FEB-09channel disk1: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t041323_0.267.678773605 tag=TAG20090214T041323 comment=NONEchannel disk1: backup set complete, elapsed time: 00:00:01Finished backup at 14-FEB-09

Starting Control File and SPFILE Autobackup at 14-FEB-09piece handle=+FRADG/sati11/autobackup/2009_02_14/s_678773606.268.678773609 comment=NONEFinished Control File and SPFILE Autobackup at 14-FEB-09released channel: disk1released channel: disk2released channel: disk3released channel: disk4

RMAN> **end-of-file**

10/67


Checking the Backup

RMAN> list backupset;using target database control file instead of recovery catalog

List of Backup Sets===================

BS Key Size Device Type Elapsed Time Completion Time------- ---------- ----------- ------------ ---------------6 362.00K DISK 00:00:00 14-FEB-09 BP Key: 6 Status: AVAILABLE Compressed: NO Tag: TAG20090214T041323 Piece Name: +FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t041323_0.267.678773605

List of Archived Logs in backup set 6 Thrd Seq Low SCN Low Time Next SCN Next Time ---- ------- ---------- --------- ---------- --------- 1 3 612605 14-FEB-09 613088 14-FEB-09

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------7 Full 9.36M DISK 00:00:03 14-FEB-09 BP Key: 7 Status: AVAILABLE Compressed: NO Tag: TAG20090214T041326 Piece Name: +FRADG/sati11/autobackup/2009_02_14/s_678773606.268.678773609 SPFILE Included: Modification time: 14-FEB-09 SPFILE db_unique_name: SATI11 Control File Included: Ckp SCN: 613098 Ckp time: 14-FEB-09

RMAN> list archivelog all;

11/67


List of Archived Log Copies for database with db_unique_name SATI11=====================================================================

Key Thrd Seq S Low Time------- ---- ------- - ---------3 1 3 A 14-FEB-09 Name: +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603

Checking the backup directories on ASM

[oracle@asmxpt sati11]$ asmcmd ls -l fradg/sati11/AUTOBACKUP/2009_02_14/Type Redund Striped Time Sys NameAUTOBACKUP UNPROT COARSE FEB 14 04:00:00 Y s_678773606.268.678773609

[oracle@asmxpt sati11]$ asmcmd ls -l fradg/sati11/BACKUPSET/2009_02_14/Type Redund Striped Time Sys NameBACKUPSET UNPROT COARSE FEB 14 04:00:00 Y annnf0_TAG20090214T041323_0.267.678773605

12/67


Crash Restore and Recovery Scenarios

System tablespace lossOn the first part of this scenario we will crash the database and remove the system tablespace, we will try to get the database up…The scenario is run using the script :

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/1-system-tablespace-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./1-system-tablespace-lossGenerating database crash ...

System Tablespace Datafile is +DATADGNR/sati11/datafile/system.259.678779875

ORACLE instance shut down.

Generating system tablespace loss ...State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks NameMOUNTED NORMAL N 512 4096 1048576 8188 3970 2047 961 0 DATADGNR/MOUNTED EXTERN N 512 4096 1048576 6141 4401 0 4401 0 FRADG/

SYSAUX.261.678706169SYSTEM.259.678779875UNDOTBS1.260.678706169USERS.268.678706171

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name

13/67


MOUNTED NORMAL N 512 4096 1048576 8188 5374 2047 1663 0 DATADGNR/MOUNTED EXTERN N 512 4096 1048576 6141 4401 0 4401 0 FRADG/

SYSAUX.261.678706169UNDOTBS1.260.678706169USERS.268.678706171

Trying to restart the database after the crash ...

ORACLE instance started.

Total System Global Area 351522816 bytesFixed Size 1313344 bytesVariable Size 272631232 bytesDatabase Buffers 71303168 bytesRedo Buffers 6275072 bytesDatabase mounted.ORA-01157: cannot identify/lock data file 1 - see DBWR trace fileORA-01110: data file 1: '+DATADGNR/sati11/datafile/system.259.678779875'

Restore And Recover System TablespaceThe restore of a lost system tablespace is simple if we have a good backup; it is implemented running the following commands:

• Restore datafile 1;• Recover database;

The restore and recover from system tablespace loss is run using this script:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/1- restore-from-system-tablespace-loss

14/67


[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./1-restore-from-system-tablespace-loss

Executing Command : RESTORE and RECOVER SYSTEM DATAFILE

Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 06:30:37 2009


connected to target database: SATI11 (DBID=124143874, not open)

RMAN>Starting restore at 14-FEB-09using target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=152 device type=DISK

channel ORA_DISK_1: restoring datafile 00001input datafile copy RECID=4 STAMP=678773597 file name=+FRADG/sati11/datafile/system.262.678773299destination for restore of datafile 00001: +DATADGNR/sati11/datafile/system.259.678779875channel ORA_DISK_1: copied datafile copy of datafile 00001output file name=+DATADGNR/sati11/datafile/system.259.678781847 RECID=0 STAMP=0Finished restore at 14-FEB-09

RMAN>Starting recover at 14-FEB-09using channel ORA_DISK_1

starting media recovery

15/67


archived log for thread 1 with sequence 3 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603archived log for thread 1 with sequence 4 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175archived log for thread 1 with sequence 5 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567archived log for thread 1 with sequence 6 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_6.257.678780233archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603 thread=1 sequence=3archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175 thread=1 sequence=4media recovery complete, elapsed time: 00:00:06Finished recover at 14-FEB-09

RMAN>database opened

RMAN>

Recovery Manager complete.

SQL*Plus: Release 11.1.0.7.0 - Production on Sat Feb 14 06:33:11 2009



16/67


SQL>FILE_NAME--------------------------------------------------------------------------------+DATADGNR/sati11/datafile/system.259.678781847+DATADGNR/sati11/datafile/sysaux.261.678706169+DATADGNR/sati11/datafile/undotbs1.260.678706169+DATADGNR/sati11/datafile/users.268.678706171

SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options

17/67


User datafile loss

On this scenario the database does not crash, we will set the users tablespace offline to be able to remove it from ASM. Restore and recovery can also take place while the database is open.When the application is executed the following error is returned:

ERROR at line 1:ORA-00376: file 4 cannot be read at this timeORA-01110: data file 4: '+DATADGNR/sati11/datafile/users.268.678706171'

The scenario is run using this script :

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/2-user-datafile-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./2-user-datafile-lossPreparing User Application ...

User dropped.User created.Grant succeeded.Table created.31 rows updated.Commit complete.

USERNAME------------------------------CUSTOMER_SYSTEMCUSTOMER_SYSCUSTOMER_MGMT_VIEW

18/67


CUSTOMER_SYSMANCUSTOMER_DBSNMPCUSTOMER_APPSCUSTOMER_OUTLNCUSTOMER_WKSYSCUSTOMER_OLAPSYSCUSTOMER_SI_INFORMTN_SCHEMACUSTOMER_OWBSYSCUSTOMER_FLOWS_030000CUSTOMER_ORDPLUGINSCUSTOMER_WKPROXYCUSTOMER_XDBCUSTOMER_ANONYMOUSCUSTOMER_CTXSYSCUSTOMER_WK_TESTCUSTOMER_WMSYSCUSTOMER_EXFSYSCUSTOMER_ORDSYSCUSTOMER_MDSYSCUSTOMER_FLOWS_FILESCUSTOMER_SPATIAL_WFS_ADMIN_USRCUSTOMER_SPATIAL_CSW_ADMIN_USRCUSTOMER_APEX_PUBLIC_USERCUSTOMER_DIPCUSTOMER_MDDATACUSTOMER_XS$NULLCUSTOMER_TSMSYSCUSTOMER_ORACLE_OCM

31 rows selected.

19/67


Generating user datafile remove ...Setting tablespace users offline ...Tablespace altered.

Removing tablespace users datafile ...State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks NameMOUNTED NORMAL N 512 4096 1048576 8188 3970 2047 961 0 DATADGNR/MOUNTED EXTERN N 512 4096 1048576 6141 4374 0 4374 0 FRADG/

SYSAUX.261.678706169SYSTEM.259.678781847UNDOTBS1.260.678706169USERS.268.678706171

SYSAUX.261.678706169SYSTEM.259.678781847UNDOTBS1.260.678706169State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks NameMOUNTED NORMAL N 512 4096 1048576 8188 3982 2047 967 0 DATADGNR/MOUNTED EXTERN N 512 4096 1048576 6141 4374 0 4374 0 FRADG/

Checking application ...select username from apps.customers *ERROR at line 1:ORA-00376: file 4 cannot be read at this timeORA-01110: data file 4: '+DATADGNR/sati11/datafile/users.268.678706171'

20/67


Restore and Recover users tablespace

To restore and recover the users tablespace it is not necessary to bring the database down. The commands used in this case from within Rman are:

• Restore tablespace users;• Recover tablespace users;• Alter tablespace users online

This script was used to execute the restore and recovery process:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/2-restore-from-user-datafile-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./2-restore-from-user-datafile-lossRestoring and Recovering Tablespace Users ...





21/67


channel ORA_DISK_1: restoring datafile 00004input datafile copy RECID=1 STAMP=678773321 file name=+FRADG/sati11/datafile/users.265.678773315destination for restore of datafile 00004: +DATADGNR/sati11/datafile/users.268.678706171channel ORA_DISK_1: copied datafile copy of datafile 00004output file name=+DATADGNR/sati11/datafile/users.268.678797585 RECID=0 STAMP=0Finished restore at 14-FEB-09

RMAN>Starting recover at 14-FEB-09using channel ORA_DISK_1


archived log for thread 1 with sequence 3 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603archived log for thread 1 with sequence 4 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175archived log for thread 1 with sequence 5 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567archived log for thread 1 with sequence 6 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_6.257.678780233archived log for thread 1 with sequence 7 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_7.256.678781955archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603 thread=1 sequence=3archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175 thread=1 sequence=4archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567 thread=1 sequence=5

22/67


media recovery complete, elapsed time: 00:00:05Finished recover at 14-FEB-09

RMAN>sql statement: alter tablespace users online

RMAN>


Checking application ...

USERNAME------------------------------CUSTOMER_SYSTEMCUSTOMER_SYSCUSTOMER_MGMT_VIEWCUSTOMER_SYSMANCUSTOMER_DBSNMPCUSTOMER_APPSCUSTOMER_OUTLNCUSTOMER_WKSYSCUSTOMER_OLAPSYSCUSTOMER_SI_INFORMTN_SCHEMACUSTOMER_OWBSYSCUSTOMER_FLOWS_030000CUSTOMER_ORDPLUGINSCUSTOMER_WKPROXYCUSTOMER_XDBCUSTOMER_ANONYMOUSCUSTOMER_CTXSYS

23/67


CUSTOMER_WK_TESTCUSTOMER_WMSYSCUSTOMER_EXFSYSCUSTOMER_ORDSYSCUSTOMER_MDSYSCUSTOMER_FLOWS_FILESCUSTOMER_SPATIAL_WFS_ADMIN_USRCUSTOMER_SPATIAL_CSW_ADMIN_USRCUSTOMER_APEX_PUBLIC_USERCUSTOMER_DIPCUSTOMER_MDDATACUSTOMER_XS$NULLCUSTOMER_TSMSYSCUSTOMER_ORACLE_OCM

31 rows selected.

24/67


Online redologs loss

In this step we will simulate a database crash that led to loss the online redologs; when the database is restarted the following errors are displayed:

ORA-00313: open failed for members of log group 1 of thread 1ORA-00312: online log 1 thread 1:'+DATADGNR/sati11/onlinelog/group_1.269.678706577'ORA-00312: online log 1 thread 1:'+DATADGNR/sati11/onlinelog/group_1.262.678706585'

This script was used to simulate the online redolog loss:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/3-online-redolog-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./3-online-redolog-lossGenerating database crash ...

asmcmd rm +DATADGNR/sati11/onlinelog/group_1.269.678706577asmcmd rm +DATADGNR/sati11/onlinelog/group_1.262.678706585

Database log mode Archive ModeAutomatic archival EnabledArchive destination USE_DB_RECOVERY_FILE_DESTOldest online log sequence 7Next log sequence to archive 9Current log sequence 9

SEQUENCE#

25/67


---------- 8

GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- 1 1 7 52428800 2 YES INACTIVE 676896 14-FEB-09 2 1 8 52428800 2 YES INACTIVE 708745 14-FEB-09 3 1 9 52428800 2 NO CURRENT 747092 14-FEB-09

Database closed.Database dismounted.ORACLE instance shut down.

Removing online redo log group ...

Starting database after the crash ...


Total System Global Area 351522816 bytesFixed Size 1313344 bytesVariable Size 272631232 bytesDatabase Buffers 71303168 bytesRedo Buffers 6275072 bytesDatabase mounted.ORA-00313: open failed for members of log group 1 of thread 1ORA-00312: online log 1 thread 1:'+DATADGNR/sati11/onlinelog/group_1.269.678706577'ORA-00312: online log 1 thread 1:'+DATADGNR/sati11/onlinelog/group_1.262.678706585'

26/67


Restore and Recover from Online Redolog lossIn this step we perform the restore and recovery of the database when online redologs are missing.

When a redolog is missing a full database restore must be done.

The script will request the sequence # to recover using the until sequence clause. This sequence can be obtained from the database alert.log

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./3-restore-and-recover-from-online-redo-loss

Executing Full Database Restore ...

Please check the last archived sequence of the database.please enter sequence number to restore to ...8



connected to target database: SATI11 (DBID=124143874, not open)

RMAN>database is already started

RMAN>Starting restore at 14-FEB-09using target database control file instead of recovery catalog

27/67


allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=145 device type=DISK

channel ORA_DISK_1: restoring datafile 00001input datafile copy RECID=4 STAMP=678773597 file name=+FRADG/sati11/datafile/system.262.678773299destination for restore of datafile 00001: +DATADGNR/sati11/datafile/system.259.678781847channel ORA_DISK_1: copied datafile copy of datafile 00001output file name=+DATADGNR/sati11/datafile/system.259.678781847 RECID=0 STAMP=0channel ORA_DISK_1: restoring datafile 00002input datafile copy RECID=3 STAMP=678773597 file name=+FRADG/sati11/datafile/sysaux.263.678773303destination for restore of datafile 00002: +DATADGNR/sati11/datafile/sysaux.261.678706169channel ORA_DISK_1: copied datafile copy of datafile 00002output file name=+DATADGNR/sati11/datafile/sysaux.261.678706169 RECID=0 STAMP=0channel ORA_DISK_1: restoring datafile 00003input datafile copy RECID=2 STAMP=678773528 file name=+FRADG/sati11/datafile/undotbs1.264.678773305destination for restore of datafile 00003: +DATADGNR/sati11/datafile/undotbs1.260.678706169channel ORA_DISK_1: copied datafile copy of datafile 00003output file name=+DATADGNR/sati11/datafile/undotbs1.260.678706169 RECID=0 STAMP=0channel ORA_DISK_1: restoring datafile 00004input datafile copy RECID=1 STAMP=678773321 file name=+FRADG/sati11/datafile/users.265.678773315destination for restore of datafile 00004: +DATADGNR/sati11/datafile/users.268.678797585channel ORA_DISK_1: copied datafile copy of datafile 00004output file name=+DATADGNR/sati11/datafile/users.268.678797585 RECID=0 STAMP=0Finished restore at 14-FEB-09RMAN>

28/67


Starting recover at 14-FEB-09using channel ORA_DISK_1


archived log for thread 1 with sequence 3 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603archived log for thread 1 with sequence 4 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175archived log for thread 1 with sequence 5 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567archived log for thread 1 with sequence 6 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_6.257.678780233archived log for thread 1 with sequence 7 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_7.256.678781955archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603 thread=1 sequence=3archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175 thread=1 sequence=4archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567 thread=1 sequence=5archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_6.257.678780233 thread=1 sequence=6archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_7.256.678781955 thread=1 sequence=7media recovery complete, elapsed time: 00:00:19Finished recover at 14-FEB-09RMAN>database opened

29/67


RMAN>

Recovery Manager complete.Checking Database after online redolog loss and database restore and recoverSQL*Plus: Release 11.1.0.7.0 - Production on Sat Feb 14 18:46:31 2009



SQL> SQL> SQL> GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- 1 1 1 52428800 2 NO CURRENT 708746 14-FEB-09 2 1 0 52428800 2 YES UNUSED 0 3 1 0 52428800 2 YES UNUSED 0

SQL>MEMBER------------------------------------------------+DATADGNR/sati11/onlinelog/group_1.262.678825909+DATADGNR/sati11/onlinelog/group_1.269.678825917+DATADGNR/sati11/onlinelog/group_2.264.678825925+DATADGNR/sati11/onlinelog/group_2.263.678825931+DATADGNR/sati11/onlinelog/group_3.266.678825939+DATADGNR/sati11/onlinelog/group_3.265.678825943

6 rows selected.

30/67



Before starting the new scenario we will make a full backup

[oracle@asmxpt ~/RMAN-SCRIPTS]$ rman target /Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 18:53:36 2009



RMAN> @backup-incr-lev0-plus-recover.rmnRMAN> # backup-incr-lev0-plus-recover.rmn2> # ---------------------------------3> # This script on the first run makes a level 0 backup if no one exists4> # In the second run run makes an incremental level 15> # On every subsequent run:6> # 1) Applies the previous level 1 to the backupset and7> # 2) Creates a new level 1 backup8> RUN9> {10> ALLOCATE CHANNEL disk1 DEVICE TYPE DISK ;11> ALLOCATE CHANNEL disk2 DEVICE TYPE DISK ;12> ALLOCATE CHANNEL disk3 DEVICE TYPE DISK ;13> ALLOCATE CHANNEL disk4 DEVICE TYPE DISK ;14> RECOVER COPY OF DATABASE15> WITH TAG 'INCREMENTAL_DAILY_UPDATED' ;

31/67


16> BACKUP INCREMENTAL LEVEL 117> FOR RECOVER OF COPY WITH TAG 'INCREMENTAL_DAILY_UPDATED'18> DATABASE PLUS ARCHIVELOG ;19> }using target database control file instead of recovery catalogallocated channel: disk1channel disk1: SID=126 device type=DISK




Starting recover at 14-FEB-09Finished recover at 14-FEB-09

Starting backup at 14-FEB-09current log archivedchannel disk1: starting archived log backup setchannel disk1: specifying archived log(s) in backup setinput archived log thread=1 sequence=8 RECID=8 STAMP=678818997channel disk1: starting piece 1 at 14-FEB-09channel disk2: starting archived log backup setchannel disk2: specifying archived log(s) in backup setinput archived log thread=1 sequence=7 RECID=7 STAMP=678781959channel disk2: starting piece 1 at 14-FEB-09

32/67


channel disk3: starting archived log backup setchannel disk3: specifying archived log(s) in backup setinput archived log thread=1 sequence=3 RECID=3 STAMP=678773603input archived log thread=1 sequence=4 RECID=4 STAMP=678779177channel disk3: starting piece 1 at 14-FEB-09channel disk4: starting archived log backup setchannel disk4: specifying archived log(s) in backup setinput archived log thread=1 sequence=1 RECID=10 STAMP=678826433channel disk4: starting piece 1 at 14-FEB-09channel disk1: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.274.678826435 tag=TAG20090214T185354 comment=NONEchannel disk1: backup set complete, elapsed time: 00:00:11channel disk1: starting archived log backup setchannel disk1: specifying archived log(s) in backup setinput archived log thread=1 sequence=5 RECID=5 STAMP=678779566input archived log thread=1 sequence=6 RECID=6 STAMP=678780233channel disk1: starting piece 1 at 14-FEB-09channel disk2: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.275.678826437 tag=TAG20090214T185354 comment=NONEchannel disk2: backup set complete, elapsed time: 00:00:11channel disk2: starting archived log backup setchannel disk2: specifying archived log(s) in backup setinput archived log thread=1 sequence=9 RECID=9 STAMP=678825907channel disk2: starting piece 1 at 14-FEB-09channel disk3: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.276.678826441 tag=TAG20090214T185354 comment=NONEchannel disk3: backup set complete, elapsed time: 00:00:08channel disk4: finished piece 1 at 14-FEB-09

33/67


piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.277.678826443 tag=TAG20090214T185354 comment=NONEchannel disk4: backup set complete, elapsed time: 00:00:07channel disk1: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.278.678826447 tag=TAG20090214T185354 comment=NONEchannel disk1: backup set complete, elapsed time: 00:00:01channel disk2: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.279.678826449 tag=TAG20090214T185354 comment=NONEchannel disk2: backup set complete, elapsed time: 00:00:02Finished backup at 14-FEB-09

Starting backup at 14-FEB-09channel disk1: starting incremental level 1 datafile backup setchannel disk1: specifying datafile(s) in backup setinput datafile file number=00001 name=+DATADGNR/sati11/datafile/system.259.678781847channel disk1: starting piece 1 at 14-FEB-09channel disk2: starting incremental level 1 datafile backup setchannel disk2: specifying datafile(s) in backup setinput datafile file number=00002 name=+DATADGNR/sati11/datafile/sysaux.261.678706169channel disk2: starting piece 1 at 14-FEB-09channel disk3: starting incremental level 1 datafile backup setchannel disk3: specifying datafile(s) in backup setinput datafile file number=00003 name=+DATADGNR/sati11/datafile/undotbs1.260.678706169channel disk3: starting piece 1 at 14-FEB-09channel disk4: starting incremental level 1 datafile backup setchannel disk4: specifying datafile(s) in backup setinput datafile file number=00004 name=+DATADGNR/sati11/datafile/users.268.678797585channel disk4: starting piece 1 at 14-FEB-09channel disk4: finished piece 1 at 14-FEB-09

34/67


piece handle=+FRADG/sati11/backupset/2009_02_14/nnndn1_tag20090214t185410_0.283.678826477 tag=TAG20090214T185410 comment=NONEchannel disk4: backup set complete, elapsed time: 00:01:05channel disk3: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/nnndn1_tag20090214t185410_0.282.678826473 tag=TAG20090214T185410 comment=NONEchannel disk3: backup set complete, elapsed time: 00:02:10channel disk1: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/nnndn1_tag20090214t185410_0.280.678826453 tag=TAG20090214T185410 comment=NONEchannel disk1: backup set complete, elapsed time: 00:02:20channel disk2: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/nnndn1_tag20090214t185410_0.281.678826459 tag=TAG20090214T185410 comment=NONEchannel disk2: backup set complete, elapsed time: 00:02:20Finished backup at 14-FEB-09

Starting backup at 14-FEB-09current log archivedchannel disk1: starting archived log backup setchannel disk1: specifying archived log(s) in backup setinput archived log thread=1 sequence=2 RECID=11 STAMP=678826592channel disk1: starting piece 1 at 14-FEB-09channel disk1: finished piece 1 at 14-FEB-09piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185632_0.285.678826593 tag=TAG20090214T185632 comment=NONEchannel disk1: backup set complete, elapsed time: 00:00:01Finished backup at 14-FEB-09

Starting Control File and SPFILE Autobackup at 14-FEB-09piece handle=+FRADG/sati11/autobackup/2009_02_14/s_678826594.286.678826597 comment=NONE

35/67


Finished Control File and SPFILE Autobackup at 14-FEB-09released channel: disk1released channel: disk2released channel: disk3released channel: disk4

RMAN> # -------------------------------------2> # eof backup-incr-lev0-plus-recover.rmn3> **end-of-file**

36/67


Controlfile loss

This scenario simulates a database crash that lead to a missing controlfile, when the database is restarted the followingerror is displayed:

ORA-00205: error in identifying control file, check alert log for more info

The script that generates this scenario is this:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/4-controlfile-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./4-controlfile-lossGenerating database crash ...

asmcmd rm +DATADGNR/sati11/controlfile/current.258.678705843asmcmd rm +DATADGNR/sati11/controlfile/current.257.678705845

Database closed.Database dismounted.ORACLE instance shut down.Checking databases up ...oracle 4146 1 0 11:35 ? 00:00:00 asm_smon_+ASM

Removing controlfiles ...

Starting databases after the crash ...


Total System Global Area 351522816 bytes

37/67


Fixed Size 1313344 bytesVariable Size 272631232 bytesDatabase Buffers 71303168 bytesRedo Buffers 6275072 bytesORA-00205: error in identifying control file, check alert log for more info

Restore And Recover From Controlfile LossTo restore the controlfile we will need to set the DBID on rman. You can check the DBID of the database on the backup directory we used to build this database: /u01/app/oracle/BACKUP/sati11/dbid_124143874.lst.gz

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./4-restore-and-recover-from-controlfile-loss

Executing Controlfile Restore

Please check the DBID on this file:

-rw-r--r-- 1 oracle dba 104 Feb 12 18:08 /u01/app/oracle/BACKUP/sati11/dbid_124143874.lst.gz

please enter DBID number of the database to restore the controlfile124143874



connected to target database: SATI11 (not mounted)

38/67


RMAN>executing command: SET DBID

RMAN>database is already started


recovery area destination: +FRADGdatabase name (or database unique name) used for search: SATI11channel ORA_DISK_1: AUTOBACKUP +fradg/SATI11/AUTOBACKUP/2009_02_14/s_678826594.286.678826597 found in the recovery areachannel ORA_DISK_1: looking for AUTOBACKUP on day: 20090214channel ORA_DISK_1: restoring control file from AUTOBACKUP +fradg/SATI11/AUTOBACKUP/2009_02_14/s_678826594.286.678826597channel ORA_DISK_1: control file restore from AUTOBACKUP completeoutput file name=+DATADGNR/sati11/controlfile/current.257.678832083output file name=+DATADGNR/sati11/controlfile/current.258.678832085Finished restore at 14-FEB-09

RMAN>database mountedreleased channel: ORA_DISK_1

RMAN>Starting recover at 14-FEB-09Starting implicit crosscheck backup at 14-FEB-09

39/67


allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=156 device type=DISKCrosschecked 16 objectsFinished implicit crosscheck backup at 14-FEB-09

Starting implicit crosscheck copy at 14-FEB-09using channel ORA_DISK_1Crosschecked 4 objectsFinished implicit crosscheck copy at 14-FEB-09

searching for all files in the recovery areacataloging files...cataloging done

List of Cataloged Files=======================File Name: +fradg/SATI11/ARCHIVELOG/2009_02_14/thread_1_seq_3.287.678828379File Name: +fradg/SATI11/AUTOBACKUP/2009_02_14/s_678826594.286.678826597

using channel ORA_DISK_1


archived log for thread 1 with sequence 3 is already on disk as file +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.287.678828379archived log for thread 1 with sequence 4 is already on disk as file +DATADGNR/sati11/onlinelog/group_1.262.678825909archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.287.678828379 thread=1 sequence=3archived log file name=+DATADGNR/sati11/onlinelog/group_1.262.678825909 thread=1 sequence=4

40/67


media recovery complete, elapsed time: 00:00:01Finished recover at 14-FEB-09


RMAN>

Recovery Manager complete.Checking Database after controlfile loss and restore

SQL> select name from v$controlfile;NAME--------------------------------------------------------------------------------+DATADGNR/sati11/controlfile/current.257.678832083+DATADGNR/sati11/controlfile/current.258.678832085

41/67


Total Database Loss

This scenario simulates a crash that lead to the complete loss of the database, the whole database directory is removedthe ASM diskgroup DATADG, all online logs, controlfiles, spfile and datafiles are lost.

Before the crash is simulated a complete set of information, that is required to recover the database, is displayed on the screen. Keep this information at hand to be used during the restore steps. This information includes controlfile and spfile backupset pieces, DBID and last archived sequences.

After the crash when trying to open the database the following errors are returned:

ORA-01078: failure in processing system parametersORA-01565: error in identifying file '+DATADGNR/sati11/spfilesati11.ora'ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-15056: additional error messageORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-15173: entry 'sati11' does not exist in directory '/'ORA-06512: at line 4

This is the script that executes this scenario:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/5-total-database-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./5-total-database-lossGenerating database crash ...

Piece Name: +FRADG/sati11/autobackup/2009_02_15/s_678882001.261.678882005 Piece Name: +FRADG/sati11/autobackup/2009_02_15/s_678884598.271.678884601

42/67


Database log mode Archive ModeAutomatic archival EnabledArchive destination USE_DB_RECOVERY_FILE_DESTOldest online log sequence 3Next log sequence to archive 5Current log sequence 5

NAME DBID--------- ----------SATI11 124143874

SEQUENCE#---------- 4

GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- 1 1 4 52428800 2 YES INACTIVE 774566 15-FEB-09 2 1 5 52428800 2 NO CURRENT 774613 15-FEB-09 3 1 3 52428800 2 YES INACTIVE 772993 15-FEB-09

asmcmd rm -rf +DATADGNR/SATI11Database closed.Database dismounted.ORACLE instance shut down.

Preparing to crash and burn database ...

43/67


Starting database after the crash ...ORA-01078: failure in processing system parametersORA-01565: error in identifying file '+DATADGNR/sati11/spfilesati11.ora'ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-15056: additional error messageORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-15173: entry 'sati11' does not exist in directory '/'ORA-06512: at line 4

Evaluating damage ...

Listing directories on ASM Data diskgroup

Listing directories on ASM Flash Recovery Area diskgroup

ARCHIVELOG/AUTOBACKUP/BACKUPSET/CHANGETRACKING/DATAFILE/

Restore And Recover From Total Loss

On this step we will restore the database after a total loss.

44/67


In first term we will recreate the database metadata using the backup we used before to build the database.

In this case we will restore only the metadata for diskgroup DATADGNR; this is necessary to be able to restore the spfile and controlfile on the first stages of the restore procedure.

The instance is initially started by rman with dummy parameters; once the spfile is restored the instance is restarted in nomount mode using the spfile.

Once the controlfile is successfully restored the database can be restored, recovered and opened with the resetlogs option.

This is the script that implement the restore and recovery steps:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/5-restore-and-recover-from-total-database-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./5-restore-and-recover-from-total-database-loss

Executing Controlfile Restore

please enter DBID number of the database to restore the controlfile124143874

Please check the last archived sequence of the database.please enter sequence number to restore to ...4

Please enter the full path to the controlfile and spfile backup to restore them

+FRADG/sati11/autobackup/2009_02_15/s_678884598.271.678884601

Rebuilding Database Directory on Data Diskgroup

45/67


Current Diskgroup being restored: DATADGNRSystem template TEMPFILE modified!System template DUMPSET modified!System template XTRANSPORT modified!System template DATAFILE modified!System template ARCHIVELOG modified!System template ASM_STALE modified!System template CHANGETRACKING modified!System template ONLINELOG modified!System template FLASHBACK modified!System template AUTOBACKUP modified!System template DATAGUARDCONFIG modified!System template PARAMETERFILE modified!System template CONTROLFILE modified!System template BACKUPSET modified!Directory +DATADGNR/SATI11 re-created!Directory +DATADGNR/SATI11/CONTROLFILE re-created!Directory +DATADGNR/SATI11/ONLINELOG re-created!Directory +DATADGNR/SATI11/DATAFILE re-created!Directory +DATADGNR/SATI11/TEMPFILE re-created!Directory +DATADGNR/SATI11/PARAMETERFILE re-created!SATI11/

Executing Rman Restore and Recovery Steps

Recovery Manager: Release 11.1.0.7.0 - Production on Sun Feb 15 11:19:08 2009


46/67


connected to target database (not started)

RMAN>executing command: SET DBID

RMAN>startup failed: ORA-01078: failure in processing system parametersORA-01565: error in identifying file '+DATADGNR/sati11/spfilesati11.ora'ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-15056: additional error messageORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.oraORA-15173: entry 'spfilesati11.ora' does not exist in directory 'sati11'ORA-06512: at line 4

starting Oracle instance without parameter file for retrieval of spfileOracle instance started


Fixed Size 1312052 bytesVariable Size 67111628 bytesDatabase Buffers 88080384 bytesRedo Buffers 2514944 bytes


47/67


channel ORA_DISK_1: restoring spfile from AUTOBACKUP +FRADG/sati11/autobackup/2009_02_15/s_678884598.271.678884601channel ORA_DISK_1: SPFILE restore from AUTOBACKUP completeFinished restore at 15-FEB-09

RMAN>Oracle instance started


Fixed Size 1313344 bytesVariable Size 272631232 bytesDatabase Buffers 71303168 bytesRedo Buffers 6275072 bytes

RMAN>Starting restore at 15-FEB-09allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=152 device type=DISK

channel ORA_DISK_1: restoring control filechannel ORA_DISK_1: restore complete, elapsed time: 00:00:09output file name=+DATADGNR/sati11/controlfile/current.257.678885579output file name=+DATADGNR/sati11/controlfile/current.265.678885581Finished restore at 15-FEB-09

RMAN>database mountedreleased channel: ORA_DISK_1

RMAN> 2> 3> 4> 5>

48/67


executing command: SET until clause

Starting restore at 15-FEB-09Starting implicit crosscheck backup at 15-FEB-09allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=152 device type=DISKCrosschecked 10 objectsFinished implicit crosscheck backup at 15-FEB-09

Starting implicit crosscheck copy at 15-FEB-09using channel ORA_DISK_1Crosschecked 4 objectsFinished implicit crosscheck copy at 15-FEB-09

searching for all files in the recovery areacataloging files...cataloging done

List of Cataloged Files=======================File Name: +fradg/SATI11/AUTOBACKUP/2009_02_15/s_678884598.271.678884601

using channel ORA_DISK_1

channel ORA_DISK_1: restoring datafile 00001input datafile copy RECID=19 STAMP=678881962 file name=+FRADG/sati11/datafile/system.262.678773299destination for restore of datafile 00001: +DATADGNR/sati11/datafile/system.260.678880089channel ORA_DISK_1: copied datafile copy of datafile 00001output file name=+DATADGNR/sati11/datafile/system.259.678885603 RECID=0 STAMP=0channel ORA_DISK_1: restoring datafile 00002

49/67


input datafile copy RECID=20 STAMP=678881962 file name=+FRADG/sati11/datafile/sysaux.263.678773303destination for restore of datafile 00002: +DATADGNR/sati11/datafile/sysaux.261.678880175channel ORA_DISK_1: copied datafile copy of datafile 00002output file name=+DATADGNR/sati11/datafile/sysaux.261.678885687 RECID=0 STAMP=0channel ORA_DISK_1: restoring datafile 00003input datafile copy RECID=18 STAMP=678881962 file name=+FRADG/sati11/datafile/undotbs1.264.678773305destination for restore of datafile 00003: +DATADGNR/sati11/datafile/undotbs1.259.678880249channel ORA_DISK_1: copied datafile copy of datafile 00003output file name=+DATADGNR/sati11/datafile/undotbs1.260.678885763 RECID=0 STAMP=0channel ORA_DISK_1: restoring datafile 00004input datafile copy RECID=17 STAMP=678881962 file name=+FRADG/sati11/datafile/users.265.678773315destination for restore of datafile 00004: +DATADGNR/sati11/datafile/users.265.678880295channel ORA_DISK_1: copied datafile copy of datafile 00004output file name=+DATADGNR/sati11/datafile/users.258.678885807 RECID=0 STAMP=0Finished restore at 15-FEB-09

Starting recover at 15-FEB-09using channel ORA_DISK_1


archived log for thread 1 with sequence 2 is already on disk as file +FRADG/sati11/archivelog/2009_02_15/thread_1_seq_2.279.678881853archived log for thread 1 with sequence 3 is already on disk as file +FRADG/sati11/archivelog/2009_02_15/thread_1_seq_3.276.678881967archived log file name=+FRADG/sati11/archivelog/2009_02_15/thread_1_seq_2.279.678881853 thread=1 sequence=2

50/67


archived log file name=+FRADG/sati11/archivelog/2009_02_15/thread_1_seq_3.276.678881967 thread=1 sequence=3media recovery complete, elapsed time: 00:00:08Finished recover at 15-FEB-09


RMAN>


Checking Database after total database loss, restore and recover

SQL*Plus: Release 11.1.0.7.0 - Production on Sun Feb 15 11:26:37 2009



SQL> SQL> GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- 1 1 1 52428800 2 NO CURRENT 774567 15-FEB-09 2 1 0 52428800 2 YES UNUSED 0 3 1 0 52428800 2 YES UNUSED 0

51/67


MEMBER--------------------------------------------------------------------------------+DATADGNR/sati11/onlinelog/group_1.262.678885823+DATADGNR/sati11/onlinelog/group_1.269.678885829+DATADGNR/sati11/onlinelog/group_2.264.678885837+DATADGNR/sati11/onlinelog/group_2.263.678885845+DATADGNR/sati11/onlinelog/group_3.266.678885917+DATADGNR/sati11/onlinelog/group_3.268.678885927

NAME---------SATI11


Scripts

backup-incr-lev0-plus-recover.rmn

# backup-incr-lev0-plus-recover.rmn# ---------------------------------# This script on the first run makes a level 0 backup if no one exists# In the second run run makes an incremental level 1# On every subsequent run:# 1) Applies the previous level 1 to the backupset and

52/67


# 2) Creates a new level 1 backupRUN{ALLOCATE CHANNEL disk1 DEVICE TYPE DISK ;ALLOCATE CHANNEL disk2 DEVICE TYPE DISK ;ALLOCATE CHANNEL disk3 DEVICE TYPE DISK ;ALLOCATE CHANNEL disk4 DEVICE TYPE DISK ;RECOVER COPY OF DATABASEWITH TAG 'INCREMENTAL_DAILY_UPDATED' ;BACKUP INCREMENTAL LEVEL 1FOR RECOVER OF COPY WITH TAG 'INCREMENTAL_DAILY_UPDATED'DATABASE PLUS ARCHIVELOG ;}# -------------------------------------# eof backup-incr-lev0-plus-recover.rmn

53/67


1-system-tablespace-loss

#!/bin/tcsh# 1-system-tablespace-loss# ------------------------# This script crash the database and removes the system tablespace# ----------------------------------------------------------------source ./set-environmentecho Generating database crash ...echoset v_rmf=`echo 'select file_name from dba_data_files where file_id=1;' | sqlplus -s / as sysdba | grep system`setenv ORACLE_SID +ASMsqlplus -s $dbauser/$dbapwd as sysdba <<eofshutdown abort;eofsqlplus -s $dbauser/$dbapwd as sysdba <<eofstartupeofechoecho Generating system tablespace loss ...echoasmcmd lsdgechoasmcmd ls +datadgnr/sati11/datafileechoasmcmd rm -rf $v_rmfechoasmcmd lsdgecho

54/67


asmcmd ls +datadgnr/sati11/datafileechoecho Trying to restart the database after the crash ...echosqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eofstartupeofexit# ----------------------------------------------------------------# eof 1-system-tablespace-loss

1-restore-from-system-tablespace-loss

#!/bin/tcsh# 1-restore-from-system-tablespace-loss# ------------------------# This script restore the system tablespace after a crash that removed the system tablespace# -------------------------------------------------------------------------------source ./set-environmentechoecho Executing Command : RESTORE and RECOVER SYSTEM DATAFILEechosetenv ORACLE_SID $datadbrman TARGET / <<eofrestore datafile 1;recover datafile 1;alter database open;eofsqlplus $dbauser/$dbapwd@$datadb as sysdba <<eof

55/67


select file_name from dba_data_files;exiteofexit# ----------------------------------------------------------------# eof 1-restore-from-system-tablespace-loss

2-user-datafile-loss

#!/bin/tcsh# 2-user-datafile-loss# ------------------------# This script create an application user then removes the user tablespace datafile# --------------------------------------------------------------------------------source ./set-environmentsetenv ORACLE_SID $datadbclearechoecho Preparing User Application ...echo# connect to the database, create user a apps and a dummy table, select from it# to simulate an application running# -----------------------------------------------------------------------------sqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eofdrop user apps cascade;create user apps identified by apps default tablespace users temporary tablespace temp;grant dba to apps;connect apps/apps@$datadbcreate table customers as select * from dba_users;update customers set username='CUSTOMER_'||USERNAME;

56/67


commit;select username from customers;exiteof# generate the command to remove user's tablespace datafile from ASM# the tablespace is set offline to be able to remove it# -------------------------------------------------------------------echoecho Generating user datafile remove ...echoset v_rmf=`echo "select file_name from dba_data_files where tablespace_name='USERS';" | sqlplus -s / as sysdba | grep users`echoecho Setting tablespace users offline ...echosqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eofalter tablespace users offline;eof# removing the users tablespace datafile from ASM# -----------------------------------------------echoecho Removing tablespace users datafile ...echosetenv ORACLE_SID +ASMasmcmd lsdgechoasmcmd ls +datadgnr/$datadb/datafileechoasmcmd rm -rf $v_rmfechoasmcmd ls +datadgnr/$datadb/datafile

57/67


echoasmcmd lsdgechoecho Checking application ...echosqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eofselect username from apps.customers;exiteof# ----------------------------------------------------------------# eof 2-user-datafile-loss

2-restore-from-user-datafile-loss

#!/bin/tcsh# 2-restore-from-user-datafile-loss# ------------------------# This script restore the users tablespace while the database is still running# ----------------------------------------------------------------------------source ./set-environmentecho Restoring and Recovering Tablespace Users ...echosetenv ORACLE_SID $datadbrman TARGET / <<eofrestore tablespace users;recover tablespace users;sql 'alter tablespace users online' ;eofechoecho Checking application ...

58/67


echosqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eofselect username from apps.customers;exiteofexit# ----------------------------------------------------------------# eof 2-restore-from-user-datafile-loss

3-online-redolog-loss

#!/bin/tcsh# 3-online-redolog-loss# ---------------------# This script simulates a database crash followed by the loss of all# online redologs# ------------------------------------------------------------------source ./set-environmentecho Generating database crash ...echoset v_logf=v\$logfileset v_logs=v\$logset v_logh=v\$log_historysetenv ORACLE_SID $datadb

# Create a script to remove online redologs# -----------------------------------------sqlplus -s / as sysdba <<eofset pages 50000 lines 120 echo off head off veri off flush off ti offspool rmonlnlog.shselect 'asmcmd rm '||member||'' from $v_logf where group#=1;

59/67


spool offset echo on head on veri onset pages 50000 lines 120spool redolog_miss_status-before_crash.logarchive log list;select SEQUENCE# from $v_logh where FIRST_TIME=(select max(FIRST_TIME) from $v_logh);select * from $v_logs;spool offSHUTDOWN IMMEDIATE;eofecho# Remove online redologs# ----------------------echo Removing online redo log group ...echosetenv ORACLE_SID +ASMchmod 700 ./rmonlnlog.sh./rmonlnlog.shrm ./rmonlnlog.sh# Restart the database after the simulated crash and check the error# ------------------------------------------------------------------echoecho Starting database after the crash ...echosetenv ORACLE_SID $datadbsqlplus -s / as sysdba <<eofSTARTUPeof# -------------------------# eof 3-online-redolog-loss

60/67


3-restore-and-recover-from-online-redo-loss

#!/bin/tcsh# 3-restore-and-recover-from-online-redo-loss# -------------------------------------------# this script executes the procedure to restore and recover the database# after the loss of all online logs# ----------------------------------------------------------------------source ./set-environmentset v_logf=v\$logfileset v_log=v\$logecho Executing Full Database Restore ...echoecho Please check the last archived sequence of the database.echo please enter sequence number to restore to ...set v_seq = $<echosetenv ORACLE_SID $datadbrman TARGET / <<eofSTARTUP MOUNT;RESTORE DATABASE;RECOVER DATABASE UNTIL SEQUENCE $v_seq THREAD 1;ALTER DATABASE OPEN RESETLOGS;eofecho Checking Database after online redolog loss and database restore and recoverechosqlplus / as sysdba <<eofset pages 50000 lines 200col member for a55select * from $v_log;select member from $v_logf;

61/67


exiteof# -----------------------------------------------# eof 3-restore-and-recover-from-online-redo-loss

4-restore-and-recover-from-controlfile-loss

#!/bin/tcsh# 4-restore-and-recover-from-controlfile-loss# -------------------------------------------source ./set-environmentecho Executing Controlfile Restoreechoecho Please check the DBID on this file:echols -l /u01/app/oracle/BACKUP/sati11/dbid_124143874.lst.gzechoecho please enter DBID number of the database to restore the controlfileset v_dbid = $<echo# ------------------------------------------------# Restore the controlfile and recover the database# ------------------------------------------------setenv ORACLE_SID $datadbrman TARGET / <<eofSET DBID $v_dbid;STARTUP NOMOUNT;RESTORE CONTROLFILE FROM AUTOBACKUP;ALTER DATABASE MOUNT;RECOVER DATABASE;ALTER DATABASE OPEN RESETLOGS;

62/67


eofset v_ctlf=v\$controlfileecho Checking Database after controlfile loss and restoreechosqlplus / as sysdba <<eofset pages 50000 lines 120select name from $v_ctlf ;exiteof# -----------------------------------------------# eof 4-restore-and-recover-from-controlfile-loss

5-total-database-loss

#!/bin/tcsh# 5-total-database-loss# ---------------------source ./set-environmentecho Generating database crash ...echoset v_par=v\$parameterset v_dba=v\$databaseset v_logs=v\$logset v_logh=v\$log_historysetenv ORACLE_SID $datadb# ------------------------------------------------# Get backup to restore controlfile and spfile# ------------------------------------------------echorman target / @listbk | grep autobackupecho

63/67


# ------------------------------------------------# Get actual sequence before crashing the database# ------------------------------------------------sqlplus -s / as sysdba <<eofset echo on head on veri on pages 50000 lines 120spool redolog_miss_status-before_crash.logarchive log list;select name,dbid from $v_dba;select SEQUENCE# from $v_logh where FIRST_TIME=(select max(FIRST_TIME) from $v_logh);select * from $v_logs;spool offset pages 50000 lines 120 echo off head off veri off flush off ti offspool rmdbs.shselect 'asmcmd rm -rf '||a.value||'/'||b.namefrom $v_par a, $v_dba bwhere a.name='db_create_file_dest';spool offSHUTDOWN IMMEDIATEeof# ------------------------------------------------# Remove all database files# ------------------------------------------------echoecho Preparing to crash and burn database ...echosetenv ORACLE_SID +ASMchmod 700 ./rmdbs.sh./rmdbs.shrm rmdbs.shechoecho Starting database after the crash ...

64/67


echosetenv ORACLE_SID $datadbsqlplus -s / as sysdba <<eofSTARTUPeof# ------------------------------------------------# Evaluate damage# ------------------------------------------------echoecho Evaluating damage ...echoecho Listing directories on ASM Data diskgroupechosetenv ORACLE_SID +ASMasmcmd ls +datadgnrechoecho Listing directories on ASM Flash Recovery Area diskgroupechoasmcmd ls +fradg/$datadbecho# -------------------------# eof 5-total-database-loss

5-restore-and-recover-from-total-database-loss

#!/bin/tcsh# --------------------------------------------------# 5-restore-and-recover-from-total-database-loss# --------------------------------------------------source ./set-environment

65/67


echo Executing Controlfile Restoreechoecho please enter DBID number of the database to restore the controlfileset v_dbid = $<echoecho Please check the last archived sequence of the database.echo please enter sequence number to restore to ...set v_seq = $<echoecho Please enter the full path to the controlfile and spfile backup to restore themechoset v_bkp = $<echo# -----------------------------------# Rebuild database diskgroup metadata# -----------------------------------echo Rebuilding Database Directory on Data Diskgroupechosetenv ORACLE_SID +ASMasmcmd md_restore -b /u01/app/oracle/BACKUP/sati11/asm_md_backup -t nodg -g DATADGNRasmcmd ls +DATADGNRechoecho Executing Rman Restore and Recovery Stepsecho# -----------------------------------# Restore the database# -----------------------------------setenv ORACLE_SID $datadbrman TARGET / <<eofSET DBID $v_dbid;STARTUP NOMOUNT;

66/67


RESTORE SPFILE FROM '$v_bkp';STARTUP FORCE NOMOUNT;RESTORE CONTROLFILE FROM '$v_bkp';ALTER DATABASE MOUNT;run {set until sequence $v_seq thread 1;restore database;recover database;}ALTER DATABASE OPEN RESETLOGS;eofset v_log=v\$logset v_logf=v\$logfileset v_dbs=v\$databaseecho Checking Database after total database loss, restore and recoverecho# --------------------------------------------# Check the database after restore and recover# --------------------------------------------sqlplus $dbauser/$dbapwd@$datadb as sysdba <<eofset pages 50000 lines 120select * from $v_log;elect member from $v_logf;select name from $v_dbs;exiteof# --------------------------------------------------# eof 5-restore-and-recover-from-total-database-loss

End of Lab 13

67/67

Lab13 ASM and Rman Crash Restore and Recovery Scenarios

Documents

Transcript of Lab13 ASM and Rman Crash Restore and Recovery Scenarios