Linux Oracle RAC Cluster backup solution with Tivoli ... · Linux Oracle RAC Cluster backup...

Practical Guide

Linux Oracle RAC Cluster backup solution with

Tivoli Storage FlashCopy Manager and ASM

This document can be found on the web, www.ibm.com/support/techdocs Search for document number WP102501 under the category of “White Papers.

Authors: Jörg Walter Axel Westphal

© Copyright IBM Corporation, 2014 of 32

Document History

Document Location This is a snapshot of an on-line document. Paper copies are valid only on the day they are printed. Refer to the author if you are in any doubt about the currency of this document.

Revision History

Revision Number

Revision Date

Summary of Changes Changes marked

(0.1) 01.12.2014 Initial version (N) (N) (N)


Contents 1. Abstract .................................................................................................................. 5

1.1 Overview of FlashCopy Manager .................................................................................................... 5

1.2 Overview of Oracle ASM ................................................................................................................. 6

1.3 IBM SAN Volume Controller ........................................................................................................... 6

2. Hardware Setup of the Solution .............................................................................. 8 2.1 Overview Diagram........................................................................................................................... 8

2.2 Solution Components ...................................................................................................................... 9

2.3 Software Versions ......................................................................................................................... 10

2.4 TCP Port Addressing Scheme ...................................................................................................... 10

3. Preparation ........................................................................................................... 11 3.1 Physical vs. Virtual DB Servers .................................................................................................... 11

3.2 Dedicated vs. Shared Backup Proxy Servers ............................................................................... 11

3.3 ASMLib .......................................................................................................................................... 12

3.4 Space in TSFCM Install Directories .............................................................................................. 12

3.5 Separate User ID for Oracle DB Instance(s) ................................................................................ 12

3.6 RMAN catalog database ............................................................................................................... 12

3.7 TSM client configuration ............................................................................................................... 13

3.8 FlashCopy Target Disks ................................................................................................................ 13

3.9 Zoning ........................................................................................................................................... 14

4. TSFCM Installation ............................................................................................... 15 4.1 Create TSFCM instance user and modify profile .......................................................................... 15

4.2 Copy TSFCM code to PS and BS ................................................................................................. 15

4.3 Install TSFCM base and patch ...................................................................................................... 15

5. TSFCM Configuration ........................................................................................... 17 5.1 Preparing Files and Scripts ........................................................................................................... 17

5.1.1 Target Volume File ................................................................................................................ 17

5.1.2 FC Backup Script ................................................................................................................... 17

5.1.3 TSM Backup Script ................................................................................................................ 17

5.1.4 RMAN Script .......................................................................................................................... 18

5.1.5 RMAN Restore Script ............................................................................................................ 18

5.2 Running the Config Wizard on Production Server (PS) ................................................................ 18

5.3 Preparing the Backup Server (BS) ................................................................................................ 22

5.3.1 Copy fcmselfcert.arm from PS to BS ..................................................................................... 22


5.3.2 Modify initSID.ora .................................................................................................................. 23

5.4 Running the configuration Wizard on the Backup Server (BS) ..................................................... 24

6. Test Scenarios ...................................................................................................... 26 6.1 Disk-Only Backup (FlashCopy) ..................................................................................................... 26

6.2 TSM Tape Offload ......................................................................................................................... 27

6.3 Disk-Only Restore (FlashBack) ..................................................................................................... 29

6.4 TSM Tape Restore ........................................................................................................................ 30

7. References ........................................................................................................... 32


1. Abstract Backups of large Oracle RAC databases are normally performed within a window during batch processing because the impact of the backup load on a database or application can be best tolerated. Compressing all backup activities into this backup window causes an extremely high peak workload for the system infrastructure (network, servers, tape drives). Because of growing database volumes and the business need for 24-by-7 application availability, the backup window can become a key obstacle in meeting service-level requirements. IBM Tivoli Storage FlashCopy Manager provides the operational flexibility necessary for today‘s IT infrastructure. Zero impact FlashCopy backups can be started any time during the daily operation of an application system, because the FlashCopy technique minimizes the impact on the application (server-processor load, database-backup window). Fast FlashCopy restores can minimize the downtime required for data restore. FlashCopy Manager supports manifold combinations of operating systems, database applications and storage systems. This practical guide describes a Tivoli Storage FlashCopy Manager (TSFCM) installation and configuration as well as the test procedures that have been performed for an Oracle database instance “SID”, which is running on a two-node RAC cluster with Oracle Automatic Storage Management (ASM) on Red Hat Enterprise Linux, using IBM San Volume Controller as the storage subsystem.

1.1 Overview of FlashCopy Manager Tivoli Storage FlashCopy Manager uses the copy services capabilities of intelligent disk subsystems to create application aware point-in-time copy (FlashCopy or snapshot) of the production data. This copy is then retained on disk as backup allowing for a very fast restore operation. In addition to restoring from a point-in-time copy, Tivoli Storage FlashCopy Manager allows mounting the copy on an auxiliary server (backup server) as a logical copy. This copy (instead of the original data on the production server) can then be accessed for further processing, such as creating a subsequent tape backup or performing backup verification functions. Tivoli Storage FlashCopy Manager for Unix or TSFCM as we will be calling it in this paper supports the following database applications and environments:

• DB2 with SAP and non-SAP applications • DB2 HADR environments • SAP for Oracle environments • Native Oracle environments • Native Oracle ASM running on raw physical volumes • Custom applications such as file systems or other non-DB2 or non-Oracle databases

This document refers to Oracle Automatic Storage Management environments. Automatic Storage Management environments will be referred as ASM in the following text. Tivoli Storage FlashCopy Manager provides its own backup and recovery interface for native Oracle databases that creates full database backups and restores. Optionally, the database control files can be backed up into the Tivoli Storage FlashCopy Manager backup repository. Tivoli Storage FlashCopy Manager supports these backup functions:

• Full database backups at a volume level • Back up of database control files into the Tivoli Storage FlashCopy Manager repository

after a full backup completes. • Database restores with or without restoring database control files.

More generic information about TSFCM configuration, HW and SW prerequisites etc., can be found in the IBM Infocenter:

http://pic.dhe.ibm.com/infocenter/tsminfo/v7r1/index.jsp?topic=%2Fcom.ibm.itsm.fcm.unx.doc%2Ft_protect_fcmul.html


1.2 Overview of Oracle ASM

Basically, ASM could be considered as an evolution in file system and volume management functionality for Oracle database files. Storage related tasks such as provisioning of volumes to the databases became much more simplified with ASM, since ASM takes over the provisioning of file system and volume manager capabilities built into the oracle database kernel. This eliminates the need for third-party volume manager and file systems for managing the Oracle database files. ASM enables disk management by using either well known SQL commands (create/alter/drop) or the so called ASMCMD which provides the corresponding operations over a command line interface. Among many other benefits, ASM improves file system scalability and performance, manageability, and availability for single instance databases as well as for Oracle Real Application Clusters (RAC) environments. ASM is easier to manage and faster than conventional file systems, and it provides I/O balancing, large file support, automatic on line redistribution after the incremental addition or removal of storage capacity. Further, for migration purposes, a new RMAN command enables non-ASM managed files to be relocated to an ASM disk group. Figure 1depicts the architecture of ASM databases. In the bottom the storage subsystems are shown, on top of them the ASM disk groups are built. Finally the Oracle Database Servers reside in those disk groups.

Figure 1: ASM Overview (Oracle Technical White Paper, June 2007)

1.3 IBM SAN Volume Controller The IBM System Storage SAN Volume Controller® (SVC) is a storage virtualization system. It enables a single point of control for storage resources to improve business application availability and resource usage. SVC helps increase


available storage capacity to host applications. By pooling capacity from multiple disk systems in the storage area network (SAN), storage administrators can reach beyond traditional islands of SAN storage. The FlashCopy function creates an almost instant copy of active data for backup and cloning. SVC supports incremental FlashCopy operations that copy only the portions of the source or target virtual disk updated since the last FlashCopy function, and also backs up cascaded operations where the target of one FlashCopy relationship is further copied. These abilities can help maintain and update a test environment based on production data.


2. Hardware Setup of the Solution This chapter gives an overview of the implemented solution with TSFCM.

The environment described in this document consists of a two-node Oracle Real Application Cluster (RAC) with Automated Storage Management (ASM), as shown in Figure 2. The used ASM disk groups (DG) consist of two failure groups (FG) each, where one of the FGs is located on a SAN Volume Controller (SVC) in datacentre 1 and the other FG is located on a SVC in datacentre 2.

In each of the DCs a TSFCM Backup server (BS) is located which is used to mount the FlashCopy target disks and to offload the disk backups to Tivoli Storage Manager.

All SVC disks (FC source and target) are thin-provisioned, therefore the FlashCopy type “NOCOPY” is used. As a FlashCopy backup can be created only on one storage system at a time, the FC backups are performed in an alternating manner between both DCs, 1 and 2.

Therefore, also a FlashCopy restore (FlashBack) can be performed in one DC at a time. This results in one of the ASM FGs to be offline after a restore. This FG needs to be synced manually afterwards.

Note: The database instance SID is used as a reference, but the solution can be applied to any other database in the same way.

2.1 Overview Diagram

Figure 2: Hardware Setup


2.2 Solution Components The FlashCopy backup solution consists of the following components:

Component Function

Database Server(s) The Oracle databases to be protected are running on xSeries servers with LINUX platform.

The databases use the Oracle RAC feature to build a cluster on the application layer, where Oracle ASM is used as the Storage Manager layer. Failure Groups are used for storage HA.

The RAC nodes are distributed between two datacentres and are called racnode1 and racnode2

SAN Volume Controller(s) The disks used by the database servers are located on two SAN Volume Controllers, one in each data center (DC).

Each of the SVCs is equipped with an embedded CIM agent, which is contacted by TSFCM to create the FlashCopies.

Storage Subsystem IDs of the SVCs DC1 DC2

SVC_01 SVC_02 Backup Proxy Server(s) The FC target disks are mapped to backup servers in order to create TSM

offload backups to tape. There is one BS per data centre, each mounting the FC target disks from the local SVC.

Names of the Backup Servers as they are registered in the SVCs DC1 DC2

Proxy01 Proxy02 TSM Server(s) The implemented FlashCopy solution is intended to provide fast restore

capabilities in case of logical errors. It does NOT provide any DR protection, as the backup images reside on the same storage systems as the source data.

In order to protect the system against HW failures or a disaster, a “conventional” TSM tape backup is required.

Another reason for having TSM configured in parallel is the fact that TSFCM is only creating point-in-time copies of the database, while other file systems, binaries and especially DB archive logs need to be stored to TSM.


2.3 Software Versions The following software products are used for the FlashCopy Manager implementation:

• Tivoli Storage FlashCopy Manager for Unix 4.1.0.2

• Tivoli Storage Manager BA client v7.1 + API

• Tivoli Storage Agent 6.2.4.200

• Tivoli Storage Manager for Oracle (TDPO) 6.3.0.0

Note: On systems using SAP as an application, TSM4ERP might be the preferable TSM client. Be aware that this is not supported when offloaded backups have to be created. See chapter “TSM Client Configuration” for further details.

2.4 TCP Port Addressing Scheme Tivoli FlashCopy Manager uses a central management deamon (acsd) that controls the overall FC workflow. For each database instance in scope of TSFCM, a dedicated acsd instance is required.

As multiple acsd instances might run on the same host, a TCP listener port scheme should be used in the TSFCM configuration to avoid conflicts, e.g.:

Database Instance TCP Port

SID1 57328 (default)

SID2 57329

SID3 57330

SID4 57331


3. Preparation FlashCopy Manager has some prerequisites to be met, before the solution can be implemented. A complete list of prereqs can be found here (see attached Pre-Installation checklist):

http://www-01.ibm.com/support/docview.wss?uid=swg21650875

The Pre-Requisite checklist above contains detailed requirements for all applications, OS and storage subsystems supported by TSFCM. The most important prerequisites for the given environment (Oracle RAC/ASM on Linux) are listed briefly below.

Nevertheless – it is highly recommended to review also the appropriate sections in the overall Pre-Req document!

3.1 Physical vs. Virtual DB Servers It is preferred to perform FC backups from DB systems based on physical servers (e.g. in order to have an option to do LANfree TSM backups for very large DBs).

Nevertheless – it is supported to install and run TSFCM for Unix/Oracle also in a VMware guest. In this case, the disks used by the database need to be located either on:

• pRDMs (physical Raw Device Mappings), or

• iSCSI LUNs, or

• NFS mounts on N series storage

3.2 Dedicated vs. Shared Backup Proxy Servers In order to remove load for performing LANfree TSM tape backups from the production database servers, additional backup servers (BS) have to be implemented. These servers will mount the FlashCopy target disks, start the Oracle database in no-mount mode and perform “offloaded” TSM tape backups.

The backup servers have to be physical machines, with the same operating system and Oracle levels as the production servers (PS).

It is possible to share the backup servers for multiple “source” databases, but it needs to be ensured that

• the backup servers can take the load to offload all source DBs within a given timeframe / backup window

• the source systems will not get in conflict with differing OS or Oracle levels

The BS need to have the same Oracle instance setup (also same UIDs and GIDs) and TSM client configuration as the PS.

Note: Ensure that the +DATA diskgroup(s) DO NOT contain a copy of the Oracle control file! Control files can be located e.g. in the OLOG, RECO or ARCH disk groups.

For each of the Oracle instances on the backup servers, at least one of these disk groups need to be available in order to do a control file restore, which is required to start the DB in no-mount mode for offloaded backups. These DGs need to have the same name, but not necessarily the same size as the DGs on the PS.


3.3 ASMLib On Linux, TSFCM supports only ASM environments setup with Oracle ASMLib since Oracle recommends ASMLib over raw devices for ease-of-use and performance reasons. Raw devices are not supported. ASMLib 2.0 is required by TSFCM.

Note: The use of ASMLib is not optional.

3.4 Space in TSFCM Install Directories Ensure that about 1GB of free space is available in /tmp and /opt/tivoli file systems before starting the installation (500MB would be ok, too, when TSFCM base level 4.1.0.0 and patch level 4.1.0.2 are installed sequentially).

Ensure that about 500MB is available in /home/fcm<sid> .

3.5 Separate User ID for Oracle DB Instance(s) In an Oracle environment with ASM, two types of “instances” exists – an ASM instance and one- or more database instances.

In many Oracle environments, both instances run under the same user ID: “oracle”, where aliases are defined to switch ORACLE_SID to different values on demand (the default SID of the “oracle” user points to the ASM instance).

FlashCopy Manager deamons need to run under a user ID that have the environment variable ORACLE_SID permanently set to the value that describes the database instance it shall protect. A “dynamic” change of this variable is not supported.

Therefore, a dedicated user needs to be created per database instance in scope of flashcopy. This user needs to be available with same UID and GIDs on the RAC database nodes, as well on all backup proxies that should be able to create an offload backup for this instance.

The user name is “fcm{sid}” (e.g. fcmsid) and has home directory /home/fcm{sid}. The instance-specific copy of the TSFCM software will be installed to /home/fcm{sid}/acs.

3.6 RMAN catalog database To run FlashCopy Manager with Oracle, an RMAN catalog database needs to be configured (even for disk only backups). It should not reside on the PS (because it would be lost in the case of a disaster) but must be accessible from both PS and BS.

The Oracle DB instances need to be registered to the RMAN catalog and appropriate listeners need to be established on all PS and BS before the TSFCM configuration is started.


3.7 TSM client configuration The TSFCM tape offload backups require having the TSM API as well as TDP for Oracle (TDPO) being configured properly on both backup servers.

For archiving of Redo logs, and in case of a required TSM tape restore, the TSM API and TDPO need to be configured on both production servers, too.

In order to backup or restore large DB systems in a reasonable amount of time, the TSM for SAN (StorageAgent) needs to be installed and configured on the PS and BS.

As the TSM node used for backing up a specific Oracle DB instance will potentially contact the TSM server from four different hosts (both PS and both BS), the TSM password needs to be kept in sync between all systems. This can either be achieved by configuring non-expiring passwords, or by using the TSM Proxy node concept, where each host uses its own TSM node name and password and acts as an “agent” for a single “target”.

Note: As the Oracle databases are used by SAP, the most appropriate way to create a TSM tape backup would be to use the Tivoli Storage Manager for Enterprise Resource Planning (TSM4ERP) which implements the BACKINT interface and perfectly integrates with the BR*Tools.

As the BACKINT interface is not supported for the TSFCM device types util_vol and util_vol_online when ASM is used as the storage manager, we cannot use the BR*Tools in combination with TSFCM.

Due to this fact, the whole backup mechanism needs to be changed for all Oracle databases in scope of FlashCopy Manager: Instead of TSM4ERP, Oracle RMAN with TDPO has to be used!

In order to optimize the DB backup for the ProtecTIER VTL, ensure that the RMAN script uses multiple, parallel sessions and DOES NOT use multiplexing. Use FilesPerSet=1 to achieve this.

3.8 FlashCopy Target Disks All disks that belong to the ASM +DATA disk groups are in scope of FlashCopy. Therefore, for each +DATA disk a set of FlashCopy target disks needs to be created (same number and size!).

As the FlashCopies shall be performed in both data centers, a set of target disks needs to be available for each ASM failure group / in each DC.

The FC target disks need to reside within the same storage subsystem as the corresponding source disks.

When using direct attached DS8000 storage for the DB servers, there are some constraints:

• only one single LUN is supported per DATA disk group – the DB may not be spread over multiple LUNS!

• When using thin-provisioned DS8000 FC target disks, a reverse FlashCopy restore is not possible. This type of disks would only be supported to create offloaded TSM backups.

For SVC storage, the restrictions above do not apply.


3.9 Zoning The source disks need to be visible to the production DB servers (PS) only, while the FC target disks need to be visible to the backup servers (BS) only - the zoning needs to be setup accordingly.

Note: FlashCopy Manager will establish the mapping of all target disks to the backup proxy on demand, so it is not necessarily required to do the mapping in advance. Just ensure that the zoning would allow to map the target disks from the SVC to the BS and that a host entry exists in the SVC for the BS.


4. TSFCM Installation This chapter describes the Tivoli Storage FlashCopy Manager installation for one database SID.

Note: The commands listed below might contain user specific parameters, e.g. UID, which need to be changed when the same steps are performed for another database instance!

4.1 Create TSFCM instance user and modify profile On all involved servers (production servers (PS) and backup servers (BS)) add a user for TSFCM:

useradd -m -u 1321 -g oinstall -G asmdba,dba,oper,sapinst,asmadmin,asmoper fcmsid

Set a password for the user with the command passwd.

su - fcmsid

On both production servers (PS), add the following line to the profile (vi /home/fcmsid/.bashrc):

export ORAENV_ASK=NO;export ORACLE_SID=SID002;. oraenv;export ORAENV_ASK=YES;env|grep ORA

On both backup servers (PS), add the following line to the profile (vi /home/fcmsid/.bashrc):

export ORAENV_ASK=NO;export ORACLE_BASE=/oracle/SID;export ORACLE_HOME=/oracle/SID/112;export ORACLE_SID=SID;export ORAENV_ASK=YES;env|grep ORA

export PATH=$ORACLE_HOME/bin:$PATH

4.2 Copy TSFCM code to PS and BS Copy the TSFCM installation binaries to all involved servers (PS and BS), e.g. to /tmp/tsfcm and make the files executable.

The TSFCM 4.1.0.0 base code (including the license) and the 4.1.0.2 patch are required:

• 4.1.0.0-TIV-TSFCM-Linux.bin

• 4.1.0.2-TIV-TSFCMP-Linux.bin

4.3 Install TSFCM base and patch As user root, install the base level by executing the binary and follow the instructions of the install wizard.

When asked, use the following parameters in the install wizard:

Product: 2- IBM Tivoli Storage FlashCopy (R) Manager (Oracle) NOT SAP!

Install folder: /opt/tivoli/tsfcm/acs_4.1.0.0

Database Instance directory: /home/fcmsid


This will install the code to directory /opt/tivoli/tsfcm/acs_4.1.0.0 and create an “Instance specific” copy to /home/fcmsid/acs.

When finished, perform the same steps for the patch level. Use the directory /opt/tivoli/tsfcm/acs_4.1.0.2 as installation folder.


5. TSFCM Configuration

5.1 Preparing Files and Scripts Before starting the configuration wizard, some files and scripts have to be prepared, which will be used in the configuration later. The main configuration takes place on the production server.

5.1.1 Target Volume File The Target Volume file contains the names of FC target LUNs, grouped by TSFCM device classes:

fcmsid@racnode1:/oracle/SID/112/acs % cat fcm-target-volumes.fct

>>> TARGET_SET DC1_1

DEVICE_CLASS SVC_DC1

TARGET_VOLUME Proxy01_SID_DATA_VOL001


<<<

>>> TARGET_SET DC2_1

DEVICE_CLASS SVC_DC2



<<<

5.1.2 FC Backup Script The FlashCopy backup is executed by the acsora command and needs to be followed by a subsequent LOG switch. This is handled by a small script:

fcmsid@ racnode1:/home/fcmsid/acs % cat fcm_backup.sh

#!/bin/bash

/home/fcmsid/acs/acsora -f backup

sqlplus / as sysdba << EOF

ALTER SYSTEM ARCHIVE LOG CURRENT;

EXIT;

EOF

5.1.3 TSM Backup Script The offloaded TSM backup is triggered by a TSFCM command which is also included in a small script (for consistency reasons):

fcmsid@racnode1:/home/fcmsid/acs % cat tsm_backup.sh

#!/bin/bash


/home/fcmsid/acs/fcmcli -f tape_backup

5.1.4 RMAN Script The TSM offload command will connect to the Backup Proxy server (via TSFCM internal communication) and perform an RMAN tape backup. Therfore, a RMAN script needs to be created:

fcmsid@racnode1:/home/fcmsid/acs % cat rman_script.sh

run

{

allocate channel t1 type 'sbt_tape' parms 'ENV=(TDPO_OPTFILE=/oracle/rman_sid002/tdpo/tdpo.db.opt)';


backup

FILESPERSET=1

format 'df_%t_%s_%p'

(database);

release channel t1;

release channel t2;

}

5.1.5 RMAN Restore Script In case a TSM restore from tape would be required later, a RMAN restore script is required:

fcmsid@racnode1:/home/fcmsid/acs % cat rman_restore_script.sh

run

{



restore database;

recover database;

alter database open;

release channel t1;

release channel t2;

}

5.2 Running the Config Wizard on Production Server (PS) When all the files above have been prepared on the PS, the config wizard can be started.

As user fcmsid, execute the following command:

fcmsid@racnode1:/home/fcmsid/acs % ./setup_ora.sh


This will start the configuration wizard: fcmsid@racnode1:/home/fcmsid/acs % ./setup_ora.sh Select one of these configurations: (1) On-Site Production System configuration with optional remote Backup System configuration (2) On-Site Backup System configuration Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager on the production server with the option to remotely synchronize the configuration of one or multiple backup systems using Secure Shell. Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager on the backup system as a 'separate installation'. [1] Preparing to install on Production System Found profile /home/fcmsid/acs/profile Do you want to: (c)reate a new profile (r)euse the profile unchanged (m)odify the profile Please enter one of the strings in parentheses: m The management daemon seems to be running. Stopping IBM Tivoli Storage FlashCopy(R) Manager management daemon. Successfully stopped upstart job fcmsid-acsd. Successfully stopped upstart job fcmsid-acsgend. Deactivating IBM Tivoli Storage FlashCopy(R) Manager management daemon. Successfully uninstalled upstart job fcmsid-acsd. Successfully uninstalled upstart job fcmsid-acsgend. Select one of these configurations: (1) backup only (2) cloning only (3) backup and cloning Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager for backup only. Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager for cloning only. Enter '3' to configure IBM Tivoli Storage FlashCopy(R) Manager for backup and cloning. [1] Currently off-lodaded backups are configured. Do you want to keep this setting? [y|n] [y] Profile configuration ===================== Special commands: '?' -> Show help for current parameter '!d' -> Delete current parameter instance / delete user input FMM8421I Modifying existing profile '/home/fcmsid/acs/profile' for application 'ORACLE' ... ****** Profile parameters for section GLOBAL : ****** Path of the IBM Tivoli Storage FlashCopy(R) Manager directory {ACS_DIR} = [/home/fcmsid/acs] Hostname and port of machine running Management Agent {ACSD} (<hostname> <port>) = [racnode1 57328] ****** Profile parameters for section ACSD : ****** Path to the repository directory {ACS_REPOSITORY} = [/home/fcmsid/acs/acs_repository] ****** Profile parameters for section CLIENT : ****** Suspend target database {TARGET_DATABASE_SUSPEND} (NO|YES|OFFLINE) = [YES] Used volume manager {VOLUME_MGR} (LVM|ASM) = [ASM] Create backup from snapshot {TSM_BACKUP} (NO|YES|TSM_ONLY|MANDATE|LATEST [USE_FOR <list of device classes>]) = [LATEST] Do you want to add another instance of this parameter? [y|n] [n] Maximum number of snapshot versions {MAX_VERSIONS} (<number>|ADAPTIVE) = [ADAPTIVE] Device class(es) {DEVICE_CLASS} (<list of dev classes> [USE_AT <weekdays>] [FROM <time> TO <time>]) = [SVC_DC1 SVC_DC2] ****** Profile parameters for section ORACLE : ****** Recovery catalog connect string {CATALOG_DATABASE_CONNECT_STRING} = [CATALOGNAME] Username for Recovery Catalog database {CATALOG_DATABASE_USERNAME} = [CATALOGUSER]


Fully qualified file name of the RMAN backup script on backup server {DATABASE_BACKUP_SCRIPT_FILE} = [/home/fcmsid/acs/rman_script.sh] User name of the ASM instance owner {ASM_INSTANCE_USER} (AUTO|<string>) = [oracle] SID of the ASM instance {ASM_INSTANCE_ID} = [+ASM2] Role when connecting to ASM instance {ASM_ROLE} (SYSDBA|SYSASM) = [SYSASM] ****** Profile parameters for section OFFLOAD : ****** Synchronize database configuration file on the BS with PS {OVERWRITE_DATABASE_PARAMETER_FILE} (YES|NO) = [NO] Level of backup {DATABASE_BACKUP_INCREMENTAL_LEVEL} (<number>) = [0] ****** Profile parameters for section DEVICE_CLASS SVC_DC2: ****** Type of storage system or type of filesystem {COPYSERVICES_HARDWARE_TYPE} (DS8000|SVC|XIV|SAN_NSERIES|NAS_NSERIES|GPFS) = [SVC] Server name/address of primary storage device {COPYSERVICES_PRIMARY_SERVERNAME} = [SVC_2_IP] Username for primary storage device {COPYSERVICES_USERNAME} = [fcmuserDC2] Priority of background copy process {SVC_COPY_RATE} (<0-100> ) = [80] Is the FlashCopy/Snapshot taken from the mirror volumes {COPYSERVICES_REMOTE} (YES|NO) = [NO] Communication protocol {COPYSERVICES_COMMPROTOCOL} (HTTPS|HTTP) = [HTTPS] Flashcopy type {FLASHCOPY_TYPE} (INCR|COPY|NOCOPY) = [NOCOPY] Storage system ID of referred cluster for mirrored environments {STORAGE_SYSTEM_ID} = [SVC_02] Hostname of backup host {BACKUP_HOST_NAME} = [Proxy02] Target volumes set {TARGET_SETS} (VOLUMES_DIR|VOLUMES_FILE|<list of target set names>) = [VOLUMES_FILE] Filename of the target volumes file {VOLUMES_FILE} = [/home/fcmsid/acs/fcm-target-volumes.fct] ****** Profile parameters for section DEVICE_CLASS SVC_DC1: ****** Type of storage system or type of filesystem {COPYSERVICES_HARDWARE_TYPE} (DS8000|SVC|XIV|SAN_NSERIES|NAS_NSERIES|GPFS) = [SVC] Server name/address of primary storage device {COPYSERVICES_PRIMARY_SERVERNAME} = [SVC_1_IP] Username for primary storage device {COPYSERVICES_USERNAME} = [fcmuserDC1] Priority of background copy process {SVC_COPY_RATE} (<0-100> ) = [80] Is the FlashCopy/Snapshot taken from the mirror volumes {COPYSERVICES_REMOTE} (YES|NO) = [NO] Communication protocol {COPYSERVICES_COMMPROTOCOL} (HTTPS|HTTP) = [HTTPS] Flashcopy type {FLASHCOPY_TYPE} (INCR|COPY|NOCOPY) = [NOCOPY] Storage system ID of referred cluster for mirrored environments {STORAGE_SYSTEM_ID} = [SVC_01] Hostname of backup host {BACKUP_HOST_NAME} = [Proxy01] Target volumes set {TARGET_SETS} (VOLUMES_DIR|VOLUMES_FILE|<list of target set names>) = [VOLUMES_FILE] Filename of the target volumes file {VOLUMES_FILE} = [/home/fcmsid/acs/fcm-target-volumes.fct] FMM8426I Saving profile '/home/fcmsid/acs/profile' ... FMM1555I Profile successfully created. Do you want to continue by specifying passwords for the defined sections? [Y/N] [n] y Please enter the password for authentication with the ACS daemon: [***] Please re-enter password for verification: Please enter the password for primary storage device configured in section(s) SVC_DC2: [***] FMM8437I Verifying password... FMM0005I Start of program at: Wed 16 Jul 2014 03:01:47 PM UTC. FMM4184I CIM Agent version for SVC: '7.2.0'. FMM0020I End of program at: Wed 16 Jul 2014 03:01:47 PM UTC. FMM0021I Elapsed time: 00 sec. FMM6959I Script 'LD_LIBRARY_PATH=/home/fcmsid/acs/xpyv/lib /home/fcmsid/acs/fmcima -l /home/fcmsid/acs -f checkpw' returned with code 0. Please enter the password required for connecting to ORACLE catalog database: [***] FMM8437I Verifying password... Recovery Manager: Release 11.2.0.3.0 - Production on Wed Jul 16 15:01:55 2014 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. connected to target database: SID (DBID=136954576) connected to recovery catalog database RMAN> Recovery Manager complete. Please enter the password for primary storage device configured in section(s) SVC_DC1: [***] FMM8437I Verifying password... FMM0005I Start of program at: Wed 16 Jul 2014 03:02:04 PM UTC. FMM4184I CIM Agent version for SVC: '7.2.0'. FMM0020I End of program at: Wed 16 Jul 2014 03:02:04 PM UTC. FMM0021I Elapsed time: 00 sec. FMM6959I Script 'LD_LIBRARY_PATH=/home/fcmsid/acs/xpyv/lib /home/fcmsid/acs/fmcima -l /home/fcmsid/acs -f checkpw' returned with code 0. Creating password file at /home/fcmsid/acs/shared/pwd.acsd. A copy of this file needs to be available to all components that connect to acsd. TLS/SSL configuration: GSKit is installed correctly with the supported version (8.0.14.28). Found key database. Consistency check ok. Found FCM server certificate in key database.


IBM Tivoli Storage FlashCopy(R) Manager requires at least two daemon processes to be running always. IBM Tivoli Storage FlashCopy(R) Manager can create and start upstart jobs for the necessary daemon processes. Alternatively, you can choose to start and stop these processes yourself, for instance if you want to include the processes in your HA policy. Do you want IBM Tivoli Storage FlashCopy(R) Manager to create and start the upstart jobs for you? [y|n] [y] The offload backup to Tivoli Storage Manager is performed by the IBM Tivoli Storage FlashCopy(R) Manager 'tsm4acs' offload agent. Specify YES to offload the backup immediately after the FlashCopy backup finishes. This process creates and starts an upstart job configuration for 'tsm4acs' offload agent. The default is to run 'tsm4acs' as a daemon process on the production server. In High Availability environments, an upstart job is not created since the 'tsm4acs' agent must be added to the High Availability script. Specify NO to schedule offload backups individually with a crontab entry. Do you want off-loaded tape backups to begin immediately after the snapshot? [y|n] [n] Successfully installed upstart job fcmsid-acsd. Successfully installed upstart job fcmsid-acsgend. Currently no backup system is setup. To configure a backup system please select option n. n) configure a new backup system q) quit configuration Please select one of the options above. q Quit configuration

Note: In the example above, the parameter for synchronizing the DB configuration between PS and BS is set to “NO”: Synchronize database configuration file on the BS with PS {OVERWRITE_DATABASE_PARAMETER_FILE} (YES|NO) = [NO] During the very first configuration, the parameter needs to be set to YES instead – this will copy an SPfile extract from the PS to the BS in order to startup the database there for an offloaded backup.

When this was done, the SPfile copy needs to be changed manually on the BS, as some parameters are valid on the PS only. For details, see the BS configuration section of this document.

When the wizard completes, it writes down all information to /home/fcmsid/acs/profile .

The profile looks like this: fcmsid@racnode1:/home/fcmsid/acs % cat profile >>> GLOBAL ACS_DIR /home/fcmsid/acs ACSD racnode1 57328 # ENFORCE_TLS12 NO # TRACE NO <<< >>> ACSD ACS_REPOSITORY /home/fcmsid/acs/acs_repository # REPOSITORY_LABEL TSM # SYNCHRONOUS_RECONCILE RESTORE_AND_DELETE <<< >>> CLIENT # BACKUPIDPREFIX ORCL__ APPLICATION_TYPE ORACLE TARGET_DATABASE_SUSPEND YES VOLUME_MGR ASM TSM_BACKUP LATEST # MAX_VERSIONS ADAPTIVE # NEGATIVE_LIST NO_CHECK # TIMEOUT_FLASH 120 # GLODC1_SYSTEM_IDENTIFIER DEVICE_CLASS SVC_DC1 SVC_DC2 <<< >>> ORACLE CATALOG_DATABASE_CONNECT_STRING CATALOGNAME CATALOG_DATABASE_USERNAME CATALOGUSER DATABASE_BACKUP_SCRIPT_FILE /home/fcmsid/acs/rman_script.sh # TARGET_DATABASE_PARAMETER_FILE # DATABASE_CONTROL_FILE_RESTORE NO ASM_INSTANCE_USER oracle ASM_INSTANCE_ID +ASM2 ASM_ROLE SYSASM <<< >>> OFFLOAD BACKUP_METHOD ORACLE OVERWRITE_DATABASE_PARAMETER_FILE NO


# DATABASE_BACKUP_INCREMENTAL_LEVEL 0 ASM_INSTANCE_USER oracle ASM_INSTANCE_ID +ASM ASM_ROLE SYSASM <<< >>> DEVICE_CLASS SVC_DC2 COPYSERVICES_HARDWARE_TYPE SVC COPYSERVICES_PRIMARY_SERVERNAME SVC_2_IP COPYSERVICES_USERNAME fcmuserDC2 # CLONE_DATABASE NO # SVC_COPY_RATE 80 # SVC_CLEAN_RATE 50 # SVC_GRAIN_SIZE 256 # COPYSERVICES_REMOTE NO # COPYSERVICES_COMMPROTOCOL HTTPS # COPYSERVICES_CERTIFICATEFILE NO_CERTIFICATE # COPYSERVICES_SERVERPORT 5989 FLASHCOPY_TYPE NOCOPY # COPYSERVICES_TIMEOUT 6 # RESTORE_FORCE NO STORAGE_SYSTEM_ID SVC_02 # RECON_INTERVAL 12 BACKUP_HOST_NAME Proxy02 TARGET_SETS VOLUMES_FILE VOLUMES_FILE /home/fcmsid/acs/fcm-target-volumes.fct <<< >>> DEVICE_CLASS SVC_DC1 COPYSERVICES_HARDWARE_TYPE SVC COPYSERVICES_PRIMARY_SERVERNAME SVC_1_IP COPYSERVICES_USERNAME fcmuserDC1 # CLONE_DATABASE NO # SVC_COPY_RATE 80 # SVC_CLEAN_RATE 50 # SVC_GRAIN_SIZE 256 # COPYSERVICES_REMOTE NO # COPYSERVICES_COMMPROTOCOL HTTPS # COPYSERVICES_CERTIFICATEFILE NO_CERTIFICATE # COPYSERVICES_SERVERPORT 5989 FLASHCOPY_TYPE NOCOPY # COPYSERVICES_TIMEOUT 6 # RESTORE_FORCE NO STORAGE_SYSTEM_ID SVC_01 # RECON_INTERVAL 12 BACKUP_HOST_NAME Proxy01 TARGET_SETS VOLUMES_FILE VOLUMES_FILE /home/fcmsid/acs/fcm-target-volumes.fct <<<

5.3 Preparing the Backup Server (BS) We assume that the fcmsid user was created and the .bashrc was modified on the BS already, as described in chapter “TSFCM Installation” already.

In environments where ssh login as root is possible, the BS software installation and configuration can be performed from the production server, using the same wizard as described above.

While root ssh connects to servers are not allowed in many environments, the BS needs to be prepared, installed and configured manually:

5.3.1 Copy fcmselfcert.arm from PS to BS FlashCopy Manager uses the GSKit to encrypt communication between TSFCM processes running on different servers (PS and BS). During the configuration of the PS, a certificate file is created which needs to be copied manually to the /home/fcmsid/acs directory on the BS.

The certificate is imported to the local keystore and deleted afterwards when the configuration wizard is started.


5.3.2 Modify initSID.ora As described before, an extract of the database configuration is copied from PS to BS during the first offload backup run (which is triggered on the PS).

Ensure to have the target directories for the db config file available on the BS with proper permissions:

mkdir /oracle/SID/saptrace/audit (if it does not exist already) chmod g+w /oracle/SID/112/dbs

chmod g+w /oracle/SID/saptrace/diag/rdbms/sid/SID002/trace

When the first offload was triggered and the DB config file was copied, it needs to be adapted to the BS properties: vi /oracle/SID/112/dbs/initSID002.ora_fromSPfile

Change: *.cluster_database=TRUE to: *.cluster_database=FALSE

Remove unused or unreachable listeners. This depends on the Oracle configuration and is different for every environment.


5.4 Running the configuration Wizard on the Backup Server (BS) The BS hosts only one TSFCM component – the so called “Mount-Agent” (acsgen), which is responsible to mount the FC target disks to the BS, and to communicate with the local Oracle DB instance for startup and backup to TSM tape.

All main configuration resides on the production DB server, where the central TSFCM management deamon (acsd) is running.

Therefore the configuration wizard will create a profile with very limited content – it only tells the local acsgen where to find the remote acsd (IP and port).

This is how the config wizard is started on the BS: [fcmsid@Proxy01 acs]$ ./setup_ora.sh Select one of these configurations: (1) On-Site Production System configuration with optional remote Backup System configuration (2) On-Site Backup System configuration Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager on the production server with the option to remotely synchronize the configuration of one or multiple backup systems using Secure Shell. Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager on the backup system as a 'separate installation'. [2] Going to install on backup server Deactivating IBM Tivoli Storage FlashCopy(R) Manager management daemon. Select one of these configurations: (1) backup only (2) cloning only (3) backup and cloning Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager for backup only. Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager for cloning only. Enter '3' to configure IBM Tivoli Storage FlashCopy(R) Manager for backup and cloning. [1] Currently off-loaded backups are not configured. Do you want to configure off-loaded backups? [y|n] [n] After the configuration of the off-loaded backups, the CLIENT section will be removed from the profile. If you want to review the settings of the Client section to configure the .utl file, please have a look at the profile with the .bck file name extension in the directory where the profile is stored. Profile configuration ===================== Special commands: '?' -> Show help for current parameter '!d' -> Delete current parameter instance / delete user input FMM8421I Modifying existing profile '/home/fcmsid/acs/profile' for application 'n/a' ... ****** Profile parameters for section GLOBAL : ****** Path of the IBM Tivoli Storage FlashCopy(R) Manager directory {ACS_DIR} = [/home/fcmsid/acs] Hostname and port of machine running Management Agent {ACSD} (<hostname> <port>) = [racnode1 57328] FMM8426I Saving profile '/home/fcmsid/acs/profile' ... FMM1555I Profile successfully created. Recovery catalog connect string (CATALOG_DATABASE_CONNECT_STRING) = CATALOGNAME Username for Recovery Catalog database (CATALOG_DATABASE_USERNAME) = CATALOGUSER Please enter the password for authentication with the ACS daemon: Please re-enter password for verification: Please enter the password required for connecting to ORACLE catalog database: FMM8437I Verifying password... Recovery Manager: Release 11.2.0.3.0 - Production on Thu Jul 17 09:37:52 2014 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. connected to target database (not started) connected to recovery catalog database


RMAN> Recovery Manager complete. Creating password file at /home/fcmsid/acs/shared/pwd.acsd. A copy of this file needs to be available to all components that connect to acsd. TLS/SSL configuration: GSKit is installed correctly with the supported version (8.0.14.28). Found key database. Consistency check ok. Found FCM server certificate in key database. Enter the device classes to use for this Backup System. Make sure each device class is separated by a comma. SVC_DC1 IBM Tivoli Storage FlashCopy(R) Manager requires at least two daemon processes to be running always. IBM Tivoli Storage FlashCopy(R) Manager can create and start upstart jobs for the necessary daemon processes. Alternatively, you can choose to start and stop these processes yourself, for instance if you want to include the processes in your HA policy. Do you want IBM Tivoli Storage FlashCopy(R) Manager to create and start the upstart jobs for you? [y|n] [y] Successfully installed upstart job fcmsid-acsgenmntd.

This is the “minimalistic” profile on the backup server: [fcmsid@Proxy01 acs]$ cat profile >>> GLODC1 ACS_DIR /home/fcmsid/acs ACSD <racnode1-IP> 57328 # ENFORCE_TLS12 NO # TRACE NO <<<


6. Test Scenarios Before testing backup and restore with TSFCM, ensure that:

• The TSFCM configuration was completed

• All TSFCM daemons are up and running

o On Production Server (acsd and acsgen for SID are running in this example) fcmsid@racnode1:/home/fcmsid % ps -ef |grep acs root 14487 1 0 09:23 ? 00:00:00 /home/fcmsid/acs/acsgen -D fcmsid 18451 1 0 Jul16 ? 00:00:24 /home/fcmsid/acs/acsd

o On Backup Server (acsgen to handle “DC1” device classes on DC1 BS) [fcmsid@Proxy01 ~]$ ps -ef |grep acs root 30064 1 0 09:38 ? 00:00:00 /home/fcmsid/acs/acsgen -D -M -s SVC_DC1

The following examples show a complete backup and restore cycle using the TC target disks in DC1.

6.1 Disk-Only Backup (FlashCopy)

FlashCopy Backup to Target Disks: fcmsid@racnode1:/home/fcmsid/acs % ./fcm_backup.sh IBM Tivoli Storage FlashCopy(R) Manager for Oracle - Version 4, Release 1, Level 0.2 for Linux x86_64 - Build: 649J generated on Apr 25 2014 (c) Copyright IBM Corporation, 2000, 2014, All Rights Reserved. FMM0005I Start of program at: Tue 15 Jul 2014 09:28:40 AM UTC. FMM8800I The command is: backup FMM6201I Checking status of database. FMM6225I Create database parameter file '/oracle/SID/112/dbs/initSID002.ora_fromSPfile' from SPfile. FMM6223I Detected control file: +OLOG_SID/sid/cntrlsid.dbf FMM6223I Detected control file: +RECO_SID/sid/cntrlsid.dbf FMM6223I Detected control file: +ARCH_SID/sid/cntrlsid.dbf FMM6224I Create control file copy: /oracle/SID/saptrace/diag/rdbms/sid/SID002/trace/cntrl_copy_A0HXN0PK8R FMM1553I RMAN is logging to /home/fcmsid/acs/logs/rman.SID002.20140715092840.log FMM1510I New connection received from host racnode1. FMM1514I *****> Device client connected. FMM6219I Backup to Tivoli Storage Manager: YES LATEST FMM1582I The target set DC1_1 will be used for the current backup. FMM6901I Response to Init request. FMM6902I Response to Partition request. FMM4184I CIM Agent version for SVC: '7.2.0'. FMM0340I The following backup items were not assigned to corresponding snapshot items: 60050768019807B7280000000000014D 60050768019807B7280000000000014E FMM6269I The diskgroup 'DATA_SID' with 'NORMAL' redundancy and '2' failure groups has been partitioned with failure groups 'FG_DC1 '. FMM6903I Response to Prepare Flash request. FMM0357I FlashCopy type is 'NOCOPY'. FMM0368I Stopping the consistency group 'FCMHX308W0YSCRB' and all its older dependents. FMM6230I Set table space files in backup mode. FMM6212I Suspend database. FMM6905I Response to Flash request. FMM0589I Flushing the buffers to disk... FMM4183I Performing NOCOPY FlashCopy of source volume racnode1_VOL001 to target volume Proxy01_VOL001. FMM4183I Performing NOCOPY FlashCopy of source volume racnode1_VOL002 to target volume Proxy01_VOL002. FMM0142I Snapshot started ... FMM0143I Snapshot successful. FMM6214I Resume database. FMM6231I End backup mode for table space files. FMM6906I Response to Verify request. FMM6959I Script 'LD_LIBRARY_PATH=/home/fcmsid/acs/xpyv/lib /home/fcmsid/acs/fmcima -l /home/fcmsid/acs -N DEVICE_CLASS:SVC_DC1.0 -T 20140715092847 -f backup' returned with code 0. #SAVED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/psapsr3db.258.843651165


#SAVED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/psapsr3db.259.843651155 #SAVED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/psapundo001.256.843655413 #SAVED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/system.260.843650263 #SAVED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/sysaux.263.843650265 #SAVED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/psapundo002.264.843655439 #SAVED ORCL__A0HXN0PK8R +DATA_SID/sid/tempfile/psaptemp.261.843650269 FMM6217I Database switched to next logfile. FMM0020I End of program at: Tue 15 Jul 2014 09:29:17 AM UTC. FMM0021I Elapsed time: 37 sec. FMM0024I Return code is: 0. SQL*Plus: Release 11.2.0.3.0 Production on Tue Jul 15 09:29:17 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> System altered. SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options fcmsid@racnode1:/home/fcmsid/acs %

6.2 TSM Tape Offload The tsm offload is triggered on the production server (PS), and also the log file is located on the PS. Nevertheless, some rman specific log is created on the BS in parallel.

This is how the tape offload is triggered on PS:

fcmsid@racnode1:/home/fcmsid/acs % ./tsm_backup.sh IBM Tivoli Storage FlashCopy(R) Manager offload agent - Version 4, Release 1, Level 0.2 for Linux x86_64 - Build: 649J generated on Apr 25 2014 (c) Copyright IBM Corporation, 2000, 2014, All Rights Reserved. FMM0005I Start of program at: Tue 15 Jul 2014 09:48:39 AM UTC. FMM8800I The command is: tape_backup FMM6501I Initializing 'mount' request. FMM6502I Executing 'mount' request. FMM6901I Response to Init request. FMM4184I CIM Agent version for SVC: '7.2.0'. FMM0188I Trying to set the semaphore for the critical part of importing/exporting ... FMM0158I Start of listing of importing volume groups/mounting file systems ... FMM0189I Semaphore released. FMM0159I End of listing. FMM0138I Time stamp: 07/15/14-09:49:20. FMM6959I Script 'LD_LIBRARY_PATH=/home/fcmsid/acs/xpyv/lib /home/fcmsid/acs/fmcima -l /home/fcmsid/acs -N mount -f mount' returned with code 0. FMM6555I Selected snapshot backup with ID 'SID002,ORCL__,A0HXN0PK8R'. FMM6503I Terminating 'mount' request. FMM6557I The 'mount' request for database 'ORCL__' processed successfully. FMM6547I Do not overwrite database parameter file. FMM6520I Starting database instance 'SID002'. FMM6205I Changing Oracle mode to: startup nomount. FMM6521I Database instance 'SID002' was started successfully. FMM6545I Write control file +OLOG_SID/sid/cntrlsid.dbf FMM6545I Write control file +RECO_SID/sid/cntrlsid.dbf FMM6545I Write control file +ARCH_SID/sid/cntrlsid.dbf FMM1553I RMAN is logging to /home/fcmsid/acs/logs/rman.SID002.20140715094921.log FMM6548I Start backup of database instance 'SID002'. FMM1553I RMAN is logging to /home/fcmsid/acs/logs/rman.SID002.20140715094921.log FMM6549I Finished backup of database instance 'SID002' successfully. FMM6524I Stopping database instance 'SID002'. FMM6525I Database instance 'SID002' was stopped successfully. FMM6501I Initializing 'unmount' request. FMM6502I Executing 'unmount' request. FMM6901I Response to Init request.


FMM0188I Trying to set the semaphore for the critical part of importing/exporting ... FMM0189I Semaphore released. FMM0159I End of listing. FMM0138I Time stamp: 07/15/14-09:55:17. FMM4184I CIM Agent version for SVC: '7.2.0'. FMM0159I End of listing. FMM6959I Script 'LD_LIBRARY_PATH=/home/fcmsid/acs/xpyv/lib /home/fcmsid/acs/fmcima -l /home/fcmsid/acs -N unmount -f unmount' returned with code 0. FMM6555I Selected snapshot backup with ID 'SID002,ORCL__,A0HXN0PK8R'. FMM6503I Terminating 'unmount' request. FMM6557I The 'unmount' request for database 'ORCL__' processed successfully. FMM0020I End of program at: Tue 15 Jul 2014 09:55:23 AM UTC. FMM0021I Elapsed time: 06 min 44 sec. FMM0024I Return code is: 0.

This is the rman log from the BS: [fcmsid@Proxy01 logs]$ cat rman.SID002.20140715094921.log Recovery Manager: Release 11.2.0.3.0 - Production on Tue Jul 15 09:49:23 2014 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. connected to target database: SID (not mounted) connected to recovery catalog database RMAN> 2> 3> 4> RMAN> 2> 3> 4> allocated channel: c1 channel c1: SID=24 device type=DISK Starting restore at 15-JUL-14 channel c1: copied control file copy output file name=+OLOG_SID/sid/cntrlsid.dbf output file name=+RECO_SID/sid/cntrlsid.dbf Finished restore at 15-JUL-14 released channel: c1 RMAN> RMAN> Recovery Manager complete. Recovery Manager: Release 11.2.0.3.0 - Production on Tue Jul 15 09:49:31 2014 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. connected to target database: SID (DBID=136954576, not open) connected to recovery catalog database RMAN> 2> 3> 4> 5> 6> 7> 8> 9> RMAN> 2> 3> 4> 5> 6> 7> 8> 9> allocated channel: t1 channel t1: SID=24 device type=SBT_TAPE channel t1: Data Protection for Oracle: version 6.3.0.0 Starting backup at 15-JUL-14 Starting implicit crosscheck backup at 15-JUL-14 Finished implicit crosscheck backup at 15-JUL-14 Starting implicit crosscheck copy at 15-JUL-14 Crosschecked 50 objects Finished implicit crosscheck copy at 15-JUL-14 searching for all files in the recovery area cataloging files... no files cataloged RMAN-06554: WARNING: file 1 is in backup mode RMAN-06554: WARNING: file 2 is in backup mode RMAN-06554: WARNING: file 4 is in backup mode RMAN-06554: WARNING: file 5 is in backup mode RMAN-06554: WARNING: file 6 is in backup mode RMAN-06554: WARNING: file 7 is in backup mode channel t1: starting incremental level 0 datafile backup set channel t1: specifying datafile(s) in backup set input datafile file number=00006 name=+DATA_SID/sid/datafile/psapundo001.256.843655413 input datafile file number=00007 name=+DATA_SID/sid/datafile/psapundo002.264.843655439 input datafile file number=00004 name=+DATA_SID/sid/datafile/psapsr3db.259.843651155 input datafile file number=00005 name=+DATA_SID/sid/datafile/psapsr3db.258.843651165 input datafile file number=00002 name=+DATA_SID/sid/datafile/sysaux.263.843650265 input datafile file number=00001 name=+DATA_SID/sid/datafile/system.260.843650263 channel t1: starting piece 1 at 15-JUL-14 channel t1: finished piece 1 at 15-JUL-14 piece handle=df_852976181_2276_1 tag=TAG20140715T094941 comment=API Version 2.0,MMS Version 6.3.0.0 channel t1: backup set complete, elapsed time: 00:05:15 Finished backup at 15-JUL-14 Starting backup at 15-JUL-14 channel t1: starting full datafile backup set


channel t1: specifying datafile(s) in backup set input control file copy name=/oracle/SID/saptrace/diag/rdbms/sid/SID002/trace/cntrl_copy_A0HXN0PK8R channel t1: starting piece 1 at 15-JUL-14 channel t1: finished piece 1 at 15-JUL-14 piece handle=75pdeorj_1_1 tag=TAG20140715T095458 comment=API Version 2.0,MMS Version 6.3.0.0 channel t1: backup set complete, elapsed time: 00:00:07 Finished backup at 15-JUL-14 RMAN-06497: WARNING: control file is not current, control file AUTOBACKUP skipped released channel: t1 RMAN> RMAN> Recovery Manager complete. [fcmsid@Proxy01 logs]$

6.3 Disk-Only Restore (FlashBack) Before starting the restore, the database has to be stopped: root@racnode1:/root % su - oracle oracle@racnode1:+ASM2> crsctl stop resource ora.DATA_SID.dg CRS-2673: Attempting to stop 'ora.DATA_SID.dg' on 'racnode1' CRS-2677: Stop of 'ora.DATA_SID.dg' on 'racnode1' succeeded oracle@racnode1:+ASM2> crsctl status resource ora.DATA_SID.dg NAME=ora.DATA_SID.dg TYPE=ora.diskgroup.type TARGET=OFFLINE STATE=OFFLINE

The restore from disk is executed by a TSFCM command: fcmsid@racnode1:/home/fcmsid/acs % ./acsora -f restore IBM Tivoli Storage FlashCopy(R) Manager for Oracle - Version 4, Release 1, Level 0.2 for Linux x86_64 - Build: 649J generated on Apr 25 2014 (c) Copyright IBM Corporation, 2000, 2014, All Rights Reserved. FMM0005I Start of program at: Tue 15 Jul 2014 01:46:08 PM UTC. FMM8800I The command is: restore FMM6232I Looking for the latest backup. FMM6233I Restoring backup with ID ORCL__A0HXN0PK8R. FMM1510I New connection received from host racnode1. FMM1514I *****> Device client connected. FMM6901I Response to Init request. FMM6904I Response to Restore request. FMM4184I CIM Agent version for SVC: '7.2.0'. FMM0511I ====>Performing IBM Tivoli Storage FlashCopy(R) Manager PREPARE SNAP RESTORE command. FMM6905I Response to Flash request. FMM0511I ====>Performing IBM Tivoli Storage FlashCopy(R) Manager RESTORE command. FMM0138I Time stamp: 07/15/14-13:46:13. FMM0236I Disabling the volumes and filesystems ... FMM0188I Trying to set the semaphore for the critical part of importing/exporting ... #UNMOUNTING_FS FMM4183I Performing NOCOPY FlashCopy of source volume Proxy01_VOL001 to target volume racnode1_VOL001. FMM4183I Performing NOCOPY FlashCopy of source volume Proxy01_VOL002 to target volume racnode1_VOL002. FMM0142I Snapshot started ... FMM0143I Snapshot successful. FMM0225I Enabling the volumes and filesystems ... FMM0158I Start of listing of importing volume groups/mounting file systems ... FMM0591I Bringing up the volume groups... FMM0189I Semaphore released. FMM0159I End of listing. #FS_MOUNTED FMM0138I Time stamp: 07/15/14-13:46:31. FMM6907I Response to Complete Restore request. FMM6959I Script 'LD_LIBRARY_PATH=/home/fcmsid/acs/xpyv/lib /home/fcmsid/acs/fmcima -l /home/fcmsid/acs -N 0 -T 20140715134609 -f restore' returned with code 0. #RESTORED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/psapsr3db.258.843651165 #RESTORED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/psapsr3db.259.843651155 #RESTORED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/psapundo001.256.843655413 #RESTORED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/system.260.843650263 #RESTORED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/sysaux.263.843650265 #RESTORED ORCL__A0HXN0PK8R +DATA_SID/sid/datafile/psapundo002.264.843655439 #RESTORED ORCL__A0HXN0PK8R +DATA_SID/sid/tempfile/psaptemp.261.843650269 FMM0020I End of program at: Tue 15 Jul 2014 01:46:33 PM UTC. FMM0021I Elapsed time: 25 sec. FMM0024I Return code is: 0.


Afterwards, the database needs to be started and recovered: fcmsid@racnode1:/home/fcmsid % srvctl start instance -d SID -i SID002 -o mount fcmsid@racnode1:/home/fcmsid % sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Tue Jul 15 13:49:34 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> recover database Media recovery complete. SQL> alter database open; Database altered.

A small test can be performed to check that the DB is back online: fcmsid@racnode1:/home/fcmsid % sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Tue Jul 15 13:50:53 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> select count(*) from dba_users; COUNT(*) ----------

12 fcmsid@racnode1:/home/fcmsid % srvctl status database -d SID Instance SID002 is running on node racnode1

Note:

During a disk based restore (FlashBack), only one Failure Group (FG) is restored, while the other FG is set OFFLINE.

The FGs need to be re-synced after a restore!

6.4 TSM Tape Restore Before doing a restore from tape, the database has to be stopped. root@racnode1:/root % su - oracle oracle@racnode1:+ASM2> crsctl stop resource ora.DATA_SID.dg CRS-2673: Attempting to stop 'ora.DATA_SID.dg' on 'racnode1' CRS-2677: Stop of 'ora.DATA_SID.dg' on 'racnode1' succeeded oracle@racnode1:+ASM2> crsctl status resource ora.DATA_SID.dg NAME=ora.DATA_SID.dg TYPE=ora.diskgroup.type TARGET=OFFLINE STATE=OFFLINE

The tape restore is triggered from the PS by using a basic rman script in this example: root@racnode1:/root % su - oracle oracle@racnode1:+ASM2> crsctl stop resource ora.DATA_SID.dg CRS-2673: Attempting to stop 'ora.DATA_SID.dg' on 'racnode1' CRS-2677: Stop of 'ora.DATA_SID.dg' on 'racnode1' succeeded oracle@racnode1:+ASM2> crsctl status resource ora.DATA_SID.dg NAME=ora.DATA_SID.dg TYPE=ora.diskgroup.type TARGET=OFFLINE STATE=OFFLINE


fcmsid@racnode1:/home/fcmsid/acs % rman target / rcvcat USER/PASSWORD cmdfile rman_restore_script.sh mylog Recovery Manager: Release 11.2.0.3.0 - Production on Tue Jul 15 14:14:46 2014 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. connected to target database: SID (DBID=136954576, not open) connected to recovery catalog database RMAN> run 2> { 3> allocate channel t1 type 'sbt_tape' parms 'ENV=(TDPO_OPTFILE=/oracle/rman_sid002/tdpo/tdpo.db.opt)'; 4> 5> restore database; 6> recover database; 7> alter database open; 8> 9> release channel t1; 10> } 11> 12> allocated channel: t1 channel t1: SID=139 instance=SID002 device type=SBT_TAPE channel t1: Data Protection for Oracle: version 6.3.0.0 Starting restore at 15-JUL-14 channel t1: starting datafile backup set restore channel t1: specifying datafile(s) to restore from backup set channel t1: restoring datafile 00001 to +DATA_SID/sid/datafile/system.260.843650263 channel t1: restoring datafile 00002 to +DATA_SID/sid/datafile/sysaux.263.843650265 channel t1: restoring datafile 00004 to +DATA_SID/sid/datafile/psapsr3db.259.843651155 channel t1: restoring datafile 00005 to +DATA_SID/sid/datafile/psapsr3db.258.843651165 channel t1: restoring datafile 00006 to +DATA_SID/sid/datafile/psapundo001.256.843655413 channel t1: restoring datafile 00007 to +DATA_SID/sid/datafile/psapundo002.264.843655439 channel t1: reading from backup piece df_852976181_2276_1 channel t1: piece handle=df_852976181_2276_1 tag=TAG20140715T094941 channel t1: restored backup piece 1 channel t1: restore complete, elapsed time: 00:04:13 Finished restore at 15-JUL-14 Starting recover at 15-JUL-14 starting media recovery media recovery complete, elapsed time: 00:00:02 Finished recover at 15-JUL-14 database opened released channel: t1 Recovery Manager complete.


7. References

These Web sites provide useful references to supplement the information contained in this paper:

IBM Tivoli Storage FlashCopy Manager for UNIX and Linux:

http://www-01.ibm.com/support/knowledgecenter/SSGSG7_7.1.0/com.ibm.itsm.fcm.unx.doc/t_protect_fcmul.html

IBM Tivoli Storage FlashCopy Manager Hard and Software Requirements:

http://www-01.ibm.com/support/docview.wss?uid=swg21650875

IBM SAN Volume Controller:

http://www-03.ibm.com/systems/storage/software/virtualization/svc/index.html

Administering ASM Disk Groups:

https://docs.oracle.com/cd/B28359_01/server.111/b31107/asmdiskgrps.htm#OSTMG137

Oracle ASMLib:

http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html

Configuring Oracle ASMLib on Multipath Disks:

http://www.oracle.com/technetwork/topics/linux/multipath-097959.html

Linux Oracle RAC Cluster backup solution with Tivoli ... · Linux Oracle RAC Cluster backup...

Documents

Transcript of Linux Oracle RAC Cluster backup solution with Tivoli ... · Linux Oracle RAC Cluster backup...