EMC Solutions for Oracle Database 10g/11g for … Celerra Unified Storage Platform. ... Chapter 6...

EMC Solutions for Oracle Database 10g/11gfor Midsize Enterprises

EMC Celerra Unified Storage Platform

Applied Technology Guide

www.EMC.com

EMC Global Solutions 42 South Street

Hopkinton, MA 01748-9103 1-508-435-1000 www.EMC.com

http://www.emc.com/

Copyright © 2008, 2009 EMC Corporation. All rights reserved.

Published February 2009

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated.

All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly.

EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute.

No warranty of system performance or price/performance is expressed or implied in this document. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All other trademarks used herein are the property of their respective owners.

EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform


P/N H4161.4

2 EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform


Contents

About this Document .................................................................................................................... 11

Chapter 1 Executive Summary ...................................................................................................................... 13 Introduction ................................................................................................................................... 13 Tested configurations .................................................................................................................... 14 Typographic conventions .............................................................................................................. 14 Physically booted RAC solutions .................................................................................................. 15 Virtualized single-instance solutions ............................................................................................. 16 Pure NFS VMware HA cluster virtualized single-instance solution ............................................. 18 Solution diagrams .......................................................................................................................... 19

Chapter 2 Passwordless Authentication Using ssh ........................................................................................ 25 Multiple hosts/same user ............................................................................................................... 26 Single host/different user ............................................................................................................... 28

Chapter 3 Configuring EMC Replication Manager ....................................................................................... 31 Introduction ................................................................................................................................... 31 Components overview ................................................................................................................... 31 SnapView snapshot ....................................................................................................................... 32 Overview ............................................................................................................................... 32 SnapView snapshot replication ..................................................................................................... 32 SnapView clone ............................................................................................................................. 33 Prerequisites for SnapView clones ................................................................................................ 33 Configuring CLARiiON storage ................................................................................................... 34 Configuring Replication Manager ................................................................................................. 46 Adding a host ............................................................................................................................... 47 Configuring storage ....................................................................................................................... 49 Creating a storage pool .................................................................................................................. 54 Creating an application set for replication ..................................................................................... 58

Chapter 4 Logical and Physical Storage Backup: NFS .................................................................................. 65 Overview ....................................................................................................................................... 65 Logical storage backup benefits .................................................................................................... 66 Physical storage backup benefits ................................................................................................... 66 Logical storage backup using Celerra SnapSure ........................................................................... 67 Logical storage process overview ................................................................................................. 67

EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform 3 Applied Technology Guide

Contents

Initializing the logical storage backup process .............................................................................. 68 Iterating the logical backup process .............................................................................................. 75

Chapter 5 Logical and Physical Storage Backup: FCP .................................................................................. 79 Overview ....................................................................................................................................... 79 Benefits .......................................................................................................................................... 79 Logical storage backup benefits .................................................................................................... 79 Physical storage backup benefits ................................................................................................... 80 Logical storage backup using CLARiiON SnapView ................................................................... 80 Logical storage process overview .................................................................................................. 80 Logical storage backup using Replication Manager ...................................................................... 81 Physical storage backup using RMAN .......................................................................................... 89

Chapter 6 Restore and Recovery: NFS .......................................................................................................... 93 Overview ....................................................................................................................................... 93 Restore and recovery from logical storage backup ........................................................................ 93 Using rootfs_ckpt .......................................................................................................................... 93 Using file system commands ......................................................................................................... 96

Chapter 7 Restore and Recovery: FCP ........................................................................................................... 97 Overview ....................................................................................................................................... 97 Restore and recovery from a SnapView snapshot using Replication Manager ............................. 97 Restore and recovery from physical storage backup using RMAN ............................................. 103

Chapter 8 RMAN Physical Backup Using a De-duplication Array ............................................................ 105 Overview ..................................................................................................................................... 105 Setup ........................................................................................................................................ 105 Backup procedure ........................................................................................................................ 106 Conclusions ................................................................................................................................. 106

Chapter 9 Memory Configuration ................................................................................................................ 107 Tuning Huge Pages...................................................................................................................... 107 Memory configuration: Oracle 11g ............................................................................................. 108

Chapter 10 Creating a Test/Dev Solution Using SnapView Clone ................................................................ 111 Overview ..................................................................................................................................... 111 Creating and executing a Replication Manager job for SnapView clone .................................... 112 Mount and recovery of a target clone database using Replication Manager ............................... 118

Chapter 11 Creating a Test/Dev Solution Using Celerra SnapSure ............................................................... 123 Overview ..................................................................................................................................... 123 Use case scenarios ....................................................................................................................... 123 Prerequisites ............................................................................................................................. 124 Creating a test/dev database using Celerra writeable checkpoints .............................................. 124

Chapter 12 Advanced Protect Solution Using MirrorView and Data Guard ................................................. 135 MirrorView .................................................................................................................................. 135 Data Guard .................................................................................................................................. 138

Chapter 13 Direct NFS ................................................................................................................................... 141 Overview: Direct NFS and Oracle 11g ........................................................................................ 141 Comparison: Kernel NFS and DNFS .......................................................................................... 141 Kernel NFS ............................................................................................................................. 141 DNFS ............................................................................................................................. 141



Contents

Advantages of DNFS .................................................................................................................. 142 Performance ............................................................................................................................. 142 Data integrity ............................................................................................................................. 142 Management and setup ................................................................................................................ 142 EMC Celerra and DNFS .............................................................................................................. 143 Tested configuration .................................................................................................................... 143 IP storage server environment ..................................................................................................... 143 Network connections ................................................................................................................... 143 Enabling DNFS ........................................................................................................................... 143 Verifying that DNFS has been enabled ....................................................................................... 144

Chapter 14 Networking .................................................................................................................................. 147 Networking issues ....................................................................................................................... 147 Jumbo frames ............................................................................................................................. 147 Ethernet trunking and link aggregation ....................................................................................... 148 Configuring the public and private network ................................................................................ 149

Chapter 15 Oracle on VMware ...................................................................................................................... 151 Single-instance virtualized Oracle database server scale-out ...................................................... 151 Configuration .............................................................................................................................. 152 ESX server configuration ............................................................................................................ 154 Migrating a single-instance Oracle database using VMotion ...................................................... 158 Discovering SAN storage on VMware ........................................................................................ 161 VMware High Availability cluster configuration ........................................................................ 173 Network configuration ................................................................................................................ 173 NFS storage networking in a VMware HA cluster context ......................................................... 174

Chapter 16 Conclusion ................................................................................................................................... 177

EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform 5


Contents



Figures

Figure 1 Pure NFS physically booted solution: Oracle RAC and Celerra ................................ 19 Figure 2 Blended FCP/NFS physically booted solution: Oracle RAC and Celerra ................. 20 Figure 3 Pure NFS virtualized solution: Oracle RAC and Celerra ........................................... 21 Figure 4 Blended FCP/NFS virtualized solution: Oracle RAC and Celerra ............................. 22 Figure 5 Pure NFS VMware HA cluster solution: Oracle RAC and Celerra ........................... 23 Figure 6 Reserved LUN Pool (snap cache) status .................................................................... 33 Figure 7 Create Storage Group ................................................................................................. 34 Figure 8 Assign Storage Group Name ..................................................................................... 35 Figure 9 Storage group creation > success screen .................................................................... 36 Figure 10 Select LUNs ............................................................................................................... 37 Figure 11 Storage Group Properties > LUNs ............................................................................. 38 Figure 12 EMC Replication Storage > LUNs ............................................................................ 39 Figure 13 Clone Private LUNs ................................................................................................... 40 Figure 14 Create Storage Group > Clone target LUNs .............................................................. 41 Figure 15 Storage Group > Clone target LUNs .......................................................................... 42 Figure 16 Mount Host SG > Select target LUNs ....................................................................... 43 Figure 17 Storage Group Properties > Select target LUNs ........................................................ 44 Figure 18 Mount Host SG > LUNs ............................................................................................ 45 Figure 19 Add host to clone target storage group ...................................................................... 46 Figure 20 Select New Host ......................................................................................................... 47 Figure 21 Assign Host Name ..................................................................................................... 47 Figure 22 Discover Arrays > Discover and Update ................................................................... 48 Figure 23 Discover Arrays > Array Discovery complete ........................................................... 48 Figure 24 Hosts .......................................................................................................................... 49 Figure 25 Add Storage ............................................................................................................... 49 Figure 26 Confirmation screen ................................................................................................... 50 Figure 27 Add Storage Wizard > Welcome ............................................................................... 50 Figure 28 Add Storage Wizard > Select Storage Services ......................................................... 51 Figure 29 Array Connections ..................................................................................................... 51 Figure 30 Discover Storage ........................................................................................................ 52 Figure 31 Add Storage Wizard > Discovery complete ............................................................... 52 Figure 32 Add Storage Wizard > Select Target Devices ............................................................ 53 Figure 33 Completing the Add Storage Wizard ......................................................................... 53 Figure 34 Completing the Add Storage Wizard ......................................................................... 54


Figures

Figure 35 Completing the Add Storage Wizard ......................................................................... 54 Figure 36 New Storage Pool ...................................................................................................... 55 Figure 37 New Pool .................................................................................................................... 55 Figure 38 Add Storage to Pool ................................................................................................... 56 Figure 39 New Pool > LUNs ...................................................................................................... 56 Figure 40 New Pool > Storage > Devices in Pool ...................................................................... 57 Figure 41 New Application Set .................................................................................................. 58 Figure 42 Application Set Wizard > Welcome screen ............................................................... 59 Figure 43 Application Set Wizard > Welcome screen ............................................................... 59 Figure 44 Application Credentials > Oracle database credentials .............................................. 60 Figure 45 Application Credentials > ASM credentials ............................................................... 60 Figure 46 Application Set Wizard > New database instance ...................................................... 61 Figure 47 Application Set Wizard > Assign application set name ............................................. 61 Figure 48 Application Set Wizard > Completing the Application Set Wizard ........................... 62 Figure 49 Application set created ............................................................................................... 62 Figure 50 Logical backup process .............................................................................................. 67 Figure 51 New Job ..................................................................................................................... 81 Figure 52 Job Wizard > welcome screen .................................................................................... 82 Figure 53 Job Wizard > Job Name and Settings ........................................................................ 82 Figure 54 Job Wizard > Mount options ...................................................................................... 83 Figure 55 Job Wizard > Starting the Job .................................................................................... 83 Figure 56 Job Wizard > Users to be Notified ............................................................................. 84 Figure 57 Job Wizard > Completing the Job Wizard ................................................................. 84 Figure 58 Jobs ............................................................................................................................ 85 Figure 59 Selecting a job to run ................................................................................................. 86 Figure 60 Running a job > confirmation .................................................................................... 86 Figure 61 Executing a job .......................................................................................................... 87 Figure 62 Job executed successfully .......................................................................................... 87 Figure 63 Job status > Successful ............................................................................................... 88 Figure 64 Verifying Snapshot Session information .................................................................... 88 Figure 65 Using Navisphere to verify snapshot session information ......................................... 89 Figure 66 Restoring a snapshot .................................................................................................. 98 Figure 67 Restore Wizard > Replica to Restore ......................................................................... 99 Figure 68 Restore Wizard > Replica to Restore ......................................................................... 99 Figure 69 Confirmation screen > restore replica ...................................................................... 100 Figure 70 Restore Replica ........................................................................................................ 100 Figure 71 Restore replica completed successfully .................................................................... 101 Figure 72 Create New Job ........................................................................................................ 112 Figure 73 Job Wizard > Welcome screen ................................................................................. 113 Figure 74 Job Name and Settings ............................................................................................. 113 Figure 75 Replication Storage .................................................................................................. 114 Figure 76 Mount Options ......................................................................................................... 115 Figure 77 Starting the Job ........................................................................................................ 115 Figure 78 Users to be Notified ................................................................................................. 116 Figure 79 Users to be Notified ................................................................................................. 116 Figure 80 Cloning job created .................................................................................................. 117 Figure 81 Executing a clone job ............................................................................................... 117 Figure 82 Executing a clone job ............................................................................................... 118



Figures

Figure 83 Mount replication job ............................................................................................... 119 Figure 84 Mount Wizard > Select replica to mount ................................................................. 120 Figure 85 Mount Wizard > Mount Options .............................................................................. 121 Figure 86 Mount Replica > progress screen ............................................................................. 122 Figure 87 DNFS solution: EMC Celerra and Oracle 11g ......................................................... 143 Figure 88 Total physical memory available on ESX server ..................................................... 154 Figure 89 Total number of CPUs available on ESX server ...................................................... 155 Figure 90 Four VMs on the ESX server ................................................................................... 156 Figure 91 VMwareBoot ........................................................................................................... 156 Figure 92 Configure additional NICs ....................................................................................... 157 Figure 93 Mounting Celerra file systems using NFS on VMware hosts .................................. 158 Figure 94 Private IP address assigned for VMotion on ESX server mteoraesx5 ..................... 159 Figure 95 Private IP address assigned for VMotion on ESX server mteoraesx4 ..................... 159 Figure 96 Virtual switch name for mteoraesx5 ........................................................................ 160 Figure 97 Virtual switch name for mteoraesx4 ........................................................................ 161 Figure 98 mteoraesx4.solutions1.rtp.dg.com is connected ....................................................... 162 Figure 99 LUNs are accessible ................................................................................................. 163 Figure 100 LUNs are discovered................................................................................................ 164 Figure 101 Virtual Machine Properties ...................................................................................... 165 Figure 102 Select Device Type .................................................................................................. 166 Figure 103 Select Raw Device Mappings for your disk ............................................................. 167 Figure 104 Select the target LUN ............................................................................................... 168 Figure 105 Select Store with Virtual Machine as the datastore .................................................. 169 Figure 106 Select Physical as the compatibility mode ............................................................... 170 Figure 107 Select the Virtual Device Node ................................................................................ 171 Figure 108 Complete the configuration ...................................................................................... 172 Figure 109 Typical VMware HA cluster network configuration ............................................... 173 Figure 110 Using NFS to store vmdk files within a VMware HA cluster .................................. 175


Figures



About this Document

This document covers the procedures for validating various solutions of Oracle RAC 10g and 11g on the EMC Celerra Unified Storage Platform using both the NFS protocol and the FCP protocol. The areas covered include:

♦ Backup and recovery, including the use of a de-duplication array as a backup target

♦ Memory configuration

♦ Networking issues, such as the use of trunked Ethernet connections and jumbo frames

♦ Test/dev using writeable snapshots and cloning

♦ Virtualization

Purpose

Information in this document can be used as the basis for a solution build, white paper, best practices document, or training. Information in this document can also be used by other EMC organizations (for example, the technical services or sales organization) as the basis for producing documentation for a technical services or sales kit.

Audience

The primary target audiences for this document are database administrators, system administrators, storage administrators, and architects who analyze, design, implement, and maintain robust database and storage systems. Readers should be familiar with Oracle RAC 10g/11g software, basic Oracle Enterprise Linux and Red Hat Enterprise Linux system administration, basic networking, and EMC products. As such, readers should already be familiar with the installation and administration of their server operating environment and the Oracle RAC 10g/11g software. This document is intended for internal EMC personnel, EMC partners, and customers.

Related documents

EMC

The following documents provide additional, relevant information. Access to these documents is based on your login credentials. If you do not have access to the following content, contact your EMC representative:


About this Document



♦ Managing Celerra Volumes and File Systems Manually

♦ EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises Physically Booted Solutions EMC Celerra NS40 Unified Storage Platform – Reference Architecture

♦ EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises Virtualized Solutions EMC Celerra NS40 Unified Storage Platform – Reference Architecture

♦ EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform – Best Practices Planning

♦ EMC Solutions for Oracle Database 10g/11g for Midsize Enterprise EMC Celerra Unified Storage Platform Physically Booted Blended FCP/NFS Solution: RecoverPoint with CX Splitters – Applied Technology

♦ Celerra NS20/40 FC Series Network Server documentation

♦ CLARiiON CX4 series storage array documentation

Oracle

♦ The Oracle Technology Network

♦ Metalink, the Oracle support website

http://www.oracle.com/technology/index.html

http://metalink.oracle.com/

Chapter 1 Executive Summary

This chapter presents the following topics:

Introduction ................................................................................................................................... 13 Tested configurations .................................................................................................................... 14 Typographic conventions .............................................................................................................. 14 Physically booted RAC solutions .................................................................................................. 15 Virtualized single instance solutions ............................................................................................. 16 Solution diagrams .......................................................................................................................... 18

Introduction The EMC® Celerra® Unified Storage Platform is a remarkably versatile device. It provides both a world-class NAS device providing NFS access as well as a world-class midrange SAN device, through the front-end ports on the CLARiiON® CX4-240 backend storage array. In this solution, EMC has provided a unique solution that combines the high performance of FCP with the manageability of NFS in the context of Oracle RAC 10g/11g on Linux. This document contains detailed instructions on how to implement this solution in a customer environment.

The topics covered include:

♦ Passwordless authentication using ssh

♦ Backup and recovery using snapshots and a specialized de-duplication array

♦ Test/dev solution (SnapView™ clone and Celerra writeable SnapSure™ checkpoint)

♦ Configuration of Replication Manager for:

♦ Advanced Backup using SnapView snapshot

♦ Test/dev solutions using SnapView clone

♦ Advanced Protect solution (MirrorView™, Oracle Data Guard, and RecoverPoint with a CX splitter driver)

♦ Memory configuration

♦ Configuring the SAN with VMware and other virtualization issues


Executive Summary

These topics are presented in the context of Oracle RAC 10g/11g for Red Hat Enterprise Linux or Oracle Enterprise Linux and EMC Celerra over FCP and NFS.

Tested configurations Five separate configurations were tested:

♦ Pure NFS on a physically booted OS

♦ Blended FCP/NFS on a physically booted OS

♦ Pure NFS on a virtualized OS

♦ Blended FCP/NFS on a virtualized OS

♦ Pure NFS VMware High Availability (HA) cluster

In addition, both Oracle RAC 10g and 11g were tested on all configurations. Unless otherwise indicated, the comments in this document apply to both versions of Oracle RAC software.

Typographic conventions In this document, many steps are listed in the form of terminal output. This is referred to as a code listing. For example:

[root@prodcelerracs root]# nas_fs -name clone_archive1 \

> -type uxfs -create samesize=archive1 pool=clar_r5_performance

[root@prodcelerracs root]#

Note the following about code listings:

♦ Commands you type are shown in bold.

♦ For lengthy commands the backslash “\” character is used to show line continuation. While this is a common UNIX convention, it may not work in all cases. You should enter the command on one line.

♦ The use of ellipses (…) in the output indicates that lengthy output was deleted for brevity.

If a Celerra or Linux command is referred to in text it is indicated in bold and lowercase, like this: The fs_copy command. If a SQL or RMAN command is referred to in text, it is indicated in uppercase, like this: The ALTER DATABASE RENAME FILE command. A special font is not used in this case.



Executive Summary

Physically booted RAC solutions The following components were tested:

Table 1 Solution components: Physically booted RAC

Component Description Scale-Up OLTP • In a scale-up scenario, every customer of the database must be

able to see every row in every table. This is distinct from a scale-out scenario, in which the database is federated, and not all rows can be seen by all customers.

• In a single-database image context, scalability is provided by adding storage, network, and database-server CPU and memory.

• In a RAC context, scalability is provided by adding nodes to the RAC cluster.

• Real-world performance and capacity testing. • Utilizes an industry-standard OLTP database performance

benchmark, while providing only real-world tunings on a reasonably priced and configured platform.

Basic Backup and Recovery • Uses only the functionality provided by the database server and the operating system software to perform backup and recovery.

• Uses the database server’s CPU, memory, and I/O channels for all backup, restore, and recovery operations.

Advanced Backup and Recovery (snapshot)

• Uses additional software components at the storage layer to free up the database server’s CPU, memory, and I/O channels from the effects of operations relating to backup, restore, and recovery.

• Provides high-performance backup and restore operations, improved space efficiency, or other benefits in comparison to Basic Backup and Recovery.

Advanced Backup and Recovery (de-duplication)

• Saves acquisition, power, space and cooling costs by increasing the density of storage of Oracle database backups by using a specialized hardware de-duplication array.

Basic Protect • Uses tools provided by the operating system and database server software (in the same sense as Basic Backup) to provide disaster recovery.

• Uses the database server’s CPU, memory, and I/O channels for all operations relating to the disaster recovery configuration.

Advanced Protect • Uses additional software components at the storage layer to enable disaster recovery, thereby freeing up the database server’s CPU, memory, and I/O channels from the effects of these operations.

• Enables the creation of a writeable copy of the production database on the disaster recovery target, allowing this database to be used for operations such as backup, test/dev, and data warehouse staging.

Resiliency • Every significant layer of the solution is tested by introducing faults in an effort to cause the solution to fail. In the process, the entire solution is shown to be resilient to faults at every layer, including database clustering, networking, and storage.


Executive Summary



Component Description Test/dev • A running production OLTP database is cloned with minimal, if

any, performance impact on the production server, as well as no downtime. The resulting dataset is provisioned on another server for use for testing and development. This is a critical capability for many midsize enterprise customers.

• Both of the following scenarios have been validated: o Bringing up the target database as a single-instance

database o Bringing up the target database as a RAC

Virtualized single-instance solutions The following components were tested:

Table 2 Solution components: Virtualized single instance

Component Description Scale-Out OLTP • Real-world performance and capacity testing.

• Utilizes an industry-standard OLTP database performance benchmark, while providing only real-world tunings on a reasonably priced and configured platform.

• Scalability is provided by adding additional database instances that are not clustered and that access their own physical database. This assumes that the database application can be broken down into many small, independent databases, and that no single user needs to see the data of any other user outside of the database associated with that user. A typical example would be Software as a Service (SaaS).



Executive Summary


Component Description Advanced Backup and Recovery (snapshot)


• Provides high-performance backup and restore operations, improved space efficiency, or other benefits in comparison to basic backup and recovery.

Advanced Backup and Recovery (de-duplication)1

• Saves acquisition, power, space and cooling costs by increasing the density of storage of Oracle database backups by using a specialized hardware de-duplication array.

Basic Protect • Uses tools provided by the operating system and database server software (in the same sense as basic backup) to provide disaster recovery.


Advanced Protect • Uses additional software components at the storage layer to enable disaster recovery, thereby freeing up the database server’s CPU, memory, and I/O channels from the effects of these operations.

• Enables the creation of a writeable copy of the production database on the disaster recovery target, allowing this database to be used for operations such as backup, test/dev, and data warehouse staging.

Resiliency2 • Every significant layer of the solution is tested by introducing faults in an effort to cause the solution to fail. In the process, the entire solution is shown to be resilient to faults at every layer, including database clustering, networking, and storage.

Test/dev • A running production OLTP database is cloned with minimal, if any, performance impact on the production server, as well as no downtime. The resulting dataset is provisioned on another server for use for testing and development. This is a critical capability for many midsize enterprise customers.

1 Virtualized pure NFS solution only 2 Virtualized pure NFS solution only

Executive Summary

Pure NFS VMware HA cluster virtualized single-instance solution The following components were tested:

Table 3 Solution components: Pure NFS VMware HA cluster

Component Description Scale-Out OLTP • Real-world performance and capacity testing.

• Utilizes an industry-standard OLTP database performance benchmark, while providing only real-world tunings on a reasonably priced and configured platform.

• Scalability is provided by adding additional database instances that are not clustered and that access their own physical database. This assumes that the database application can be broken down into many small, independent databases, and that no single user needs to see the data of any other user outside of the database associated with that user. A typical example would be Software as a Service (SaaS).



Advanced Backup and Recovery (snapshot)


• Provides high-performance backup and restore operations, improved space efficiency, or other benefits in comparison to basic backup and recovery.

Basic Protect • Uses tools provided by the operating system and database server software (in the same sense as basic backup) to provide disaster recovery.


Test/dev • A running production OLTP database is cloned with minimal, if any, performance impact on the production server, as well as no downtime. The resulting dataset is provisioned on another server for use for testing and development. This is a critical capability for many midsize enterprise customers.



Executive Summary

Solution diagrams Figure 1 provides an overview of the pure NFS physically booted solution for Oracle RAC and Celerra:

Figure 1 Pure NFS physically booted solution: Oracle RAC and Celerra


Executive Summary

Figure 2 provides an overview of the blended FCP/NFS physically booted solution for Oracle RAC and Celerra:

Figure 2 Blended FCP/NFS physically booted solution: Oracle RAC and Celerra



Executive Summary

Figure 3 provides an overview of the pure NFS virtualized solution for Oracle RAC and Celerra.

Figure 3 Pure NFS virtualized solution: Oracle RAC and Celerra


Executive Summary

Figure 4 provides an overview of the blended FCP/NFS virtualized solution for Oracle RAC and Celerra.

Figure 4 Blended FCP/NFS virtualized solution: Oracle RAC and Celerra



Executive Summary

Figure 5 provides an overview of the pure NFS VMware HA cluster solution for Oracle RAC and Celerra.

Figure 5 Pure NFS VMware HA cluster solution: Oracle RAC and Celerra


Executive Summary



Chapter 2 Passwordless Authentication Using ssh


Multiple hosts/same user ............................................................................................................... 26 Single host/different user ............................................................................................................... 28

A fundamental concept to make successful use of Oracle RAC 10g or 11g with Celerra is the use of passwordless authentication using ssh. The ssh passwordless authentication methodology relies on the three files shown in Table 4.

Table 4 ssh files

File Created by Purpose ~/.ssh/id_dsa.pub ssh-keygen Contains the host’s dsa key

for ssh authentication (functions as the proxy for a password)

~/.ssh/authorized_keys ssh Contains the dsa keys of hosts that are authorized to log in to this server without issuing a password

~/.ssh/known_hosts ssh Contains the dsa key and hostname of all hosts that are allowed to log in to this server using ssh

Of these three files, the most important is id_dsa.pub. Once you have established a passwordless authentication for a host onto another host, if the id_dsa.pub file is re-created, the passwordless authentication will cease to work. Therefore, if ssh-keygen is run and it discovers that id_dsa.pub already exists, do not accept the option to overwrite this file.


Passwordless Authentication Using ssh

To enable passwordless authentication using ssh:

1. Create the dsa_id.pub file using ssh-keygen.

2. Copy the key for the host for which authorization is being given to the authorized_keys file of the host that allows the login.

3. Complete a login so that ssh knows about the host that is logging in (that is, record the host’s key and hostname in the known_hosts file).

Multiple hosts/same user Complete steps 2 and 3 (above) on a single host, and then copy the authorized_keys and known_hosts files to the other hosts.

To use this script, copy and paste the text into a new file on the Linux server, edit the variable definitions at the top of the script, chmod the script to allow it to be executed, and run the script. At the end of this process, all of the equivalent users on the set of hosts will be able to log in to all of the other hosts without issuing a password.

The ssh_multi_handler.bash script accomplishes this. Use the ssh_multi_handler.bash script to complete the steps for passwordless authentication for the same user on a set of hosts. This is a very common task when setting up Oracle RAC 10g prior to installation of Oracle Clusterware.

Code Listing 1: ssh multi-host handler

#!/bin/bash

#-----------------------------------------------------------#

# Script: ssh_multi_handler.bash #

# Purpose: Handles creation of authorized_keys #

#-----------------------------------------------------------#

ALL_HOSTS="rtpsol347 rtpsol348 rtpsol349 rtpsol350"

THE_USER=root

mv -f ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak

mv -f ~/.ssh/known_hosts ~/.ssh/known_hosts.bak

for i in ${ALL_HOSTS}

do

ssh ${THE_USER}@${i} "ssh-keygen -t dsa"

ssh ${THE_USER}@${i} "cat ~/.ssh/id_dsa.pub" \

>> ~/.ssh/authorized_keys

ssh ${THE_USER}@${i} date

done




for i in $ALL_HOSTS

do

scp ~/.ssh/authorized_keys ~/.ssh/known_hosts \

${THE_USER}@${i}:~/.ssh/

done

for i in ${ALL_HOSTS}

do

for j in ${ALL_HOSTS}

do

ssh ${THE_USER}@${i} "ssh ${THE_USER}@${j} date"

done

done

mv -f ~/.ssh/authorized_keys.bak ~/.ssh/authorized_keys

mv -f ~/.ssh/known_hosts.bak ~/.ssh/known_hosts

exit

On our systems with the settings noted previously, this script produced the following effect:

Code Listing 2: ssh multi-host output

[root@rtpsol347 ~]# ./ssh_multi_handler.bash

Enter file in which to save the key (/root/.ssh/id_dsa):

Generating public/private dsa key pair.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_dsa.

Your public key has been saved in /root/.ssh/id_dsa.pub.

The key fingerprint is:

f8:21:61:55:55:92:15:ed:0a:62:89:c5:ed:93:5f:27 [email protected]

root@rtpsol347's password:

Tue Aug 8 22:21:31 EDT 2006


...(additional similar output not shown)

authorized_keys 100% 5008 4.9KB/s 00:00

known_hosts 100% 2189 2.1KB/s 00:00




...<repeated 3 times>

Tue Aug 8 22:22:05 EDT 2006

...<repeated 15 times>

[root@rtpsol347 ~]#

The 16 date outputs, without any requests for passwords, indicate that the passwordless authentication files on all root users among these four hosts have been successfully created.

Single host/different user The other common task is to set up passwordless authentication across two users between two hosts. For example, enable the Oracle user on the database server to run commands as the root or nasadmin user on the Celerra Control Station. The ssh_single_handler.bash script accomplishes this:

Code Listing 3: ssh single host handler

#!/bin/bash

#-----------------------------------------------------------#

# Script: ssh_single_handler.bash #

# Purpose: Handles creation of authorized_keys #

#-----------------------------------------------------------#

THE_USER=root

THE_HOST=rtpsol33

ssh-keygen -t dsa

KEY=`cat ~/.ssh/id_dsa.pub`

ssh ${THE_USER}@${THE_HOST} "echo ${KEY} >> \

~/.ssh/authorized_keys"

ssh ${THE_USER}@${THE_HOST} date

exit

The effect of this script is to create passwordless authentication from the presently logged in user to the root user on the Celerra Control Station. This script produces the following output:

Code Listing 4: ssh single host output

[oracle@rtpsol347 scripts]$ ./ssh_single_handler.bash

Generating public/private dsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:




Your identification has been saved in /home/oracle/.ssh/id_dsa.

Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.

The key fingerprint is:

09:13:4d:7d:20:0c:9a:c4:4e:35:c9:c9:11:9e:30:31 [email protected]

Wed Aug 9 09:40:01 EDT 2006

[oracle@rtpsol347 scripts]$

The date output without a password request indicates that the passwordless authentication files have been created.


Chapter 3 Configuring EMC Replication Manager


Introduction ................................................................................................................................... 31 Components overview ................................................................................................................... 31 SnapView snapshot ....................................................................................................................... 32 SnapView clone ............................................................................................................................. 33 Configuring Replication Manager ................................................................................................. 46

Introduction EMC Replication Manager automates the creation and management of EMC disk-based point-in-time replicas. Replication Manager integrates with the Oracle database server and provides an easy interface to create and manage Oracle replicas.

This chapter provides information on how to configure Replication Manager.

Components overview Replication Manager consists of the following software components:

Replication Manager Server

♦ This component manages replicas, controls replication jobs, and stores data about each replica. The server component resides on the server host.

♦ In the solutions described in this white paper, Replication Manager Server was installed on a VM running Microsoft Windows.

Replication Manager Agent

♦ This component prepares a host machine to create or mount a replica. It consists of an interface to communicate with the database or file system that is being replicated.The agent software is installed on each host that participates in the replication process, including hosts that manage production data and hosts that are used to mount replicas.

♦ In the solutions covered in this white paper, Replication Manager Agent was installed on each of the physically-booted RAC nodes.


Configuring EMC Replication Manager



Replication Manager Console

♦ This component is the graphical user interface that controls the server and agent software locally or remotely. The console is a portable Java application and must be installed on a Windows machine.

♦ In the solutions described in this white paper, Replication Manager Console was installed on a VM running Microsoft Windows.

SnapView snapshot Overview

A snapshot is a virtual point-in-time copy of a LUN. This virtual copy is assembled by using a combination of data in the source LUN, and the before images of updated blocks that are stored on the CLARiiON backend array in the Reserved LUN Pool (RLP).

Note: In Replication Manager, the RLP is referred to as the snap cache. For the purposes of this white paper, the RLP and the snap cache will be considered to be synonymous. However, we will adhere to Replication Manager terminology and use the term “snap cache.”

SnapView snapshot replication

Guidelines for configuring free LUNs in the snap cache

The number and size of LUNs in the snap cache varies based on the requirements. The main factors you should consider when configuring the LUNs in the snap cache are as follows:

♦ Rate of change of source LUN data

♦ Number of concurrent sessions

♦ Expected duration of SnapView sessions

♦ Snapshot write rate

We recommend the following best practices:

♦ Allocate at least one LUN for each source LUN.

♦ It is more efficient to allocate a large number of LUNs of relatively small size, than to allocate a smaller number of LUNs of large size.

Preparing for SnapView snapshot replication

Before beginning SnapView snapshot replication, carry out the following steps:

1. Verify that free LUNs are configured in the snap cache.

2. Dedicate enough free LUNs in the snap cache, as outlined in the guidelines above, to support the associated replications.

Note: It is essential to have an adequate number of LUNs in the snap cache because the CLARiiON backend array will terminate the SnapView session if the snap cache runs out of space.

Figure 6 shows the status of the snap cache. Note that in Navisphere®, the snap cache is refered to as the Reserved LUN Pool.


Figure 6 Reserved LUN Pool (snap cache) status

SnapView clone Prerequisites for SnapView clones

Before creating SnapView clones, carry out the following steps:

♦ Create a storage group named “EMC Replication Storage” and populate it with target LUNs created to store replicas.

♦ The source LUNs (the LUNs that are used to store the source data on the production server) and the target LUNs (the LUNs that are used to store the cloned data on the target server) must be identical in size and must contain an identical number of blocks.

♦ Verify that there are enough clone private LUNs, and that they are the required minimum size.

♦ The minimum size for a clone private LUN is 1 GB per storage processor (SP) for CX4 series CLARiiON arrays, and 128 MB per SP for all other CLARiiON arrays.

♦ Create a mount storage group for each mount host and make sure that the storage group contains at least one LUN, and that the LUN is visible to the mount host.

♦ In the case of VMware hosts, move the target LUNs to the mount storage group.



Configuring CLARiiON storage

Carry out the following steps to configure CLARiiON storage to use Replication Manager and SnapView to create clones for backup and recovery, and test/dev:

1. Create a Storage Group

1. In Navisphere Manager, click the Storage tab, right-click Storage Groups and select Create Storage Group as shown in Figure 7.

Figure 7 Create Storage Group




2. Type “EMC Replication Storage” in the Storage Group Name field, as shown in Figure 8.

The name “EMC Replication Storage” must be used in order for Replication Manager to work correctly. The name is case sensitive.

Figure 8 Assign Storage Group Name

3. Click Apply, then click OK.



4. An alert screen is displayed to notify you that the storage group has been successfully created, as shown in Figure 9.

Figure 9 Storage group creation > success screen




2. Add LUNs to the Storage Group

1. In Navisphere Manager, under Storage Groups right-click EMC Replication Storage and choose Select LUNs, as shown in Figure 10.

Figure 10 Select LUNs



2. The Storage Group Properties tab is displayed.

3. In the LUNs tab, select all the LUNs that are used to store the database. Click Apply.

Figure 11 Storage Group Properties > LUNs




4. The selected LUNs are now visible under the EMC Replication Storage group as shown in Figure 12.

Figure 12 EMC Replication Storage > LUNs



5. To enable the CLARiiON cloning feature, two LUNs with a minimum capacity of 1 GB (for CX4 series) must be designated as clone private LUNs. These LUNs will be used during cloning. Figure 13 shows LUNs that are configured as clone private LUNs.

Figure 13 Clone Private LUNs




6. Create a separate storage group for clone target LUNs:

a. In Navisphere Manager, click the Storage tab, right-click Storage Groups and select Create Storage Group.

b. Type “Mount Host SG” in the Storage Group Name field, as shown in Figure 14.

Figure 14 Create Storage Group > Clone target LUNs



7. The storage group created for clone target LUNs is shown in Figure 15.

Figure 15 Storage Group > Clone target LUNs




8. Add target LUNs to the Mount Host SG storage group:

a. Select Mount Host SG and right-click.

b. Click Select LUNs.

Figure 16 Mount Host SG > Select target LUNs



9. The Storage Group Properties tab is displayed, as shown in Figure 17.

10. Select the target LUNs that are to be used for cloning and then click Apply.

Figure 17 Storage Group Properties > Select target LUNs




11. The target LUNs to be used for cloning now reside in the storage group named “Mount Host SG” as shown in Figure 18.

Figure 18 Mount Host SG > LUNs





12. Add the host to the clone target storage group, so that the clone target LUNs can be mounted on the target storage group to bring up the test/dev copy of the database.

Figure 19 Add host to clone target storage group

Configuring Replication Manager Note: The procedure for preparing and installing Replication Manager components is outside the scope of this document.

After installing Replication Manager Server and Agents, basic configuration involves:

1. Adding a host

2. Adding storage

3. Creating an application set


Adding a host

1. In the Replication Manager console, select and right-click Hosts then select New Host as shown in Figure 20.

Figure 20 Select New Host

2. Type a name for the host (database server name), as shown in Figure 21. Click OK.

Figure 21 Assign Host Name



3. The Discover and Update progress bar is displayed as shown in Figure 22.

Figure 22 Discover Arrays > Discover and Update

4. When the progress bar reaches 100% as shown in Figure 23, click Close.

Figure 23 Discover Arrays > Array Discovery complete




5. The new host is now visible in the Hosts list as shown in Figure 24.

Figure 24 Hosts

6. To add all other hosts (database servers) and the mount host server to Replication Manager, repeat steps 1 to 5.

Configuring storage

After you have added the hosts and the mount host, you must then add the storage.

1. In Replication Manager, select Storage Services, right-click and select Add Storage as shown in Figure 25.

Figure 25 Add Storage



2. Click Yes, as shown in Figure 26.

Figure 26 Confirmation screen

3. The welcome screen of the Add Storage Wizard is displayed as shown in Figure 27. Click Next.

Figure 27 Add Storage Wizard > Welcome




4. The Select Storage Services screen is displayed, as shown in Figure 28.

Figure 28 Add Storage Wizard > Select Storage Services

5. Select the appropriate storage service and double-click.

6. The Array Connections screen is displayed as shown in Figure 29.

Type the login credentials for the storage array, then click OK.

Figure 29 Array Connections



7. The Discover Storage screen and progress bar are displayed as shown in Figure 30.

Figure 30 Discover Storage

8. When the progress bar reaches 100% as shown in Figure 31, click Close.

Figure 31 Add Storage Wizard > Discovery complete




9. The Select Target Devices screen is displayed as shown in Figure 32.

Figure 32 Add Storage Wizard > Select Target Devices

10. Select the snap cache for snapshots and the LUNs. Click Next.

11. Click Finish.

Figure 33 Completing the Add Storage Wizard



12. The Add Storage progress screen is displayed, as shown in Figure 34. When the progress bar reaches 100%, click Close.


13. The storage array is now visible in the Storage Services list as shown in Figure 35.


Creating a storage pool

A dedicated storage pool for cloning must be created so that the specified LUNs in that pool can be selected as cloning targets.

To create a new storage pool carry out the following steps:




1. In Replication Manager, select Storage Pools and right-click and select New Storage Pool as shown in Figure 36.

Figure 36 New Storage Pool

2. The New Pool screen is displayed.

Type a pool name and a description, then click Add as shown in Figure 37.

Figure 37 New Pool



3. Select the LUNs that you want to include in the storage pool, as shown in Figure 38. Click OK.

Figure 38 Add Storage to Pool

4. The LUNs are then updated as shown in Figure 39. Click Add.

Figure 39 New Pool > LUNs




5. Click Close.

Figure 40 New Pool > Storage > Devices in Pool



Creating an application set for replication

After you have added the storage to Replication Manager, you must then create an application set using the Replication Manager console.

1. In Replication Manager, select Application Sets, right-click and select New Application Set as shown in Figure 41.

Figure 41 New Application Set




2. The welcome screen of the Application Set Wizard is displayed, as shown in Figure 42. Click Next.

Figure 42 Application Set Wizard > Welcome screen

3. Click Add Instance to create a database instance, as shown in Figure 43.

Figure 43 Application Set Wizard > Welcome screen



4. The Application Credentials screen is displayed.

5. Type the Oracle database credentials (as shown in Figure 44) or the ASM credentials (as shown in Figure 45) and click OK.

Figure 44 Application Credentials > Oracle database credentials

Figure 45 Application Credentials > ASM credentials




6. The database instance created is shown in Figure 46.

Figure 46 Application Set Wizard > New database instance

7. Select the newly created instance and type a name for the application set that will be created, as shown in Figure 47. Click Next.

Figure 47 Application Set Wizard > Assign application set name



8. The Completing the Application Set Wizard screen is created, as show in Figure 48. To create the application set click Finish.

Figure 48 Application Set Wizard > Completing the Application Set Wizard

9. The application set is created as shown in Figure 49.

Figure 49 Application set created




This completes steps that are required to configure CLARiiON storage to use Replication Manager and SnapView to create clones for backup and recovery, and test/dev.


Chapter 4 Logical and Physical Storage Backup: NFS


Overview ....................................................................................................................................... 65 Logical storage backup benefits .................................................................................................... 66 Physical storage backup benefits ................................................................................................... 66 Logical storage backup using Celerra SnapSure ........................................................................... 67 Logical storage process overview ................................................................................................. 67 Initializing the logical storage backup process .............................................................................. 68 Iterating the logical backup process .............................................................................................. 75

Overview In the context of this document, the term “logical storage backup” is distinguished from the term “logical backup” which is generally used within the Oracle community and which is a backup performed using the import/export utilities. A logical storage backup is a backup that does not physically exist. Rather, it consists of the blocks in the active file system, combined with blocks in a SavVol, an area where the original versions of the updated blocks are retained. The effect of a logical storage backup is that a view of the file system as of a certain point in time can be assembled.

A logical storage backup is also distinguished from a “physical storage backup”, which is a full and complete copy of the database to a different physical media. Unlike a physical storage backup, a logical storage backup can be taken very rapidly, and requires very little space to store (typically a small fraction of the size of a physical storage backup).

An intermediary case is the use of a de-duplication array. In this case, the physical backup exists, but duplicate blocks are not stored. The de-duplication array detects the presence of duplicate blocks, and stores only one copy of each block in a manner transparent to the user. The use of a de-duplication array can save significantly in storage density which leads to savings in cost of acquisition, power, space, cooling and the like. The use of the de-duplication array to perform physical backup is covered in “RMAN Physical Backup Using a De-duplication Array.”


Logical and Physical Storage Backup: NFS

Logical storage backup benefits At the center of the Celerra backup approach is the Celerra SnapSure feature, which uses the checkpoint command (fs_ckpt). This command allows a database administrator (DBA) to capture a logical storage backup of the entire file system. Both logical storage and physical storage backups are helpful to the DBA.

Logical storage backups have the following advantages:

• Because the operation is very lightweight, a logical storage backup can be taken very frequently. Most customers report that they cannot perceive the performance impact of this operation because it is so slight.

• Restoring from a logical storage backup can occur very quickly, depending on the amount of data changes. This dramatically reduces mean time to recovery (MTTR) over that which can be achieved restoring from a physical backup.

• Due to the high frequency of backups, a small number of archived redo log files need to be applied if a recovery is needed. This further reduces mean time to recovery.

Of course, a logical storage backup does not provide protection from hardware failures. That is the purpose of the physical storage backup, which is covered in the next section. However, it does protect against logical corruption of the database, as well as accidental file deletion, and other similar human errors.

Physical storage backup benefits Taking logical storage backups is not enough to protect the database from all risks. Physical storage backups are also required to protect the database against double disk failures and other hardware failures at the storage layer.




Logical storage backup using Celerra SnapSure A logical storage backup is described as a virtual copy of the datafiles. This is enabled by the Celerra SnapSure checkpoint feature. Figure 50 illustrates the process.

Figure 50 Logical backup process

The datafile copies can be used in the same manner as any other backup of the datafiles.

Logical storage process overview

A typical backup scheme would use six logical storage backups per day, at four-hour intervals, combined with one physical storage backup per day. The example scripts shown can be integrated into the Oracle Enterprise Manager job scheduling process or cron, having them execute every four hours.

Documentation on scheduling a job in this way can be found at: http://download-east.oracle.com/docs/cd/B19306_01/server.102/b14231/jobtosched.htm

The process documented here assumes a four-hour interval.

To initialize the process, create six checkpoints using a naming convention that allows the process of refreshing these checkpoints to be coordinated with the current time, as reported by the operating system. Also, create read-only mount points for these checkpoints on the database server, and mount them. The iterative process (which will occur at regular intervals, as defined in the settings within the Oracle job scheduler) of backing up an Oracle database using Celerra SnapSure checkpoint logical storage backups is as follows:

Snapshot

Datafile storage

Oracle 10g/11g database server

EMC Celerra

Datafiles

Datafile copies


http://download-east.oracle.com/docs/cd/B19306_01/server.102/b14231/jobtosched.htm


1. Take the entire database into backup mode.

2. Refresh the appropriate checkpoint for that interval. Again, this uses the current time as reported by the operating system.

3. Take the database out of backup mode. Since the checkpoint refreshes very quickly, the time spent in backup mode is therefore minimal.

The scripts shown in these examples assume that the passwordless authentication is set up using ssh between the oracle user account and the Celerra Control Station. Instructions on how to accomplish this can be found in the “Passwordless Authentication Using ssh” section. This allows the oracle user account to issue commands to the Control Station within a script.

The following section provides a detailed illustration of the logical storage backup process.

Initializing the logical storage backup process

The example script in this section initializes a logical storage backup process using a four-hour interval, or six daily checkpoints. To do this, it performs the following steps:

1. Create six checkpoints on the Celerra for the datafile volume.

2. Create mount points for the checkpoint file systems on the Celerra.

3. The checkpoint file systems will be automatically mounted with read-only permission on the Celerra.

4. Export the checkpoint file systems from the Celerra.

5. Create read-only mount points for the checkpoint file systems on the Oracle database server.

6. Create the mount point directories for the checkpoint file systems, and mount them on the Oracle database server.

The following script, log_bkup_init.bash, accomplishes this:

Code Listing 5: Logical backup initialization script

#!/bin/bash

#-----------------------------------------------------------#

# Script: log_bkup_init.bash #

# Purpose: Initializes fs_ckpt Oracle hot backup process #

# This script must be run as root #

#-----------------------------------------------------------#

THE_NS=mteorans40-1

THE_DM=192.168.4.25

THE_USER=root

THE_FS=datafs

echo "Creating six checkpoints for production file system datafs"




for i in 0 1 2 3 4 5

do

echo "Creating checkpoint for production file system datafs"

ssh ${THE_USER}@${THE_NS} "export NAS_DB=/nas; \

/nas/bin/fs_ckpt ${THE_FS} -name ${THE_FS}_ckpt${i} \

-Create"

echo "Checkpoint created will automatically get mounted with read only permission on the datamover on which filesystem datafs is mounted"

ssh ${THE_USER}@${THE_NS} "export \

NAS_DB=/nas;/nas/bin/server_mount server_2 | grep \

${THE_FS}_chkpt${i}"

echo "Exporting the checkpoint filesystem to database server


/nas/bin/server_export server_2 -o \

root=192.168.4.51:192.168.4.52:192.168.4.53:192.168.4.54 \

/${THE_FS}_ckpt${i}"

grep -q ${THE_FS}_ckpt${i} /etc/fstab || echo \

"${THE_DM}:/${THE_FS}_ckpt${i} /mnt/${THE_FS}_ckpt${i} \

nfs \

ro,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0" >> /etc/fstab

[ ! -d /mnt/${THE_FS}_ckpt${i} ] && mkdir -p \

/mnt/${THE_FS}_ckpt${i}

done

mount -a

mount

Remember that the lines ending with the backslash (\) symbol should be entered on one line. Also remember that this script must be run as root. Running this script on our system produces the following results (output elided for brevity):

Code Listing 6: Logical backup initialization output

[root@mteoradb51 scripts]# ./log_bkup_init.bash

Creating six checkpoints for production file system datafs which will be refreshed later based on the current time



Creating checkpoint for production file system datafs

"EMC" is a registered trademark of EMC Corporation, and "Linux"

is a registered trademark of Linus Torvalds. "Celerra" is a

trademark of EMC.

EMC Celerra Control Station Linux 1.0

operation in progress (not interruptible)...


operation in progress (not interruptible)...id = 118

name = datafs

acl = 0

in_use = True

type = uxfs

worm = off

volume = v252

pool = datapool

member_of = root_avm_fs_group_35

rw_servers= server_2

ro_servers=

rw_vdms =

ro_vdms =

auto_ext = no,virtual_provision=no

ckpts = datafs_chkpt0

stor_devs = APM00064500161-0020,APM00064500161-001F,APM00064500161-0023,APM00064500161-0016,APM00064500161-0011

disks = d20,d27,d29,d33,d8

disk=d20 stor_dev=APM00064500161-0020 addr=c0t2l0 server=server_2


disk=d27 stor_dev=APM00064500161-001F addr=c16t1l15 server=server_2











id = 126

name = datafs_chkpt0

acl = 0

in_use = True

type = ckpt

worm = off

volume = vp259

pool = snap_pool

member_of =

rw_servers=

ro_servers= server_2

rw_vdms =

ro_vdms =

checkpt_of= datafs Fri Aug 31 19:03:30 EDT 2007

used = 1%

full(mark)= 90%

stor_devs = APM00064500161-0024

disks = d22



Checkpoint created will automatically get mounted with read only permission on the datamover on which filesystem datafs is mounted



trademark of EMC.


datafs_chkpt0 on /datafs_chkpt0 ckpt,perm,ro



Exporting the checkpoint filesystem to database server



trademark of EMC.


server_2 : done

Mounting the checkpoint filesystem exported from Celerra on database server

(Similar output for the other checkpoints)

(Below is the output for last checkpoint created)

Creating checkpoint for production file system datafs



trademark of EMC.



name = datafs

acl = 0

in_use = True

type = uxfs

worm = off

volume = v252

pool = datapool



ro_servers=

rw_vdms =

ro_vdms =


ckpts = datafs_chkpt0,datafs_chkpt1,datafs_chkpt2,datafs_chkpt3,datafs_chkpt4,datafs_chkpt5


disks = d20,d27,d29,d33,d8














id = 131


acl = 0

in_use = True

type = ckpt

worm = off

volume = vp259

pool = snap_pool

member_of =

rw_servers=


rw_vdms =

ro_vdms =

checkpt_of= datafs Fri Aug 31 19:03:36 EDT 2007

used = 6%

full(mark)= 90%

stor_devs = APM00064500161-0024

disks = d22





Checkpoint created will automatically get mounted with read only permission on the datamover on which filesystem datafs is mounted



trademark of EMC.


datafs_chkpt5 on /datafs_chkpt5 ckpt,perm,ro

Exporting the checkpoint filesystem to database server



trademark of EMC.


server_2 : done

Mounted the checkpoint filesystem exported from Celerra on the database server.

/dev/sda1 on / type ext3 (rw)

none on /proc type proc (rw)

none on /sys type sysfs (rw)

none on /dev/pts type devpts (rw,gid=5,mode=620)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

none on /dev/shm type tmpfs (rw)

ip-dart-qa:/pdd2/ip-dart-qa on /mnt/ip-dart-qa type nfs (rw,tcp,addr=128.222.1.24)

10.6.24.97:/results on /mnt/results type nfs (rw,tcp,addr=10.6.24.97)

192.168.4.25:/datafs/oradata on /u02/oradata type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)

192.168.4.25:/datafs/crs on /u02/crs type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)

192.168.4.25:/log1fs/oradata on /u03/oradata type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)

192.168.4.25:/log1fs/crs on /u03/crs type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)




192.168.4.25:/log2fs/oradata on /u04/oradata type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)

192.168.4.25:/log2fs/crs on /u04/crs type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)

192.168.4.25:/archfs on /u05 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)

192.168.4.25:/flashfs on /u06 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)

192.168.4.25:/backupfs on /u07 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.25)

192.168.4.25:/datafs_chkpt0 on /checkpoint/datafs_chkpt0 type nfs (ro,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25)





192.168.4.25:/datafs_chkpt5 on /checkpoint/datafs_chkpt5 type nfs

(ro,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.4.25

[root@mteoradb51 scripts]#

Iterating the logical backup process

Using the Oracle job scheduler, a job that performs the following steps every four hours can be created as follows:

1. From the database server, place the database in hot backup mode.

2. From the Celerra, refresh the SnapSure checkpoint, created during the initialization process, relating to the current hour. The code in the example script is executed on the Oracle database server, using passwordless ssh authentication between the oracle user account and the Control Station’s root user. Again, the instructions on how to create passwordless ssh authentication can be found in the section “Passwordless Authentication Using ssh”. In the example script, a variable named $SNAPNAME stores the checkpoint name for the current hour. The date command, with the +%k format string, outputs the current hour in 0 - 23 format. Finally, integer division (the “/”



operator) divides the current hour by four and discards the remainder. The result is a number from zero to five for any given hour. Test this code yourself to see the behavior.

3. Exit hot backup mode on the Oracle database server.

4. If RMAN integration is required, from the database server, using a SQL-on-SQL approach, create two RMAN scripts: one to uncatalog the datafile copies and the other to catalog them. The SQL Plus script in the second code listing accomplishes this. This script frees the user from the burden of maintaining and updating a list of datafiles that must be backed up, because the list is obtained from the data dictionary when the backup occurs. The heart of the SQL Plus script is the SELECT statements. The first part of each statement is a literal that outputs a snippet of RMAN script code. The CHR (39) statement outputs the single-quote character. The idiom “&1” is a parameter passed into the script. This parameter contains the file system path to the checkpoint file system, as it is mounted on the Oracle database server. This should include all directories necessary to reach the datafiles. The statement SUBSTR (name, INSTR (name,'/',-1, 1), LENGTH (name)) simply produces the names of all of the datafiles, not including any of the path information prior to the filename (but including the leading slash). Again, pass the path information to the checkpoint file system in the parameter given in the script. Note that the last trailing slash is omitted from the parameter that is passed in.

This is the shell script that drives the process:

Code Listing 7: Logical backup iteration shell script

#!/bin/bash

#-----------------------------------------------------------#

# Script: log_bkup_iter.bash #

# Purpose: Iterates fs_ckpt Oracle hot backup process #

# This script can be run as oracle if passwordless ssh is #

# set up between the oracle user account and the NS Series #

# Control Station root account #

#-----------------------------------------------------------#

THE_NS=mteorans40-1

THE_USER=root

THE_FS=datafs

echo "Putting the database server in hot backup mode"

sqlplus /nolog <<EOF1

connect / as sysdba

alter database begin backup;

exit

EOF1

SNAPNAME=${THE_FS}'_ckpt'$((`date +%k` / 4))

echo "SNAPNAME is ${SNAPNAME}"

ssh ${THE_USER}@${THE_NS} "export NAS_DB=/nas; /nas/bin/fs_ckpt \




${SNAPNAME} -refresh"

echo "Taking the database out of hotbackup mode"


connect / as sysdba

alter database end backup;

exit

EOF2

Executing this script on our system produces the following effect:

Code Listing 8: Logical backup iteration output

[oracle@mteoradb51 scripts]# ./log_bkup_iter.bash

Putting the database server in hot backup mode

SQL*Plus: Release 10.2.0.2.0 - Production on Thu Sep 6 4:11:42 2007

Copyright (c) 1982, 2005, Oracle. All Rights Reserved.

SQL> Connected.

SQL>

Database altered.

SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

Thu Sep 6 4:22:35 EDT 2007

SNAPNAME is datafs_chkpt1

Refreshing the SNAPNAME datafs_chkpt1



trademark of EMC.




acl = 0





in_use = True

type = ckpt

worm = off

volume = vp259

pool = snap_pool

member_of =

rw_servers=


rw_vdms =

ro_vdms =

checkpt_of= datafs Thu Sep 6 08:25:28 EDT 2007

used = 75%

full(mark)= 90%

stor_devs = APM00064500161-0024

disks = d22



Thu Sep 6 4:22:36 EDT 2007

SQL*Plus: Release 10.2.0.2.0 - Production on Thu Sep 6 4:22:36 2007


SQL> Connected.

SQL>

Database altered.

[oracle@mteoradb51 scripts]#

As a best practice during backup you should also switch log files, archive all log files, and back up the control file. The instructions on how to do these operations are beyond the scope of this document. However these steps should be incorporated into the script shown in “Code Listing 8: Logical backup iteration output.” Also, the script log_bkup_iter.bash should be incorporated into some scheduling mechanism (say, the Oracle Enterprise Manager job control process), which will cause it to be executed approximately every four hours.

Chapter 5 Logical and Physical Storage Backup: FCP


Overview ....................................................................................................................................... 79 Benefits ......................................................................................................................................... 79 Logical storage backup benefits .................................................................................................... 79 Physical storage backup benefits ................................................................................................... 80 Logical storage backup using CLARiiON SnapView ................................................................... 80 Logical storage process overview ................................................................................................. 80 Logical storage backup using Replication Manager ...................................................................... 81 Physical storage backup using RMAN .......................................................................................... 89

Overview In the context of this document, the term “logical storage backup” is distinguished from the term “logical backup,” which is generally used within the Oracle community and which is a backup performed using the import/export utilities. A logical storage backup is a backup that does not physically exist. Rather, it consists of the blocks in the active LUN, combined with blocks in a SavVol, an area where the original versions of updated blocks are retained. The effect of a logical storage backup is that you can assemble a view of the LUN as of a certain point in time. A logical storage backup is distinguished also from a “physical storage backup,” which is a full and complete copy of the database objects made to different physical media. Unlike a physical storage backup, a logical storage backup can be taken very rapidly, and requires very little space to store (typically a small fraction of the size of a physical storage backup).

Benefits Logical storage backup benefits

At the center of the logical backup approach is the CLARiiON SnapView feature, which uses the Navisphere snapview command. This command allows a database administrator (DBA) to capture a logical storage backup of an entire LUN, or with consistency technology, a group of LUNs. Both logical storage and physical storage backups are helpful to the DBA.

Logical storage backups have certain advantages:


Logical and Physical Storage Backup: FCP

1. The performance impact of logical backup is modest. The database is in hot backup for a few seconds at most.

2. Restoring from a logical storage backup can be done very quickly, depending on the amount of data changes. This dramatically reduces mean time to recovery (MTTR) over that which can be achieved restoring from a physical backup.

3. Due to the high frequency of backups, a small number of archived redo log files need to be applied if a recovery is needed. This further reduces mean time to recovery.

Of course, a logical storage backup does not protect you from hardware failures; that is the purpose of the physical storage backup covered in the next section. However, it does protect against logical corruption of the database, as well as accidental file deletion, and other similar human errors.

Physical storage backup benefits

Taking logical storage backups is not enough to protect the database from all risks. Physical storage backups are also required to protect the database against double disk failures and other hardware failures at the storage layer.

Logical storage backup using CLARiiON SnapView A logical storage backup consists of virtual copies of all LUNs being used to store datafiles. This is enabled by the CLARiiON SnapView feature. Using EMC consistency technology, multiple LUNs can be used to store an automated storage management (ASM) disk group and snapshots of all of those LUNs can be created in a consistent manner. This technique is demonstrated and documented in this section.

Logical storage process overview

A typical backup scheme would use six logical storage backups per day, at four-hour intervals, combined with one physical storage backup per day.

The procedure described next can be integrated with the Oracle Enterprise Manager job scheduling process or cron. This can be used to execute a logical storage backup once every four hours.

The following subsection provides a detailed description of the logical storage backup process using Replication Manager.




Logical storage backup using Replication Manager The steps used to perform a backup using SnapView snapshots are as follows:

Creating a Replication Manager job for SnapView snapshot:

1. Select Jobs, right-click and select New Job as shown in Figure 51.

Figure 51 New Job



2. The welcome page of the Job Wizard is displayed, as shown in Figure 52. Click Next.

Figure 52 Job Wizard > welcome screen

3. The Job Name and Settings screen is displayed, as shown in Figure 53.

Type in details for: the job name, replication source, replication technology, and the number of replicas to be created. Click Next.

Figure 53 Job Wizard > Job Name and Settings





4. Select the mount options, as shown in Figure 54. Click Next. Note: You do not have to select the mount options at this point in the process. If necessary, you can select the mounting options at a later point in time.

Figure 54 Job Wizard > Mount options

5. The Starting the Job screen is displayed. Choose how you would like to start the job, as shown in Figure 55. Click Next.

Figure 55 Job Wizard > Starting the Job


6. The Users to be Notified screen is displayed, as shown in Figure 56. Type the email addresses of the users that you would like to be notified when the job has completed. Click Next.

Figure 56 Job Wizard > Users to be Notified

7. The Completing the Job Wizard screen is displayed, as shown in Figure 57. Click Finish.

Figure 57 Job Wizard > Completing the Job Wizard




8. The job is now visible in the Jobs list as shown in Figure 58.

Figure 58 Jobs



Executing a Replication Manager job:

1. In Replication Manager, select Jobs from the navigation panel.

2. Select a job, right-click and select Run to execute the job, as shown in Figure 59.

Figure 59 Selecting a job to run

3. Click Yes to confim that you want to run the job, as shown in Figure 60.

Figure 60 Running a job > confirmation




4. The job begins to execute as shown in Figure 61.

Figure 61 Executing a job

5. When the progress bar reaches 100%, as shown in Figure 62, click Close.

Figure 62 Job executed successfully



6. The status of the job now displays as Successful, as shown in Figure 63.

Figure 63 Job status > Successful

7. To verify the Snapshot Session information, select the the appropriate item under Storage Services as shown in Figure 64.

Figure 64 Verifying Snapshot Session information




8. You can also use Navisphere to view the snapshot sessions that were created using Replication Manager, as shown in Figure 65:

Figure 65 Using Navisphere to verify snapshot session information

As a best practice, during backup you should switch the log files, archive all the log files, and back up the control file. The instructions on how to carry out these operations are beyond the scope of this document.

Physical storage backup using RMAN Physical backup of the Celerra array can be accomplished using RMAN. The backup target is typically SATA or LCFC disks on the Celerra array. If tape is used with a product that includes a media management layer, such as EMC NetWorker®, Oracle Secure Backup must be used. Normal RMAN semantics apply to this backup method. This is thoroughly covered on the Oracle Technology Network website and will not be included in this document.

Code Listing 11: RMAN backup script

#!/bin/bash

#-----------------------------------------------------------#

# Script: rmanbkp.bash #

# Purpose: It creates rman backup #

#-----------------------------------------------------------#

echo "This is rmanbkp.bash"

echo “Starting RMAN Backup”



rman<<EOF1

connect target /

backup database;

backup current controlfile;

exit

EOF1

echo "End of RMAN backup rmanbkp.bash"

Code Listing 12: RMAN backup script output

[oracle@mteoradb1 ~]$ ./rmanbkp.bash

This is rmanbkp.bash

Starting RMAN Backup

Recovery Manager: Release 10.2.0.2.0 - Production on Fri Aug 31 06:53:00 2007

Copyright (c) 1982, 2005, Oracle. All rights reserved.

RMAN>

connected to target database: MTERAC5 (DBID=815179535)

RMAN>

Starting backup at 31-AUG-07

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=4166 instance=mterac51 devtype=DISK

channel ORA_DISK_1: starting full datafile backupset

channel ORA_DISK_1: specifying datafile(s) in backupset

input datafile fno=00008 name=/u02/oradata/mterac5/test01.dbf













ncluding current control file in backupset




channel ORA_DISK_1: starting piece 1 at 31-AUG-07

channel ORA_DISK_1: finished piece 1 at 31-AUG-07

piece handle=/u06/oradata/MTERAC5/backupset/2007_08_31/o1_mf_ncnnf_TAG20070831T065724_3fj98bcs_.bkp tag=TAG20070831T065724 comment=NONE

channel ORA_DISK_1: backup set complete, elapsed time: 01:10:00

Finished backup at 31-AUG-07

RMAN>

End of RMAN backup rmanbkp.bash


Chapter 6 Restore and Recovery: NFS


Overview ....................................................................................................................................... 93 Restore and recovery from logical storage backup ........................................................................ 93 Using rootfs_ckpt .......................................................................................................................... 93 Using file system commands ......................................................................................................... 96

Overview Backup is worthless without the ability to restore and recover the database. Thus, the ability to restore and recover the database successfully must be tested and verified. This section discusses various methods to restore and recover the database.

Restore and recovery from logical storage backup Using rootfs_ckpt

If there is a valid logical-storage backup of the database, the most efficient way to restore the database is by using the rootfs_ckpt -Restore command. The following script accomplishes this by allowing the user to pass in the name of a SnapSure checkpoint as the first parameter:

Code Listing 13: Logical restore/recovery shell script

#!/bin/bash

#-----------------------------------------------------------#

# Script: log_bkup_restore.bash #

# Purpose: Restores and performs full database recovery #

# using backup taken with fs_ckpt. #

# This script can be run as oracle if passwordless ssh is #

# set up between the oracle user account and the NS Series #


Restore and Recovery: NFS

# Control Station root account #

#-----------------------------------------------------------#

THE_NS=rtpsol33

THE_USER=root

SNAPNAME=datafs_chkpt1

echo "SNAPNAME is ${SNAPNAME}"

THE_DATAMNT=/u02/oradata/mterac5

cp -fp ${THE_DATAMNT}/*.ctl /tmp


/nas/sbin/rootfs_ckpt ${SNAPNAME} -Restore -Force"

cp -fp /tmp/*.ctl ${THE_DATAMNT}


connect / as sysdba

startup mount;

set autorecovery on;

recover database;

alter database open;

exit

EOF2

date

The following are the steps to be carried out for the restore and recovery operation:

1. Shut down all the database instances by typing the following command:

[oracle@mteoradb51 oracle]$ srvctl stop database -d mterac5

2. Verify that all database instances have been shut down by typing the following command:

[oracle@mteoradb51 oracle]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....51.lsnr application ONLINE ONLINE mteoradb51

ora....b51.gsd application ONLINE ONLINE mteoradb51

ora....b51.ons application ONLINE ONLINE mteoradb51

ora....b51.vip application ONLINE ONLINE mteoradb51
















ora.mterac5.db application OFFLINE OFFLINE

ora....51.inst application OFFLINE OFFLINE




3. On our test system, running this restore script generated the following output:

[oracle@mteoradb51 oracle]$ ./log_bkup_restore.bash

SNAPNAME is datafs_chkpt1



trademark of EMC.




----------------------------------

----------------------------------(output deleted for brevity)

stor_devs = APM00064500161-0024

disks = d22



SQL*Plus: Release 10.2.0.2.0 - Production on Tue Sep 25 10:13:05 2007






SQL>Connected to an idle instance.

SQL> ORACLE instance started.

Total System Global Area 2.1475E+10 bytes

Fixed Size 2109584 bytes

Variable Size 2382368624 bytes

Database Buffers 1.9076E+10 bytes

Redo Buffers 14663680 bytes

Database mounted.

Media recovery complete.

SQL>

Database altered.

SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

Thu Sep 06 10:45:57 EDT 2007

[oracle@mteoradb51 oracle]$

==============================================================

Using file system commands

Once a checkpoint file system is mounted as a normal (albeit read-only) file system on the Oracle database server, the datafile is copied to the checkpoint file system and can be treated as if it were any other user-managed backup of the Oracle database. That is, the normal file system semantics can be used to restore these files into the active file system. Often, this is a very reasonable decision. However, bear in mind that full database restore using the rootfs_ckpt -Restore command will often be more efficient in terms of time and I/O. This is because the rootfs_ckpt -Restore command requires minimal I/O (basically restoring changed blocks) in order to complete the restore operation. On the other hand, using the file system commands like cp, dd, tar, and so forth will round-trip the network, reading the blocks off of the Celerra within the checkpoint file system, and then writing the blocks back out to the Celerra into the active file system. Since the rootfs_ckpt -Restore command is limited to restoring the entire file system, for restore operations that affect a small subset of the database, and where downtime on the entire database cannot be allowed, an operation using file system semantics may be the best choice.

Chapter 7 Restore and Recovery: FCP


Overview ....................................................................................................................................... 97 Restore and recovery from a SnapView snapshot using Replication Manager ............................. 97 Restore and recovery from physical storage backup using RMAN ............................................. 103

Overview Backup is worthless without the ability to restore and recover your database. Thus, you must thoroughly test and verify the ability to restore and recover the database successfully. This section discusses various methods to restore and recover your database.

Restore and recovery from a SnapView snapshot using Replication Manager This test validated the full restore and recovery of a database from a CLARiiON SnapView hot backup using Replication Manager. The backup created in Chapter 5 was used as the source for this restore.

1. Stop the normal database instances on all nodes:

[root@mteoradb70 ~]# srvctl stop database -d mterac21

2. Stop and disable the ASM instances on all nodes except the node where the restore operation will be performed:

[root@mteoradb70 ~]# srvctl stop asm -n mteoradb68



[root@mteoradb70 ~]# srvctl disable asm -n mteoradb70



3. Start the database instance on the first node:


Restore and Recovery: FCP

[oracle@mteoradb67 ~]$ srvctl start database -d mterac21

4. Verify the status of the diskgroups and dismount the DATA diskgroup:

SQL> select name,state from v$asm_diskgroup;

NAME STATE

------------------------------ -----------

DATA MOUNTED

LOG1 MOUNTED

LOG2 MOUNTED

SQL> alter diskgroup data dismount;

Diskgroup altered.

5. Open the Replication Manager console.

6. Choose the appropriate application set and select the snapshot that you want to restore. Right-click and select Restore, as shown in Figure 66.

Figure 66 Restoring a snapshot




7. The Restore Wizard > Replica to Restore screen is displayed; it lists all the replicas that exist on the storage, as shown in Figure 67. Select the replica that you want to restore, and click Next.

Figure 67 Restore Wizard > Replica to Restore

8. The Objects to be Restored screen is displayed, as shown in Figure 68. Expand the tree and select the objects that you want to restore. Click Finish.

Figure 68 Restore Wizard > Replica to Restore



9. To restore the replica click Yes, as shown in Figure 69.

Figure 69 Confirmation screen > restore replica

10. The Restore Replica screen is displayed, as shown in Figure 70. Verify that the restore operation takes place without any errors.

Figure 70 Restore Replica




11. When the progress bar reaches 100% click Close, as shown in Figure 71.

Figure 71 Restore replica completed successfully

12. After the restore operation is successfully completed, verify that the diskgroups are mounted:

SQL> select name,state from v$asm_diskgroup;

NAME STATE

DATA MOUNTED

LOG1 MOUNTED

LOG2 MOUNTED

13. When you have verified that the diskgroups are mounted, mount the database:

[oracle@mteoradb67 ~]$ export ORACLE_SID=mterac21

[oracle@mteoradb67 ~]$ sq

SQL*Plus: Release 10.2.0.3.0 - Production on Fri Nov 14 06:41:45 2008


Connected to an idle instance.

SQL> startup mount pfile='/home/oracle/initmterac21.ora';

ORACLE instance started.

Total System Global Area 1.7180E+10 bytes



Database Buffers 1.5016E+10 bytes




Database mounted.

14. When the database is mounted, perform a recovery on the database:

SQL> set autorecovery on;

SQL> recover database;

ORA-00279: change 16187378 generated at 11/14/2008 06:18:14 needed for thread 1

ORA-00289: suggestion : /u05/MTERAC21/1_7_670694846.dbf

ORA-00280: change 16187378 for thread 1 is in sequence #7










Log applied.

Media recovery complete.

15. When the recovery of database is complete, open the database:

SQL> alter database open;

Database altered.




Restore and recovery from physical storage backup using RMAN Restore and recovery from physical backup is accomplished using RMAN:

Code Listing 14: RMAN restore and recovery shell script

#!/bin/bash

#-----------------------------------------------------------#

# Script: rmanrstr.bash #

# Purpose: It restore rman backup #

#-----------------------------------------------------------#

echo "This is rmanrstr.bash"

echo “Starting RMAN Restore”

rman<<EOF1

connect target /

shutdown immediate;

startup mount;

restore database;

exit

EOF1

echo "End of RMAN restore rmanrstr.bash

Code Listing 15: RMAN restore and recovery shell script output

[oracle@mteoradb1 ~]$ ./rmanrstr.bash

This is rmanrstr.bash

Starting RMAN Restore

Recovery Manager: Release 10.2.0.2.0 - Production on Mon Sep 3 13:57:39 2007

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database: MTERAC5 (DBID=815179535, not open)

RMAN>

Starting restore at 03-SEP-07

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=5480 instance=mterac51 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set





restoring datafile 00001 to /u02/oradata/mterac5/system01.dbf

restoring datafile 00002 to /u02/oradata/mterac5/undotbs01.dbf

restoring datafile 00003 to /u02/oradata/mterac5/sysaux01.dbf

restoring datafile 00004 to /u02/oradata/mterac5/users01.dbf




restoring datafile 00008 to /u02/oradata/mterac5/test01.dbf

……………………………………………………………………

……………………………………………………………………



channel ORA_DISK_1: reading from backup piece /u06/oradata/MTERAC5/backupset/2007_08_31/o1_mf_nnndf_TAG20070831T065724_3fhx9q0h_.bkp

channel ORA_DISK_1: restored backup piece 1

piece handle=/u06/oradata/MTERAC5/backupset/2007_08_31/o1_mf_nnndf_TAG20070831T065724_3fhx9q0h_.bkp tag=TAG20070831T065724

channel ORA_DISK_1: restore complete, elapsed time: 05:49:44

Finished restore at 03-SEP-07

RMAN>

End of RMAN restore rmanrstr.bash

Chapter 8 RMAN Physical Backup Using a De-duplication Array


Overview ..................................................................................................................................... 105 Setup ........................................................................................................................................ 105 Backup procedure ........................................................................................................................ 106 Conclusions ................................................................................................................................. 106

Overview The Quantum DXi5500 disk array optimizes the storage space required to store backups by using a data de-duplication technique. The DXi5500 backup solution addresses some of the main challenges faced by the customer such as:

♦ Time required to complete the backup

♦ Time required to restore the backup

♦ Space required to store the Oracle backup sets

The array detects when a duplicate block is being stored, and simply stores a pointer in the place of the duplicate block. In the process, significant space savings can be achieved. While this requires a modest increase in backup time, no additional impact is imposed on the performance of the backup. There are also significant improvements in restore time with this device. This section explains the advantages of this solution component in full.

Setup The Quantum DXi5500 device is set up using a GUI from a web browser. Full information on the setup procedure can be found in the Quick Start Guide. We used the array as an NFS server and mounted the backup target file system as an NFS mount point. From the perspective of the database server, the procedure for mounting the target file system as an NFS mount point is identical to setting up the NFS for basic storage of Oracle database files. The only difference is that the Quantum array is used to export the NFS mount point, not a Celerra. Finally, we set the flashback recovery area target to be the mount point on the Quantum array.


http://downloads.quantum.com/dxi3500/DXi3500_5500_Quick_Start_Guide_81-81700-02_B01.pdf

RMAN Physical Backup Using a De-duplication Array



Backup procedure Performing a backup to an Oracle database to a Quantum DXi5500 de-duplication array is identical to performing a backup to a Celerra using RMAN. Simply back up the database to the Quantum DXi5500 array using RMAN. The de-duplication array will remove duplicate blocks in a manner transparent to you. For more information refer to “Physical storage backup using RMAN.”

To back up an Oracle database to a Quantum DXi5500 array:

1. Set up the Quantum DXi5500 to be a backup target.

2. To begin the RMAN backup, type the command BACKUP DATABASE;

3. Perform a restore/recovery operation using the Quantum DXi5500 as the backup source.

Conclusions The Quantum DXi5500 offers significant benefits with respect to increasing the amount of backup data that can be retained on the same amount of disk and also with faster restore operations.

Using the Quantum DXi5500 to back up data is slightly slower than using the Celerra, however, the space savings are significant. The Quantum DXi5500 uses significantly less space than a Celerra to store the same data. Also, during backup, there is no significant performance difference between the Celerra and the Quantum.

In our testing, the Quantum took an hour less than the Celerra to perform a restore operation on a database approximately 150 GB in size. During the restore operation, of course, the database is down, and therefore this is the most critical operation that needs to be optimized in most cases.

Chapter 9 Memory Configuration


Tuning Huge Pages ..................................................................................................................... 107 Memory configuration: Oracle 11g ............................................................................................. 108

Tuning Huge Pages The Linux 2.6 kernel includes a new feature called Huge Pages. This feature allows you to specify the number of physically contiguous large memory pages that will be allocated and pinned in RAM for shared memory segments like the Oracle System Global Area (SGA). These pre-allocated pages can only be used for shared memory, and must be large enough to accommodate the entire SGA. This means that unused Huge Pages will not be available for use other than for shared memory allocations even if the system runs out of memory and starts swapping. Hence, a wrongly configured Huge Pages setting may result in a poor performance. This may even make the machine unusable. Therefore, Huge Pages must be tuned carefully and set correctly.

Huge Pages can create a very significant performance improvement for the Oracle RAC 10g database servers. Thus, the payoff of enabling Huge Pages is significant as well.

This section provides a straightforward procedure for tuning Huge Pages. If this procedure is followed carefully, the system will be correctly configured. The steps are as follows:

1. Start with a system that has adequate memory. For the test systems, a machine with 24 GB of RAM and a 20 GB SGA was used.

2. Set the Huge Pages parameters to a size into which the SGA will fit comfortably (These settings are in /etc/sysctl.conf). For example:

root@mteoradb51 ~]# grep Huge /proc/meminfo

HugePages_Total: 11241

HugePages_Free: 11241

Hugepagesize: 2048 kB


Memory Configuration

In this case, the Huge Pages pool size is about 21 GB. The SGA should fit comfortably.

3. Start the instance, and check the Huge Pages parameters again.

[root@mteoradb51 ~]# grep Huge /proc/meminfo


HugePages_Free: 1000

Hugepagesize: 2048 KB

4. The value of HugePages_Free is the number of Huge Pages that are being wasted. Subtract this number from the HugePages_Total. In this case, the resulting number of pages is 10241. Set the HugePages_Total parameters to that amount, reboot, and start the instance again. Check the Huge Pages parameters again:

[root@mteoradb51 ~]# grep Huge /proc/meminfo


HugePages_Free: 0

Hugepagesize: 2048 KB

When HugePages_Free is equal to zero, the tuning is correct. Refer to metalink article 361323.1 and http://www.puschitz.com/TuningLinuxForOracle.shtml for more information on enabling and tuning Huge Pages.

Memory configuration: Oracle 11g Memory configuration in 32-bit Oracle 11g is possibly the most challenging aspect of configuring the database server. If the memory is not configured the performance of the database server will be very poor. If it is incorrectly configured the database server will be unstable. The database may not open at all. If it does, you may experience errors due to lack of shared pool space. In an OLTP context, the size of the shared pool is frequently the limitation on performance of the database.

The performance on 32-bit Oracle 11g can be comparable to 64-bit Oracle 10g if memory is configured correctly.

Memory address space is an issue when running Oracle 11g in a 32-bit environment. Because the memory is being mapped in a 32-bit operating system, the amount of shared memory that can be configured is constrained to 4 GB (which is 2 ^ 32, or 2 to the power of 32). The operating system, all processes, and any shared memory structures must fit into that space. The solution to this issue is to configure shared memory as a file system mounted under /dev/shm, and have the SGA stored there. In addition, the memory model in Oracle 11g 64 bit (Release 1) also requires /dev/shm. Therefore these techniques are still required in the 64-bit version of the product. This section explains the methodology to set up this configuration. The critical files are shown in Table 5.



http://www.puschitz.com/TuningLinuxForOracle.shtml


Table 5 Memory configuration files

File Created by Description /etc/rc.local Linux installer Commands contained in this file

will be run after all of the commands in /etc/inittab are parsed and executed. This is where the shared memory file system must be configured.

/etc/sysctl.conf Linux installer Contains the shared memory parameters for the Linux operating system. This file must be configured in order for Oracle to create the SGA with shared memory.

/etc/security/limits.conf Linux installer Contains the limits imposed by Linux on users’ use of resources. This file must be configured correctly in order for Oracle to use shared memory for the SGA.

Oracle parameter file Oracle installer, dbca, or DBA who creates the database

Contains the parameters used by Oracle to start an instance. This file must contain the correct parameters in order for Oracle to start an instance using shared memory.

These files were configured in our test system as follows:

Code Listing 16: /etc/rc.local (part of the file relevant to Oracle)

umount /dev/shm

mount -t tmpfs shmfs -o size=18g /dev/shm

chown oracle:oinstall /dev/shm

These three lines configure the /dev/shm shared memory file system. The sizing of this file system is critical. We set the size to 18 GB, as shown. Our system had 24 GB of memory, and used a 13 GB buffer cache. Be sure to set the size of the file system so that it is appropriate for the system you are configuring. In addition, you will probably need to try several combinations in order to tune to the correct size.

Code Listing 17: /etc/sysctl.conf (part of the file relevant to Oracle)

# Oracle parameters

kernel.shmall = 2097152

kernel.shmmax = 4294967295

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 6553600

net.ipv4.ip_local_port_range = 1024 65000

net.core.rmem_default = 4194304





Note:

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 262144 The size of kernel.shmmax has been set to (2 ^ 32) - 1 (that is 2 to the power of 32

minus 1). As stated above, (2 ^ 32) - 1 is the theoretical limit of a 32-bit address space. If you set the value of kernel.shmmax incorrectly (for instance to 4294967296, one greater than the value we set), it will wrap to zero, and you will have effectively no value for this parameter. We did this and the result was very poor performance. Be sure to set this value correctly. We did not find that there was any reason to set the value to less than 4294967295.

Code Listing 18: /etc/security/limits.conf (part of the file relevant to Oracle)

# Oracle parameters

oracle soft nofile 131072

oracle hard nofile 131072

oracle soft nproc 131072

oracle hard nproc 131072

oracle soft core unlimited

oracle hard core unlimited

oracle soft memlock 3145728

oracle hard memlock 3145728

Note: Make sure that the memlock has been configured. This is required for the shared memory file system. This is not covered in the Oracle Database 11g Installation Guide, so be sure to set this parameter. If you do not set this parameter, your database will behave uncharacteristically.

Code Listing 19: Oracle parameter (part of the file relevant to memory configuration)

use_indirect_data_buffers=true

db_block_buffers=1664000

shared_pool_size=2800M

The first parameter is required in order to use the shared memory file system to store the SGA.

Note: The memory_target and memory_max_target have not been configured, and neither has sga_target or sga_max_target.

These automatic memory settings do not appear to work in the 32-bit version of Oracle 11g. This will probably be corrected in the 64-bit version. We attempted to configure memory_max_target and memory_target instead, and got much worse performance than tuning db_block_buffers and shared_pool_size manually. Also, our limitation in terms of the number of users was constrained by the size of the shared pool. However, if we configured a shared pool larger than 2.8 GB, we got an out-of-memory error message when starting the instance. This also appears to be a limitation of the 32-bit operating system.

Again, greater scalability and simplicity will be afforded by the 64-bit version of Oracle 11g when it arrives. In the meantime, this configuration provides acceptable performance (although less than 64-bit Oracle 10g) in our test configuration.

Chapter 10 Creating a Test/Dev Solution Using SnapView Clone

This chapter presents these topics:

Overview ..................................................................................................................................... 111 Creating and executing a Replication Manager job for SnapView clone .................................... 112 Mount and recovery of a target clone database using Replication Manager ............................... 118

Overview You can use CLARiiON SnapView clone with consistency technology to create a Test/dev solution using a restartable copy of a running Oracle RAC 10g production database. Doing so has minimal impact on the production database. You can either use SnapView snap, or SnapView clone to create the copy. In the case of SnapView snap, the I/Os that are performed on the database copy will be handled by the same physical hardware as the production database. This can have an impact on the production database performance. In the case of SnapView clone, the cloned LUNs can be created on a completely different set of RAID groups from the original LUNs. As a result, I/O performed on the database copy is handled by different physical hardware from the production database, and thus the database copy’s I/O does not impact the production database in any way. However, the SnapView clone approach requires a full copy of the production database, whereas the SnapView snap approach does not. The read I/O to the production LUNs that is required to create the clone LUNs does impact the production database. However, the ongoing operation of the copy database then has no appreciable impact on the production database. We assumed that the SnapView clone approach was preferable for most customers.

A second set of database servers is required to mount the LUNs and manage them within the oracleasm kernel module. This is because ASM writes a unique signature to each LUN, and will not open a LUN containing the same signature as an existing LUN. As the use of the same database servers would have an obvious impact on the production database, we assumed this was the preferable approach.

The use of consistency technology is the key to this approach. Consistency technology allows the customer to create a copy (either virtual or physical) of a set of LUNs on either a single array or multiple arrays with the writes to all of the LUNs being in perfect order. That means that the copy LUNs are identical to the state in which the production LUNs would be in if the database server was powered off. This allows the customer to create a restartable database copy, as the database


Creating a Test/Dev Solution Using SnapView Clone

engine will perform crash recovery on this copy using the online redo logs in exactly the same manner as a power loss.

Because of this unique functionality, the use of backup mode or RMAN cloning is not required, which is advantageous to the customer as both of these approaches use host and array resources and can have an impact on production database performance.

A restartable copy should not be considered to be a substitute for a backup. Database restart following database server crash is not guaranteed by Oracle. Therefore, a crash consistent image is not a reliable backup. Further, recovery cannot be performed on a crash consistent image. The image can be restarted in the state it was in at the time the copy occurred. It cannot be recovered to the state of a later time. Therefore, the customer should use normal backup semantics to back up the production Oracle database, using either SnapView snap or RMAN, as described elsewhere in this document.

This does not mean, however, that a restartable copy is useless; it is very useful. For example, a restartable copy can be used for testing and development purposes. Since the creation of the restartable copy has low impact on the production database, it can be created fairly often, typically daily.

We cloned a four-node RAC production database, and then mounted the cloned database on a VMware host as a single-instance database. The cloning and mounting of the cloned database on the target VM host was performed using Replication Manager. The following sections of this chapter describe the main steps that were followed.

Creating and executing a Replication Manager job for SnapView clone 1. In Replication Manager select Jobs, right-click and select New Job as shown in Figure

72.

Figure 72 Create New Job




2. The Welcome screen of the Job Wizard is displayed, as shown in Figure 73. Choose the application set that you want to replicate, then click Next.

Figure 73 Job Wizard > Welcome screen

3. The Job Name and Settings screen is displayed, as shown in Figure 74. Type the job name and select a replication source, a replication technology name, and the number of replicas to be created. Click Next.

Figure 74 Job Name and Settings



4. The Replication Storage screen is displayed, as shown in Figure 75. Select the storage pool that you want to use for the replica. Click Next.

Figure 75 Replication Storage





5. The Mount Options screen is displayed. Select the mount options, and click Next.

Note: You do not have to select the mount options at this point in the process. You can select the mount options as a later point in time if required

Figure 76 Mount Options

6. Choose how you would like to start the job. Click Next.

Figure 77 Starting the Job


7. The Users to be Notified screen is displayed, as shown in Figure 78. Type the email addresses of the users that you would like to be notified when a job is completed. Click Next.

Figure 78 Users to be Notified

8. The Completing the Job Wizard screen is displayed, as shown in Figure 79. To save the job, click Finish.

Figure 79 Users to be Notified




9. Verify that the Replication Manager job has been created without any errors as shown in Figure 80. If errors occur during the creation of the job, this will be indicated in the Latest Status column.

Figure 80 Cloning job created

10. After the clone job has been created, select the job, right-click and select Run to execute the job. The complete logs will be displayed during the execution as shown in Figure 81.

Figure 81 Executing a clone job



11. Verify that the job executed successfully, as shown in Figure 82.

Figure 82 Executing a clone job

Mount and recovery of a target clone database using Replication Manager Replication Manager generates its own init.ora file as part of the replication process. This file is then placed in the directory specified by the ERM_TEMP_BASE variable (/tmp by default). This init.ora file is usually sufficient to start the database, but it does not necessarily contain all the parameters from the original init.ora file. We used the procedure described below to customize the init.ora file that is generated by Replication Manager.

1. On the mount host, cd to the RM client install directory:

[root@mteoraesx1-vm5 ~]# cd /opt/emc/rm/client/bin/

2. Create a new directory using the SID name for the mount, then cd to that directory:

[root@mteoraesx1-vm5 bin]# mkdir mterac211

[root@mteoraesx1-vm5 bin]# cd mterac211/

3. Create a new init<SID>.ora file using the same SID name that is specified above.

[root@mteoraesx1-vm5 mterac211]# vi initmterac211.ora





4. Customize the parameter file as required.

Note: These parameters will be appended to the init.ora file that was generated by Replication Manager, so they must follow the correct Oracle syntax.

For more information on parameter file customization refer to EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform – Best Practices Planning.

5. In Replication Manager, select the clone replication job, right-click and select Mount as shown in Figure 83.

6. Specify the correct SID name. The new init<SID>.ora file will be picked up on the fly. Those parameters will be used to start the database on the mount host in conjunction with the parameter file generated by Replication Manager.

Figure 83 Mount replication job


7. The Mount Wizard is displayed. Select the replica to be mounted as shown in Figure 84. Click Next.

Figure 84 Mount Wizard > Select replica to mount




8. The Mount Options screen is displayed as shown in Figure 85.

a. Under Path options, select Original path. This ensures that the database can be mounted using the same path on the target host as well.

b. Under Oracle, select Recover Database. This ensures that the target clone database will be automatically recovered after mounting by Replication Manager.

c. Click Finish.

Figure 85 Mount Wizard > Mount Options



9. Verify that the mount/recovery of the clone database is started without any errors.

Figure 86 Mount Replica > progress screen

10. Once the job is completed successfully, verify that the clone database is open in read/write mode:

SQL> select name,open_mode from v$database;

NAME OPEN_MODE

MTERAC21 READ WRITE

In this way, Replication Manager can be used to automate the complete process of cloning the LUNs, mounting the cloned LUNs on the target host, and restoring and recovering the clone database.



Chapter 11 Creating a Test/Dev Solution Using Celerra SnapSure


Overview ..................................................................................................................................... 123 Use case scenarios ....................................................................................................................... 123 Prerequisites ............................................................................................................................. 124 Creating a test/dev database using Celerra writeable checkpoints .............................................. 124

Overview A separate test/dev environment is created by replicating the production database using the EMC Celerra SnapSure writeable checkpoints feature. The replicated database is a read/writeable copy that can be used by developers and testers without impacting on the production database.

In the case of Celerra SnapSure writeable snapshots, the unchanged blocks will be shared between the production database and the test/dev database, so care must be taken to ensure that I/O to the test/dev database does not impose an unreasonable burden on the production database.

The procedure to create the test/dev database is very similar to the procedure that is used to create a logical storage backup using Celerra SnapSure. “Logical storage backup using Celerra SnapSure” provides more details.

Use case scenarios The following user case scenarios have been validated as part of this solution.

1. Create a Celerra SnapSure writeable checkpoint when the Benchmark factory work load is running on the four-node RAC production database. Use the checkpoint to bring up the test/dev database as a single instance on a VMware host.

2. Create a Celerra SnapSure writeable checkpoint when the Benchmark factory work load is running on the four-Node RAC production database.


Creating a Test/Dev Solution Using Celerra SnapSure

Use the checkpoint to bring up the test/dev database as a RAC database on another set of four physical servers.

Prerequisites

♦ The target clone nodes must have access to the Celerra system that is hosting the production database file systems.

♦ Create and export two new file systems “snap_log1” and “snap_log2” to store the redo logs of the clone target database.

♦ Install the database software on the target clone nodes. Ensure that the software is installed at the same location on the target clone nodes as on the production database servers.

♦ In the case of the target being a RAC, cluster services must be up and running on the RAC nodes.

Creating a test/dev database using Celerra writeable checkpoints The steps to create a test/dev database using Celerra writeable checkpoints are as follows:

1. Place the production database in archive log mode:

MTERAC15> archive log list

Database log mode Archive Mode

Automatic archival Enabled

Archive destination /u05/MTERAC15

Oldest online log sequence 7

Next log sequence to archive 8

Current log sequence 8

MTERAC15>

2. Place the production database in hot backup mode:

MTERAC15> alter database begin backup;

Database altered.

3. Create a writeable checkpoint of the database data file system (datafs):

[root@mteorans40-1 ~]# fs_ckpt datafs -Create -readonly y


name = datafs

acl = 0

in_use = True

type = uxfs

worm = off

volume = v540

pool = datapool






ro_servers=

rw_vdms =

ro_vdms =


ckpts = datafs_ckpt1


disks = d20,d27,d29,d33,d8











id = 350

name = datafs_ckpt1

acl = 0

in_use = True

type = ckpt

worm = off

volume = vp859

pool = datapool

member_of =

rw_servers=




rw_vdms =

ro_vdms =

checkpt_of= datafs Tue Apr 22 16:20:37 EDT 2008

used = 3%

full(mark)= 90%


disks = d20,d27,d29,d33,d8











4. Make the checkpoint writeable:

[root@mteorans40-1 ~]#

[root@mteorans40-1 ~]# fs_ckpt datafs_ckpt1 -Create -readonly n


name = datafs_ckpt1

acl = 0

in_use = True

type = ckpt

worm = off

volume = vp859

pool = datapool

member_of =




rw_servers=


rw_vdms =

ro_vdms =

checkpt_of= datafs Tue Apr 22 16:20:37 EDT 2008

ckpts = datafs_ckpt1_writeable1

used = 6%

full(mark)= 90%


disks = d20,d27,d29,d33,d8











id = 351

name = datafs_ckpt1_writeable1

acl = 0

in_use = True

type = wckpt

worm = off

volume = vp859

pool = datapool

member_of =




ro_servers=

rw_vdms =

ro_vdms =

checkpt_of= datafs

baseline_ckpt = datafs_ckpt1 Tue Apr 22 16:20:37 EDT 2008

used = 6%

full(mark)= 90%


disks = d20,d27,d29,d33,d8











5. Take the production database out of hot backup mode:

MTERAC15>alter database end backup;

Database altered.

6. Archive the current redo log file:

MTERAC15> alter system archive log current;

System altered.

7. List the current active log files:

MTERAC15> SELECT member FROM v$log l, v$logfile f WHERE l.group# = f.group# AND l.status= CURRENT';

MEMBER




--------------------------------------------------------------------------------

/u03/oradata/MTERAC15/onlinelog/o1_mf_1_40qxykv4_.log

/u04/oradata/MTERAC15/onlinelog/o1_mf_1_40qy1834_.log

/u03/oradata/MTERAC15/onlinelog/o1_mf_8_40qyhd6o_.log

/u04/oradata/MTERAC15/onlinelog/o1_mf_8_40qyl3bb_.log

/u03/oradata/MTERAC15/onlinelog/o1_mf_6_40qysor5_.log

/u04/oradata/MTERAC15/onlinelog/o1_mf_6_40qyw7yj_.log

/u03/oradata/MTERAC15/onlinelog/o1_mf_3_40qyyvnh_.log

/u04/oradata/MTERAC15/onlinelog/o1_mf_3_40qz1r89_.log

8 rows selected.

8. To create a writeable checkpoint of the database archive log file system (archfs), repeat steps 1 to 7.

9. Create nfs exports for the data and arch writeable checkpoints:

[root@mteorans40-1 ~]# server_export server_2 -P nfs \

/datafs_ckpt1_writeable1

server_2 : done

[root@mteorans40-1 ~]# server_export server_2 -P nfs \

/archfs_ckpt1_writeable1

server_2 : done

10. Mount the data and arch file system writeable checkpoints and snap_log1 and snap_log2 file systems, on the test/dev database server, to the same directory structure as the respective database components are placed on the production database servers.

[root@mteoraesx1-vm6 ~]# cat /etc/fstab

---Relevant portion only shown

192.168.4.101:/datafs_ckpt1_writeable1/oradata /u02/oradata nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0

192.168.4.101:/snap_log1/oradata /u03/oradata nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0

192.168.4.101:/snap_log2/oradata /u04/oradata nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0

192.168.4.101:/archfs_ckpt1_writeable1 /u05 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0

[root@mteoraesx1-vm6 ~]# mount /u02/oradata







[root@mteoraesx1-vm6 ~]# mount /u05

11. On the test/dev database server, set the environment parameters ORACLE_SID and ORACLE_HOME to the same settings as on the production database servers, or set them as per the Oracle installation procedure.

[oracle@mteoraesx1-vm6 ~] $export ORACLE_SID=mterac155

[oracle@mteoraesx1-vm6 ~] $export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1

12. Create the required database dump directories on the test/dev database server:

[oracle@mteoraesx1-vm6 ~]$ mkdir $ORACLE_HOME/admin

[oracle@mteoraesx1-vm6 ~]$ mkdir $ORACLE_HOME/admin/mterac155

[oracle@mteoraesx1-vm6 ~]$ mkdir $ORACLE_HOME/admin/ \

mterac155/adump


mterac155/bdump


mterac155/cdump


mterac155/dpdump


mterac155/hdump


mterac155/udump


mterac155/pfile

13. Configure the listener.ora and tnsnames.ora files on the test/dev database server.

Note: If the clone target is RAC, ensure that steps 9 to 12 are repeated on each of the RAC nodes of the target clone database.

14. Create a pfile for the test/dev instance. Use the parameter file from the production instance as a base template

15. Modify the parameter file depending on whether the target is a single-instance database or a RAC database. Ensure that the following parameters such as control_files, db_recovery_file_dest, db_recovery_file_dest_size, etc. are updated. In the case of the clone target being a RAC database, make sure that the parameter file is updated on all RAC nodes.

16. Copy the control file from the production database server to the corresponding location on the test/dev database server.

[oracle@mteoradb51 ~]$ scp /u03/oradata/MTERAC15/controlfile/o1_mf_40qxyh25_.ctl oracle@mteoraesx1-vm6:/u03/oradata/MTERAC15/controlfile/



oracle@mteoraesx1-vm6's password:

o1_mf_40qxyh25_.ctl 100% 15MB 14.6MB/s 00:01

[oracle@mteoradb51 ~]$

17. List the available archive logs. This list is required for recovering the test/dev database.

[oracle@mteoraesx1-vm6 dbs]$ ls -ltr /u05/MTERAC15/

total 15147480

-rw-rw---- 1 oracle oinstall 1185634304 Apr 22 2008 3_10_652572689.dbf















[oracle@mteoraesx1-vm6 dbs]$

18. Configure the target clone RAC database by adding database instances.

Note: This step is required only if the clone target is a RAC database. The target clone database must be configured by adding the appropriate database and corresponding instances.

[oracle@mteoradb63 ~]$ srvctl add database -d mterac20 -o /u01/app/oracle/product/10.2.0/db_1


[oracle@mteoradb63 ~]$ srvctl add instance -d mterac20 -i mterac201 -n mteoradb63




[oracle@mteoradb63 ~]$ srvctl config database

mterac20

19. Start up the test/dev database in mount mode. In the case of a RAC database, execute the following from the first node:

MTERAC155> startup mount


Total System Global Area 662700032 bytes



Database Buffers 50331648 bytes


Database mounted.

MTERAC155>

20. Recover the test/dev database. Specify the filenames of the archive logs from the list displayed in step 17. At the end, specify the current log file name from the same thread.

MTERAC155> recover database until cancel;




Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

/u05/MTERAC15/1_17_652572689.dbf





/u05/MTERAC15/2_4_652572689.dbf








/u05/MTERAC15/3_13_652572689.dbf




Apr 22 23:30 4_18_652572689.dbf


/u05/MTERAC15/4_21_649662783.dbf




ORA-00278: log file '/u05/4_17_652572689.dbf' no longer needed for this recovery


cancel

Media recovery cancelled.

21. Open the test/dev database with the resetlogs option.

MTERAC155> alter database open resetlogs;

22. Verify that the test/dev database instance is opened in read-write mode on the clone target.

MTERAC155> select name, open_mode from v$database;

NAME OPEN_MODE

------------ -------------------

PRIMDB READ WRITE

23. If the clone target is a RAC database, shut down and restart both the database and the cluster services. Verify that the cluster services are up and that all the RAC database instances are online:

[oracle@mteoradb63 ~]$ crs_stat -t

Name Type Target State Host





------------------------------------------------------------

















ora....ac20.db application ONLINE ONLINE mteoradb63

ora....01.inst application ONLINE ONLINE mteoradb63




[oracle@mteoradb63 ~]$

The test/dev database is now open in read write mode and contains a copy of the production database. This can be used for testing and development, data warehouse staging, backup, or any other purpose for which you need a copy of the production database. Remember, however, that unchanged blocks are shared with the production database, and reads of those blocks will impact production database performance as well.

Chapter 12 Advanced Protect Solution Using MirrorView and Data Guard


MirrorView ................................................................................................................................. 135 Data Guard .................................................................................................................................. 138

The use of Oracle Data Guard for creating a disaster recovery configuration is an established best practice. MirrorView/A over iSCSI is commonly used as a way of seeding the database for the Data Guard configuration. Once the production database has been copied to the target location, then redo log shipping can be established using Data Guard.

The use of MirrorView over iSCSI requires specific network configuration. Various means can be used to bridge an iSCSI network over a WAN connection, so that the data on an iSCSI network can be transmitted over long distances. We will not cover those technologies here. We assume that the customer has established a mechanism to transmit the data from the source to the target array over an IP network using the iSCSI protocol.

The advantages of MirrorView are:

♦ The data can be replicated over a long distance. Local access is not required.

♦ No downtime on the source database is required.

MirrorView Before starting the mirroring procedure:

♦ The source and target arrays must be in the same CLARiiON domain. We designated the source CLARiiON as the master, and the target CLARiiON as a domain member. Once this is done, both the source and target arrays appear in the same Navisphere GUI and can be managed together. This is mandatory for MirrorView mirroring to be established.

♦ Reserved LUN pool LUNs are required for MirrorView mirroring. You should configure a large number of small LUNs for best results. We configured 20 LUNs of 10 GB in size.


Advanced Protect Solution Using MirrorView and Data Guard

Establishing MirrorView as a solution for Data Guard seeding involves the following steps:

1. Create a consistency group on the source array with the following command:

[root@mteoradb1 db_root]# naviseccli -h 10.6.24.205 mirror –async \

-creategroup –name ConsistentDBGroup –o

You are about to create a new mirror group with the name ConsistentDBGroup.

Do you want continue with the creation of this group (y/n)? y

2. Create a mirror group for the source LUNs, for example:

[root@mteoradb59 ~]# naviseccli -h 10.6.27.65 mirror -async -create -name LUN1_LOG1_mirror -lun 1 -requiredimages 1

Warning! Make sure you are done enabling paths among all arrays. If not, exit and do so.

Do you want to continue with the creation of this mirror (y/n)? y

3. Verify that all the mirror groups were created successfully using the following command:

[root@mteoradb59 ~]# naviseccli -h 10.6.27.65 mirror -async -list

4. Use the following script to add the target LUNs to the mirror groups and also to the consistency group:

Code Listing 22: MirrorView/A Data Guard seeding script

[root@mteoradb59 db_root]# cat mirror_new.bash

echo "This is mirror.bash"

DATA_LUNS="3 4 5 6 7 8"

LOG_LUNS="1 2"

SPA=10.6.27.65

echo "Add the target LOG LUNs for mirroring"

for i in ${LOG_LUNS}

do

echo "Now adding lun LUN${i} of target Clarion to mirror"

naviseccli -address ${SPA} mirror -async -addimage -name LUN${i}_LOG${i}_mirror -arrayhost 10.6.27.138 -lun 5${i} -recoverypolicy auto -syncrate high

done

echo "Add the target DATA LUNs for mirroring"

for i in ${DATA_LUNS}

do




echo "Now adding lun LUN${i} of target Clarion to mirror"

naviseccli -address ${SPA} mirror -async -addimage -name LUN${i}_DATA_mirror -arrayhost 10.6.27.138 -lun 5${i} -recoverypolicy auto -syncrate high

done

echo "Now adding mirror for LOG LUNS to consistent group."

for i in ${LOG_LUNS}

do

naviseccli -address ${SPA} mirror -async -addtogroup -name ConsistentDBGroup -mirrorname LUN${i}_LOG${i}_mirror

done

echo "Now adding mirror for DATA LUNS to consistent group."

for i in ${DATA_LUNS}

do

naviseccli -address ${SPA} mirror -async -addtogroup -name ConsistentDBGroup -mirrorname LUN${i}_DATA_mirror

done

26644394

echo "Now exiting mirror.bash"

Code Listing 23: The output from the MirrorView/A script

[root@mteoradb59 db_root]# ./mirror_new.bash

This is mirror.bash

Add the target LOG LUNs for mirroring

Now adding lun LUN51 of target Clarion to mirror


Add the target DATA LUNs for mirroring







Now adding mirror for LOG LUNS to consistent group.

Now adding mirror for DATA LUNS to consistent group.

Now exiting mirror.bash



5. Verify the mirroring status by executing the following command:

[root@mteoradb59 ~]# naviseccli -h 10.6.27.65 mirror -async \

-list -images | grep Progress

Synchronizing Progress(%): 100








6. Once the mirrors are synchronized, they can be fractured by using the following command. The mirror LUNs will become available for I/O only after they are fractured.

[root@mteoradb59 ]# naviseccli -h 10.6.27.65 mirror -async \

-fracturegroup -name ConsistentDBGroup -o

7. Once the seeding of database is complete, the remaining tasks for Data Guard can be performed to do redo log shipping.

Data Guard The configuration steps for shipping the redo logs and bringing up the standby database are accomplished using Oracle Data Guard. The semantics are covered thoroughly on the Oracle Technology Network website, and will not be included here.

The Data Guard Failover operation was also performed in MAXIMUM AVAILABILITY mode using the following steps:

1. Shut down all the database instances at the production/primary site.

MTERAC71> shutdown abort

ORACLE instance shut down.







2. The standby database is changed to primary by issuing the following commands:

[MTERAC7 – Production Database & MTERAC7-SB – Standby Database]

MTERAC7_SB>SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM $ARCHIVE_GAP;




no rows selected

MTERAC7_SB>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;

Database altered.

MTERAC7_SB>ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

Database altered.

MTERAC7_SB>shutdown immediate;

ORA-01109: database not open

Database dismounted.


mteora7_sb>set sqlprompt ‘NEW_PRIMARY> ‘

NEW_PRIMARY>startup


Total System Global Area 171966464 bytes



Database Buffers 50331648 bytes


Database mounted.

Database opened.

NEW_PRIMARY> select DATABASE_ROLE, SWITCHOVER_STATUS, GUARD_STATUS from v$DATABASE;

DATABASE_ROLE SWITCHOVER_STATUS GUARD_S

---------------- -------------------- -------

PRIMARY NOT ALLOWED NONE


Chapter 13 Direct NFS


Overview: Direct NFS and Oracle 11g........................................................................................ 141 Comparison: Kernel NFS and DNFS .......................................................................................... 141 Advantages of DNFS .................................................................................................................. 142 Management and setup ................................................................................................................ 142 EMC Celerra and DNFS .............................................................................................................. 143 Tested configuration .................................................................................................................... 143 IP storage server environment ..................................................................................................... 143 Network connections ................................................................................................................... 143 Enabling DNFS ........................................................................................................................... 143 Verifying that DNFS has been enabled ....................................................................................... 144

Overview: Direct NFS and Oracle 11g Oracle 11g includes a new feature for storing Oracle datafiles on a NAS device, referred to as Direct NFS or DNFS. This feature provides performance advantages over conventional Linux kernel NFS (or kNFS) in that fewer context switches are required to perform an I/O. This should result in reduced CPU utilization, and thus a performance benefit.

Comparison: Kernel NFS and DNFS Kernel NFS

I/O caching and performance characteristics vary between operating systems. This leads to varying NFS performance across different operating systems (for example, Linux and Solaris), and across different releases of the same operating system (for example, RHEL 6.2 and RHEL 7.2). This in turn results in varying NFS performance across your implementations.

DNFS

Because DNFS integrates the client NFS protocol into the Oracle kernel, this allows all I/O calls to be made in user space, rather than requiring a context switch to kernel space. As a result, CPU utilization associated with the I/O of the database server is reduced.


Direct NFS

Advantages of DNFS Performance

Consistent performance

Consistent NFS performance is observed across all operating systems.

Improved caching and I/O management

The DNFS kernel is designed for improved caching and management of the I/O patterns that are typically experienced in database environments, that is, larger and more efficient reads/writes.

Asynchronous direct I/O

The DNFS kernel enables asynchronous direct I/O, which is typically the most efficient form of I/O for databases. Asynchronous direct I/O significantly improves database read/write performance by enabling I/O to continue while other requests are being submitted and processed.

Overcomes OS write locking

DNFS overcomes OS write locking, which can be inadequate in some operating systems and can cause I/O performance bottlenecks in others.

Reduced CPU and memory usage

Database server CPU and memory usage are reduced by eliminating the overhead of copying data to and from the OS memory cache to the database SGA cache.

Data integrity

To ensure database integrity, immediate writes must be made to the database when requested. Operating system caching delays writes for efficiency reasons, this potentially compromises data integrity during failure scenarios. DNFS uses database caching techniques and asynchronous direct I/O to ensure almost immediate data writes, thus reducing data integrity risks.

Management and setup

Simplified network setup in high availability environments

Load-balancing and high availability (HA) are managed internally within the DNFS client itself, rather than at the OS level. This greatly simplifies network setups in HA environments and reduces dependence on IT network administrators by eliminating the need to set up network subnets and bond ports, for example, LACP bonding.

Parallel network paths

DNFS allows four parallel network paths/ports to be used for I/O between the database server and the IP storage array. For efficiency and performance, these paths are managed and load-balanced by the DNFS client, not by the operating system.



Direct NFS

EMC Celerra and DNFS The Oracle RAC 11g and EMC Celerra DNFS solution enables a midsize enterprise to deploy an EMC NAS architecture with dNFS connectivity for its Oracle RAC 11g database applications with lower cost and reduced complexity than direct-attached storage (DAS) or a storage area network (SAN).

Figure 87 DNFS solution: EMC Celerra and Oracle 11g

Tested configuration IP storage server environment

A Celerra NS40 Unified Storage Platform was used as the basis for all DNFS testing.

Network connections

A Celerra NS40 was configured as the IP storage server as follows:

♦ Four one Gigabit Ethernet connections for each Data Mover

♦ Four individual ports for the dNFS connections. DNFS manages the bonding, HA, and load balancing for the connections.

♦ All four network ports were configured for full use by DNFS

♦ Three one Gigabit Ethernet connections were used for the Oracle RAC Interconnect

Enabling DNFS It is relatively easy to configure DNFS. It applies only to the storage of Oracle datafiles. Redo log files, tempfiles, control files, and the like are not affected. You can attempt to configure the mount points where these files are stored to support DNFS, but this will have no impact.

After installing the Oracle 11g database software, carry out the following steps to enable DNFS:


Direct NFS

1. Create a file called “oranfstab” at the location “$ORACLE_HOME/dbs/”:

[oracle@mteoradb55 ~]$ cat /u01/app/oracle/product/11.1.0/db_1/dbs/oranfstab

server: mteorans40-2

path: 192.168.4.99

path: 192.168.5.99

path: 192.168.2.99

path: 192.168.6.99

export: /datafs mount: /u02

export: /log1fs mount: /u03

export: /log2fs mount: /u04

export: /archfs mount: /u05

2. Oracle database uses an ODM library, libnfsodm11.so, to enable DNFS. To replace the standard ODM library ‘libodm11.so’ with the ODM NFS library ‘libnfsodm11.so’:

a. Change the directory to $ORACLE_HOME\bin

b. Shut down Oracle.

c. Execute the following commands on the database servers:

$ cp libodm11.so libodm11.stub

$ mv libodm11.so libodm11.so_stub

$ ln –s libnfsodm11.so libodm11.so

Verifying that DNFS has been enabled The following queries can be executed to verify that DNFS is enabled for the database.

Check the available DNFS storage paths

SQL> select unique path from v$dnfs_channels;

PATH

--------------------------------------------------------------------------------

192.168.4.99

192.168.6.99

192.168.5.99

192.168.2.99

Check the data files configured under DNFS

SQL> select FILENAME from V_$DNFS_FILES;



Direct NFS

FILENAME

--------------------------------------------------------------------------------

/u03/oradata/MTERAC23/controlfile/o1_mf_4mst6cxw_.ctl

/u04/oradata/MTERAC23/controlfile/o1_mf_4mst6d7s_.ctl

/u02/oradata/MTERAC23/datafile/o1_mf_system_4mst3l7g_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_sysaux_4mst3lbm_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_undotbs1_4mst3ld8_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_users_4mst3lgc_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_undotbs2_4mstl01c_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_undotbs3_4mstl17x_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_undotbs4_4mstl2co_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4msxhblb_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4msxs7fd_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4msy35pl_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4msyf49w_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4msyq3ck_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4msz12f1_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4mszc454_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4mszny17_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4mszyr40_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4mt08d5o_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4mt0l6ng_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_test_4mt0vv45_.dbf

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mstkcsg_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15nlb_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15nx6_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15o88_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15okz_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15otm_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15p3o_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15pcv_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15pn8_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15pwz_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15q79_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15qk8_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15qsl_.tmp


Direct NFS



/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15r2t_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15rcc_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15rms_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15rwr_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15s84_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15skl_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15st8_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15t3m_.tmp

/u02/oradata/MTERAC23/datafile/o1_mf_temp_4mt15td3_.tmp

/u03/oradata/MTERAC23/onlinelog/o1_mf_1_4npzyjfj_.log

/u04/oradata/MTERAC23/onlinelog/o1_mf_1_4nq0114z_.log

/u03/oradata/MTERAC23/onlinelog/o1_mf_2_4nq03db9_.log

/u04/oradata/MTERAC23/onlinelog/o1_mf_2_4nq05t20_.log

47 rows selected.

Check the server and the directories configured under DNFS

SQL> select SVRNAME, DIRNAME, MNTPORT, NFSPORT from V_$DNFS_SERVERS;

SVRNAME DIRNAME MNTPORT NFSPORT

--------------- --------------- ---------- ----------

mteorans40-2 /log1fs 1234 2049

mteorans40-2 /log2fs 1234 2049

mteorans40-2 /datafs 1234 2049

mteorans40-2 /archfs 1234 2049

Chapter 14 Networking


Networking issues ....................................................................................................................... 147 Ethernet trunking and link aggregation ....................................................................................... 148 Configuring the public and private network ................................................................................ 149

Networking issues Jumbo frames

Jumbo frames should be configured for the storage and RAC interconnect networks on this solution. For information on configuring jumbo frames for the RAC interconnect see Metalink article 300388.1.

Jumbo frames are configured for three layers:

♦ Celerra Data Mover

♦ Switch

♦ Oracle RAC 10g servers

The steps for configuring this vary. Configuration steps for the switch are not covered here, as that is vendor-specific. Check your switch documentation for details.

On the Data Mover, to configure jumbo frames, execute the following command on the Control Station:

server_ifconfig server_2 int1 mtu=9000

where:

♦ server_2 is the Data Mover

♦ int1 is the interface

For Linux servers, the command is:

ifconfig eth0 mtu 9000


Networking

Or the following statement can be placed in the network scripts in /etc/sysconfig/network-scripts:

MTU=9000

See the next section for more information.

To test whether jumbo frames are enabled, use the following command:

ping –M do –s 8192 <target>

Where target is the interface to be tested, jumbo frames must be enabled on all layers of the network for this command to succeed.

Ethernet trunking and link aggregation A trunking device is a virtual device created using two or more network devices to achieve higher performance with load-balancing capability, and high availability with failover capability.

With Ethernet trunking/link aggregation, packets traveling through the virtual device are distributed among the underlying devices to achieve higher aggregated bandwidth, based on the source MAC address.

EMC recommends that you configure an Ethernet trunking interface with two Gigabit Ethernet ports to the same switch. To enable trunking on Data Mover, use the following methods.

♦ Set up a two-port channel device using this command:

server_sysconfig server_2 -virtual -name lacp1 -create trk \

-option "cge0,cge1 protocol=lacp"

♦ Assign an IP address to the logical device:

server_ifconfig server_2 -create -Device lacp1 -name int1 \

-protocol IP x.x.x.x y.y.y.y z.z.z.z

On the database servers, network redundancy is achieved by using Linux kernel bonding. This is accomplished using the scripts contained in /etc/sysconfig/network-scripts. A typical bonded connection is shown:

DEVICE=bond0

ONBOOT=yes

BOOTPROTO=none

USERCTL=no

IPADDR=10.6.25.188

NETMASK=255.255.255.0

MTU=9000

This device consists of two Ethernet ports, whose scripts are similar to that shown:

DEVICE=eth1

MASTER=bond0

SLAVE=yes



Networking

ONBOOT=yes

BOOTPROTO=none

HWADDR=00:04:23:B9:66:F3

The result is that the Ethernet ports that show their master as bond0 are joined to the bonded connection. Modify the /etc/modprobe.conf file. The following is an example of the lines that must be added:

options bonding max_bonds=2 mode=4

alias bond0 bonding

alias bond1 bonding

Either reboot the Linux server or down and up the interfaces to enable the trunk.

Configuring the public and private network Each node should have one static IP address for the public network and one static IP address for the private cluster interconnect. The private interconnect should only be used by Oracle to transfer cluster manager and cache fusion related data. Although it is possible to use the public network for the RAC interconnect, this is not recommended as it may cause degraded database performance (reducing the amount of bandwidth for cache fusion and cluster manager traffic).

The virtual IP addresses must be defined in either the /etc/hosts file or DNS for all RAC nodes and client nodes. The public virtual IP addresses will be configured automatically by Oracle when the Oracle Universal Installer is run, which starts Oracle's Virtual Internet Protocol Configuration Assistant (vipca). All virtual IP addresses will be activated when the following command is run:

srvctl start nodeapps -n <node_name>

where node_name is the hostname/IP address that will be configured in the client's tnsnames.ora file.


Networking



Chapter 15 Oracle on VMware


Single-instance virtualized Oracle database server scale-out ...................................................... 151 Configuration .............................................................................................................................. 152 ESX server configuration ............................................................................................................ 154 Migrating a single-instance Oracle database using VMotion ...................................................... 158 Discovering SAN storage on VMware ........................................................................................ 161 VMware High Availability cluster configuration ........................................................................ 173 Network configuration ................................................................................................................ 173 NFS storage networking in a VMware HA cluster context ......................................................... 174

Single-instance virtualized Oracle database server scale-out Many database customers use Oracle in a scale-out configuration. This is defined as a scenario whereby the database can be partitioned into multiple logical databases. An example would be the model of providing software as a service (SaaS). When a company is providing software as a service over the Internet, each customer does not need to see the data of any other customer. This means that the database data can be partitioned across multiple physical servers with no loss of performance or functionality.

In this scenario, we found that virtualization provided significant advantages:

♦ Flexibility and manageability can be improved through virtualization. For example, you can use VMotion to migrate a running Oracle single instance database from one VMware ESX server to another. This enables you to carry out tasks such as hardware upgrades and software patching with no downtime on the Oracle database. This is covered in detail in “Migrating a single instance Oracle database using VMotion.”

♦ Performance can actually be improved. Our testing indicated that a single instance Oracle database on a large physically booted Linux server produced less transactional throughput than several smaller Oracle single instance databases running under virtualized Linux on an equivalent piece of server hardware running VMware ESX.

The support limitations of Oracle prohibit the use of virtualization with Oracle RAC 10g. For this reason, the solution validated by EMC is limited to single instance Oracle Database 10g.


Oracle on VMware

Configuration A total of four VMs were created and each VM hosted a separate single instance database. The Benchmark Factory tests were executed simultaneously on all four databases to capture the scale-out performance. The main steps for configuring the ESX server, VM, and Storage are explained in this section.

Storage Configuration

The same Celerra system was used to store each of the four databases. The file systems for storing data, redo logs, archive, and flash were created and exported to each of the four VMs as shown below.

[root@rtpsol37a root]# nas_fs -l

id inuse type acl volume name server

1 n 1 0 10 root_fs_1

2 y 1 0 12 root_fs_2 1

3 y 1 0 14 root_fs_3 2

4 n 1 0 16 root_fs_4

5 n 1 0 18 root_fs_5

6 n 1 0 20 root_fs_6

7 n 1 0 22 root_fs_7

8 n 1 0 24 root_fs_8

9 n 1 0 26 root_fs_9

10 n 1 0 28 root_fs_10

11 n 1 0 30 root_fs_11

12 n 1 0 32 root_fs_12

13 n 1 0 34 root_fs_13

14 n 1 0 36 root_fs_14

15 n 1 0 38 root_fs_15

16 y 1 0 40 root_fs_common 2,1

17 n 5 0 73 root_fs_ufslog

18 n 5 0 76 root_panic_reserve

19 n 5 0 77 root_fs_d3

20 n 5 0 78 root_fs_d4

21 n 5 0 79 root_fs_d5

22 n 5 0 80 root_fs_d6

23 y 1 0 121 log1fs 1

25 y 1 0 123 log2fs 1

27 y 1 0 128 datafs 1

29 y 1 0 130 archfs 1



Oracle on VMware

31 y 1 0 132 flashfs 1

===============================================================

[root@rtpsol37a root]# server_export server_2

server_2 :

export "/flashfs" root=192.168.4.10:192.168.4.20:192.168.4.30:192.168.4.40

export "/archfs" root=192.168.4.10:192.168.4.20:192.168.4.30:192.168.4.40

export "/log2fs" root=192.168.4.10:192.168.4.20:192.168.4.30:192.168.4.40

export "/log1fs" root=192.168.4.10:192.168.4.20:192.168.4.30:192.168.4.40

export "/datafs" root=192.168.4.10:192.168.4.20:192.168.4.30:192.168.4.40

export "/" anon=0 access=192.168.1.100:192.168.2.100:192.168.1.101:192.168.2.101

===============================================================

Under each of those filesystems, separate directories were created which will in turn map to the corresponding database.

===============================================================

[root@rtpsol37a server_2]# cd datafs/

[root@rtpsol37a datafs]# pwd

/mnt/server_2/datafs

[root@rtpsol37a datafs]# ls -l

total 8

drwxrwxrwx 2 500 501 8192 Mar 10 21:36 lost+found

drwxrwxr-x 4 500 502 80 Mar 19 18:56 mtedb17a

drwxrwxrwx 3 500 501 80 Mar 13 20:02 mtedb17b

drwxrwxrwx 3 500 501 80 Mar 13 21:08 mtedb17c

drwxrwxrwx 3 500 501 80 Mar 17 07:35 mtedb17d

In the case of the blended FCP/NFS solution, the ARCHIVE and FLASH file systems were configured as NFS file systems and four separate sets of LUNs (data and redo logs) were created on the backend CLARiiON CX4-240 series storage array to store redo logs and data.


Oracle on VMware

The LUN layout on the CLARiiON CX4-240 series storage array for the blended FCP/NFS solution is as shown below.

Table 6 LUN layout for blended solution

Database files LUNs for VM1 LUNs for VM2 LUNs for VM3 LUNs for VM4 REDO LOG1 LUN1 LUN11 LUN21 LUN31 REDO LOG2 LUN2 LUN12 LUN22 LUN32 DATA LUN3 LUN13 LUN23 LUN33 DATA LUN4 LUN14 LUN24 LUN34 DATA LUN5 LUN15 LUN25 LUN35 DATA LUN6 LUN16 LUN26 LUN36 DATA LUN7 LUN17 LUN27 LUN37 DATA LUN8 LUN18 LUN28 LUN38

ESX server configuration 1. Capture the total physical memory that is available on the ESX server.

Figure 88 Total physical memory available on ESX server



Oracle on VMware

2. Capture the total number of CPUs that is available on the ESX server.

Figure 89 Total number of CPUs available on ESX server


Oracle on VMware

3. Create four VMs on the ESX server. Distribute the memory and CPUs available equally to each of the VMs.

Figure 90 Four VMs on the ESX server

4. Assign a VMkernel IP (10.6.115.144) to each ESX server so that it can be used to mount NFS storage. All the VMs need to be located on common storage and this is mandatory for performing VMotion.

Figure 91 VMwareBoot



Oracle on VMware

5. Configure four additional NICs on the ESX server; dedicate each NIC to a VM as shown in Figure 92.

Figure 92 Configure additional NICs

These additional NICs were used to configure the dedicated private network connection to Celerra where the database files resided.

The Celerra file systems for storing the DATA files, REDO log files, ARCHIVE, and FLASH files were directly mounted using NFS on the VMware hosts as shown in Figure 93.


Oracle on VMware

Figure 93 Mounting Celerra file systems using NFS on VMware hosts

In the case of the blended FCP/NFS solution, the redo log and data LUNs were presented to the VM using Raw Device Mapping (RDM). Please refer to “Discovering SAN Storage on VMware” for a detailed procedure for mapping the LUNs on VM using RDM. The archive and flash file systems were still mounted using NFS as explained previously.

Migrating a single-instance Oracle database using VMotion As stated in the previous section, VMotion can be used to migrate a running Oracle Database 10g single instance database from one VMware ESX server to another. Doing so has no impact on the performance of the running Oracle database.

In order to use VMotion in this manner, the following prerequisites must be in place:

1. Make sure that you have a private IP assigned for VMotion on both ESX servers. For example, on the ESX server “mteoraesx5”, the IP address was assigned as 192.168.75.1.



Oracle on VMware

Figure 94 Private IP address assigned for VMotion on ESX server mteoraesx5

Similarly, on the “mteoraesx4” server the VMotion IP address was set to 192.168.75.2 as shown below.

Figure 95 Private IP address assigned for VMotion on ESX server mteoraesx4


Oracle on VMware

2. If NFS storage is used, the databases for each of the four VMs must reside on a single Celerra NS40 storage system. The NFS file systems are directly mounted on the VMs. So, to perform VMotion, it is critical that there is connectivity to storage on both ESX servers. The switch configuration should be exactly the same including the name of the virtual switch. In this example, the virtual switch used for storage connectivity on VM “mteoraesx5-vm6” is named as “mteoraesx5-vm6”. Refer to the following diagram.

Figure 96 Virtual switch name for mteoraesx5



Oracle on VMware

Similarly, on the other ESX server “mteoraesx4”, a virtual switch with the same name is created and connectivity is established to the NS storage through the same port. The network configuration on “mteoraesx4” also has the same named switch, “mteoraesx5-vm6,” as shown below.

Figure 97 Virtual switch name for mteoraesx4

3. Make sure that the VMware tools are loaded on any VM that will be migrating using VMotion.

If these prerequisites are met, the VM containing an Oracle Database 10g instance can be migrated easily and with no perceptible performance impact on the running production Oracle database.

Discovering SAN storage on VMware Virtualized Oracle database servers can be used as targets for Test/dev, Advanced and Basic Protect solutions. These servers are more conveniently managed as virtual machines than as physically booted Oracle database servers. The advantages of consolidation, flexible migration and so forth, which are the mainstays of virtualization, apply to these servers very well.

A single VMware Linux host was used as the target for both the Test/dev and Protect solutions. The target database in the case of the Test/dev solution was brought up as a single-instance database on the VMware host. Similarly, the standby database in the case of the Protect solution was a single-instance database running on a VMware host.

The procedure for discovering the backend CLARiiON CX4-240 series storage array LUNs on the VMware host using RDM is as follows:


Oracle on VMware



Note: Before starting the mirroring procedure, ensure that the FC connections are made from the ESX server to the backend storage and that appropriate zoning is configured so that the ESX server can see the required storage.

1. Make sure that the ESX server is connected to the storage group where the LUNs are going to be located.

Figure 98 mteoraesx4.solutions1.rtp.dg.com is connected

As shown in Figure 98, the ESX server mteoraesx4.solutions1.rtp.dg.com is connected to the storage group standby_stg.

Oracle on VMware

2. Move the LUNs into the storage group to which the ESX server is connected. As shown in Figure 99, all the LUNs moved into this storage group are accessible from the ESX server.

Figure 99 LUNs are accessible

3. Using the Virtual Infrastructure Client, log in to the ESX server.


Oracle on VMware

4. Go to the Configuration tab and click Rescan to discover the storage. After the rescan, you will see the LUNs being discovered on the ESX server.

Figure 100 LUNs are discovered

5. Click the VMware machine where the target database needs to be mounted by mapping the ESX discovered storage, then click Edit Settings.



Oracle on VMware

Figure 101 Virtual Machine Properties

6. In the dialog box displayed, click Add and you will see the option to add the type of device.


Oracle on VMware

7. Select Hard Disk and click Next, as shown in Figure 102.

Figure 102 Select Device Type



Oracle on VMware

8. In the dialog box that appears, select Raw Device Mappings, as shown in Figure 103.

Figure 103 Select Raw Device Mappings for your disk


Oracle on VMware

9. Select the target LUN and click Next, as shown in Figure 104.

Figure 104 Select the target LUN



Oracle on VMware

10. Select the Store with Virtual Machine option, as shown in Figure 105.

Figure 105 Select Store with Virtual Machine as the datastore


Oracle on VMware

11. Select the compatibility mode as Physical.

Figure 106 Select Physical as the compatibility mode



Oracle on VMware

12. Select the default value for the Virtual Device Node.

Figure 107 Select the Virtual Device Node


Oracle on VMware

13. Click Finish to complete the configuration.

Figure 108 Complete the configuration

14. Once you log in to the VMware machine, you will see the new PowerPath device as well as the corresponding ASM disk discovered for the newly added LUN.

15. Once the ASM disks are discovered, the corresponding ASM diskgroups can be mounted and the database can be mounted, recovered, and opened on the target.



Oracle on VMware

VMware High Availability cluster configuration This section describes the configuration of a VMware HA cluster with a view towards implementing Oracle Database 10g or 11g.

Network configuration

Figure 109 shows the network configuration that should be applied to each of the nodes in a VMware HA cluster:

Figure 109 Typical VMware HA cluster network configuration

The critical components are the storage network (required for the pure NFS solution) and the VMkernel HA Cluster Network. You must apply an identical configuration to each of the ESX hosts in the cluster (other than the physical IP address of course). Each node must contain exactly the same vSwitches and they must be accessible to each other over IP.

Although Figure 109 shows a RAC Interconnect Network, it is not required for this solution. RAC is currently not supported on a VMware HA cluster.


Oracle on VMware

NFS storage networking in a VMware HA cluster context

The guest OS should use NFS through a direct connection to the NFS server, and not by an indirect connection through the ESX server. There are huge manageability and performance advantages to configuring the solution in this manner. The configuration of NFS on the guest OS is identical to a physically booted Linux host, that is, the /etc/fstab file should be configured to mount the NFS exports directly onto the guest OS. The following listing from the /etc/fstab file is typical:

mteorans5-dm2:/datafs /u02 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101 0 0

mteorans5-dm2:/log1fs /u03 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101 0 0

mteorans5-dm2:/log2fs /u04 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101 0 0

mteorans5-dm2:/archfs /u05 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101 0 0

mteorans5-dm2:/flashfs /u06 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101 0 0

When the /etc/fstab file is configured this way, the mount command returns the following:

[root@mteoradbvm1 ~]# mount

....

mteorans5-dm2:/archfs on /u05 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101)

mteorans5-dm2:/flashfs on /u06 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101)

mteorans5-dm2:/datafs on /u02 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101)

mteorans5-dm2:/log1fs on /u03 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101)

mteorans5-dm2:/log2fs on /u04 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=192.168.4.101)

[root@mteoradbvm1 ~]#

Physical connectivity to the NFS server is of course required; this must be provided in the ESX context by use of a vSwitch as described in “Network configuration.”



Oracle on VMware

The use of NFS to store the vmdk files (containing the guest OS images) is both supported and encouraged. NFS provides a simple and convenient way to create the necessary shared file system to allow automatic failover of a VM within a VMware HA cluster.

Figure 110 illustrates our use of NFS to store vmdk files for the solution described in this white paper.

Figure 110 Using NFS to store vmdk files within a VMware HA cluster

In Figure 110, the NFS mount point called “vm” on server 192.168.4.101 is used to store the guest OS images. This enables the automatic failover feature of VMware HA cluster through the shared file system provided by NFS.


Oracle on VMware



Chapter 16 Conclusion

The EMC Celerra Unified Storage Platform provides a high-performance, high-availability solution for storing Oracle databases. The use of NFS, combined with Celerra's many unique capabilities, provides advantages to the DBA in terms of convenience, robustness, backup and recovery, simplicity, data protection, and cloning. In addition, the EMC CLARiiON CX4-240 series backend array is available to store data via the FCP protocol.


Conclusion



EMC Solutions for Oracle Database 10g/11g for … Celerra Unified Storage Platform. ... Chapter 6...

Documents

Transcript of EMC Solutions for Oracle Database 10g/11g for … Celerra Unified Storage Platform. ... Chapter 6...