h8100 Unified Storage Oracle Celerra Psg

7/25/2019 h8100 Unified Storage Oracle Celerra Psg

1/95

EMC Unified Storage for Oracle Database 11gPerformance Enabled by EMC Celerra Using DNFS or ASM Proven Solution Guide

1

EMC Unified Storage forOracle Database 11g

Performance Enabled by EMC Celerra

Using DNFS or ASM

Proven Solution Guide


2/95


2

Copyright 2011 EMC Corporation. All rights reserved.

Published February 2011

EMC believes the information in this publication is accurate of its publication date. The information issubject to change without notice.

The information in this publication is provided as is. EMC Corporation makes no representations orwarranties of any kind with respect to the information in this publication, and specifically disclaims impliedwarranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicablesoftware license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

VMware, ESX, and VMware vSphere are registered trademarks or trademarks of VMware, Inc. in theUnited States and/or other jurisdictions. All other trademarks used herein are the property of theirrespective owners.

Part number: H8100


3/95

Table of Contents


3

Table of Contents

Chapter 1: About this document ............................................................................... 6

Audience and purpose ...................................................................................................................... 7

Scope ................................................................................................................................................ 8

Business challenge ........................................................................................................................... 8

Technology solution .......................................................................................................................... 9

Objectives........................................................................................................................................ 10

Reference Architecture ................................................................................................................... 11

Validated environment profile ......................................................................................................... 13

Hardware and software resources .................................................................................................. 13

Unified storage platform environment ............................................................................................. 15

Prerequisites and supporting documentation.................................................................................. 17

Terminology .................................................................................................................................... 18

Chapter 2: Storage Design ...................................................................................... 19

Overview ......................................................................................................................................... 19

Concepts ......................................................................................................................................... 20

Storage setup .................................................................................................................................. 20

Best practices .................................................................................................................................. 21

Data Mover parameters setup ........................................................................................................ 23

Data Mover failover ......................................................................................................................... 24

RAID group layout ........................................................................................................................... 25

Chapter 3: File System ............................................................................................. 28

Overview ......................................................................................................................................... 28

Limitations ....................................................................................................................................... 29

File system layout ........................................................................................................................... 30

Chapter 4: Oracle Database Design........................................................................ 31

Overview ......................................................................................................................................... 31

Considerations ................................................................................................................................ 32

Database file layout ......................................................................................................................... 33

Oracle ASM ..................................................................................................................................... 34


4/95

Table of Contents

EMC Unified Storage for Oracle Database 11g- PerformanceEnabled by EMC Celerra Using DNFS or ASM Proven Solution Guide (Draft)

4

Oracle 11gDNFS ............................................................................................................................ 35

Memory configuration for Oracle 11g.............................................................................................. 38

HugePages ..................................................................................................................................... 39

Chapter 5: Network Design ..................................................................................... 41

Overview ......................................................................................................................................... 41

Concepts ......................................................................................................................................... 42

Best practices and recommendations ............................................................................................. 42

SAN network layout ......................................................................................................................... 43

IP network layout ............................................................................................................................. 43

Virtual LANs .................................................................................................................................... 44

Jumbo frames ................................................................................................................................. 45

Public and private networks ............................................................................................................ 46

Oracle RAC 11gserver network architecture ................................................................................. 47

Chapter 6: Installation and Configuration .............................................................. 48

Overview ......................................................................................................................................... 48

Task 1: Install and configure EMC PowerPath ............................................................................... 49

Task 2A: Set up and configure NAS for Celerra ............................................................................. 53

Task 2B: Set up and configure ASM for CLARiiON ........................................................................ 53

Task 3: Set up and configure database servers ............................................................................. 56

Task 4: Configure NFS client options ............................................................................................. 57

Task 5: Install Oracle grid infrastructure and Oracle RAC .............................................................. 59

Task 6: Configure database server memory options ...................................................................... 59

Task 7: Configure and tune HugePages ......................................................................................... 61

Task 8: Set database initialization parameters ............................................................................... 63

Task 9: Configure the Oracle DNFS client ...................................................................................... 65

Task 10: Verify that DNFS has been enabled................................................................................. 68

Task 11: Configure Oracle Database control files and logfiles ....................................................... 70

Task 12: Enable passwordless authentication using ssh (optional) ............................................... 71

Chapter 7: Testing and Validation .......................................................................... 75

Overview ......................................................................................................................................... 75

Testing tools .................................................................................................................................... 76

Test procedure ................................................................................................................................ 77


5/95

Table of Contents


5

Test results ...................................................................................................................................... 78

DNFS configuration test results ...................................................................................................... 79

ASM configuration test results ........................................................................................................ 82

Chapter 8: Conclusion ............................................................................................. 92

Overview ......................................................................................................................................... 92

Findings and conclusion .................................................................................................................. 92

Supporting Information ........................................................................................... 94

Overview ......................................................................................................................................... 94

Managing and monitoring EMC Celerra .......................................................................................... 94


6/95

Chapter 1: About this document


6


Introduction tounified storage

This Proven Solution Guide summarizes a series of best practices that EMCdiscovered, validated, or otherwise encountered during the validation of a solution forusing an EMC

Celerra

NS-960 unified storage platform, EMC CLARiiON

CX4-960

built-in as a back-end storage array, and Oracle Database 11g on Linux using Oracle

Direct Network File System (DNFS) or Oracle Automatic Storage Management(ASM).

EMC's commitment to consistently maintain and improve quality is led by the TotalCustomer Experience (TCE) program, which is driven by Six Sigma methodologies.As a result, EMC has built Customer Integration Labs in its Global Solutions Centersto reflect real-world deployments in which TCE use cases are developed andexecuted. These use cases provide EMC with an insight into the challenges currentlyfacing its customers.

Use case

definition

A use case reflects a defined set of tests that validates the reference architecture for

a customer environment. This validated architecture can then be used as a referencepoint for a Proven Solution.

Contents The content of this chapter includes the following topics.

Topic See Page

Audience and purpose 7

Scope 8

Business challenge 8

Technology solution 9

Objectives 10

Reference Architecture 11

Validated environment profile 13

Hardware and software resources 13

Unified storage platform environment 15

Prerequisites and supporting documentation 17

Terminology 18


7/95



7

Audience and purpose

Audience The intended audience for the Proven Solution Guide is:

Internal EMC personnel

EMC partners

Customers

Purpose This purpose of this proven solution is to detail the use of two storage networkingtechnologies with Oracle: Oracle Automatic Storage Management (ASM) over FibreChannel (FC) and Oracle Direct NFS (DNFS) over Internet Protocol (IP). EMC fieldpersonnel and account teams can use this as a guide for designing the storage layerfor Oracle environments. (Oracle DNFS is an implementation of Oracle where theNFS client is embedded in the Oracle kernel. This makes the NFS implementationOS agnostic, and it is specifically tuned for Oracle database workloads.)

The purpose of this Proven Solution Guide is to highlight the functionality,performance, and scalability of DNFS and ASM in the context of an onlinetransaction processing (OLTP) workload. When testing both storage technologies,the EMC Celerra NS-960 was used for storage.

In the case of ASM, the database servers were connected directly to the host-side FC ports on the CX4-960, which is the back end to the NS-960.

In the case of DNFS, the database servers were connected to the NS-960sData Movers using a 10 GbE storage network. Oracle RAC 11gon Linux forx86-64 was used for the database environment.


8/95



8

Scope

Scope The use cases tested in the context of an OLTP workload are:

Oracle ASM over FC

Oracle DNFS over IP

The scope of this guide is limited to the performance, functionality, and scalability ofthese two storage technologies in the context of Oracle RAC 11gfor Linux onx86-64. The cluster used for testing these use cases consisted of four nodes, eachcontaining 16 cores, with 128 GB of RAM. A 10 GbE storage network was used forthe IP storage use case, and FC was used for ASM.

A reasonable amount of storage and database tuning was performed in order toachieve the results documented in this guide. However, undocumented parameterswere not used. The goal of this testing was to establish the real-world performancethat could be expected in a production customer environment. For this reason, thestorage was designed to support robustness and reliability, at the cost of

performance. This is consistent with the use of Oracle in a production, fault-tolerantcontext.

Not in scope The testing performed in these use cases did not include:

Backup and recovery

Disaster recovery

Remote replication

Test/dev cloning

EMC has previously documented these use cases on Celerra and CLARiiONplatforms. Refer to the documents listed in the Prerequisites and supportingdocumentationsection.

Business challenge

Overview Customers face multiple challenges in maintaining and scaling up their storagesystems including balancing cost with performance and manageability.

Traditionally, mission-critical Oracle database applications have been positioned to

run on block-based storage over FC-connected SAN. However, SAN may not be thebest choice for every customer. Customers have different skills, infrastructures, andbudgetary constraints. These factors affect the choice of protocol used in theirOracle environments. In many cases, customers may choose multiple protocols, andthis is supported very well by the EMC unified storage architecture.

The Celerra NS-960 unified storage platform offers total flexibility in terms ofprotocols, network connection types, and applications. This solution demonstrates acouple of alternatives that give customers the same type of performance with


9/95



9

different storage options:

DNFS on IP networks provides a low-cost, high-performing, and scalablesolution for customers.

ASM on FC provides a high-performing and scalable solution for customers.

Technology solution

Overview This solution demonstrates how organizations can:

Use Celerra NS-960 with DNFS to:

Simplify network setup and management by taking advantage of DNFSautomated management of tasks, such as IP port trunking, and tuning ofLinux NFS parameters

Increase the capacity and throughput of their existing infrastructure through

the use of 10 GbE networking Use CLARiiON CX4-960 with ASM over FC to:

Use FC as the protocol for storage connectivity with the Oracle ASM filesystem

This solution will:

Use a Celerra NS-960 and a high-speed 10 GbE network to chart the limits ofperformance and user scalability in an Oracle RAC 11gDNFS OLTPenvironment

Demonstrate that network-attached storage is competitive on cost andperformance as compared to a traditional storage infrastructure

Use a CLARiiON CX4-960 high-speed 4 Gb/s FC fabric to chart the limits ofperformance and user scalability in an Oracle RAC 11gASM OLTPenvironment

Demonstrate that storage area networking is competitive on performanceand may be the best choice for customers who prefer that storage networktype


10/95



10

Objectives

Objectives EMCs solution includes the objectives outlined inTable 1.

Table 1. Solution objectives

Objective Details

Performance Demonstrate the baseline performance of the CelerraNS-960 running over NFS with Oracle RAC 11gR2DNFS on a 10 GbE network.

Demonstrate the baseline performance of the CelerraNS-960 running over FC with an Oracle RAC 11gASMenvironment.

Scale the workload and show the database performanceachievable on the array over NFS and over ASM.


11/95



11

Reference Architecture

CorrespondingReferenceArchitecture

This use case has a corresponding Reference Architecture document that isavailable on Powerlinkand EMC.com. Refer to EMC Unified Storage for OracleDatabase 11g - Performance Enabled by EMC Celerra Using DNFS or ASMfor

details.

If you do not have access to this content, contact your EMC representative.

ReferenceArchitecturediagram forDNFS

Figure 1 depicts the solutions overall physical architecture for the DNFS over IPimplementation.

Figure 1. Physical architecture for the DNFS over IP implementation

This implementation uses Celerra NS-960 and a 10 GbE network fabric to build aphysical four-node Oracle Real Application Cluster (RAC) 11gR2 DNFS OLTPenvironment.


12/95



12

Referencearchitecturediagram forASM

Figure 2 depicts the solutions overall physical architecture for the ASM over FCimplementation.

Figure 2. Physical architecture for the ASM over FC implementation


13/95



13

Validated environment profile

Profilecharacteristics

EMC used the environment profile defined inTable 2 to validate this solution.

Table 2. Profile characteristics

Profile characteristic Value

Database characteristic OLTP

Benchmark profile Quest Benchmark FactoryTPC-C-like benchmark

Response time < 2 seconds

Read/write ratio 70/30

Database scale The Quest Benchmark scale is 11,500, which keeps thesystem running within agreed performance limits.

Size of databases 1 TBNumber of databases 1

Array drives: size and speed FC: 300 GB; 15k rpm

SATA: 1 TB; 7,200 rpm

Hardware and software resources

Hardware Table 3 lists the hardware used to validate this solution.

Table 3. Solution hardware

Equipment Quantity Configuration

EMC Celerra NS-960 unified storageplatform

(included an EMC CLARiiON CX4-960back-end storage array)

1 2 storage processors

3 Data Movers

1 Control Station

4 x 10 GbE network connections per DataMover

7 FC shelves

2 SATA shelves

105 x 300 GB 15k FC disks30 x 1 TB SATA disks

10 Gigabit Ethernet switches

(Brocade 8000)

2 24 CEE ports (10 Gb/s)

8 FC ports (8 Gb/s)

FC switches

(QLogic SANbox 5602)

2 16 ports (4 Gb/s)


14/95



14

Equipment Quantity Configuration

Database servers

(Oracle RAC 11g servers)

(Fujitsu PRIMERGY RX600 S4)

4 4 x 3 GHz Intel Nahalem quad-coreprocessors

128 GB of RAM

2 x 146 GB 15k internal SCSI disks2 onboard GbE Ethernet NICs

2 additional CNA cards

2 additional 8 Gb host bus adapters (HBAs)

Software Table 4 lists the software that EMC used to validate this solution.

Table 4. Solution software

Software Version

Oracle Enterprise Linux 5.5

VMware vSphere 4.0

Microsoft Windows Server 2003 Standard Edition 2003

Oracle RAC 11gEnterprise Edition 11.2.0.1

Quest Benchmark Factory for Databases 5.8.1

EMC Celerra Manager Advanced Edition 5.6

EMC NavisphereAgent 6.29.5.0.37

EMC FLARE 04.29.000.5.006 (OS for CLARiiON)

EMC DART 5.6.49-3 (OS for Celerra NAS head)

EMC Navisphere Management Suite 6.29


15/95



15

Unified storage platform environment

Introduction tothe unifiedstorage

platformenvironment

This solution tested a unified storage platform with two different environments.

DNFS environment

With DNFS over IP, all database objects are accessed using the Celerra Data Moverand accessible through an NFS mount. Datafiles, tempfiles, control files, online redologfiles, and archived log files are accessed using DNFS over the IP protocol.

ASM environment

With ASM over FC, all database objects, including datafiles, tempfiles, control files,online redo logfiles, and archived log files, are stored on ASM disk groups that resideon SAN storage.

Solutionenvironment

The solution environment consists of:

A Celerra NS-960 unified storage array, which includes a CLARiiON CX4-960back-end storage array

In the case of DNFS, the storage array is connected to the Oracle RAC 11gservers through an IP production storage network.

In the case of ASM, the back-end storage array is directly connected to theOracle RAC 11gservers through an FC production storage network.

A four-node Oracle RAC 11gcluster

Storage layout To test the unified storage platform solution with different protocols and different diskdrives, the database was built in two different configurations. The back-end storagelayout is the same for both except for the file-system type.

Table 5 shows the storage layouts for both environments.

The DNFS implementation has a RAID-protected NFS file system.

The ASM implementation has a RAID-protected ASM disk group.

Table 5. Storage layouts

What Where

Oracle datafiles and tempfiles

FC disk

Oracle online redo logfiles

Oracle control files

Voting disk

OCR files

Archived logfiles SATA II


16/95



16

Fast recovery area (FRA)

Backup target

For the DNFS environment, all files are accessed using DNFS.

RAID-protected NFS file systems are designed to satisfy the I/O demands ofparticular database objects. For example, RAID 5 is sometimes used for thedatafiles and tempfiles, but RAID 1 is always used for the online redo logfiles.

For more information, refer to:EMC Celerra NS-960 Specification Sheet

Oracle datafiles and online redo logfiles reside on their own NFS file system.Online redo logfiles are mirrored across two different file systems usingOracle software multiplexing. Three NFS file systems are used - one filesystem for datafiles and tempfiles, and two file systems for online redologfiles.

Oracle control files are mirrored across the online redo logfile NFS filesystems.

Networkarchitecture

The design implements the following physical connections:

4 Gb/s FC SAN connectivity for the ASM implementation

10 GbE network connectivity for the DNFS implementation

10 GbE network VLAN for the RAC interconnect

1 GbE network connectivity for the client network

DNFS provides file system semantics for Oracle RAC 11gon NFS over IP

ASM provides file system semantics for Oracle RAC 11gon SAN over FCThe RAC interconnect and storage networks are 10 GbE. Jumbo frames are enabledon these networks.
http://www.emc.com/collateral/hardware/specification-sheet/h6035-celerra-ns960-ss.pdfhttp://www.emc.com/collateral/hardware/specification-sheet/h6035-celerra-ns960-ss.pdfhttp://www.emc.com/collateral/hardware/specification-sheet/h6035-celerra-ns960-ss.pdfhttp://www.emc.com/collateral/hardware/specification-sheet/h6035-celerra-ns960-ss.pdf


17/95



17

Prerequisites and supporting documentation

Technology This paper assumes that you have a general knowledge of:

EMC Celerra unified storage platform

EMC CLARiiON

Oracle Database

Oracle Enterprise Linux

Supportingdocuments

The following documents, located on Powerlink or EMC.com, provide additional,relevant information. Access to documents on Powerlink depends upon your logincredentials. If you do not have access to the following content, contact your EMCrepresentative.

EMC Celerra NS-960 System Installation Guide

EMC CLARiiON CX4 Model 960 (CX4-960) Storage System Setup Guide

EMC Celerra Network Server Command Reference Manual

EMC CLARiiON Best Practices for Performance and Availability: Release 30Firmware Update

EMC PowerPath 5.1 Product Guide

EMC PowerPath for Linux Installation and Administration Guide

EMC Backup and Recovery for Oracle Database 11gSAN PerformanceEnabled by EMC CLARiiON CX4-120 Using the Fibre Channel Protocol andOracle Automatic Storage Management (ASM)A Detailed Review

EMC Backup and Recovery for Oracle Database 11g without Hot BackupMode using DNFS and Automatic Storage Management on Fibre ChannelADetailed Review

EMC Unified Storage for Oracle Database 11gEnabled by EMC CLARiiONand EMC Celerra Using FCP and NFS Reference Architecture

EMC Business Continuity for Oracle Database 11gEnabled by EMC CelerraUsing DNFS and NFS Reference Architecture

Third-partydocuments

The following documents are available on the Oracle website:

Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux

Oracle Real Application Clusters Installation Guide 11g Release 2 (11.2) forLinux and UNIX

Large SGA on Linux
http://download.oracle.com/docs/cd/E11882_01/install.112/e17212/toc.htmhttp://download.oracle.com/docs/cd/E11882_01/install.112/e17214/toc.htmhttp://download.oracle.com/docs/cd/E11882_01/install.112/e17214/toc.htmhttp://www.oracle-base.com/articles/linux/LargeSGAOnLinux.phphttp://www.oracle-base.com/articles/linux/LargeSGAOnLinux.phphttp://download.oracle.com/docs/cd/E11882_01/install.112/e17214/toc.htmhttp://download.oracle.com/docs/cd/E11882_01/install.112/e17214/toc.htmhttp://download.oracle.com/docs/cd/E11882_01/install.112/e17212/toc.htm


18/95



18

Terminology

Terms anddefinitions

Table 6 defines the terms used in this document.

Table 6. Terminology

Term Definition

Automatic StorageManagement (ASM)

Oracle ASM is a volume manager and a file system for Oracle Databasefiles. It supports single-instance Oracle Database and Oracle RealApplication Clusters (Oracle RAC) configurations. ASM uses block-levelstorage.

Direct NFS (DNFS) A feature of the Oracle Database 11gsoftware stack in which theNetwork File Storage (NFS) client protocol is embedded in the Oracle11g database kernel. The NFS implementation is OS agnostic and tunedfor database I/O patterns.

Fast recovery area (FRA) Fast recovery area (formerly called flash recovery area) is a specific

area of disk storage that is set aside for all recovery-related files andactivities in an Oracle database. In this solution, EMC put the databasebackups, which are used to refresh the database environment afterseveral test runs, into the FRA.

Kernel NFS (KNFS) A standard feature of all Linux and UNIX operating systems in which theNetwork File Storage (NFS) client protocol is embedded in the operatingsystem kernel.

Online transactionprocessing (OLTP)

A type of processing in which a computer responds to requests. Eachrequest is called a transaction.

Oracle Real ApplicationCluster (RAC)

Oracle RAC allows multiple concurrent instances to share a singlephysical database.

Scale-up OLTP Using an industry-standard OLTP benchmark against a single databaseinstance, comprehensive performance testing is performed to validate themaximum achievable performance using the solution stack of hardwareand software.

Serial AdvancedTechnology Attachment(SATA) drive

SATA is a standard for connecting hard drives into computer systems.SATA is based on serial-signaling technology.


19/95

Chapter 2: Storage Design


19


Overview

Introduction tostorage design

The environment consists of a four-node Oracle RAC 11gcluster that accesses asingle production database. The four RAC nodes communicate with each otherthrough a dedicated private network that includes a Brocade 8000 FCoE switch. Thiscluster interconnection synchronizes cache across various database instances

between user requests. An FC SAN is provided by two QLogic SANbox 5602

switches.

For the SAN configuration, EMC PowerPathis used in this solution and works withthe storage system to manage I/O paths. For each server, PowerPath manages fouractive I/O paths to each device and four passive I/O paths to each device.

Contents This chapter contains the following topics:

Topic See Page

Concepts 20

Storage setup 20

Best practices 21

Data Mover parameters setup 23

Data Mover failover 24

RAID group layout 25


20/95



20

Concepts

High

availability andfailover

EMC Celerra has built-in high-availability (HA) features. These HA features allow theCelerra to survive various failures without a loss of access to the Oracle database.These HA features protect against the following:

Data Mover failure

Network port failure

Power loss affecting a single circuit connected to the storage array

Storage processor failure

FC switch failure

Disk failure

Storage setup

Setting upCLARiiON (CX)storage

To set up CLARiiON (CX) storage, the steps in the following table must be carriedout:

Step Action

1 Configure zoning.

2

Configure RAID groups and bind LUNs.

3 Allocate hot spares.

4 Create storage groups.

5

Discover FCP LUNs from the database servers.

Setting up NASstorage

To set up NAS storage, the steps in the following table must be carried out:

Step Action

1

Create RAID groups.

2

Allocate hot spares.

3

Create user-defined pools.

4

Create file systems and file system NFS exports.


21/95



21

Best practices

Disk drives The following are the general recommendations for disk drives:

Because of significantly better performance, FC drives are always

recommended for storing datafiles, tempfiles, control files, and online redo logfiles.

Serial Advanced Technology-Attached (SATA II) drives have slower responseand rotational speed, and moderate performance with random I/O. However,they are less expensive than the FC drives for the same or similar capacity.

SATA II drives SATA II drives are frequently the best option for storing archived redo logs and thefast recovery area. In the event of high-performance requirements for backup andrecovery, FC drives can also be used for this purpose.

RAID types andfile types

Table 7 describes the recommendations for RAID types corresponding toOracle file types.

Table 7. Recommended RAID types

1 The use of FC disks for archived logs is relatively rare. However, if many archivedlogs are being created, and the I/O requirements for archived logs exceed a

reasonable number of SATA II disks, this may be a more cost-effective solution.

Description RAID 10/FC RAID 5/FC RAID 5/SATA II

Datafiles/tempfiles Recommended Recommended Avoid

Control files Recommended Recommended Avoid

Online redo logs Recommended Avoid Avoid

Archived logs Possible (applytuning) 1

Possible (applytuning) 1

Recommended

Fast recovery area OK OK Recommended

OCR file/voting disk OK OK Avoid


22/95



22

Tempfiles,undo, andsequential tableor index scans

In some cases, if an application creates a large amount of temp activity, placing yourtempfiles on RAID 10 devices may be faster due to RAID 10s superior sequentialI/O performance. This is also true for undo. Further, an application that performsmany full table scans or index scans may benefit from these datafiles being placedon separate RAID 10 devices.

Online redologfiles

Online redo log files should be put on RAID 1 or RAID 10 devices. You should notuse RAID 5 because sequential write performance of distributed parity (RAID 5) isnot as high as that of mirroring (RAID 1).

RAID 1 or RAID 10 provides the best data protection; protection of online redo logfiles is critical for Oracle recoverability.

OCR files andvoting disk files

You should use FC disks for OCR files and voting disk files; unavailability of thesefiles for any significant period of time (due to disk I/O performance issues) may

cause one or more of the RAC nodes to reboot and fence itself off from the cluster.

The LUN/RAID group layout images in the RAID group layoutsection, show twodifferent storage configurations that can be used for Oracle RAC 11gdatabases on aCelerra. That section can help you to determine the best configuration to meet yourperformance needs.

Stripe size forCelerra storagepool

EMC recommends a stripe size of 32 KB for all types of database workloads.

The default stripe size for all file systems on FC shelves (redo logs and data) shouldbe 32 KB. Similarly, the recommended stripe size for the file systems on SATA II

shelves (archive and flash) should be 256 KB.

Shelfconfiguration

The most common error when planning storage is designing for storage capacityrather than for performance. The single most important storage parameter forperformance is disk latency. High disk latency is synonymous with slowerperformance; low disk counts lead to increased disk latency.

The recommendation is a configuration that produces average database I/O latency(the Oracle measurement db file sequential read) of less than or equal to 20 ms. Intodays disk technology, the increase in storage capacity of a disk drive hasoutpaced the increase in performance. Therefore, performance must be the standard

rather than storage capacitywhen planning an Oracle databases storageconfiguration, not disk storage capacity.

The number of disks that should be used is determined first by the I/O requirements,then by capacity. This is especially true for datafiles and tempfiles. Consult with yourEMC sales representative for specific sizing recommendations for your workload.


23/95



23

Data Mover parameters setup

Noprefetch EMC recommends that you turn off file-system read prefetching for an OLTPworkload. Leave it on for a Decision Support System (DSS) workload.

Prefetch will waste I/Os in an OLTP environment, since few, if any, sequential I/Osare performed. In a DSS, setting the opposite is true.

To turn off the read prefetch mechanism for a file system, type:

$ server_mount -option ,noprefetch

For example:

$ server_mount server_3 option rw,noprefetch ufs1 /ufs1

NFS threadcount

EMC recommends that you use the default NFS thread count of 256 for optimalperformance.

Do not set this to a value lower than 32 or to a value higher than 512.

For more information about these parameter, see the Celerra Network ServerParameters Guide on Powerlink. If you do not have access to this content, contactyour EMC representative.

file.asyncthreshold

EMC recommends that you use the default value of 32 for the parameterfile.asyncthreshold. This provides optimum performance for databases.

For more information about these parameter, see the Celerra Network ServerParameters Guide on Powerlink. If you do not have access to this content, contactyour EMC representative.


24/95



24

Data Mover failover

Highavailability

The Data Mover failover capability is a key feature unique to the Celerra. Thisfeature offers redundancy at the file-server level, allowing continuous data access. Italso helps to build a fault-resilient RAC architecture.

Configuringfailover

EMC recommends that you set up an auto-policy for the Data Mover, so that if aData Mover fails, either due to hardware or software failure, the Control Stationimmediately fails the Data Mover over to its partner. The standby Data Moverassumes the faulted Data Movers identities:

Network identity: IP and MAC addresses of all its network interface cards(NICs)

Storage identity: File systems controlled by the faulted Data Mover

Service identity: Shares and exports controlled by the faulted Data Mover

This ensures continuous file sharing transparently for the database without requiringusers to unmount and remount the file system. The NFS applications and NFSclients do not see any significant interruption in I/O.

Pre-conditionsfor failover

Data Mover failover occurs if any of these conditions exists:

Failure (operation below the configured threshold) of both internal networkinterfaces by the lack of a heartbeat (Data Mover timeout)

Power failure within the Data Mover (unlikely, as the Data Mover is typicallywired into the same power supply as the entire array)

Software panic due to exception or memory error

Data Mover hang

Events that donot causefailover

Data Mover failover does not occur under these conditions:

Removing a Data Mover from its slot

Manually rebooting a Data Mover

Manual failover Because manual rebooting of Data Mover does not initiate a failover, EMCrecommends that you initiate a manual failover before taking down a Data Mover formaintenance.


25/95



25

RAID group layout

RAID grouplayout design

Two sets of RAID and disk configurations have been tested. These are described inthe following sections.

For this solution, the FC disks are designed to hold all other database files, forexample, datafiles, control files, online redo log files. As per EMC best practices,online redo log files are put on RAID 10, while datafiles and control files are put onRAID 5.

The SATA disks are only designed to hold the archive logs and backup files.Therefore, there is no impact on the performance and scalability testing performedby EMC.

For a customers production environment, EMC recommends RAID 6 instead ofRAID 5 for archive logs and backup files when using SATA. RAID 6 provides extraredundancy; its performance is almost the same as RAID 5 for read, but is slower forwrite.

For more information about RAID 6, see EMC CLARiiON RAID 6 TechnologyADetailed Review.


26/95



26

RAID grouplayout for ASM

The RAID group layout for seven-FC shelf RAID 5/RAID 1 and two-SATA II RAID 5using user-defined storage pools is shown inFigure 3.

Figure 3. RAID group layout for ASM


27/95



27

RAID grouplayout for NFS

The RAID group layout for seven-FC shelf RAID 5/RAID 1 and two-SATA II RAID 5using user-defined storage pools is shown inFigure 4.

Figure 4. RAID group layout for NFS


28/95

Chapter 3: File System


28


Overview

Introduction tothe file system

For NFS, in addition to the configuration on the CLARiiON, a few configuration stepsare necessary for the Celerra, which provides the file system path to the hosts.Seven file systems need to be created to hold database files, online redo logs,archive logs, backup files, and CRS files. For database files, two file systems arecreated to use two Data Movers for better performance. For online redo log files, twofile systems are created to use two Data Movers for multiplexing and betterperformance.

For ASM, no additional configuration on Celerra is required.

There are many different ways of organizing software and services in the file systemthat all make sense. However, standardization on a workable single layout across allservices has more advantages than picking a layout that is well suited for a particularapplication.


Topic See Page

Limitations 29

File system layout 30


29/95



29

Limitations

Limitationswith AutomaticVolume

Management

An Automatic Volume Management (AVM) system-defined pool can provide amaximum of eight stripes, which is only 40 disks.

For example, there are 40 disk volumes on the storage system. AVM takes eightdisk volumes, creates stripe1, slice1, metavolume1, then creates the file systemufs1. Therefore, if you have more than eight RAID groups, you should createthe volume on those RAID groups manually.

In this solution, the database datafiles are spread over 80 disks, which farexceed what the disks AVM system-defined pool can support. To resolve thislimitation, EMC did not use the system-defined pool but created the storagepool manually, which allows for a wide striping AVM pool. Two file systems arecreated on this wide stripe by using different Data Movers, and the files arespread across 80 disks instead of 40.


30/95



30

File system layout

File systemlayouts for NFS

The file systems shown inTable 8 were created on MVM user-defined pools,exported on the Celerra, and mounted on the database servers.

ASM does not utilize a file system.

Table 8. File system layouts for NFS

File system/export AVM user-defined pool Volumes

/crs

log1pool(user-defined storage pool createdusing log1stripe volume)

/crsfs

/datafs1

datapool(user-defined storage pool createdusing datastripe volume)

data1stripe(metavolume consisting of allavailable FC 4+1 RAID 5groups)

/datafs2 datapool(user-defined storage pool createdusing datastripe volume)

data2stripe(metavolume consisting of allavailable FC 4+1 RAID 5groups)

/log1fs


log1stripe(metavolume using oneRAID 10 group)

/log2fs


log2stripe(metavolume using oneRAID 10 group)

/archfs

archpool

(user-defined storage pool createdusing archstripe volume)

archstripe

(metavolume using the SATA6+1 RAID 5 group)1

/frafs

frapool(user-defined storage pool createdusing frastripe volume)

frastripe(metavolume using the SATA6+1 RAID 5 group)1

EMC strongly recommends using RAID 6 with high-capacity SATA drives. High capacity is 1 TB or greater incapacity.


31/95

Chapter 4: Oracle Database Design


31


Overview

Introduction toOracledatabasedesign

This chapter provides guidelines on the Oracle 11gRAC database design used forthis validated solution. The design and configuration instructions apply to the specificrevision levels of components used during the development of the solution.

Before attempting to implement any real-world solution based on this validatedscenario, you must gather the appropriate configuration documentation for therevision levels of the hardware and software components. Version-specific releasenotes are especially important.

Contents This section contains the following topics:

Topic See Page

Considerations 32

Database file layout 33

Oracle ASM 34

Oracle 11gDNFS 35

Memory configuration for Oracle 11g 38

HugePages 39


32/95



32

Considerations

Heartbeatmechanisms

The synchronization services component (CSS) of Oracle Clusterware maintains twoheartbeat mechanisms:

The disk heartbeat to the voting disk The network heartbeat across the RAC interconnects that establishes and

confirms valid node membership in the cluster

Both of these heartbeat mechanisms have an associated time-out value. For moreinformation on Oracle Clusterware misscountand disktimeoutparameters, seeOracle MetaLink Note 294430.1.

EMC recommends setting the disk heartbeat parameter disktimeout to160 seconds. You should leave the network heartbeat parametermisscount at the default of 60 seconds.

Rationale

These settings will ensure that the RAC nodes do not evict when the active DataMover fails over to its partner.

The command to configure this option is:

$ORA_CRS_HOME/bin/crsctl set css disktimeout 160

Note In Oracle RAC 11gR2, the default value of disktimeouthas been set to200. Therefore, there is no need to manually change the value to 160. Youmust check the current value of the parameter before you make anychanges by executing the following command:

$GRID_HOME/bin/crsctl get css disktimeout

Oracle ClusterReady Services

Oracle Cluster Ready Services (CRS) are enabled on each of the Oracle RAC 11gservers. The servers operate in active/active mode to provide local protection againsta server failure and to provide load balancing.

Provided that the required mount-point parameters are used, CRS-required files(including the voting disk and the OCR file) can reside on NFS volumes.

For more information on the mount-point parameters required for the OracleClusterware files, see Chapter 6: Installation and Configuration > Task 4: ConfigureNFS client options.

NFS client In the case of the Oracle RAC 11gdatabase, the embedded Oracle DNFS protocolis used to connect to the Celerra storage array. DNFS runs over TCP/IP.

Oracle binaryfiles

The Oracle RAC 11g binary files, including the Oracle CRS, are installed on thedatabase servers' local disks.
https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=294430.1https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=294430.1


33/95



33

Database file layout

Database filelayout for NFS

Datafiles, online redo log files, archive log files, tempfiles, control files, and CRS filesreside on Celerra NFS file systems. These file systems are designed (in terms of theRAID level and number of disks used) to be appropriate for each type of file.

Table 9 lists each file or activity type and indicates where it resides.

Table 9. Location of files and activities for NFS

Content Location

Database binary files Database servers local disk

(or vmdk file for virtualized servers)

Datafiles, tempfiles Spread across /datafs1 and /datafs2

Online redo log files, control files Multiplexed across /log1fs and /log2fs

Archived log files /archfs

Fast recovery area (FRA) /frafs

OCR and voting disk files /crs

Database filelayout for ASM

Datafiles, online redo log files, archive log files, tempfiles, control files, and CRS filesreside on CLARiiON storage that is managed by Oracle ASM. The database wasbuilt with six distinct ASM disk groups: +DATA, +LOG1, +LOG2, +ARCH, +FRA, and+CRS.

Table 10 lists each file or activity type and indicates where it resides.

Table 10. Location of files and activities for ASM

Content Location

Database binary files Database servers local disk

(or vmdk file for virtualized servers)

Datafiles, tempfiles +DATA

Online redo log files, control files Multiplexed across +LOG1, +LOG2

Archived log files +ARCH

Fast recovery area (FRA) +FRA

OCR and voting disk files +CRS


34/95



34

Oracle ASM

ASMLib Oracle has developed a storage management interface called the ASMLib API.ASMLib is not required to run ASM; it is an add-on module that simplifies themanagement and discovery of ASM disks. The ASMLib provides an alternative, to

the standard operating system interface, for ASM to identify and access blockdevices.

The ASMLib API provides two major feature enhancements over standard interfaces:

Disk discovery Providing more information about the storage attributes tothe database and the database administrator (DBA)

I/O processing To enable more efficient I/O

Best practicesfor ASM and

databasedeployment

ASM provides out-of-the-box enablement of redundancy and optimal performance.However, the following items should be considered to increase either performance or

availability, or both: Implement multiple access paths to the storage array using two or more HBAs

or initiators.

Deploy multipathing software over these multiple HBAs to provide I/O load-balancing and failover capabilities.

Use disk groups with similarly sized and performing disks. A disk groupcontaining a large number of disks provides a wide distribution of dataextents, thus allowing greater concurrency for I/O, and reduces theoccurrence of hotspots. Since a large disk group can easily sustain variousI/O characteristics and workloads, a single (database area) disk group can beused to house database files, logfiles, and controlfiles.

Use disk groups with four or more disks and ensure these disks span severalback-end disk adapters.

For example, a common deployment can be four or more disks in a database diskgroup (for example, DATA disk group) spanning all back-end disk adapters/directors,and eight to ten disks for the FRA disk group. The size of the FRA area will dependon what is stored and how much, that is, full database backups, incrementalbackups, flashback database logs, and archive logs.

Note An active copy of the controlfile and one member of each of the redo loggroups are stored in the FRA.


35/95



35

Oracle 11gDNFS

Overview ofOracle DirectNFS

Oracle 11gincludes a feature for storing Oracle datafiles on a NAS device, referredto as Direct NFS or DNFS. DNFS integrates the NFS client directly inside thedatabase kernel instead of the operating system kernel.

As part of this solution, the storage elements for Oracle RAC 11gwere accessedusing the DNFS protocol. It is relatively easy to configure DNFS. It applies only to thestorage of Oracle datafiles. Redo log files, tempfiles, control files, and so on are notaffected. You can attempt to configure the mount points where these files are storedto support DNFS, but this will have no impact.

DNFS provides performance advantages over conventional Linux kernel NFS (orKNFS) because fewer context switches are required to perform an I/O. BecauseDNFS integrates the client NFS protocol into the Oracle kernel, this allows all I/Ocalls to be made in user space, rather than requiring a context switch to kernelspace. As a result, CPU utilization associated with the I/O of the database server isreduced.

Disadvantages of KNFS

I/O caching and performance characteristics vary between operating systems. Thisleads to varying NFS performance across different operating systems (for example,Linux and Solaris), and across different releases of the same operating system (forexample, RHEL 4.7 and RHEL 5.4). This in turn results in varying NFS performanceacross implementations.

DNFS and EMCCelerra

Using the DNFS configuration, the Oracle RAC 11g and EMC Celerra solutionenables you to deploy an EMC NAS architecture with DNFS connectivity for itsOracle RAC 11g database applications that have lower cost and reduced

complexity than direct-attached storage (DAS) or storage area network (SAN).

Figure 5 illustrates how DNFS can be used to deploy an Oracle 11g andEMC Celerra solution.

Figure 5. DNFS and Celerra unified storage


36/95



36

DNFSperformanceadvantage

Table 11 describes the performance advantages that can be gained by usingDNFS.

Table 11. DNFS performance advantages

Advantage

Details

Consistent performance

Consistent NFS performance is observed across alloperating systems.

Improved caching andI/O management

The DNFS kernel is designed for improved cachingand management of the I/O patterns that are typicallyexperienced in database environments, that is, largerand more efficient reads/writes.

Asynchronous direct I/O

The DNFS kernel enables asynchronous direct I/O,which is typically the most efficient form of I/O fordatabases. Asynchronous direct I/O significantlyimproves database read/write performance by

enabling I/O to continue while other requests arebeing submitted and processed.

Overcomes OS writelocking

DNFS overcomes OS write locking, which can beinadequate in some operating systems and can causeI/O performance bottlenecks in others.

Reduced CPU andmemory usage

Database server CPU and memory usage arereduced by eliminating the overhead of copying datato and from the OS memory cache to the databaseSGA cache.

Included in 11g DNFS is included free of charge with the Oracle 11g.

Enhanced dataintegrity

To ensure database integrity, immediate writes must be made to the database whenrequested. Operating system caching delays writes for efficiency reasons; thispotentially compromises data integrity during failure scenarios.

DNFS uses database caching techniques and asynchronous direct I/O to ensurealmost immediate data writes, thus reducing data integrity risks.

Load balancingand highavailability

Load balancing and high availability (HA) are managed internally within the DNFSclient itself, rather than at the OS level. This greatly simplifies network setups in HAenvironments and reduces dependence on IT network administrators by eliminating

the need to set up network subnets and bond ports, for example, LACP bonding.

DNFS allows multiple parallel network paths/ports to be used for I/O between thedatabase server and the IP storage array. For each node, two paths were used inthe testing performed for this solution. For efficiency and performance, these pathsare managed and load balanced by the DNFS client, not by the operating system.The four paths should be configured in separate subnets for effective load balancingby DNFS.


37/95



37

Less tuningrequired

Oracle 11gDNFS requires little additional tuning, other than the tuningconsiderations necessary in any IP storage environment with Oracle. In anunchanging environment, once tuned, DNFS requires no ongoing maintenance.


38/95



38

Memory configuration for Oracle 11g

Memoryconfigurationand

performance

Memory configuration in Oracle 11g is one of the most challenging aspects ofconfiguring the database server.

If the memory is not configured correctly, the performance of the database server willbe very poor, and:

The database server will be unstable.

The database may not open at all; if it does open, you may experience errorsdue to lack of shared pool space.

In an OLTP context, the size of the shared pool is frequently the limitation onperformance of the database.

For more information, see Effects of Automatic Memory Management onperformance.

AutomaticMemoryManagement

A feature called Automatic Memory Management was introduced in Oracle 11g64bit (Release 1). The purpose of Automatic Memory Management is to simplify thememory configuration process for Oracle 11g.

For example, in Oracle 10g, the user is required to set two parameters,SGA_TARGET and PGA_AGGREGATE_TARGET, so that Oracle can manageother memory-related configurations such as buffer cache and shared pool. Whenusing Oracle 11g-style Automatic Memory Management, the user does not set theseSGA and PGA parameters. Instead, the following parameters are set:

MEMORY_TARGET

MEMORY_MAX_TARGET

Once these parameters are set, Oracle 11gcan, in theory, handle all memorymanagement issues, including both SGA and PGA memory. However, the AutomaticMemory Management model in Oracle 11g64 bit (Release 1) requires configurationof shared memory as a file system mounted under /dev/shm. This adds an additionalmanagement burden to the DBA/system administrator.

Effects ofAutomaticMemoryManagementon performance

Decreased database performance

EMC observed a significant decrease in performance when the Oracle 11gAutomatic Memory Management feature was enabled.

Linux HugePages are not supportedLinux HugePages are not supported when the Automatic Memory Managementfeature is implemented. When Automatic Memory Management is enabled, the entireSGA memory should fit under /dev/shm and, as a result, HugePages are not used.For more information, seeOracle MetaLink Note 749851.1.

On Oracle 11g, tuning HugePages increases the performance of the databasesignificantly. It is EMCs opinion that the performance improvements of HugePages,with no requirement for a /dev/shm file system, makes the Oracle 11gautomatic
https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=294430.1https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=294430.1https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=294430.1https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=294430.1


39/95



39

memory model a poor choice.

EMC recommendations

To achieve optimal performance on Oracle 11g, EMC recommends the following:

Disable the Automatic Memory Management feature

Use the Oracle 10gstyle of memory management on Oracle 11g

The memory management configuration procedure is described in the previoussection. This provides optimal performance and manageability per our testing.

HugePages

HugePages The Linux 2.6 kernel includes a feature called HugePages. This feature allows you tospecify the number of physically contiguous large memory pages that will beallocated and pinned in RAM for shared memory segments like the Oracle System

Global Area (SGA).

The pre-allocated memory pages can only be used for shared memory and must belarge enough to accommodate the entire SGA. HugePages can create a verysignificant performance improvement for Oracle RAC 11gdatabase servers. Theperformance payoff for enabling HugePages is significant.

Warning

HugePages must be tuned carefully and set correctly. Unused HugePages can onlybe used for shared memory allocations - even if the system runs out of memory andstarts swapping. Incorrectly configured HugePages settings may result in poorperformance and may even make the machine unusable.

HugePagesparameters

The HugePages parameters are stored in /etc/sysctl.conf. You can change the valueof HugePages parameters by editing the systctl.conf file and rebooting the instance.

Table 12 describes the HugePages parameters.

Table 12. HugePages parameters

Parameter Description

HugePages_Total

Total number of HugePages that are allocated for sharedmemory segments

(This is a tunable value. You must determine how to set thisvalue.)

HugePages_Free

Number of HugePages that are not being used

HugePagesSize

Size of each HugePage


40/95



40

Optimum values for HugePages parameters

The amount of memory allocated to HugePages must be large enough toaccommodate the entire SGA:

HugePages_Total x HugesPagesSize = Amount of memory allocated toHugePages

To avoid wasting memory resources, the value of HugePages_Free should be zero.

Note The value of vm.nr_hugepages should be set to a value that is at least equalto kernel.shmmax/2048. When the database is started, the HugePages_Freeshould show a value close to zero to reflect that memory is tuned.

For more information on tuning HugePages, see Chapter 6: Installation andconfiguration >Task 7: Configure and tune HugePages.


41/95

Chapter 5: Network Design


41


Overview

Introduction tonetwork design

This chapter focuses on the network design and layout for this solution. It includesthe technology details of SAN and IP network configuration as well as the RACinterconnect network. To maximize the network performance, jumbo frames havebeen enabled on different layers.

Contents This section contains the following topics:

Topic See Page

Concepts 42

Best practices and recommendations 42

SAN network layout 43

IP network layout 43

Virtual LANs 44

Jumbo frames 45

Public and private networks 46

Oracle RAC 11gserver network architecture 47


42/95



42

Concepts

Jumbo frames Maximum Transfer Unit (MTU) sizes of greater than 1,500 bytes are referred to asjumbo frames.

Jumbo frames require Gigabit Ethernet across the entire network infrastructure server, switches, and database servers.

VLAN Virtual local area networks (VLANs) logically group devices that are on differentnetwork segments or sub-networks.

Best practices and recommendations

GigabitEthernet

EMC recommends that you use Gigabit Ethernet for the RAC interconnects if RAC isused. If 10 GbE is available, that is better.

Jumbo framesand the RACinterconnect

For Oracle RAC 11ginstallations, jumbo frames are recommended for the privateRAC interconnect. This boosts the throughput as well as possibly lowers the CPUutilization due to the software overhead of the bonding devices. Jumbo framesincrease the device MTU size to a larger value (typically 9,000 bytes).

VLANs EMC recommends that you use VLANs to segment different types of traffic to

specific subnets. This provides better throughput, manageability, applicationseparation, high availability, and security.


43/95



43

SAN network layout

SAN networklayout for thevalidated

scenario

The SAN network layout is configured as follows:

Two Brocade 8000 switches are used for the test bed.

Two connections from each database server are connected to the Brocade8000 switches.

One FC port from SPA and SPB is connected to each of the two FC switchesat 4 Gb/s.

Zoning Each FC port from the database servers are zoned to both SP ports. According toEMCs best practices, single initiator zoning was used, meaning one HBA/one SPport per zone.

IP network layout

Network designfor thevalidatedscenario

The IP network layout is configured as follows:

TCP/IP provides network connectivity.

DNFS provides file system semantics for Oracle RAC 11g.

Client virtual machines run on a VMwareESXserver. They are connected

to a client network.

Client, RAC interconnect, and redundant TCP/IP storage networks consist ofdedicated network switches and VLANs.

Jumbo frames are enabled on the RAC interconnect and storage networks.

The Oracle RAC 11gservers are connected to the client, RAC interconnect,WAN, and production storage networks.


44/95



44

Virtual LANs

Virtual LANs This solution uses four VLANs to segregate network traffic of different types. Thisimproves throughput, manageability, application separation, high availability, andsecurity.

Table 13 describes the database server network port setup.

Table 13. Database server network port setup

VLAN ID Description CRS setting

1 Client network Public

2 RAC interconnect Private

3 Storage Private

4 Storage Private

Client VLAN The client VLAN supports connectivity between the physically booted OracleRAC 11gservers, the virtualized Oracle Database 11g, and the client workstations.

The client VLAN also supports connectivity between the Celerra and the clientworkstations to provide network file services to the clients. Control and managementof these devices are also provided through the client network.

RACinterconnectVLAN

The RAC interconnect VLAN supports connectivity between the Oracle RAC 11gservers for network I/O required by Oracle CRS. One NIC is configured on eachOracle RAC 11gserver to the RAC interconnect network.

Storage VLAN The storage VLAN uses the NFS protocol to provide connectivity between serversand storage. Each database server connected to the storage VLAN has two NICsdedicated to the storage VLAN. Link aggregation is configured on the servers toprovide load balancing and port failover between the two ports.

For validating DNFS, link aggregation is removed. DNFS was validated using one-,two-, three-, and four-port configurations. Link aggregation is not required on DNFSbecause Oracle 11ginternally manages load balancing and high availability.

Redundantswitches

In addition to VLANs, separate redundant storage switches are used.

The RAC interconnect connections are also on a dedicated switch. For real-worldsolution builds, it is recommended that these switches support GbE connections,jumbo frames, and port channeling.


45/95



45

Jumbo frames

Introduction tojumbo frames

Jumbo frames are configured for three layers:

Celerra Data Mover

Oracle RAC 11gservers

Switch

Note Configuration steps for the switch are not covered here, as that is vendor-specific. Check your switch documentation for details.

Celerra DataMover

To configure jumbo frames on the Data Mover, execute the following command onthe Control Station:

server_ifconfig server_2 int1 mtu=9000

Where:

server_2is the Data Mover

int1 is the interface

Linux servers To configure jumbo frames on a Linux server, execute the following command:

ifconfig eth0 mtu 9000

Alternatively, place the following statement in the network scripts in/etc/sysconfig/network-scripts:

MTU=9000

RACinterconnect

Jumbo frames should be configured for the storage and RAC interconnect networksof this solution to boost the throughput, as well as possibly lowering the CPUutilization due to the software overhead of the bonding devices.

Typical Oracle database environments transfer data in 8 KB and 32 KB block sizes,which require multiple 1,500 frames per database I/O, while using an MTU size of1,500. Using jumbo frames, the number of frames needed for every large I/O requestcan be reduced, thus the host CPU needed to generate a large number of interruptsfor each application I/O is reduced. The benefit of jumbo frames is primarily acomplex function of the workload I/O sizes, network utilization, and Oracle database

server CPU utilization, and so is not easy to predict.

For information on using jumbo frames with the RAC interconnect, see OracleMetaLink Note 300388.1.
https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=300388.1https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=300388.1https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=300388.1https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=300388.1https://support.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=300388.1


46/95



46

Verifying thatjumbo framesare enabled

To test whether jumbo frames are enabled, use the following command:

ping M do s 8192

Where:

target is the interface to be tested

Jumbo frames must be enabled on all layers of the network for this command tosucceed.

Public and private networks

Public andprivatenetworks

Each node should have:

One static IP address for the public network

One static IP address for the private cluster interconnect

The private interconnect should only be used by Oracle to transfer cluster managerand cache fusion related data.

Although it is possible to use the public network for the RAC interconnect, this is notrecommended as it may cause degraded database performance (reducing theamount of bandwidth for cache fusion and cluster manager traffic).

Configuringvirtual IPaddresses

The virtual IP addresses must be defined in either the /etc/hosts file or DNS for allRAC nodes and client nodes. The public virtual IP addresses will be configuredautomatically by Oracle when the Oracle Universal Installer is run, which startsOracle's Virtual Internet Protocol Configuration Assistant (vipca).

All virtual IP addresses will be activated when the following command is run:

srvctl start nodeapps -n

Where:

node_name is the hostname/IP address that will be configured in the client'stnsnames.ora file.


47/95



47

Oracle RAC 11gserver network architecture

Databaseserver networkinterfaces

Each Oracle RAC 11g server has four network interfaces:

Two interfaces connect to the storage network.

One interface connects the server to the RAC interconnect network, enablingthe heartbeat and other network I/O required by Oracle CRS.

One interface connects to the client.

Oracle RAC 11gserver networkinterfaces -DNFS

Table 14 lists each interface and describes its use for the Oracle 11gDNFSconfiguration.

Table 14. Interfaces for DNFS configuration

Interface port ID Description

eth0 Client networketh1 RAC interconnect

eth6 Storage network

eth7 Storage network

Oracle RAC 11gserver networkinterfaces -ASM

Table 15 lists each interface and describes its use for the Oracle 11gASMconfiguration.

Table 15. Interfaces for ASM configuration

Interface port ID Description

eth0 Client network

eth1 RAC interconnect

eth6 Unused

eth7 Unused


48/95

Chapter 6: Installation and Configuration


48


Overview

Introduction toinstallation andconfiguration

This chapter provides procedures and guidelines for installing and configuring thecomponents that make up the validated solution scenario. The installation andconfiguration instructions presented in this chapter apply to the specific revisionlevels of components used during the development of this solution.

Before attempting to implement any real-world solution based on this validatedscenario, gather the appropriate installation and configuration documentation for therevision levels of the hardware and software components planned in the solution.Version-specific release notes are especially important.

Note Where tasks are not divided into NAS or ASM, they are the same for bothconfigurations.


Topic See Page

Task 1: Install and configure EMC PowerPath 49

Task 2A: Set up and configure NAS for Celerra 53

Task 2B: Set up and configure ASM for CLARiiON 53

Task 3: Set up and configure database servers 56

Task 4: Configure NFS client options 57

Task 5: Install Oracle grid infrastructure and Oracle RAC 59

Task 6: Configure database server memory options 59Task 7: Configure and tune HugePages 61

Task 8: Set database initialization parameters 63

Task 9: Configure the Oracle DNFS client 65

Task 10: Verify that DNFS has been enabled 68

Task 11: Configure Oracle Database control files and logfiles 70

Task 12: Enable passwordless authentication using ssh (optional) 71


49/95



49

Task 1: Install and configure EMC PowerPath

Overview ofTask 1

EMC PowerPath provides I/O multipath functionality. With PowerPath, a node canaccess the same SAN volume via multiple paths (HBA ports), which enables bothload balancing across the multiple paths and transparent failover between the paths.

Install EMCPowerPath

InstallationThe installation is very straightforward. In the solution environment, EMC runs thefollowing command on the four nodes:

rpm -i EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm

PowerPath license

To register the PowerPath license, run the following command:

emcpreg -install

Type the 24-character alphanumeric sequence found on the License Key Carddelivered with the PowerPath media kit.

Licensing type

To set the licensing type, choose one of the following:

Standard Edition for back-end failover only

Base Edition for end-to-end failover

Full PowerPath for failover and load balancing

Load-balancing policies

To set the PowerPath load-balancing policies to CLARiiON Optimize, run the

following command:

powermt set policy=co

New device/path

To reconfigure the new device/path, run the following command:

Powermt config

Use this command when scanning for new devices. Add those new devices to thePowerPath configuration, configure all detected paths to PowerPath devices, thenadd paths to the existing devices.

For more information on prerequisites and installing PowerPath, see the EMCPowerPath for Linux Installation and Administration Guide.


50/95



50

Configure EMCPowerPath

After installation, you should be able to see pseudo devices using this command:

powermt display dev=all

To start and stop PowerPath, run the command as below:

/etc/init.d/PowerPath start/etc/init.d/PowerPath stop

All ASM disk groups are then built using PowerPath pseudo names.

Note A pseudo name is a platform-specific value assigned by PowerPath to thePowerPath device.

Because of the way in which the SAN devices are discovered on each node, there isa possibility that a pseudo device pointing to a specific LUN on one node might pointto a different LUN on another node. The emcpadm command is used to ensureconsistent naming of PowerPath devices on all nodes as shown in Figure 6.

Figure 6. PowerPath pseudo device names in use

Using theemcpadmutility

Enhancements to the emcpadmutility allow you to preserve and restore PowerPathpseudo-device-to-array logical-unit bindings. The new commands simplify theprocess of renaming pseudo devices in a cluster environment. For example, you canrename pseudo devices on one cluster node, export the new device mappings, thenimport the mappings on another cluster node.


51/95



51

For examples that describe how to rename, import, and export device mappings, seeTable 16 andTable 17 or the EMC PowerPath 5.1 Product Guide.

Table 16. emcpadm commands

Command Description

emcpadm check_mappings [-

v] -f

Displays a comparison of the currentlyconfigured mappings and the mappings in. Thedisplay lists the currently configured devicesalong with the device remappings that willoccur if you import .

emcpadm export_mappings -f

Saves the currently configured mappings to.

emcpadm import_mappings[-v] -f

Replaces the currently configured mappingswith the mappings in . If differences exist amongthe current mappings and the file mappings,the mappings in take precedence. When youimport the file mappings, current hostdevices are remapped according to the filemappings, where differences exist.

Table 17. emcpadm arguments

Argument Description

-f

Specifies the filename and location for.

-v Specifies verbose mode.

Note Before importing new mappings on a node or server, you must:

1. Preview changes with emcpadm check_mappings.

2. Shut down all applications and database systems.

3. Unmount file systems.

4. Deport VxVM disk groups.

Example:

On NodeA of a cluster, to make every other node in the cluster have the samePowerPath configuration, run the following command:

emcpadm export_mappings -f NodeA.map


52/95



52

On NodeB of the cluster, copy the NodeA.map file and compare it with the currentconfiguration:

emcpadm check_mappings -v -f NodeA.map

This shows a comparison of the two configurations and what changes will be made ifthis mapping file is imported. To proceed, run the following on NodeB:

emcpadm import_mappings -v -f NodeA.map

powermt save


53/9

h8100 Unified Storage Oracle Celerra Psg

Documents

Transcript of h8100 Unified Storage Oracle Celerra Psg