Implementing Oracle11g Database over NFSv4 from … Oracle11g Database over. NFSv4 from a Shared...

Storage Developer Conference 2008 www.storage-developer.org

Implementing Oracle11g Database overNFSv4 from a Shared Backend Storage

Bikash Roy Choudhury


Overview

Client ArchitectureWhy NFS for a Database?Oracle Database 11g RAC SetupMount Options Used Database Tuning Netapp and the Linux Community


Linux NFS Client Architecture


Linux NFSv4 Client in the 2.6.18-88 Kernel

Support NFS v4NFSv4 ACLs support

use nfs4-acl-tools package or download from http://www.citi.umich.edu/projects/nfsv4/linux/

Converts the POSIX ACLs to NFSv4

Read and write delegationsKerberos 5/5i

Features not in 2.6.18 kernelReplicationsMigration support


Less ComplexEthernet connectivity modelSimple storage provisioning & backup

Reduce the Cost of Storage ProvisioningAmortize storage costs across serversFlexClone® helps cloning master DBs for Test & Dev. Areas

Improved Oracle AdministrationSingle repositoryRecovering from Snapshot™ quick and reliable

Why NFS for Database?


Better PerformanceData is cached just once, in user space, which saves memory – no second copy in kernel space. Metadata access for the clients are much quicker with less over-headLoad balances across multiple network interfaces, if they are available.

Why NFS for Database?

Oracle Prefers NFS/NAS


Performance comparison with different Protocols


Why Oracle11g over NFSv4

NFSv4 is the building block for all scale out implementations of Oracle11g over NFSLeased-based locking

Helps to clear or recover locks on event of a network or Oracle datafile outages.

Referrals will allow a storage grid and a compute grid to mutually optimize I/O paths.

The redirection feature allows a storage grid and a compute grid to mutually optimize I/O paths.


2 Node Oracle11g RAC over NFSv4-Reference Architecture

Storage1:/u01/crscfgStorage1:/u01/votdskStorage1:/u01/orahomeStorage1:/u01/oradataStorage2:/u02/oradata

Instance ora11

Mounted overNFSv4

Mounted overNFSv4

IBM3455 RHEL5.2 ServerOra-node1


Gigabit Switch

FAS3070c

RAID Group/Aggregate Storage

Storage1

Storage2

SharedOracle Database 11gVolume(ORCL)

Cluster RegistryVolume

CRS Voting DiskVolume

Oracle® HomeVolume

/CRS Home (Local)

/CRS Home (Local)


Instance ora12Virtual IP Connection

Private IP Connection

Public IP Connection Public IP Connection

144GB 10k RPM Fibre Channel Disks

Instance orcl1


Hardware Used for Oracle Database 11gRAC Setup

Oracle® RAC nodesx86_64 Dual Core 2.8Ghz AMD Opteron CPU 10Gb RAM80Gb HDD SATA2Gb of Swap Space

1Gb (Gigabit) SwitchNetApp® Storage

FAS3070 Cluster144Gb 10k RPM FC drives4Gb Fibre Channel back end shelf speedDATA ONTAP 7.3


Software Used for Oracle Database 11gRAC Setup

2.6.18-88.el5xen #1 SMP – x86 64 bit This kernel was used due the the recent NFS performance enhancements

Oracle® Database 11g database and clusterwareData ONTAP® 7.3 on NetApp® storageNFS Mounts are all over NFSv4


Service configuration for Oracle11g RAC Setup

Boot with non-XEN kernel“libvirt” will be disabled

Creates interface call “virbr0” that has issues with Oracle® CRS install

Disable “iptables” on the Linux® RAC nodesSynchronize Time with NTP on the RAC nodes and the NetApp® Storage


Network Transport used for Oracle 11g RAC Setup

Use the TCP transportMore reliable and low risk of data corruption and better congestion control compared to UDPRetransmission happens in the transport layer instead of application layer

Enlarge TCP window size for fast responsenet.ipv4.tcp_rmem = 4096 524288 16777216net.ipv4.tcp_wmem = 4096 524288 16777216net.ipv4.tcp_mem = 16384 16384 16384

Benefits:This will increase the speed of the cluster interconnect and public network.


Mount Options Used for Oracle 11g RAC

NFSv4 ProtocolSpecify “-t nfs4” to ensure mounting over NFSv4

Background mounts (bg)Clients can finish booting without waiting for storage systems

rsize=32768 wsize=327682.6.18-88 kernel supports 64k transfer size and up to 1Mb

NetApp StorageDATA ONTAP 7.3 uses up to 128kb block size



timeo600 is good for TCP

Hard MountDefault recommendationMandatory for data integrityMinimizes the likelihood of data loss during network and server instability



intr optionAllows users and applications to interrupt the NFS clientBe aware that this doesn’t always work in Linux® and rebooting may be necessary to recover a mount pointUse soft mount insteadOracle has verified that using “intr” instead of “nointr” can cause corruption when a database instance is signaled (during a “shutdown abort”)

“nointr” is recommended


Mount Options for only Database mounts

“noac” option Disables client side caching and keeps file attributes up to date with the NFS ServerShorthand for “actimeo=0,sync”

Bug - https://bugzilla.redhat.com/show_bug.cgi?id=446083

Patch - http://article.gmane.org/gmane.linux.nfs/20074

Set the “sunrpc.tcp_slot_table_entries” to 128Benefits:

Removes a throttle between the Linux® nodes and the backend storage systemAllows a single Linux box to drive substantially more I/O to the backend storage system

https://bugzilla.redhat.com/show_bug.cgi?id=446083

http://article.gmane.org/gmane.linux.nfs/20074


ORACLE_HOME on Shared Storage

Benefits:Redundant copies are not needed for multiple hosts.

Extremely efficient in a test/dev environment where quick access to the Oracle® binaries from a similar host system is necessary.

Disk space savings. It is easier to add nodes. Patch application for multiple systems can be completed more rapidly.

For example, if testing 10 systems that you want to all run the exact same Oracle DB versions, this is beneficial.


Reference Architecture – 2 Node Oracle Database 11g RAC over NFSv4

RAID Group/Aggregate Storage

Oracle® HomeVolume



Storage1:/u01/oradata/redo01.logStorage1:/u01/oradata/redo02.logStorage1:/u01/oradata/redo03.log

Storage1:/u01/oradata/control01.ctlStorage1:/u01/oradata/control02.ctlStorage1:/u01/oradata/control03.ctl

Storage2:/u02/oradata/redo01a.logStorage2:/u02/oradata/redo02b.logStorage2:/u02/oradata/redo03c.log

Storage2:/u02/oradata/control01.ctlStorage2:/u02/oradata/control02.ctlStorage2:/u02/oradata/control03.ctl

Instance ora11

Mounted overNFSv4

Mounted overNFSv4



Gigabit Switch

FAS3070c

Storage1

Storage2

SharedOracleDatabase 11gVolume(ORCL)

Cluster RegistryVolume

CRS Voting DiskVolume

/CRS Home (Local)

/CRS Home (Local)

Instance orcl2Virtual IP Connection

Private IP Connection

Public IP Connection Public IP Connection

144GB 10k RPM Fibre Channel Disks

Instance orcl1


Oracle Database 11g CRS Timeout Settings – Best Practices

OCR and CRS voting files have to be multiplexed

A copy of both the files has to reside on each storage

Three CSS parameters have to be setmisscount – 120 seconds (30 secs default)disktimeout – 200 seconds (default)reboottime – 3 seconds (default)


NFsv3 & NFSv4 Comparison – Performance Analysis

2,900

3,000

3,100

3,200

3,300

3,400

3,500

3,600

NFSv3 NFSv4 NFSv4 with Read Delegations

tpm

C

1.4

1.45

1.5

1.55

1.6

1.65

Ave

rage

tpm

Res

pons

e Ti

me

(sec

)

tmpC Avg tpm Response Time

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Idle 0.00% 1.10% 0.00%

IO wait 0.00% 2.67% 0.00%

User Time 89.52% 84.84% 89.15%

Sys Time 10.48% 11.39% 10.85%

NFSv3 NFSv4 NFSv4 with Read Delegations


Performance Analysis Contd.

0

5

10

15

20

25

30

35

NFSv3 NFSv4

In M

illi-S

ec

Db file sequential read Db file scattered read

21,800

22,000

22,200

22,400

22,600

22,800

23,000

NFSv3 NFSv4 NFSv4 with ReadDelegations

Netw

ork Th

roug

hput (K

B/se

c)


NetApp’s Linux CommunityNetApp’s business model depends on superior client behavior and performance

NetApp is driving Linux® Client Performance and scalability, sponsored by NetApp at CITI, Univ. of Michigan

Build expertise with Linux clients and storage systems to help our customers get the most from our products

Explore and correct Linux NFS client and OS issuesEstablish positive relationship with Linux communityDevelop internal resources for customer-facing teams


NetApp’s Linux Community

Linux Certification Testing ResultsLinux 10g/11g RAC testing over NFSv3/NFSv4Linux FCP and iSCSI testingLinux NFSv4 client supportLinux certification with NFSLinux Best Practices document

• http://www.netapp.com/library/tr/3183.pdf

http://www.netapp.com/library/tr/3183.pdf


Linux Leadership with NetApp

Mature NetApp Solution for Oracle® on Linux®Database Consolidation

High Availability

Backup and Recovery

Disaster Recovery

Oracle Database 10g/11g certification with RedHat Linux and NetApp® Storage over NFSv3/NFSv4Unbreakable and Enterprise ready–NetApp, Oracle, Oracle Enterprise Linux (OEL)

Partnership and Performance Testing ResultsRedHat partnership agreement


BACKUP SLIDES


Storage Resiliency – High Availability

Clustered Failover in the event of hardware failureLess cluster failover/giveback timesTransparent to NFS clientsNondisruptive Data ONTAP® upgrades without any user downtimeReduced TCO and maximized Storage ROI


Database Performance Tuning with FlexVol

BenefitsImproves database performance quickly and measurablyUses all available spindles for data and transaction logsSpindle sharing makes total aggregate performance available to all volumesAutomatic load shifting

Performance Improvement


Backup and Recovery

Significant time savingsStay onlineReduce system and storage overheadConsolidated backupsBack up more often

Time in Hours

Time toRecover

0 1 2 3 4 5 6 7 8

Time toBackup To Tape (60GB/Hr Best Case)

From TapeRedo Logs

Snapshot™

Redo LogsSnapRestore®

250GB Database


SnapManager for Oracle

Automated, fast, and efficientUptime AND performanceSimplify backup, restore, and cloningTight Oracle Database 10gintegration

Automated Storage Manager (ASM)RMAN

Oracle® Databases

NetApp StorageAppliance

SnapManager® (GUI)

SnapDrive®

NetApp® Appliance

NFS, FCP, or iSCSI

Implementing Oracle11g Database over NFSv4 from … Oracle11g Database over. NFSv4 from a Shared...

Documents

Transcript of Implementing Oracle11g Database over NFSv4 from … Oracle11g Database over. NFSv4 from a Shared...