Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

38
Session ID: Prepared by: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP 1197 @jmjgangler Mike Gangler – Senior Database Specialist, DBA Team Lead Secure-24 - @mjgangler [email protected] Palm D Thu, Apr 14, 2016 08:30 AM - 09:30 AM

Transcript of Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Page 1: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Session ID:

Prepared by:

Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

1197

@jmjgangler

Mike Gangler – Senior Database Specialist, DBA Team Lead Secure-24 - @mjgangler [email protected]

Palm D Thu, Apr 14, 2016 08:30 AM - 09:30 AM

Page 2: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

About Mike Gangler

•  Oracle ACE with robust database credentials •  DBA for over 28 years, working with Oracle since

version 4 •  Team Lead and Senior Database Specialist at Secure-24 •  Currently serving on the board of the Southeast Michigan

Oracle Professionals (SEMOP) group – www.meetup.com •  Charter member of the Board of Directors for the

International Oracle Users Group (IOUG) – www.ioug.org •  Follow me on my Blog http://mjgangler.wordpress.com and

on twitter! @mjgangler

2

Page 3: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

About Secure-24

3

FOUNDED

HEADQUARTERS GLOBAL

OPERATION CENTERS

DATA CENTERS

Secure-24 was founded in 2001 and since then has grown

to 500+ employees and has received

recogniPon as one of Computerworld’s Best

Places to Work in IT, 3-years running.

Secure-24 is headquartered in

Southfield, MI

Serving customers around the globe,

Secure-24 has two (2) OperaPon Centers in Michigan, one (1) in

Nevada and one (1) in Hyderabad India.

Secure-24 has three (3) data centers in

Michigan, one (1) in Nevada, plus several global partnerships. We only choose the safest locaPons for

our data centers.

Secure-24 has 15 years of experience delivering managed IT operaPons, applicaPon hosPng and cloud services to enterprises worldwide. We manage SAP, Hyperion, PeopleSo^, JD Edwards, Oracle E-Business Suite and other

mission criPcal applicaPons across all industries for businesses of every size.

Page 4: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Communi'es Educa'on

Join for as low as $150 SELECT Journal Resource Center IOUG Press Webinars & Podcasts IOUG Forum 5 Minute Briefing

Plus get access to IOUG’s content library, peer-to-peer networking, and more! Corporate options also available!

Page 5: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Oracle Conferences in Detroit Area

Southeast Michigan Oracle Professionals

http://www.meetup.com/SouthEast-Michigan-Oracle-Professionals/ Meet monthly – 2nd Tuesday of the month

Michigan Oracle User Summit November 3, 2016 http://www.mous.us

Page 6: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Great Lakes Oracle Conference

•  2016 Great Lakes Oracle Conference (GLOC)

•  May 18 & 19, 2016 Cleveland Public Auditorium

Cleveland, OH

https://www.neooug.org/gloc/

Page 7: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Today’s Discussion

7

Learn how Secure-24 leverages Oracle ZFS Storage, Oracle Intelligent

Storage Protocol (OISP), Templates and Oracle 12C RAC to implement a

Private Cloud for Oracle 12c RAC applications. This pairing of Database and

Storage hardware through Oracle’s VCA has enabled Secure-24 to take

advantage of the unique integration capabilities such as OISP which enables

automatic and dynamic tuning of storage settings such as record size.

Secure-24 Confidential – Internal – Copyright © 2015 Secure-24 and/or its affiliates. All rights reserved.

Page 8: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

•  Simplicity –  Simplify the environment with new technology

•  Scalability –  Technology should be able to grow and scale as the company grows.

•  Automation –  Technology should have the ability to be automated and repeatable process

•  Overall reduction of cost –  Technology should drive down the overall cost

•  Solution driven by Customer requirements and demands –  This is the reason for looking at the technology

Secure-24 Technology Goals

Page 9: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Business Case

•  Simplify Oracle 12c RAC deployments •  Eliminate complexity around storage tuning •  Decrease support time through transparency and

simplicity •  Fully Oracle Support technology stack

Page 10: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

10

Hardware Specifications Overview

Oracle Private Cloud Appliance

OracleX86Servers OracleZFSStorageAppliance•  Redundantcontrollers,highperformancevirtualstorage

NetworkInfrastructure•  OracleFabricInterconnectF1-15(VirtualNetwork)•  OracleSwitches•  Fullyconfiguredfabricnetworking,nosetuprequired•  Integrateswithexis'ngEthernetandstoragenetworks

Fabric Interconnect

Fabric Interconnect

Compute Nodes

Compute Nodes

Mgmt Nodes

Management Storage

InfiniBand Expansion

Ethernet Switch

OracleX86Servers•  Upto25nodes•  CPU:Upto900cores•  Memory:Upto6.4TBaggregate•  SupportsanyLinux,SolarisandWindowsapplica'ons

Netw

orking M

anagement

40Gb/sServer-to-Server

InfiniBand Expansion

•  HighspeedInfiniBandnetwork–mul'ple40Gbps

•  InfiniBandconnectedstorage

•  Dedicatedmanagementsystemsincluded

•  OracleVMandOracleLinuxincluded

•  TestedandCer'fiedbyOracle

•  So^wareDefinednetworkingthroughFabricInterconnect

Page 11: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

•  Fast “Power-on to Production” - Automation –  2 hours vs. 1 month on legacy infrastructure

•  Mixed workloads - Scalable –  Oracle DB, Oracle Applications, Microsoft, Custom apps

•  More efficient Licensing – Reduction of cost –  Trusted Partitions Licensing

•  Rapid Application Deployment – Simplicity and automation –  Pre-built Oracle VM templates

•  Simplified Support - Simplicity –  One Hand to Shake (One Throat to Choke)

•  Lower Acquisition and Deployment Costs •  Impressive performance – Customer requirements

Why We Selected Private Cloud Appliance

Page 12: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Oracle 12C RAC Design Overview

•  Oracle PCA and ZS3-2 Storage •  Single Node or multiple nodes with RAC •  Oracle 12c Enterprise Edition •  Oracle ZFS storage array with OS 8.3+ •  Database connections made from Oracle

using Direct NFS •  ZFS Storage – Flash enabled performance

storage •  OISP (Oracle Intelligent Storage Protocol)

for storage optimization •  No “ASM”

X3-2

VIR

TUA

L CO

MP

UTE

AP

PLIA

NC

E

S PREARPS

MAJ OR

MINOR

US E R

C RIT IC AL

Z FS S TORAGEZ S 3-2

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S TORAGEDE 2-24P

HDDMAP12-23

0-11

S PREARPS

MAJ OR

MINOR

US E R

C RIT IC AL

Z FS S TORAGEZ S 3-2

Page 13: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Design Overview

•  Local OS disks Stored on Oracle VM data repository

•  Data, Logs, and Voting disks presented via NFS and dNFS

VM 2

dNFS

VM

NAS,SAN,iSCSI

OracleVMServers

OracleVMServerPool

VM VMVM

SIVM 1

RAC RAC

ZFSStorage

Page 14: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

ZFS Powered

•  OISP MOS Doc ID 1943618.1 •  ZFS Admin Guide http://

docs.oracle.com/cd/E51475_01/pdf/E52872.pdf

•  ZFSHybridArraywith3typesofcaching•  DRAM,SLCWriteFLASH,

MLCReadFLASH

•  OISPOp'malRecordSize&Latency•  TheOracledNFSclientpassestheop'mal

recordsizetotheZFSStorageApplianceforeachNFSv4writerequest

•  TheOracledNFSclientpassestheop'malwritebiastotheZFSStorageApplianceforeachNFSv4writerequest..

Page 15: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

ZFS Setup •  Create our 3 shares

•  OCR Voting Disk

–  This will be raw NFS v3

–  https://docs.oracle.com/database/121/CWADD/votocr.htm#CWADD92368

–  We leave Write Bias as Latency and Record Size as 128K (Generic)

•  Data and Logs files

–  This is where the magic of OISP comes in.

–  Instead of having to create multiple different volumes for redo, archive, control, data, temp etc. We simple create 2 volumes for Logs and Data

–  Write Bias and Record Size will be set via database cue’s through OISP and optimized for the database.

Page 16: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

STANDARD NFS ARCHITECTURE

Page 17: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
Page 18: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
Page 19: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

dNFS and OISP

Page 20: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Linux Configuration Setup Direct NFS

–  Fairly simple example

•  # cat oranfstab.example

•  server: 10.10.10.10

•  nfs_version: nfsv4

•  export: /export/U02 mount: /u02

•  export: /export/U03 mount: /u03

–  dNFS is fairly straight forward in 12c setup guide

•  https://docs.oracle.com/database/121/CWLIN/storage.htm#CWLIN279

– This is where NFSv4 is setup – Many other options are

possible, but simple is possible too • Old paper, still valid - http://www.oracle.com/technetwork/articles/directnfsclient-11gr1-twp-129785.pdf

– Most dNFS guides are for 11g, 12c is the same except dNFS is Natively part of 12c

– Troubleshooting • (Doc ID 1464567.1)

Page 21: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

dNFS Linux Configuration Setup

dNFS (11G) – Add 12C dnfs_on Enable:

Ln –s $ORACLE_HOME/lib/libnfsodm11.so $ORACLE_HOME/lib/libodm11.so

Disable

Ln –s $ORACLE_HOME/lib/libodmd11.so $ORACLE_HOME/lib/libodm11.so

Page 22: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

dNFS Linux Configuration Setup

dNFS (12C) Enable:

Change directory to $ORACLE_HOME/lib

>>make –f ins_rdbms.mk dnfs_on

Disable Change directory to $ORACLE_HOME/lib

>>make –f ins_rdbms.mk dnfs_off

Page 23: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

dNFS Linux Configuration Setup

dNFS Verify•  bash-3.2$ ps -ef | grep dbw

oracle 23126 1 0 06:53 ? 00:00:00 ora_dbw0_MYINST1oracle 23128 1 0 06:53 ? 00:00:00 ora_dbw1_MYINST1oracle 23246 22377 0 07:04 pts/0 00:00:00 grep dbw

•  bash-3.2$ /usr/sbin/lsof -p 23126COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEoracle 23126 oracle cwd DIR 104,3 4096 14811513 /u02/app/oracle/product/12.1.0/dbsoracle 23126 oracle rtd DIR 104,3 4096 2 /oracle 23126 oracle txt REG 104,3 210824714 15892486 /u02/app/oracle/product/12.1.0/bin/oracle

•  ...oracle 23126 oracle 32u IPv4 322388 TCP mydbhost:64553->mynfshost:nfs (ESTABLISHED) oracle 23126 oracle 33u IPv4 322389 TCP mydbhost:21549->mynfshost:nfs (ESTABLISHED)

-- dNFS setup guide •  https://docs.oracle.com/database/121/CWLIN/storage.htm#CWLIN279

Page 24: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Linux Configuration Setup

Oracle Linux 6 – uek Kernel –  Utilize dtrace in Linux

–  Ksplice for non-disruptive kernel patching

–  UEK kernel optimization for Oracle DB

Setup Mount Points –  Setup Voting disk

• Must be NFSv3

• Not Part of dNFS

• Mount options important (Doc ID 359515.1)

•  rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0

–  Setup Logs and Data

• Must be NFSv3 (dNFSv4 setup to come later)

• Mount options important (Doc ID 359515.1)

•  rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600

Page 25: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Linux Configuration Setup - #1

Install Oracle Pre-Install RPM

•  Optional but makes life easier •  >> yum install oracle-rdbms-server-12cR1-preinstall •  This will download all required dependencies

•  create the oracle user id •  modify sysctl.conf •  modify limits.conf

•  https://blogs.oracle.com/wim/entry/easily_install_oracle_rdbms_12cr1

Page 26: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Linux Configuration Setup - #2

TEST SNMP – REQUIRED FOR OISP MOS ID – 1943618.1 # SNMPGET –v1 –c public 10.10.10.10 1.3.6.1.4.1.42.2.225.1.4.2.0 -  Looking for the following:

SNMPv2- SMI::enterprises. 42.2.225.1.4.2.0 = STRING: "Sun ZFS Storage 7330"

Page 27: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Linux Configuration Setup - #3

Setup for dNFS Oracle wants us to tune for dNFS, so we will

>> vi /etc/sysctl.conf ##7.3.4 Checking TCP Network Protocol Buffer for Direct NFS Client •  net.ipv4.tcp_rmem = 4096 87380 4194304 •  net.ipv4.tcp_wmem = 4096 16384 4194304

•  https://docs.oracle.com/database/121/CWLIN/storage.htm#CHDEFHFC

Page 28: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Linux Configuration Setup

Install Oracle Pre-Install RPM

–  Optional but makes life easier

–  # yum install oracle-rdbms-server-12cR1-preinstall

–  This will download all required dependencies, create the oracle user id, modify sysctl.conf and modify limits.conf.

–  https://blogs.oracle.com/wim/entry/easily_install_oracle_rdbms_12cr1

Test snmp (needed for OISP)

–  Doc ID 1943618.1

–  # snmpget -v1 -c public 10.10.10.10 1.3.6.1.4.1.42.2.225.1.4.2.0

SNMPv2-SMI::enterprises.42.2.225.1.4.2.0 = STRING: "Sun ZFS Storage 7330"

Setup for dNFS –  Oracle wants us to

tune for dNFS, so we will

•  # vi /etc/sysctl.conf •  # •  #

https://docs.oracle.com/database/121/CWLIN/storage.htm#CHDEFHFC

•  #7.3.4 Checking TCP Network Protocol Buffer for Direct NFS Client

•  # •  net.ipv4.tcp_rmem =

4096 87380 4194304

•  net.ipv4.tcp_wmem = 4096 16384 4194304

Page 29: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Oracle 12c Enterprise Edition Install Oracle 12.1.0.2 -  If using a private heartbeat IP that contains .255 -  add the following patch (Before root.sh) -  19777496(PatchId)

-  Or dont use .255 in the heartbeat network

Oracle 12c Installation - #1

Page 30: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Confirm dNFS and OISP setup $ egrep I '"Sun ZFS"|Direct' alert_*.log

Found NAS server Sun ZFS Storage 7420 Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0 Direct NFS: channel id [0] path [10.80.216.18] to filer [10.80.216.18] via local [] is UP Direct NFS: channel id [1] path [10.80.216.19] to filer [10.80.216.18] via local [] is UP Direct NFS: mount complete dir /export/oisp/redo on 10.80.216.18 nfsport 2049 using NFS version 4 Direct NFS: mount complete dir /export/oisp/redo on 10.80.216.18 nfsport 2049 using NFS version 4

u  “Found NAS server” indicates SNMP has found the ZFSSA. Requirement for OISP.

u ODM/Direct NFS indicates that dNFS has been activated using NFSv4.

Oracle 12c Installation - #2

Page 31: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Applyrequiredpatch(OISP)Patch19630940ResolvesissuewithDBAnaly'cs

Oracle 12c Installation - #3

Page 32: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Oracle12cEnterpriseEdi'onInstallOracle12.1.0.2

ConfirmdNFSandOISPsetup$egrepI'"SunZFS"|Direct'alert_*.log

FoundNASserverSunZFSStorage7420

OracleinstancerunningwithODM:OracleDirectNFSODMLibraryVersion3.0

DirectNFS:channelid[0]path[10.80.216.18]tofiler[10.80.216.18]vialocal[]isUP

DirectNFS:channelid[1]path[10.80.216.19]tofiler[10.80.216.18]vialocal[]isUP

DirectNFS:mountcompletedir/export/oisp/redoon10.80.216.18nfsport2049usingNFSversion4

DirectNFS:mountcompletedir/export/oisp/redoon10.80.216.18nfsport2049usingNFSversion4

“FoundNASserver”indicatesSNMPhasfoundtheZFSSA.RequirementforOISP.ODM/DirectNFSindicatesthatdNFShasbeenac'vatedusingNFSv4.

ApplyrequiredpatchPatch19630940

ResolvesissuewithDBAnaly'cs

Oracle 12c Installation

Page 33: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Typical Deployment

Typical Process For Creating and Deploying New Applications

1to3weeks

1-2days1day1-5days1-2days1-5days Start

Page 34: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

PCA Rapid Deployment

•  Pre-made, certified virtual machine templates, including application, operating system, and database available at https://edelivery.oracle.com

•  Save Weeks/Days in Configuration Time –  Pre-built, pre-configured Templates

–  Application, Middleware, and DB

–  Oracle DB, JD Edwards, EBS, PeopleSoft, and many more…. 90+ templates

Start-upinOracleVMPool

ImporttoOracleVMManager

OracleVMServers

ServerPool2Customize&Save

asGoldenImage

E-DeliveryFile

JDEdwards

DownloadfromOracle.com

File

JDE

VM1

JDE

Page 35: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Application Templates & Assemblies

DB

WebLogicServer

DeploymentPoliciesConfigura'onMetadata

SOA_PROD_1

OracleVMAssemblyProduc'onEnvironment:SOA_PROD_1

VM

VM

VM

VM

VM

VM

Deploy

Deployapplica'onintoOracleVMManageras‘GoldenImage’• Clone,ini'alize,andstart-upinstance• Upandrunninginhoursvs.daysorweekswithlitlerisk

OracleVMServers

StorageRepository

ServerPool2

Deployed:Produc'onEnvironmentSOA_PROD_1

GuestAddi'ons• Allowsdirectintegra'onbetweenguestso^wareandvirtualiza'onlayer• Assistsinorchestra'onofcomplex,mul'-VMdeployments

OVF-based

3

Page 36: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Notes

•  Oracle Database 11g RAC with ZFS Storage Appliance and dNFS on SPARC: http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/oracledb-rac-sparc-zfssa-2414324.pdf

•  Optimizing Storage for Oracle Database 11gR2 with ZFS Storage Appliance: http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/oracle11gr2-zfssa-bestprac-2255303.pdf

•  How to enable OISP on ZFS Storage Appliance for 12c: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1943618.1

ANALYST DATA

•  htp://www.oracle.com/us/corporate/analystreports/industries/zs3-storage-appliance-2033590.pdf

Page 37: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Questions

JDE

[email protected]:htps://mjgangler.wordpress.com

@mjgangler

Page 38: Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP

Please complete the session evaluation Paper – 1197 Author – Mike Gangler We appreciate your feedback and Insight

You May complete the session evaluation via the mobile app