TR Nimble Oracle Setup Guide - Nimble Storage Add the below settings to the /etc/systctl.conf file....

N I M B L E T E C H N I C A L R E P O R T : N IMB LE S TORAGE F O R ORA CL E 1

TECHNICAL REPORT

Nimble Storage Setup Guide for Single Instance Oracle 11gR2 on Oracle Linux 6.4


Document Revision

Table 1Table 1Table 1Table 1.

Date Revision Description

1/8/2013 1.0 Initial Draft

1/11/2013 1.1 Revised

2/25/2014 1.2 Revised

7/31/2014 1.3 Revised iSCSI Timeout

11/17/2014 1.4 Updated iSCSI & Multipath

THIS TECHNICAL TIP IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN

TYPOGRAPHICAL ERRORS AND TECHNICAL INACCUURACIES. THE CONTENT IS PROVIDED AS IS,

WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

Nimble Storage: All rights reserved. Reproduction of this material in any manner whatsoever

without the express written permission of Nimble is strictly prohibited.


Table of Contents

Introduction ................................................................................................................................................................................. 4

Audience ...................................................................................................................................................................................... 4

Server Setup for Oracle ............................................................................................................................................................. 4

Oracle Linux 6.4 Installation ............................................................................................................ 4

Operating System configuration for Oracle ...................................................................................... 5

Oracle Linux iSCSI Setup ................................................................................................................ 6

Multipath Setup ................................................................................................................................ 6

Nimble Storage Setup ............................................................................................................................................................... 7

Create New Initiator Group .............................................................................................................. 8

Create Volumes and Allow Access .................................................................................................. 9

Protection using Nimble Volume Collection .................................................................................. 14

Oracle 11gR2 Setup ................................................................................................................................................................17

Pre-requisites .................................................................................................................................. 17

Oracle ASM Disks Setup ................................................................................................................ 17

Install Grid Infrastructure ............................................................................................................... 19

Install Oracle Software ................................................................................................................... 19


Introduction

The purpose of this technical white paper is to describe the basic setup of Oracle Linux 6.4 and Oracle 11g Release 2

with Automatic Storage Management (ASM) when deploying on Nimble Storage.

Oracle performance tuning is beyond the scope of this paper. Please visit www.oracle.com for Oracle Performance

Tuning Guide for more information in tuning your database.

Audience

This guide is intended for Oracle database solution architects, storage engineers, system administrators and IT

managers who analyze, design and maintain a robust database environment on Nimble Storage. It is assumed that the

reader has a working knowledge of network design for iSCSI SAN, and basic Nimble Storage operations. Knowledge of

Oracle Linux operating system, Oracle Clusterware, and Oracle database is also required.

Server Setup for Oracle

Configuration for a single instance Oracle database.

Oracle Linux 6.4 Installation

During the installation of the operating system, we recommend that Logical Volume Manager (LVM) not to be

used for the boot disk. Remove all LVM configurations and use the standard partition method for the boot disk.

• /boot partition

• swap partition

• / partition

[root@oraclelinux1 ~]# df

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/sda3 32998256 22234944 9087076 71% /

tmpfs 6291456 2607808 3683648 42% /dev/shm

/dev/sda1 126931 50862 69516 43% /boot

After the installation of the operating system, the following daemons need to be disabled.

o Disable selinux o Disable NetworkManager o Disable iptables o Disable ip6tables

If not installed already, the following packages are required for Oracle 11gR2 and multipathing,

• compat-db42-4.2.52-15.el6.x86_64.rpm

• compat-db43-4.3.29-15.el6.x86_64.rpm

• compat-db-4.6.21-15.el6.x86_64.rpm

• cvuqdisk-1.0.9-1.rpm

• device-mapper-multipath-0.4.9-64.0.1.el6.x86_64.rpm

• device-mapper-multipath-libs-0.4.9-64.0.1.el6.x86_64.rpm


• ksh-20100621-19.el6.x86_64.rpm

• libaio-devel-0.3.107-10.el6.x86_64.rpm

• libsysfs-2.1.0-7.el6.x86_64.rpm

• libXp-1.0.0-15.1.el6.x86_64.rpm

• libXp-devel-1.0.0-15.1.el6.x86_64.rpm

• oracleasm-support-2.1.8-1.el6.x86_64.rpm

• sysfsutils-2.1.0-7.el6.x86_64.rpm

• sysstat-9.0.4-20.el6.x86_64.rpm

• unixODBC-2.2.14-12.el6_3.x86_64.rpm

• unixODBC-devel-2.2.14-12.el6_3.x86_64.rpm

Operating System configuration for Oracle

• Add the below settings to the /etc/systctl.conf file. Run the ‘sysctl –p’ command when done.

# For 11g, recommended value for file-max is 6815744 fs.file-max = 6815744 # For 10g, uncomment 'fs.file-max 327679', comment other entries for this parameter and re-run sysctl -p # fs.file-max:327679 kernel.msgmni = 2878 kernel.sem = 250 32000 100 142 kernel.shmmni = 4096 net.core.rmem_default = 262144 # For 11g, recommended value for net.core.rmem_max is 4194304 net.core.rmem_max = 4194304 # For 10g, uncomment 'net.core.rmem_max 2097152', comment other entries for this parameter and re-run sysctl -p # net.core.rmem_max=2097152 net.core.wmem_default = 262144 # For 11g, recommended value for wmem_max is 1048576 net.core.wmem_max = 1048576 # For 10g, uncomment 'net.core.wmem_max 262144', comment other entries for this parameter and re-run sysctl -p # net.core.wmem_max:262144 fs.aio-max-nr = 3145728 # For 11g, recommended value for ip_local_port_range is 9000 65500 net.ipv4.ip_local_port_range = 9000 65500 # For 10g, uncomment 'net.ipv4.ip_local_port_range 1024 65000', comment other entries for this parameter and re-run sysctl -p # net.ipv4.ip_local_port_range:1024 65000 # Added min_free_kbytes 50MB to avoid OOM killer on EL4/EL5 vm.min_free_kbytes = 51200 # Nimble Recommended net.core.wmem_max = 16780000 net.core.rmem_max = 16780000 net.ipv4.tcp_rmem = 10240 87380 16780000 net.ipv4.tcp_wmem = 10240 87380 16780000

• Add the settings below to the /etc/security/limits.conf file.

grid soft nofile 131072 grid hard nofile 131072 grid soft nproc 131072


grid hard nproc 131072 oracle soft nofile 131072 oracle hard nofile 131072 oracle soft nproc 131072 oracle hard nproc 131072 oracle soft core unlimited oracle hard core unlimited oracle soft memlock 50000000 oracle hard memlock 50000000

• Add the code below to the /etc/profile file.

if [ $USER = "oracle" ] || [ $USER = "grid" ] ; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi

Oracle Linux iSCSI Setup

• Leverage the native iSCSI initiator for Oracle. To do this, ensure the software iSCSI initiator shown

below has been installed.

iscsi-initiator-utils-6.2.0.872-41.0.1.el6.x86_64

• Configure iSCSI as follow:

• Edit the /etc/iscsi/iscsid.conf and modify the following parameters: o node.session.timeo.replacement_timeout = 120 o node.conn[0].timeo.noop_out_interval = 5 o node.conn[0].timeo.noop_out_timeout = 10 o node.session.nr_sessions = 4 o node.session.cmds_max = 2048 o node.session.queue_depth = 1024

• Make sure the iSCSI NICs (eth1, eth2) have been assigned iSCSI network IP addresses and NET MASK. Should look something similar to this for both iSCSI NICs.

DEVICE="eth1" BOOTPROTO=static HWADDR="00:50:56:AE:30:71" NM_CONTROLLED="no" ONBOOT="yes" TYPE="Ethernet" UUID="4705a3e9-671a-45a0-8477-8b8b98d56c09" IPADDR=172.18.128.71 NETMASK=255.255.255.0 MTU=9000

Multipath Setup

• Change directory to /etc and using your favorite editor to create a new file called ‘multipath.conf’ with the code below


defaults { user_friendly_names yes find_multipaths yes } devices { device { vendor "Nimble" product "Server" path_selector "round-robin 0" path_checker tur rr_min_io_rq 10 rr_weight priorities failback immediate path_grouping_policy group_by_serial features "1 queue_if_no_path" } }

• Start multipathing by running the following: o [root@oraclelinux1 ~]# chkconfig multipathd on o [root@oraclelinux1 ~]# modprobe dm_multipath o [root@oraclelinux1 ~]# service multipathd start

Nimble Storage Setup Creating iSCSI volumes for the Oracle server on Nimble Storage is simple.

• First, create a new Initiator Group. This Initiator Group contains the Oracle server’s iqn ID.

• Second, create new volumes for the database and allow access to the Initiator Group.

Storage volumes for running Oracle database on an Oracle Linux server


Create New Initiator Group To create a new initiator group

• Open the Nimble Management GUI

• Click on “ManageManageManageManage” and select “Initiator GroupsInitiator GroupsInitiator GroupsInitiator Groups”

• Click on “New Initiator GroupNew Initiator GroupNew Initiator GroupNew Initiator Group” button and assign a name to the Initiator Group. We recommend using the Oracle server hostname for distinction.


• Click on “Add InitiatorAdd InitiatorAdd InitiatorAdd Initiator” button. Another window will appear. Enter the Initiator Group Name (oraclelinux1 in the example below) and the Initiator Name (iqn ID) and click “OKOKOKOK”. You can find the iqn ID of the Oracle server by running the following command in the Oracle Linux server:

[root@oraclelinux1 ~]# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1988-12.com.oracle:496c4aead3e1

Create Volumes and Allow Access

• Click on “ManageManageManageManage” and select “VolumesVolumesVolumesVolumes”

N I M B L E T E C H N I C A L R E P O R T : N IMB LE S TORAGE F O R ORA CL E 1 0

• Click on “New VolumeNew VolumeNew VolumeNew Volume” button. Enter a name for the volume. Enter a short description.

• Select “Limit accessLimit accessLimit accessLimit access” radio button and then select “Limit access to iSCSI initiator groupLimit access to iSCSI initiator groupLimit access to iSCSI initiator groupLimit access to iSCSI initiator group” button and then click on the drop down arrow to select the initiator group for this server (oraclelinux1 from the earlier example). Then click “NextNextNextNext”.

• Enter the size for the volume and click “NextNextNextNext”.


• At the Protection screen, select “NoneNoneNoneNone” for now and click “FinishFinishFinishFinish”. We will discuss the Protection option later in the paper.

Make sure to create enough volumes for the Oracle server. Usually in an Oracle ASM environment, you create one OCR volume for the Clusterware and create more volumes for the DATA diskgroup and FRA diskgroup depending on how you want to architect your Oracle database. Once all volumes have been created on the Nimble array, scan for the new volumes on the Oracle server and modify the /etc/multipath.conf file in your Oracle Linux server to use the aliases for the Oracle volumes. Here is how you can setup multipath for the Oracle volumes.

• Scan for disks on the Oracle Linux server. Note that you need to scan for both iSCSI networks. If there is a single discovery IP address, you can use that IP to run the discovery. Either method, please ensure that you can see the same volume across two networks.

[root@oraclelinux1 ~]# iscsiadm -m discovery -t st -p <iSCSI IP #1> [root@oraclelinux1 ~]# iscsiadm -m discovery -t st -p <iSCSI IP #2> [root@oraclelinux1 ~]# iscsiadm -m node - -login

NoteNoteNoteNote: if you want to login for all volumes that this Linux server has access to, then run the above login command. If you want to login for a certain volume, then use this command “iscsiadm –m node –T <Target Volume IQN> - -login”.

• After the scan completes, you can run the following command to find out which disk serial number belongs to which LUN so you can alias the volumes properly. In the below output, sdb and sdc is one disk (same serial #) which I used devdata2 as an alias, sdd and sdf is one disk which I used devdata1 as an alias, and sde and sdg is one disk which I used ocr as an alias.


[root@oraclelinux1 ~]# for a in `cat /proc/partitions | awk '{print $4}' | grep sd`; do echo "### $a: `scsi_id -u -g /dev/$a`" ; done ### sda: ### sda1: ### sda2: ### sda3: ### sdb: 2fed4dc81dcf2582f6c9ce900abdb3466 ### sdc: 2fed4dc81dcf2582f6c9ce900abdb3466 ### sdd: 21ddbfcd70fbd5dae6c9ce900abdb3466 ### sde: 20995db861a86c20e6c9ce900abdb3466 ### sdf: 21ddbfcd70fbd5dae6c9ce900abdb3466 ### sdg: 20995db861a86c20e6c9ce900abdb3466

• The new /etc/multipath.conf would look something like this. Please refer to Oracle Linux documentation for specific parameter settings for other storage devices.

defaults { user_friendly_names yes find_multipaths yes } devices { device { vendor "Nimble" product "Server" path_selector "round-robin 0" path_checker tur rr_min_io_rq 10 rr_weight priorities failback immediate features "1 queue_if_no_path" path_grouping_policy group_by_serial } } blacklist { wwid 26353900f02796769 devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]" } multipaths { multipath { wwid 20995db861a86c20e6c9ce900abdb3466 alias ocr } multipath { wwid 21ddbfcd70fbd5dae6c9ce900abdb3466 alias devdata1 } multipath { wwid 2fed4dc81dcf2582f6c9ce900abdb3466 alias devdata2 } }

• Reload multipathd daemon


[root@oraclelinux1 ~]# /etc/init.d/multipathd reload Reloading multipathd: [ OK ]

• Verify multipath

[root@oraclelinux1 ~]# multipath -ll devdata2devdata2devdata2devdata2 (2fed4dc81dcf2582f6c9ce900abdb3466) dm-0 Nimble,Server size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 21:0:0:0 sdb 8:16 active ready running `- 22:0:0:0 sdc 8:32 active ready running devdata1devdata1devdata1devdata1 (21ddbfcd70fbd5dae6c9ce900abdb3466) dm-1 Nimble,Server size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 23:0:0:0 sdd 8:48 active ready running `- 24:0:0:0 sdf 8:80 active ready running ocrocrocrocr (20995db861a86c20e6c9ce900abdb3466) dm-2 Nimble,Server size=5.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 25:0:0:0 sde 8:64 active ready running `- 26:0:0:0 sdg 8:96 active ready running

• Change disk IO scheduler to “noop”

To set at boot time, add the elevator option at the kernel line in the /etc/grub.conf file: elevator=noop To manually set for all disks: root@mktg04 ~]# multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN; do echo noop > /sys/block/${LUN}/queue/scheduler ; done

• Change max_sectors_kb to “1024”

To change max_sectors_kb to 1024 for a single volume: [root@racnode1 ~]# echo 1024 > /sys/block/sd?/queue/max_sectors_kb Change all volumes: multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN do echo 1024 > /sys/block/${LUN}/queue/max_sectors_kb done

NoteNoteNoteNote: To make this change persistent after reboot, add the commands in /etc/rc.local file.

• Change vm dirty writeback and expire to “500” and “3000” respectively


To change vm dirty writeback and expire: [root@racnode1 ~]# echo 100 > /proc/sys/vm/dirty_writeback_centisecs [root@racnode1 ~]# echo 100 > /proc/sys/vm/dirty_expire_centisecs


• Use “performance” setting for all available CPUs on the host when possible.

To change CPU governor setting to performance: [root@racnode1 ~]# echo performance > /sys/devices/system/cpu/<cpu #>/cpufreq/scaling_governor Change all CPUs [root@racnode1 ~]# for a in $(ls -ld /sys/devices/system/cpu/cpu[0-9]* | awk '{print $NF}') ; do echo performance > $a/cpufreq/scaling_governor ; done


Protection using Nimble Volume Collection Once you have created all volumes for the Oracle server, you can manually add the volumes to a Volume Collection. Volume Collection allows you to take a snapshot via a schedule or ad hoc using the Nimble Storage management GUI or CLI. In an Oracle environment where multiples volumes have been created for a particular database, for consistency, you need to take a snapshot of all volumes concurrently. The Volume Collection allows you to do just that. After a snapshot has been taken using the Volume Collection feature, the snapshot(s) will be available for recovery based on your retention policy setting. The snapshot(s) can be deleted manually if you wish to do so. Please note that you cannot delete a snapshot if there is a clone volume associated with that snapshot. This particular feature is very useful in an Oracle environment. It allows you to create a clone of your production database for Test, Development, or Staging environment quickly and easily. Below are the steps to create a Volume Collection and assign the Oracle volumes to it.

• In the Nimble management GUI, click on “ManageManageManageManage” � “ProtectionProtectionProtectionProtection” � “Volume CollectionVolume CollectionVolume CollectionVolume Collection”


• Click on “New Volume CollectionNew Volume CollectionNew Volume CollectionNew Volume Collection” button

• A “Create a volume collectionCreate a volume collectionCreate a volume collectionCreate a volume collection” window appears. Enter a name for this volume collection and select Blank schedule and click “NextNextNextNext”.


• A “SynchronizationSynchronizationSynchronizationSynchronization” window appears. As of this writing, you can select the “NoneNoneNoneNone” radio button and click “NextNextNextNext”.

• A “ScheduleScheduleScheduleSchedule” window appears. You must have at least one protection (snapshot) schedule defined. Enter a name and your desired schedule/retention and click “NextNextNextNext”.


• A “VolumeVolumeVolumeVolume” window appears. This is where you need to add all of your Oracle volumes to this volume collection. When done, click “FinishFinishFinishFinish”.

Oracle 11gR2 Setup Now that the operating system and Nimble Storage have been setup, we’ll discuss installing Grid Infrastructure and Oracle software.

Pre-requisites

• Create a local directory on the boot LUN or a separate Nimble LUN for the Grid Infrastructure and Oracle software.

• Create group ID of “oinstall”

• Create group ID of “dba”

• Create user ID of “grid”

• Create user ID of “oracle”

Oracle ASM Disks Setup o Run “/etc/init.d/oracleasm configure” command

[root@oraclelinux1 ~]# /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: grid


Default group to own the driver interface []: oinstall Start Oracle ASM library driver on boot (y/n) []: y Scan for Oracle ASM disks on boot (y/n) []: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ]

o Modify the /etc/sysconfig/oracleasm file as follow:

# ORACLEASM_ENABELED: 'true' means to load the driver on boot. ORACLEASM_ENABLED=truetruetruetrue # ORACLEASM_UID: Default user owning the /dev/oracleasm mount point. ORACLEASM_UID=gridgridgridgrid # ORACLEASM_GID: Default group owning the /dev/oracleasm mount point. ORACLEASM_GID=oinstalloinstalloinstalloinstall # ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot. ORACLEASM_SCANBOOT=truetruetruetrue # ORACLEASM_SCANORDER: Matching patterns to order disk scanning ORACLEASM_SCANORDER="dmdmdmdm" # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan ORACLEASM_SCANEXCLUDE="sdsdsdsd"

o Make sure ASM module is loaded. If not, then load it.

[root@oraclelinux1 ~]# lsmod | grep oracleasm oracleasm 53352 1

o Create ASM disks

[root@oraclelinux1 ~]# /etc/init.d/oracleasm creatdisk OCR /dev/mapper/ocr [root@oraclelinux1 ~]# /etc/init.d/oracleasm creatdisk DEVDATA1 /dev/mapper/devdata1 [root@oraclelinux1 ~]# /etc/init.d/oracleasm creatdisk DEVDATA2 /dev/mapper/devdata2

o Verify ASM disks

[root@oraclelinux1 ~]# /etc/init.d/oracleasm listdisks DEVDATA1 DEVDATA2 OCR


Install Grid Infrastructure As grid user, you can now install Grid Infrastructure. Follow Oracle documentation on how to install GI at www.oracle.com. When you get to the part where you have to create ASM disk group for the Clusterware, change the asm_diskstring to “/dev/oracleasm/disks/*” and select OK. Remember that you need to create one disk group for the Clusterware and one disk group for the Oracle database. In this case, for the Clusterware, you can name the disk group as “OCRDG” and select only OCR disk. For the other two disks, create a new disk group and name it “DATADG”. For both disk groups, make sure you select “EXTERNAL REDUNDANCY”. Please note that the naming and number of disks specified in this document is an example only. You can have different names and as many disks as you’d like.

Install Oracle Software After the installation of GI, change user to oracle and install Oracle software. Again, follow Oracle documentation on how to install Oracle if you don’t already know how. It is much easier to select the option to install software only first. After the software installation, run “dbca” to create your database.


Nimble Storage, Inc.

211 River Oaks Parkway, San Jose, CA 95134

Tel: 877-364-6253) | www.nimblestorage.com | [email protected] © 2014 Nimble Storage, Inc. Nimble Storage, InfoSight, SmartStack, NimbleConnect, and CASL are trademarks or registered trademarks of Nimble Storage, Inc. All other trademarks are the property of their respective owners. TR-OG-1114

TR Nimble Oracle Setup Guide - Nimble Storage Add the below settings to the /etc/systctl.conf file....

Documents

Transcript of TR Nimble Oracle Setup Guide - Nimble Storage Add the below settings to the /etc/systctl.conf file....