Presenter: Nikhil Kumar RAC- Installing your First Cluster and Database

WHO AM I ? Nikhil Kumar (DBA Manager) 6 Years of Experience in Oracle Databases and Apps. Oracle Certified Professional Oracle 9i and 11g. Worked on Mission critical Telecom, Financial ERP, Manufacturing and Government Domain.

Agenda Introduction of RAC Installation of Clusterware. Creating diskgroup / Adding disk to Diskgroup using ASMCA. Creation of ACFS Volume. Installation of RAC Database using DBCA.

Introduction of RAC A medium to provide high availability to database. Why RAC? High availability and scalability without any limitation:- OS Patching or Schedule bounce of OS. Database maintenance patch(CPU or PSU) . Static database parameter change (Due to bug or any requirement by system). Hardware upgrade or change. Harddisk failure, power failure or system failure. Prevention from Single point of failure?

Identity`Home Node Host NodeGiven NameTypeAddressAddress Assigned By Address Resolved By Node 1 Public Node 1racnode1racnode1Public192.168.7.71FixedDNS Node 1 VIP Node 1Selected by Oracle Clusterware racnode1-vipVirtual192.168.7.41FixedDNS and hosts file Node 1 Private Node 1racnode1racnode1-privPrivate192.168.71.40FixedDNS and hosts file, or none Node 2 Public Node 2racnode2racnode2Public192.168.7.72FixedDNS Node 2 VIP Node 2Selected by Oracle Clusterware racnode2-vipVirtual192.168.7.41FixedDNS and hosts file Node 2 Private Node 2racnode2racnode2-privPrivate192.168.71.41FixedDNS and hosts file, or none SCANNoneSelected by Oracle Clusterware Racnode.linuxdc.comVirtual192.168.7.43 192.168.7.44 192.168.7.45 FixedDNS Network Configuration: For racnode1 and racnode2 Note : Manually assigning the proper IPs in /etc/host file is mandatory. Even if it resolved through DNS. This is Oracle Requirement.

Cluster Overview Two Node cluster Operating System version RHEL 6.4 Cluster and database software version 11.2.0.4.0 Cluster Name: NIOUG Raw Disk size -- 10 Luns Diskgroups (Data,FRA,OCR) Creation of empty NIOUG database using DBCA.

Prerequisite Prerequisite to followed by System/Network Admin before delivering the server to DBA. 1.

Prerequisite Cont.. 2. Verify that SELinux is running and set to ENFORCING: As the root user, # getenforce Enforcing If the system is running in PERMISSIVE or DISABLED mode, modify the /etc/sysconfig/selinux file and set SELinux to enforcing as shown below. SELINUX=enforcing The modification of the /etc/sysconfig/selinux file takes effect after a reboot. To change the setting of SELinux immediately without a reboot, run the following command: # setenforce 1

Prerequisite Cont.. 3. Need to upgrade selinux-policy rpm to make SELINUX work current version of RPM Deliver with RHEL 6.4 [root@STGW2 ~]# rpm -qa selinux-policy* selinux-policy-3.7.19-195.el6.noarch selinux-policy-targeted-3.7.19-195.el6.noarch Need to upgrade with below mentioned package:- [root@racnode1 ~]# rpm -qa selinux-policy* selinux-policy-3.7.19-231.el6.noarch selinux-policy-targeted-3.7.19-231.el6.noarch

Prerequisite Cont.. 4. Make sure the shared memory file system is big enough for Automatic Memory Manager to work. EXAMPLES: # umount tmpfs # mount -t tmpfs tmpfs -o size=12g /dev/shm ( size is based upon 90% of physical memory) Make the setting permanent by amending the "tmpfs" setting of the "/etc/fstab" file to look like this. tmpfs /dev/shm tmpfs defaults,size=12g 0 0

Prerequisite Cont.. 5. Put the below entry in /etc/hosts of both node [root@racnode1 bin]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.7.71 racnode1 192.168.7.72 racnode2 192.168.71.40 racnode1-priv 192.168.71.41 racnode2-priv 192.168.7.41 racnode1-vip 192.168.7.42 racnode2-vip

Prerequisite Cont.. 6. Kernel Parameters: Add the kernel parameters in /etc/sysctl.conf file. (Apply it using command sysctl -p /etc/sysctl.conf) kernel.shmall = shmmax / 4096 kernel.shmmax= 90% of physical memory net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 fs.aio-max-nr= 3145728 fs.file-max = 6815744 kernel.msgmax = 8192 kernel.msgmnb= 65536 kernel.msgmni = 2878 kernel.sem = 250 32000 100 142 kernel.shmall = 2097152 kernel.shmmax= 7730941132 kernel.sysrq= 1 net.core.rmem_default=4194304 net.core.rmem_max=4194304 net.core.wmem_default=262144 net.core.wmem_max=1048576 net.ipv4.ip_local_port_range=9000 65500

Prerequisite Cont.. 7. Adding Groups and users: #groupadd -g 2011 asmdba #groupadd -g 2012 asmadmin #groupadd -g 2013 asmoper #groupadd -g 2014 oper #groupadd g 2015 oinstall #groupadd g 2016 dba #useradd -s /bin/bash -d /home/grid -g oinstall -G asmdba,asmadmin,asmoper,dba grid #useradd -s /bin/bash -d /home/oracle -g oinstall -G asmdba,asmadmin,asmoper,dba oracle #usermod -a -G asmdba,oper oracle For example: # id grid uid=3010(grid) gid=2004(oinstall) groups=2000(dba),2004(oinstall),2011(asmdba),2012(asmadmin),2013(asmoper) #id oracle uid=3000(oracle) gid=2004(oinstall) groups=2000(dba),2004(oinstall),2011(asmdba),2014(oper)

Prerequisite Cont.. 8. Creating the Oracle Base directory mkdir -p /u01/app/11.2.0/grid mkdir -p /u01/app/grid chown -R grid:oinstall /u01 chmod -R 775 /u01 mkdir -p /u01/app/oracle chown oracle:oinstall /u01/app/oracle

Prerequisite Cont.. 9. Network Time Protocol Setting: If you are using NTP, you must add the "-x" option into the following line in the "/etc/sysconfig/ntpd" file. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" Then restart NTP. # chkconfig --level 2345 ntpd on Start the Name Service Cache Daemon (nscd). # chkconfig --level 2345 nscd on # service nscd start

Prerequisite Cont.. 10. Setting Resource Limits Oracle users: On each node, add the following lines to the /etc/security/limits.conf file (the following example shows the software account owners oracle and grid): cat /etc/security/limits.conf oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 oracle soft stack 10240 oracle hard stack 32768 grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 grid soft stack 10240 grid hard stack 32768

Prerequisite Cont.. 11. Setting login file: As the root user, create a backup of /etc/pam.d/login # cp /etc/pam.d/login /etc/pam.d/login.bkup As the root user, add the following line within the /etc/pam.d/login file session required pam_limits.so 12 .To install and configure ASMLib software packages: 1. Download the ASMLib packages to each node in your cluster. 2. Change to the directory where the package files were downloaded. 3. As the root user, use the rpm command to install the packages. For example: # rpm -Uvh kmod-oracleasm # rpm -Uvh oracleasmlib-2.0.4-1.el6.x86_64.rpm # rpm -Uvh oracleasm-support-2.1.8-1.el6.x86_64.rpm

Prerequisite Cont.. After you have completed these commands, ASMLib is installed on the system. 4.Repeat steps 2 and 3 on each node in your cluster. Configuring asmlib: a.) /usr/sbin/oracleasm configure -i (as root user run on all the nodes) b.) oracleasm init (Load and initialize the ASMLib driver) Load the kernel module using the following command. # /usr/sbin/oracleasm init Loading module "oracleasm": oracleasm Mounting ASMlib driver filesystem: /dev/oracleasm

Prerequisite Cont.. Using ASMLib to Create ASM Disks c.) createdisk (only on the first node) # /usr/sbin/oracleasm createdisk disk_name device_partition_name Mark the five shared disks as follows. # /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1 Writing disk header: done Instantiating disk: done If you need to unmark a disk that was used in a createdisk command, you can use the following syntax: # /usr/sbin/oracleasm deletedisk disk_name

Prerequisite Cont.. d.) oracleasm scandisks ( on all the nodes) It is unnecessary, but we can run the "scandisks" command to refresh the ASM disk configuration. # /usr/sbin/oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... e.) oracleasm listdisks We can see the disk are now visible to ASM using the "listdisks" command. # /usr/sbin/oracleasm listdisks

Prerequisite Cont.. 13. Ping to check the communication between each node in cluster ping racnode1 ping racnode2 ping racnode1-priv ping racnode2-priv Run the Cluvfy to check the prerequisite of cluster installation. (Run from Grid user) /software/grid/runcluvfy.sh stage -pre crsinst -n racnode1,racnode2 verbose

Prerequisite Cont.. 14. SCAN name should be configured by network admin before starting the installation. SCAN can be verified by two ways:- # host scan_name (It should show 3 ip address) # nslookup scan_name (Run this command 2-3 times IP should interchange)

Installation of Grid Infrastructure Clusterware

Go to /software_directory/grid run this on grid user ./runInstaller

Download Software Updates

Select the "Install and Configure Grid Infrastructure for a Cluster" option, then click the "Next" button.

Select the "Advanced Installation" option, then click the "Next" button.

Select Product Languages

Specify Cluster and SCAN name information, click the Next" button.

Enter the details of the second node in the cluster, then click the "OK" button.

Provide password of grid to configure SSH

Click on setup tab to initiate the SSH configuration between the nodes.

Check the Network interface and its segment

Select ASM for storage

Choose asm disk to create diskgroup

Provide password for ASM account

Skipping IPMI, Since we dont have hardware to support this feature

Group information for ASM

Specify directory for Clusterware files

Specify the directory for central inventory

Prerequisite check is being performed

Result from prerequisite check

Ignoring some prerequisite checks

Click on Install to init

Installation is in process

Run root.sh scripts one at a time on the node

Run root.sh scripts one at a time on the node

Setting permission for orainventory

Running root.sh on first node of the cluster

root.sh on node one is complete

root.sh on node2 complete

Go back to OUI screen and click OK

Check Clusterware services on both nodes.

Grid Clusterware installation is complete

Creating Diskgroup using ASMCA

Invoking ASMCA Utility on grid user:-

Create new DATA diskgroup

Click OK

DATA diskgroup is created

Creating ACFS Volume

Create archive volume using FRA diskgroup

After selecting size click OK

Volume archive created

Now click ASM Cluster File System Tab

Now after clicking on Create tab, Select Volume Archive Which we created earlier. Provide the input to General Purpose File System Which will be mount on the Operating System.

/archive mountpoint of OS is created

Status of ACFS mount point

Check the /archive mountpoint on both node

Installation of Oracle Binaries & Database Creation

Invoke runInstaller from Oracle user

Skipping Software Update

Use Create and configure database option to install database binaries and dummy database

Select Server Class type for installation

Select both node for installation

Establish SSH connectivity for oracle user

Select Advance install type installation

Select default English Language

Select Enterprise Edition for database

Define directory structure for database binaries

Select the type of database

Provide input to Global database name and SID

Provide memory to Instance

Provide password for ASM

Skipping the backup part

Select diskgroup where database files needs to be placed

Provide password to admin account of database

Group information

Prerequisite Check Complete

Prerequisite Check Complete

Click Install to initiate the installation

Installation is in process

Database creation is in process

Click Ok

Run root.sh script on both node of cluster

Running root.sh on database server nodes

Software installation and software creation is done

Checking Database Resource