INSTALLATION OF 11.2.0.2.3 GRID INFRASTRUCTURE FOR CLUSTER IN IRB UPGRADE PRODUCTION

This document covers the details of the tasks involved in setting up Grid Infrastructure for IRB upgrade environment. Servers h20-1-1 and h21-1-1 are procured as new IRB production database servers for IRB application upgradation.

These are HP-UX IA 64-bit servers, same processor architecture of the live production servers.

New production servers are of 11.31 OS version whereas live production servers of then were 11.23 OS version.

Overview of the setup

Setup 11.2.0.2.3 Grid Infrastructure for cluster Setup 11.2.0.2.3 RDBMS software for later upgradation of database Setup 10.2.0.5.2 RDBMS software SRDF copy of live production database to new production servers Bring up ASM and database on 10.2.0.5.2

Grid Infrastructure Setup

Pre-requisites

Initially 10.2.0.5 CRS and RDBMS was setup. Hence initial OS, oracle and network related pre-requisites were met as per 10g requirements.

Later 10g setup was cleaned and a fresh 11gR2 grid installation was setup.

10gR2 pre-requisites for RAC Set up

11gR2 grid-setup pre-requisites

Final pre-requisites that were setup was a combination of 10g and 11g pre-requisites.

Required /etc/hosts entries:

172.20.86.11 h21-1-1172.20.86.12 h20-1-1192.1.1.36 h21-1-1-priv192.1.1.40 h20-1-1-priv172.20.86.22 h20-1-1-vip

172.20.86.21 h21-1-1-vip172.20.86.25 irbscan

Verify the pre-requisites using runcluvfy.sh script:

Installation of Grid Infrastructure

Set up the environment in both the nodes:

mkdir -p /crsapp/app/oracle/product/gridmkdir -p /oracle/app/oracle

chown -R oracle:oinstall /crsapp/appchown -R oracle:oinstall /oracle/app

chmod -R 775 /crsapp/chmod -R 775 /oracle/

unset ORACLE_HOMEunset TNS_ADMINunset ORA_CRS_HOME

rm -ri /var/opt/oraclerm -ri /etc/oratab

Initialize OCR and VOTING disksdd if=/dev/zero of=/dev/rdisk/irbocrvot1 bs=314572800 count=1dd if=/dev/zero of=/dev/rdisk/irbocrvot2 bs=314572800 count=1dd if=/dev/zero of=/dev/rdisk/irbocrvot3 bs=314572800 count=1dd if=/dev/zero of=/dev/rdisk/irbocrvot4 bs=314572800 count=1dd if=/dev/zero of=/dev/rdisk/irbocrvot5 bs=314572800 count=1

Since the setup is a 2-node RAC of h20-1-1 and h21-1-1, choose the installation of grid infrastructure for a cluster.

Advances installation to choose more configuration options like ASM os groups etc.

“irbscan” is the alias host name for the scan IP address in /etc/hosts. DNS configuration was not available in this setup, hence /etc/hosts configuration was used.

Use Add option and add the second node’s information. Here h21-1-1 is the primary node and h20-1-1 is the secondary node. We have chosen the node with smaller IPs as first node and there is no other specific technical reason for this choice.

Add the cluster partner details and test ssh-connectivity.

Choose the interface name and role appropriately as per configuration set up by system admin.

Disks were renamed to match disk group name during the 2nd attempt of reinstallation. This is old screenshot. Actual disk names are /dev/rdisk/irbocrvot1, /dev/rdisk/irbocrvot2, /dev/rdisk/irbocrvot3, /dev/rdisk/irbocrvot4 and /dev/rdisk/irbocrvot5.

At this point before executing root.sh, apply 11.2.0.2.3 PSU to avoid below issue related to Multicast on private interconnect.

It has been found that using a 230.0.1.0, port 42424, network address for multicasting can be problematic with some network configurations. Therefore, Oracle has released Patch: 9974223 on top of Oracle Grid Infrastructure 11.2.0.2. This patch enables multicasting on the 224.0.0.251 multicast address (port 42424) in addition to the 230.0.1.0 (port 42424) address used by default. Multicast must generally be enabled on one of these two addresses to allow Oracle Grid Infrastructure to successfully start on all nodes configured to join a particular cluster.

References:

Oracle Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1]

11.2.0.2.3 PSU Applycd /crsapp/app/oracle/product/gridmv -i OPatch OPtach_oldcopy the patch p6880880_112000_HPUX-IA64.zip here unzip p6880880_112000_HPUX-IA64.zipcd /oracle11g/grid/oracle/product/11.2/OPatchcopy p12419353_112020_HPUX-IA64.zip here

unzip p12419353_112020_HPUX-IA64.zip

It create two directories 12419331 and 12419353

Apply one by one in FIRST node

unset ORACLE_HOMEunset ORACLE_SIDunset ORACLE_BASE

./optach version

./opatch napply -local -oh /crsapp/app/oracle/product/grid -id 12419353 ./opatch napply -local -oh /crsapp/app/oracle/product/grid -id 12419331

Repeat the steps for the other cluster nodes.

After PSU apply gets completed, execute the root scripts as prompted previously by OUI.

Root Scripts Output

root@h21-1-1: /oracle/app # /oracle/app/oraInventory/orainstRoot.shChanging permissions of /oracle/app/oraInventory.Adding read,write permissions for group.Removing read,write,execute permissions for world.

Changing groupname of /oracle/app/oraInventory to oinstall.The execution of the script is complete.

root@h20-1-1: / # /oracle/app/oraInventory/orainstRoot.shChanging permissions of /oracle/app/oraInventory.Adding read,write permissions for group.Removing read,write,execute permissions for world.

Changing groupname of /oracle/app/oraInventory to oinstall.The execution of the script is complete.

root@h21-1-1: /oracle/app # /crsapp/app/oracle/product/grid/root.shRunning Oracle 11g root script...

The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /crsapp/app/oracle/product/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:

Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root script.Now product-specific root actions will be performed.Using configuration parameter file: /crsapp/app/oracle/product/grid/crs/install/crsconfig_paramsCreating trace directoryLOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'sys'..Operation successful.OLR initialization - successful root wallet root wallet cert

root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user certAdding daemon to inittabCRS-2672: Attempting to start 'ora.mdnsd' on 'h21-1-1'CRS-2676: Start of 'ora.mdnsd' on 'h21-1-1' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'h21-1-1'CRS-2676: Start of 'ora.gpnpd' on 'h21-1-1' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'h21-1-1'CRS-2672: Attempting to start 'ora.gipcd' on 'h21-1-1'CRS-2676: Start of 'ora.gipcd' on 'h21-1-1' succeededCRS-2676: Start of 'ora.cssdmonitor' on 'h21-1-1' succeededCRS-2672: Attempting to start 'ora.cssd' on 'h21-1-1'CRS-2672: Attempting to start 'ora.diskmon' on 'h21-1-1'CRS-2676: Start of 'ora.diskmon' on 'h21-1-1' succeededCRS-2676: Start of 'ora.cssd' on 'h21-1-1' succeeded

ASM created and started successfully.

Disk Group IRBOCRVOT created successfully.

clscfg: -install mode specifiedSuccessfully accumulated necessary OCR keys.Creating OCR keys for user 'root', privgrp 'sys'..Operation successful.CRS-4256: Updating the profileSuccessful addition of voting disk 7346c784a43f4fefbf5c92be875095ec.

Successful addition of voting disk a09b8777edde4f15bff538cc055937a2.Successful addition of voting disk 1dc99ac086054fcfbff96b1d79da4640.Successfully replaced voting disk group with +IRBOCRVOT.CRS-4256: Updating the profileCRS-4266: Voting file(s) successfully replaced## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 7346c784a43f4fefbf5c92be875095ec (/dev/rdisk/irbocrvot1) [IRBOCRVOT] 2. ONLINE a09b8777edde4f15bff538cc055937a2 (/dev/rdisk/irbocrvot2) [IRBOCRVOT] 3. ONLINE 1dc99ac086054fcfbff96b1d79da4640 (/dev/rdisk/irbocrvot3) [IRBOCRVOT]Located 3 voting disk(s).

CRS-2672: Attempting to start 'ora.asm' on 'h21-1-1'CRS-2676: Start of 'ora.asm' on 'h21-1-1' succeededCRS-2672: Attempting to start 'ora.IRBOCRVOT.dg' on 'h21-1-1'CRS-2676: Start of 'ora.IRBOCRVOT.dg' on 'h21-1-1' succeededConfigure Oracle Grid Infrastructure for a Cluster ... succeededroot@h21-1-1: /oracle/app # root@h21-1-1: /oracle/app # /crsapp/app/oracle/product/grid/bin/ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2096 Available space (kbytes) : 260024 ID : 139195598 Device/File Name : +IRBOCRVOT Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

root@h20-1-1: /var/opt/oracle # /crsapp/app/oracle/product/grid/root.shRunning Oracle 11g root script...

The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /crsapp/app/oracle/product/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite.The contents of "oraenv" have not changed. No need to overwrite.The contents of "coraenv" have not changed. No need to overwrite.

Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root script.Now product-specific root actions will be performed.Using configuration parameter file: /crsapp/app/oracle/product/grid/crs/install/crsconfig_paramsCreating trace directoryLOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'sys'..Operation successful.OLR initialization - successfulAdding daemon to inittabCRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node h21-1-1, number 1, and is terminatingAn active cluster was found during exclusive startup, restarting to join the clusterConfigure Oracle Grid Infrastructure for a Cluster ... succeededroot@h20-1-1: /var/opt/oracle # root@h20-1-1: /var/opt/oracle # root@h20-1-1: /var/opt/oracle # /crsapp/app/oracle/product/grid/bin/ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2096 Available space (kbytes) : 260024 ID : 139195598 Device/File Name : +IRBOCRVOT Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

oracle@h21-1-1: /home/oracle # ps -ef|grep d.bin root 17432 1 0 21:02:10 ? 0:27 /crsapp/app/oracle/product/grid/bin/ohasd.bin reboot

root 18103 1 0 21:05:21 ? 0:49 /crsapp/app/oracle/product/grid/bin/orarootagent.bin oracle 18111 1 0 21:05:21 ? 0:00 /crsapp/app/oracle/product/grid/bin/diskmon.bin -d -f root 18318 1 0 21:07:20 ? 0:17 /crsapp/app/oracle/product/grid/bin/crsd.bin reboot oracle 18083 1 0 21:05:18 ? 0:00 /crsapp/app/oracle/product/grid/bin/gpnpd.bin oracle 18095 1 0 21:05:20 ? 0:19 /crsapp/app/oracle/product/grid/bin/gipcd.bin oracle 18107 1 0 21:05:21 ? 0:16 /crsapp/app/oracle/product/grid/bin/ocssd.bin root 18101 1 0 21:05:21 ? 0:01 /crsapp/app/oracle/product/grid/bin/cssdagent oracle 18571 1 0 21:08:29 ? 0:00 /crsapp/app/oracle/product/grid/bin/tnslsnr LISTENER_SCAN1 -inherit root 18093 1 0 21:05:20 ? 0:01 /crsapp/app/oracle/product/grid/bin/cssdmonitor oracle 20485 6510 0 21:29:04 pts/1 0:00 grep d.bin oracle 18072 1 0 21:05:16 ? 0:16 /crsapp/app/oracle/product/grid/bin/oraagent.bin oracle 18358 18327 0 21:07:24 ? 0:00 /crsapp/app/oracle/product/grid/bin/evmlogger.bin -o /crsapp/app/oracle/product/grid/evm/log/evmlogger.info -l /crsapp/app/orac root 18138 1 0 21:06:57 ? 0:07 /crsapp/app/oracle/product/grid/bin/octssd.bin oracle 18327 1 0 21:07:22 ? 0:07 /crsapp/app/oracle/product/grid/bin/evmd.bin oracle 18076 1 0 21:05:16 ? 0:00 /crsapp/app/oracle/product/grid/bin/mdnsd.bin root 18514 1 0 21:08:21 ? 0:13 /crsapp/app/oracle/product/grid/bin/orarootagent.bin oracle 18463 1 0 21:08:09 ? 0:04 /crsapp/app/oracle/product/grid/bin/oraagent.bin oracle 19243 1 0 21:20:31 ? 0:00 /crsapp/app/oracle/product/grid/bin/tnslsnr LISTENER -inherit oracle 18582 1 0 21:08:33 ? 0:00 /crsapp/app/oracle/product/grid/bin/scriptagent.binoracle@h21-1-1: /home/oracle # ps -ef|grep lsn oracle 18571 1 0 21:08:29 ? 0:00 /crsapp/app/oracle/product/grid/bin/tnslsnr LISTENER_SCAN1 -inherit oracle 20489 6510 0 21:29:16 pts/1 0:00 grep lsn oracle 19243 1 0 21:20:31 ? 0:00 /crsapp/app/oracle/product/grid/bin/tnslsnr LISTENER –inherit

root@h20-1-1: /var/opt/oracle # root@h20-1-1: /var/opt/oracle # root@h20-1-1: /var/opt/oracle # /crsapp/app/oracle/product/grid/bin/crsctl stat res -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.IRBOCRVOT.dg ONLINE ONLINE h20-1-1 ONLINE ONLINE h21-1-1 ora.asm ONLINE ONLINE h20-1-1 ONLINE ONLINE h21-1-1 Started ora.gsd OFFLINE OFFLINE h20-1-1 OFFLINE OFFLINE h21-1-1 ora.net1.network

ONLINE ONLINE h20-1-1 ONLINE ONLINE h21-1-1 ora.ons ONLINE ONLINE h20-1-1 ONLINE ONLINE h21-1-1 --------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE h21-1-1 ora.cvu 1 ONLINE ONLINE h21-1-1 ora.h20-1-1.vip 1 ONLINE ONLINE h20-1-1 ora.h21-1-1.vip 1 ONLINE ONLINE h21-1-1 ora.oc4j 1 ONLINE ONLINE h21-1-1 ora.scan1.vip 1 ONLINE ONLINE h21-1-1