Installation RAC10GR2 QFS[1]
-
Upload
vinothkumar -
Category
Documents
-
view
361 -
download
9
Transcript of Installation RAC10GR2 QFS[1]
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
SUN Microsystems GmbH Oracle RAC on Sun Cluster and QFS
Author: Michael AhrensSUN Microsystems GmbH
Created 07.06.2006 12:00:00
Last change: 09/12/2006 10:31:28 vorm.
Version: 0.1
Page 1 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Table of ContentsTable of Contents......................................................................................................................................................2Document Info...........................................................................................................................................................4
Content..............................................................................................................................................................4Hardware Installation................................................................................................................................................5
Architecture.......................................................................................................................................................5Drawing.............................................................................................................................................................5
Solaris Installation.....................................................................................................................................................6Solaris 10 u1......................................................................................................................................................6Setup slvm.........................................................................................................................................................6EIS-CD..............................................................................................................................................................7QFS 4.5..............................................................................................................................................................7
Cluster Installation....................................................................................................................................................8SunCluster 3.1U4..............................................................................................................................................8Setup IPMP groups on all nodes for all public network interfaces.................................................................11Modify NVRAM parameter:...........................................................................................................................12Create Users and Groups.................................................................................................................................12Check SSH daemon.........................................................................................................................................12Setup SSH for oracle.......................................................................................................................................12Setup userenv for oracle..................................................................................................................................13Install the RAC/OPS Service from the CD.....................................................................................................14Install udlm......................................................................................................................................................14Workaround Bug.............................................................................................................................................15Set kernel parameters......................................................................................................................................17Create cluster resourcegroup for rac...............................................................................................................17Create multi-owner disk set.............................................................................................................................18Configure QFS software on all nodes.............................................................................................................19Create global mountpoint and vfstab entries on all nodes..............................................................................20Create cluster resourcegroup for qfs...............................................................................................................20Last Systemanalyse.........................................................................................................................................22
Prepare Oracle Installation......................................................................................................................................23Share staging area............................................................................................................................................23Mount staging area..........................................................................................................................................23Perform cluster check......................................................................................................................................23
Install Clusterware 10.2.0.1.....................................................................................................................................24Run installer from ora-rac-3............................................................................................................................24Running vipca.................................................................................................................................................34Check Cluster Status.......................................................................................................................................40Install crsstat.sh...............................................................................................................................................41
Install Database Home 10.2.0.1..............................................................................................................................42Run installer from ora-rac-3............................................................................................................................42
Install Management Agent......................................................................................................................................51Run installer from ora-rac-3............................................................................................................................51
Install Patchset 10.2.0.2 on CRS Home..................................................................................................................60Stop all oracle processes.................................................................................................................................60Install binaries.................................................................................................................................................60Restart Custerware..........................................................................................................................................65
Install Patchset 10.2.0.2 on DB Home....................................................................................................................66Install binaries.................................................................................................................................................66
Install Interim Patch 5117016 on DB Home...........................................................................................................71Install with opatch...........................................................................................................................................71
Create Database ZDA..............................................................................................................................................74Run DBCA......................................................................................................................................................74
Cluster Optimiziation..............................................................................................................................................78Apply Brownout Patches.................................................................................................................................78Set the CSS misscount to match the service level...........................................................................................78Run CSS in Real Time....................................................................................................................................78Set the check_interval for VIP addresses........................................................................................................78
Optimize VIP Failover............................................................................................................................................80Adjusting Listener Configuration....................................................................................................................80
Page 2 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Addendum...............................................................................................................................................................82Metalink Note 284752.1..................................................................................................................................82Metalink Note 294430.1..................................................................................................................................83Metalink Note 294430.1..................................................................................................................................89
Page 3 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Document Info
Content
This is an installation summary of the installation and patch process of the RAC environment. It is based on the official procuct documentation provided by SUN and Oracle and cannot replace those doc-uments. The OS and Clusterinstallation was strongly based on EIS done.
Page 4 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Hardware Installation
Architecture
The cluster is based on two SF440 with 4 CPU's, 16GB RAM and 4 internal Disks. The internaldisks are mirrored with SLVM. Two disks are used for the OS and the other two are used for Oracle. Oracle will not be installed on shared disks because of the rolling upgrade capability. The shared disks are luns from a SAN and they are mirrored with SLVM/OBAN, filesystem is QFS 4.5. The clusterinterconnect is realized with four switched gigabitethernet connections.
Drawing
Page 5 of 90
4 x Interconnect
Storage A
Oracle RAC Cluster Node: ora-rac-3
Oracle RAC ClusterNode: ora-rac-4
Storage B
IPMP Groupipmp0
IPMP Groupipmp0
MpxIO MpxIO
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Solaris Installation
Solaris 10 u1
Solaris 10 u1 SUNWCXall is installed.
• Do not forget the 512Mb (SC3.1 strikly recommended) partition for /globaldevices
Mirrored by SLVM, on every clusternode another metadevicename for this filesystem must be used.
ora-rac-3 d30
ora-rac-4 d35
• For SVM reserve slice 7 with 32MB (EIS recommendation) for metaDB/replicas.
• Ensure that the SAN Foundation Suite packages are installed and patched. (SAN is necessary for fibre server/storage)
change the home of root
#mkdir /root
#vi /etc/passwd
root:x:0:0:Super-User:/root:/sbin/sh
logout,login
Setup slvm
#cat /etc/lvm/md.tab
mddb1 -c 3 /dev/dsk/c3t0d0s7
d10 -m d11
d11 1 1 /dev/dsk/c3t0d0s0
d12 1 1 /dev/dsk/c3t1d0s0
d20 -m d21
d21 1 1 /dev/dsk/c3t0d0s1
d22 1 1 /dev/dsk/c3t1d0s1
d30 -m d31
d31 1 1 /dev/dsk/c3t0d0s6
d32 1 1 /dev/dsk/c3t1d0s6
Page 6 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
EIS-CD
Setup EIS environment
cd /cdrom/eis-dvd/sun/install
./setup-standard.sh
Install actual Solaris 10 patches
cd /cdrom/eis-dvd/patch/10
unpack-patches
cd /tmp/10
./install-patches
Analyse basic systemstatus
explorer
cd /opt/SUNWexplo/output/explorer*
/cdrom/eis-dvd/sun/tools/explominer/explominer
see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and in-stall all neccessary patches.
QFS 4.5
cd /tmp/qfs_packages
pkgadd -d .
choose both qfs packages
install Patch: 122807-04
Page 7 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Cluster Installation
SunCluster 3.1U4
Configure /etc/inet/hosts.
127.0.0.1 localhost192.168.10.238 ora-rac-4-1192.168.10.237 ora-rac-4-2192.168.10.215 ora-rac-3172.16.193.1 ora-rac-3-priv192.168.10.223 ora-rac-3-vip192.168.10.217 ora-rac-4 loghost ora-rac-4.172.16.193.2 ora-rac-4-priv192.168.10.239 ora-rac-4-vip
Additionally for Solaris10 configure /etc/inet/ipnodes
Enter all cluster nodes and all the logical Hosts.
Do not configure cluster nodes as routers!
Set up file /etc/defaultrouter scinstall will touch file /etc/notrouter per default.
defaultrouter = 192.168.10.254
When using supported network adapters which use the *ce* driver for private transport, insert into file /etc/system:
set ce:ce_taskq_disable=1
To enable MpxIO on Solaris 10 start
stmsboot -e before SunCluster will be installed !!
Do not run this command if SunCluster is already installed.
For safety reasons clean the devices with:
devfsadm -C on all nodes.
Notice: For use of VLANs for the cluster transport:
• one separated VLAN per cluster transport,
• no network traffic should be possible between the different cluster transport VLANs,
• VLANs should behave like separated physical segments.
Page 8 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Configure
#ndd -set /dev/ip ip_strict_dst_multihoming 0
You should do this in a startscript. You can use S68nettune from EIS-DVD.
See also SRDB 73036
Copy the SunCluster Software from the EIS-DVD to the disk and bunzip and untar this file. /cdrom/eisdvd/suninternal/progs/SunCluster/3.1_U4/suncluster_sc31u4_sparc_EIS.tar.bz2
Edit XML-File and remove the Explorer entries, because within the software is a very old version of the explorer.
vi ./suncluster_sc31u4_sparc/Solaris_sparc/.install/config/PPXMLS/SunCluster.xml
Remove following entries:
<forward_dependency id="Explorer" base_point_product_id="Explorer" topology_category="Host-Based" subcomp_refs="Explorer">
<description>"Explorer is required for NC to be installed"</description>
<lowest_compat_version_noupgrade>
<versionInfo versionNo="4.2"/>
</lowest_compat_version_noupgrade>
</forward_dependency>
Explorer,
Use the Java ES installer for package installation.
(Optional): To use the GUI of the installer
be sure that the DISPLAY is set correctly.
# ./installer
do NOT install JES Agents at this time !
Choose 'Configure Later'
'Configure Now' does not work this time!
Install/Establish SunCluster 3.1U4 on ora-rac-3 using scinstall.
Select menu 1)
Interactive Q+A
(Partition for /globaldevices required.)
If you not install package SUNWescom disable the scsymon service:
# svcadm -v disable /system/cluster/scsymon-srv
Page 9 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install Sun Cluster patches: Note the structure under .../sun2/patch/SunCluster3.x/ ..3.1 (containing sub-directories, 8, 9, 10, 9_x86, 10_x86) ..agents (containing sub-directories common, 8, 9, 10, 10_x86)
the following core patch is required for SC3.1U4:
• 120500-04 Sun Cluster 3.1 Core Patch for Solaris 10
Page 10 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Setup IPMP groups on all nodes for all public network interfaces.
At least you need one test IP for each network adapter and one production IP for each IPMP group. e.g: IPMP group active-active with interfaces ce0 and ce4 with one physical IP.
ora-rac-3
Entry of /etc/hostname.ce0 :
ora-rac-3-1 netmask + broadcast + group ipmp0 up \
addif ora-rac-3 netmask + broadcast + deprecated -failover up
Entry of /etc/hostname.ce4 :
ora-rac-3-2 netmask + broadcast + group ipmp0 deprecated -failover up
ora-rac-4
Entry of /etc/hostname.ce0 :
ora-rac-4-1 netmask + broadcast + group ipmp0 up \addif ora-rac-3 netmask + broadcast + deprecated -failover up
Entry of /etc/hostname.ce4 :
ora-rac-4-2 netmask + broadcast + group ipmp0 deprecated -failover up
If the defaultrouter is NOT 100% available please read infodocs
70060 and 70062.
Notes:
• Do not use test IP for normal applications
• Test IP for all adapters in the same IPMP group must belong to a single IP subnet
• You need an addtional IP for each logical host.
• In the /etc/default/mpathd file, the value of
TRACK_INTERFACES_ONLY_WITH_GROUPS must be yes. (default)
Common actions for filesystems and boot devices:
Set localonly flag for the did device of the root mirror disk:
• make sure only local node in node list:
# scconf -pvv | grep dsk/d<N>• if other nodes in node list, remove them:# scconf -r -D name=dsk/d<N>,nodelist=<other_phy_host>• set localonly on root mirror disk:# scconf -c -D name=dsk/d<N>,localonly=true
NOTE: To find out the did device use scdidadm -l command!
Enter global filesystems into /etc/vfstab on all nodes. Recommendation:
mount archivelog destinaton FS as an QFS Filesystem for each instance not as an global filesystem.
Page 11 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
See later in this document for qfs. Create all mountpoints on all nodes.
Add global and logging option in /etc/vfstab if you use UFS.
Modify NVRAM parameter:
Set boot-device & diag-device to both sides of the mirror.
Suggested naming convention:
• Primary: rootdisk
• Secondary: rootmirror
Create Users and Groups
To be done on all node of the cluster
/usr/sbin/groupadd -g 61000 oinstall/usr/sbin/groupadd -g 61001 dba/usr/sbin/useradd -u 61000 -g oinstall -G dba -s /usr/bin/bash -d /export/home/oracle oracle
mkdir -p /export/home/oraclechown -R oracle:oinstall /export/home/oracle
passwd oracle> oracle> oracle
Check SSH daemon
ps -ef | grep sshd
Setup SSH for oracle
Note: This steps are different for any node in the cluster. Do complete the steps on the indicated nodes.
Steps to be done on ora-rac-3 and ora-rac-4 :
su - oracle
mkdir ~/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa
Steps to be done host on ora-rac-3:
cd .sshcat id_dsa.pub > authorized_keyscat id_rsa.pub >> authorized_keys
scp authorized_keys ora-rac-4:/export/home/oracle/.ssh/authorized_keys
Page 12 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Steps to be done host on ora-rac-4:
cat id_dsa.pub >> authorized_keyscat id_rsa.pub >> authorized_keys
Steps to be done host on ora-rac-3:
scp ora-rac-4:/export/home/oracle/.ssh/authorized_keys authorized_keys
Steps to be done on ora-rac-3 and ora-rac-4:
ssh ora-rac-3 datessh ora-rac-4 date
exec /usr/bin/ssh-agent $SHELL/usr/bin/ssh-add
Setup userenv for oracle
# add to ~/.profile
. ~/set_oracle_env.sh
PS1='${LOGNAME}:${ORACLE_SID}> 'export PS1
# Set Environment for oracle
#!/usr/bin/bash
clear
echo "Choose your Oracle Home and hit RETURN"
echo "Enter 1 to use Oracle Home DB"
echo "Enter 2 to use Oracle Home Agent"
echo "Enter 3 to use Oracle Home CRS"
echo "Hit any other key to skip Oracle Home selection"
read answer
if [[ $answer = 1 ]]
then
ORACLE_HOME=/oracle/product/db ;
ORACLE_SID=ZDA1
elif [[ $answer = 2 ]]
then
ORACLE_HOME=/oracle/product/agent/agent10g ;
elif [[ $answer = 3 ]]
then
ORACLE_HOME=/oracle/product/crs ;
else
return
fi
export PATH=$ORACLE_HOME/bin:/oracle/Opatches/OPatch:$PATH
ORACLE_BASE=/oracle
export ORACLE_BASE
Page 13 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
export ORACLE_SID
export ORACLE_HOME
EDITOR=vi
export EDITOR
if [ -t 0 ]; then
stty intr ^C
fi
# stty erase ^?
alias sql='sqlplus / as sysdba'
alias crsi='/oracle/product/crs/bin/crs_stat -t'
alias xxx=". ~/set_oracle_env.sh"
alias oh='cd $ORACLE_HOME'
alias al='vi /oracle/admin/ZDA/bdump/alert_${ORACLE_SID}.log'
echo "Tip: Enter xxx to change your current setting"
Install the RAC/OPS Service from the CD
Via pkgadd on all nodes. You always need SUNWscucm SUNWudlm SUNWudlmr.
• For SVM you require SUNWscmd.
• SUNWschwr is needed for Shared QFS for Hardware Raid Support.
Install udlm
Remark: Don't use the version from the EIS DVD, for Oracle 10G the asynchronous version is re-quired, on the EIS DVD is only the synchronous version available. See chapter „Prepair Oracle Installa-tion“ for the Installationmedias.
#cd /install/oracle/stage/cl/racpatch
#uncompress ORCLudlm.tar.Z;tar xf ORCLudlm.tar
#pkgadd -d .
#The following packages are available:
1 ORCLudlm Oracle UNIX Distributed Lock Manager
(sparc) Dev Release 06/11/04, 64bit 3.3.4.8 reentrant, async libskgxn2.so
Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:
Page 14 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Workaround Bug
The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin.
mkdir -p /usr/local/bin
ln -s /usr/bin/ssh /usr/local/bin/ssh
ln -s /usr/bin/scp /usr/local/bin/scp
If you use Solaris Secure Shell (sshd) with enabled X11 forwarding and udlm you should read Sun Al-ert 57428.
Configure 'udlm.port' and 'udlm.num_ports' in /opt/SUNWudlm/etc/udlm.conf
# Modified by rac_udlm_init on Tue May 9 09:14:16 CEST 2006
#
# ident "@(#)udlm.conf.template 1.5 03/04/15 SMI"
#
# Copyright 1999-2003 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
# This file has been created by SUNW.rac_udlm resource type
# callback programs.
#
# Please do not edit this file.
#
udlm.oracle.config.file : /etc/opt/SUNWcluster/conf/udlm.conf
udlm.schedclass : RT
udlm.schedpriority : 11
udlm.port : 7000
udlm.num_ports : 32
udlm.failfastmode : panic
udlm.start_timeout : 100
udlm.step1_timeout : 100
udlm.step2_timeout : 100
udlm.step3_timeout : 100
udlm.step4_timeout : 100
udlm.step5_timeout : 100
udlm.abort_timeout : 325
#
udlm.resource_name : rac_udlm_rs
udlm.resource_group_name : rac_rg
#
# Cluster.number.nodes should be obtained dynamically
#
Page 15 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
udlm.nodes : ${cluster.number.nodes}
#
#
udlm.clustname : boschpoc
udlm.server.0.hname : 172.16.193.1
udlm.server.1.hname : 172.16.193.2
udlm.server.2.hname : ${cluster.node.2.hahost}
udlm.server.3.hname : ${cluster.node.3.hahost}
udlm.server.4.hname : ${cluster.node.4.hahost}
udlm.server.5.hname : ${cluster.node.5.hahost}
udlm.server.6.hname : ${cluster.node.6.hahost}
udlm.server.7.hname : ${cluster.node.7.hahost}
udlm.server.8.hname : ${cluster.node.8.hahost}
udlm.server.9.hname : ${cluster.node.9.hahost}
udlm.server.10.hname : ${cluster.node.10.hahost}
udlm.server.11.hname : ${cluster.node.11.hahost}
udlm.server.12.hname : ${cluster.node.12.hahost}
udlm.server.13.hname : ${cluster.node.13.hahost}
udlm.server.14.hname : ${cluster.node.14.hahost}
udlm.server.15.hname : ${cluster.node.15.hahost}
#
# Relate CMM steps to UDLM steps
#
# Format:
# ucmm.timeout.<stepname>.<component order> : <value>
#
#
ucmm.timeout.cmmstep2.05 : ${udlm.step1_timeout}
ucmm.timeout.cmmstep4.05 : ${udlm.step2_timeout}
ucmm.timeout.cmmstep5.05 : ${udlm.step3_timeout}
ucmm.timeout.cmmstep6.05 : ${udlm.step4_timeout}
ucmm.timeout.cmmstep7.05 : ${udlm.step5_timeout}
ucmm.timeout.cmmstart.05 : ${udlm.start_timeout}
ucmm.timeout.cmmabort.05 : ${udlm.abort_timeout}
ucmm.timeout.cmmstop.05 : ${udlm.stop_timeout}
Page 16 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Set kernel parameters
* Oracle settings *
set noexec_user_stack=1
set rlim_fd_cur=8192
• end of Oracle settings *
Create cluster resourcegroup for rac
# scrgadm -a -t SUNW.rac_framework
# scrgadm -a -t SUNW.rac_udlm
# scrgadm -a -t SUNW.rac_svm
# scrgadm -a -g rac_rg -y maximum_primaries=2 -y desired_primaries=2 \
-y nodelist=ora-rac-3,ora-rac-4
# scrgadm -a -j rac_framework_rs -g rac_rg -t SUNW.rac_framework
# scrgadm -a -j rac_udlm_rs -g rac_rg -t SUNW.rac_udlm \
-y resource_dependencies=rac_framework_rs -x port=7000
# scrgadm -a -j rac_svm_rs -g rac_rg -t SUNW.rac_svm \
-y resource_dependencies=rac_framework_rs
Increase svm_step4_timeout if using SVM and RAC is managed by the RGM.
# scrgadm -c -j rac_svm_rs -x svm_step4_timeout=360
The value 360 is an example!
See FIN I1139-1. Determine the best value for your configuration!
This is necessary when your SVM shared disk group setup is large or complex.
Switch the Resource Group online.
• # scswitch -Z -g rac_rg
Page 17 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Create multi-owner disk set
#metaset -s <setname> -M -a -h <nodelist>
NOTE (only SC3.1U4): If you get the message
metaset: host1: rac-ds: node host2 is not in membership list.
you should add on both nodes to /var/run/nodelist
1 ora-rac-3 172.16.193.1
2 ora-rac-4 172.16.193.2
Add drives (diddevices) to the metaset
#metaset -s racset -a /dev/did/dsk/d4
#metaset -s racset -a /dev/did/dsk/d5
root@ora-rac-3 # cat /etc/lvm/md.tab | grep racset
racset/d40 -m racset/d41 racset/d42
racset/d41 1 1 /dev/did/dsk/d5s0
racset/d42 1 1 /dev/did/dsk/d6s0
racset/d50 -m racset/d51 racset/d52
racset/d51 1 1 /dev/did/dsk/d12s0
racset/d52 1 1 /dev/did/dsk/d13s0
racset/d60 -m racset/d61 racset/d62
racset/d61 1 1 /dev/did/dsk/d14s0
racset/d62 1 1 /dev/did/dsk/d15s0
racset/d70 -m racset/d71 racset/d72
racset/d71 1 1 /dev/did/dsk/d16s0
racset/d72 1 1 /dev/did/dsk/d17s0
Page 18 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Configure QFS software on all nodes.
Laying out partitions
• Edit /etc/opt/SUNWsamfs/mcf
# Equipment Eq Eq Family Device Additional
# Identifier Ord Type Set State Parameters
#----------- --- ---- ------ ------ ----------
#
sharedrac 20 ma sharedrac on shared
/dev/md/racset/dsk/d50 21 mm sharedrac on
/dev/md/racset/dsk/d40 22 mr sharedrac on
sharedfs 30 ma sharedfs on shared
/dev/md/racset/dsk/d70 31 mm sharedfs on
/dev/md/racset/dsk/d60 32 mr sharedfs on
• Edit /etc/opt/SUNWsamfs/hosts.sharedrac
ora-rac-3 clusternode1-priv 1 - server
ora-rac-4 clusternode2-priv 2 -
• Edit /etc/opt/SUNWsamfs/hosts.sharedfs
ora-rac-3 clusternode1-priv 1 -
ora-rac-4 clusternode2-priv 2 - server
• Edit /etc/opt/SUNWsamfs/samfs.cmd
notrace
fs = sharedrac
nosam
forcedirectio
sync_meta=1
mh_write
qwrite
nstreams=1024
stripe=1
rdlease=300
wrlease=300
aplease=300
fs = sharedfs
sync_meta=1
nstreams=1024
stripe=1
Make the new configuration on both nodes available.
Page 19 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
#samd config
Use sam-fsd command for validation
Create a QFS filesystem (only 1 node).
#sammkfs -S sharedrac
#sammkfs -S sharedfs
Create global mountpoint and vfstab entries on all nodes
mkdir -p /oradata
mkdir -p /oraarch
chown root:other /oradata /oraarch
vi /etc/vfstab
sharedrac - /oradata samfs - no shared
sharedfs - /oraarch samfs - no shared
#mount sharedrac
#mount sharedfs
Create cluster resourcegroup for qfs
Because of a timingproblem between SLVM/SC and qfs implement a hack to be shure that metaset is available befor activating qfs. See bug id: 6222648
# cd /opt/SUNWsamfs/sc/bin
# vi scqfs_boot.hacked
#!/bin/ksh
my_host=`uname -n`
while true;
do
scstat -g | grep rac_svm | grep $my_host | read r rn h status statmsg
if [ $status = "Online" ]; then break;
else sleep 10; fi
done
exec /opt/SUNWsamfs/sc/bin/scqfs_init $*
# chmod 755 scqfs_boot.hacked
# mv scqfs_boot scqfs_boot_org
# ln -s scqfs_boot.hacked scqfs_boot
# scrgadm -a -t SUNW.qfs
Page 20 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
# scrgadm -a -g qfs_rac_rg -h ora-rac-3,ora-rac-4
# scrgadm -a -g qfs_rac_rg -t SUNW.qfs -j qfs_rac_rs \
-x QFSFileSystem=/oradata
# scswitch -Z -g qfs_rac_rg
# scrgadm -a -g qfs_arch_rg -h ora-rac-4,ora-rac-3
# scrgadm -a -g qfs_arch_rg -t SUNW.qfs -j qfs_arch_rs -x \
QFSFileSystem=/oraarch
# scswitch -Z -g qfs_arch_rg
If you are not able to register SUNW.qfs then create the link:
# cd /usr/cluster/lib/rgm/rtreg
# ln -s /opt/SUNWsamfs/sc/etc/SUNW.qfs
Set ownership and permissions:
chown oracle:dba /oradata /oraarch
chmod 755 /oradata /oraarch
Page 21 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Last Systemanalyse
explorer
cd /opt/SUNWexplo/output/explorer*
/cdrom/eis-dvd/sun/tools/explominer/explominer
see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and in-stall all neccessary patches.
Page 22 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Prepare Oracle Installation
Share staging area
All Oracle software is located on Host ora-poc2:/oracle/install. The directory is shared as /install/oracle and can be mounted on every node if needed.
Share stage directory:
chown -R oracle:oinstall /oracleshare -rw /oracle/install
Mount staging area
In order to install oracle software, the staging area must be mounted on the instaation node of the clus-ter.
mkdir /installmount ora-poc2:/oracle/install /install
Perform cluster check
As oracle:
/install/10gR2/clusterware/Disk1/runcluvfy.sh stage -pre crsinst -n ora-rac-3,ora-rac-4 -verbose > ~/cluvfy.log
Page 23 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install Clusterware 10.2.0.1
Run installer from ora-rac-3
Login as oracle:
DISPLAY=192.168.10.118:0.0
export DISPLAY
/install/10gR2/clusterware/Disk1/runInstaller
Page 24 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 25 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 26 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 27 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 28 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 29 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 30 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 31 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 32 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 33 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
The given interface(s), "ce0" is not public. Public interfaces should be used to configure virtual IPs.
Workaround: Run VIPCA manually
Running vipca
>> ora-rac-3 as root
/oracle/product/db/bin/vipca
Page 34 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 35 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 36 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 37 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 38 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 39 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Check Cluster Status
ORACLE_HOME=/oracle/product/crs
export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$PATH
export PATH
$ crs_stat -t
Name Type Target State Host -----------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE ora-rac-3ora....-1.lsnr application ONLINE ONLINE ora-rac-3ora....c-1.gsd application OFFLINE OFFLINE ora....c-1.ons application ONLINE ONLINE ora-rac-3ora....c-1.vip application ONLINE ONLINE ora-rac-3ora....SM2.asm application ONLINE ONLINE ora-rac-4ora....-2.lsnr application ONLINE ONLINE ora-rac-4ora....c-2.gsd application OFFLINE OFFLINE ora....c-2.ons application ONLINE ONLINE ora-rac-4ora....c-2.vip application ONLINE ONLINE ora-rac-4
Page 40 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install crsstat.sh
Because of the unreadable output of „crs_stat -t“ setup a wrapper for crs_stat.
mkdir /oracle/tools
vi /oracle/tools/crsstat.sh
#!/usr/bin/ksh
#
# Sample 10g CRS resource status query script
#
# Description:
# - Returns formatted version of crs_stat -t, in tabular
# format, with the complete rsc names and filtering keywords
# - The argument, $RSC_KEY, is optional and if passed to the script, will
# limit the output to HA resources whose names match $RSC_KEY.
# Requirements:
# - $CRS_HOME should be set in your environment
RSC_KEY=$1
QSTAT=-u
AWK=/usr/xpg4/bin/awk
# Table header:echo ""
$AWK \
'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State";
printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'
# Table body:
$CRS_HOME/bin/crs_stat $QSTAT | $AWK \
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
state == 0 {next;}
$1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
$1~/STATE/ && state == 2 {appstate = $2; state=3;}
state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'
Output of crsstat.sh
HA Resource Target State
----------- ------ -----
ora.ora-rac-3.gsd ONLINE ONLINE on ora-rac-3
ora.ora-rac-3.ons ONLINE ONLINE on ora-rac-3
ora.ora-rac-3.vip ONLINE ONLINE on ora-rac-3
ora.ora-rac-4.gsd ONLINE ONLINE on ora-rac-4
ora.ora-rac-4.ons ONLINE ONLINE on ora-rac-4
ora.ora-rac-4.vip ONLINE ONLINE on ora-rac-4
Page 41 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install Database Home 10.2.0.1
Run installer from ora-rac-3
/install/10gR2/database/Disk1/runInstaller
Page 42 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 43 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 44 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 45 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 46 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 47 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 48 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 49 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 50 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install Management Agent
Run installer from ora-rac-3
/install/10gR2/gridcontrol/DVD/runInstaller
Page 51 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 52 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 53 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 54 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 55 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Password: oracle1
Page 56 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 57 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 58 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 59 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install Patchset 10.2.0.2 on CRS Home
Stop all oracle processes
As oracle:
srvctl stop nodeapps -n ora-rac-3srvctl stop nodeapps -n ora-rac-4
As root on every node:
/etc/init.d/init.crs stop
If the processes won't stop, disable crs and reboot the host.
/etc/init.d/init.crs disable
init 6
Install binaries
Login as oracle on ora-rac-3
Set and export DISPLAY
Run Oracle Installer
/install/10gR2/patches/p4547817/Disk1/runInstaller
Page 60 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 61 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 62 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 63 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 64 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Restart Custerware
As root (on every node)
/etc/init.d/init.crs start
If the cluster processes have been disabled before, run
/etc/init.d/init.crs enable
Page 65 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install Patchset 10.2.0.2 on DB Home
Install binaries
./runInstaller
Page 66 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 67 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 68 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 69 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 70 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install Interim Patch 5117016 on DB Home
Install with opatch
>> ora-rac-3 as user oracle
#cd /install/10gR2/5117016
#opatch apply -local
Page 71 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
>> ora-rac-4 as user oracle
#cd /install/10gR2/5117016
#opatch apply -local
Page 72 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Change /etc/init.d/init.crsd:
>> ora-rac-3 and ora-rac-4 as user root
from: exec $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*
to : exec newtask $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*
Setup system parameters on both Nodes:
>> ora-rac-3 and ora-rac-4 as user root
#projadd -p 61002 -c "Oracle RAC" -U oracle \-K 'project.max-sem-ids=(priv,1024,deny)' \-K 'process.max-sem-nsems=(priv,2048,deny)' \-K 'project.max-shm-ids=(priv,256,deny)' \-K 'project.max-shm-memory=(priv,18446744073709551615,deny)' \MAH
cat >> /etc/user_attroracle::::project=RAC<CTRL>-d
Check all by reboot
>> ora-rac-3 as user root
#scshutdown -g0 -y
>> ora-rac-3 as user root
ok boot
>> ora-rac-4 as user root
ok boot
Page 73 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Create Database ZDA
Run DBCA
>> ora-rac-3
/oracle/product/db/bin/dbca
Page 74 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 75 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 76 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Password: oracle
Page 77 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Cluster OptimiziationOverview and best practices for setting up a "zero brownout" configuration to achieve maximum high availability. More info in note 369101.1.
Note: Patch 5024639 is not yet available and skipped therefore.
Apply Brownout Patches
This patches are installed by using OPatch on DB and ASM Home:
5186679 (This became patch 4960705)
5186674 (This became patch 5016142)
5127434
5127472
5127482
5176453
Set the CSS misscount to match the service level
1. Shut down the clusterware stack on all nodes but one
2. On the remaining node, run as root:
$ORA_CRS_HOME/bin/crsctl set css misscount 5
$ORA_CRS_HOME/bin/crsctl set css disktimeout 200
3. Reboot the nodes for this to take affect.
4. Confirm the new values for the CSS parameters
$ORA_CRS_HOME/bin/crsctl get css misscount
$ORA_CRS_HOME/bin/crsctl get css disktimeout
Run CSS in Real Time
1. On each node as root use, crsctl set css priority 4
2. Restart each node in series, using crsctl stop crs followed by crsctl start crs
Set the check_interval for VIP addresses
>> ora-ra-3 as root
1. Generate the resource profile for the VIP resource
cd $ORA_CRS_HOME/crs/profile
crs_stat -p ora.ora-rac-3.vip > ora. ora-rac-3.vip.capcrs_stat -p ora.ora-rac-4.vip > ora. ora-rac-4.vip.cap
(current value of check_interval = 60 (default))
Page 78 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
2. Edit ora.<hostname>.vip.cap and set the parameter CHECK_INTERVAL to 5
vi ora.ora-rac-3.vip.capvi ora.ora-rac-4.vip.cap
3. Update the resource profile as the root user:
crs_register -u ora.ora-rac-3.vipcrs_register -u ora.ora-rac-4.vip
Note: The check interval should be long enough to sustain the refresh time for the VIP.
Page 79 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Optimize VIP Failover
Adjusting Listener Configuration
Make the local IP-address the first listening entry in the listener.ora. More infos in note 3589757 and 339107.1
These bugs are based on oracle 10.1 but are not resolved by 10.2
>> All nodes
node ora-rac-3. Before:
LISTENER_ORA-RAC-3 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-3-vip) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.215) (PORT = 1521) (IP = FIRST)) ) )
[...]
After:
LISTENER_ORA-RAC-3 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.215) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-3-vip) (PORT = 1521) (IP = FIRST)) ) )
[...]
Page 80 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
node ora-rac-4. Before:
LISTENER_ORA-RAC-4 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-4-vip) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.217) (PORT = 1521) (IP = FIRST)) ) )
[...]
After:
LISTENER_ORA-RAC-4 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.217) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-4-vip) (PORT = 1521) (IP = FIRST)) ) )
[...]
Page 81 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Addendum
Metalink Note 284752.1
Page 82 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Metalink Note 294430.1
Page 83 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 84 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 85 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 86 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 87 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 88 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Metalink Note 294430.1
Page 89 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 90 of 90