Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6 ... · PDF fileBuild two node Oracle...
Transcript of Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6 ... · PDF fileBuild two node Oracle...
Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL
6.5 using Oracle VM Virtual Box
In the article you will have a look at how to use some Oracle Virtual Box
features to build two node Oracle 12c (12.1.0.1) RAC system on Oracle
Enterprise Linux (OEL 6.5). The setup will implement a role separation with
different users for Oracle RDBMS and Oracle GI that is, user oracle and grid
are used respectively in order to split the responsibilities between DBAs and
storage administrators.
An overview to Oracle Virtualization solutions can be seen here. You can see
how to use Oracle VM Virtual Box to build a two node 11gR2 RAC cluster on
Solaris 10 and Solaris 11. For information related to building a RAC 11gR2
cluster on OEL 5.X/6.X without GNS click here and here. For information how
to build a two node Oracle RAC 11gR2 11.2.0.3 with GNS (DNS, DHCP) and HAIP
look here. For information how to upgrade Oracle RAC cluster 11.2.0.3 on OEL
6.1 to Oracle RAC cluster 12c on OEL6.5 look here.
In the article you will see how to configure Linux in an Oracle VM Virtual
Box Virtual machines, install Oracle GI, Oracle RDBMS and will create a RAC
enabled Multitenant Container Database (CDB).
The following software will be used:
1. Oracle VM Virtual Box 4.3.12 – download from here.
2. Oracle 12c (database, grid, examples) – download from OTN here.
3. Oracle OEL 6.5 – download from Oracle Software Delivery Cloud (former
Edelivery) here.
Two virtual machines will be created and used.
1. OEL65A for RAC node oel65a 2. OEL65B for RAC node oel65b
The OEL65A/OEL65B Virtual Machines will be configured for RAC nodes each
with:
4096MB RAM 80GB bootable disk (Disk space will be with a fixed size pre-
allocation using LVM) NIC – bridged for public interface in RAC with address
192.168.2.11/12 (first IP 192.168.2.11 on oel65a and second IP
192.168.2.12 on node oel65b). These are public interface in RAC. NIC – bridged for private interface in RAC with address
10.10.10.11/12 (first IP 10.10.10.11 on oel65a and second IP
10.10.10.12 on node oel65b). These are private interface in RAC. NIC – bridged for private interface in RAC with address
10.10.10.71/72 (first IP 10.10.10.71 on oel65a and second IP
10.10.10.72 on node oel65b). These are private interface in RAC.
NIC – bridged for private interface in RAC with address
10.10.10.81/82 (first IP 10.10.10.81 on oel65a and second IP
10.10.10.82 on node oel65b). These are private interface in RAC. 5 10GB attached shared disks for the ASM storage. (High
Redundancy ASM disk group DATA will be deployed for CRS/Vote and
data files). 5 2GB attached shared disks for the ASM storage. (External
Redundancy ASM disk group FRA will be deployed as flash/fast
recovery area).
The interfaces’ IP addresses will be as show in the table 1 below.
VM OEL65A RAC
node oel65a
VM OEL65B RAC
node oel65b
eth0 192.168.2.11 192.168.2.12
eth1 10.10.10.11 10.10.10.12
eth2 10.10.10.71 10.10.10.72
eth3 10.10.10.81 10.10.10.82
The /etc/hosts file for oel65a/oel65b will be as follows:
192.168.2.11 oel65a.gj.com oel65a
10.10.10.11 oel65a-priv.gj.com oel65a-priv
192.168.2.21 oel65a-vip.gj.com oel65a-vip
10.10.10.71 oel65a-priv1.gj.com oel65a-priv1
10.10.10.81 oel65a-prib2.gj.com oel65a-priv2
192.168.2.12 oel65b.gj.com oel65b
10.10.10.12 oel65b-priv.gj.com oel65b-priv
192.168.2.22 oel65b-vip.gj.com oel65b-vip
10.10.10.72 oel65b-priv1.gj.com oel65b-priv1
10.10.10.82 oel65b-prib2.gj.com oel65b-priv2
192.168.2.51 scan.gj.com scan
192.168.2.52 scan.gj.com scan
192.168.2.53 scan.gj.com scan
The article will cover the following topics
1. Create an OEL65A VM with OEL 6.5 as guest OS for node oel65a. 2. Configure the OEL61A VM to meet the prerequisites for GI and RAC
12c deployment.
3. Clone OEL61A to OEL61B and attach the shared disks. 4. Change IP and hostname on OEL65B 5. Install GI 12c on oel65a and oel65b. 6. Install RAC RDBMS 12c on oel65a and oel65b. 7. Create a Multitenant Container Database (CDB) database RACDB
oel65a and oel65b.
8. Verify database creation and create a service.
1. Create an OEL65A VM with OEL 6.5 as guest OS for node oel65a
In this section you will look at how to create a guest OEL 6.5 VM
using Oracle VM Virtual Box.
1.1 Select New icon or Machine->New…
1.2 Select the RAM size. The final value I used is 4096MB.
Larger values turned out to cause VM instability. The error
below from the VM log was avoided by reducing the VM RAM
from 5000MB to 4096MB.
run OUI
00:26:18.768975 VM: Raising runtime error 'HostMemoryLow' (fFlags=0x2)
00:26:18.770070 AssertLogRel D:\tinderbox\win-
4.3\src\VBox\VMM\VMMR3\VMM.cpp(1756) int __cdecl
VMMR3EmtRendezvous(struct VM *,unsigned int,int (__cdecl *)(struct VM
*,struct VMCPU *,void *),void *): !pVCpu->vmm.s.fInRendezvous
00:26:18.791552 Console: VM runtime error: fatal=false,
errorID=HostMemoryLow message="Unable to allocate and lock memory. The
virtual machine will be paused. Please close applications to free up
memory or close the VM"
00:26:18.825633 Changing the VM state from 'RUNNING' to
'GURU_MEDITATION'.
1.3 Select Create Virtual Hard Drive now
1.4 Select the drive type VDI
1.5 Select Dynamically allocated
1.6 Select the HD size of 80G or more.
1.7 The VM is created
1.8 Add the OEL 65 ISO image so that you can boot and install
the OEL 65.
1.9 Configure the Clipboard and mouse integration
1.10 Configure all adapters as bridged
1.11 Start the VM and select Install
1.12 Select Skip
1.13 Press Next
1.14 Select language
1.15 Select keyboard
1.16 Select Basic Storage Devices
1.17 Select discard to reformat
1.18 Enter hostname
1.19 Enter IP addresses, Net masks and gateways. For example
eth1 you will have the configuration below. Repeat the
configuration for all interfaces eth0,eth2,eth3
1.20 Click Next after configuring all interfaces ethX and the
hostname.
1.21 Select a city in the time zone.
1.22 Enter root password.
1.23 Select use all space
1.24 Write the changes to disk
1.25 I am going to do a full install and will pick all rpms.
1.26 I selected everything
1.27 Wait for the install to complete.
1.28 Press Forward
1.29 Continue
1.30 You can register any time. I opted for register later.
1.31 Press Forward
1.32 Create a user.
1.33 Setup date and time
1.34 I did not enable kdump
1.35 The login screen comes up after a reboot
1.36 Install Virtual Box additions
1.37 Press Run
1.38 Wait for the installation to completes
1.39 Once installation is done power reboot the VM
1.40 This concludes OEL65 installation
2. Configure the OEL61A VM to meet the prerequisites for GI
and RAC 12c deployment.
2.1 Modify the UEKR3 kernel and add divider=10 by editing
/etc/grub.conf. This will speed up the OEL guest VM.
[root@oel65a ~]# cat /etc/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this
file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/vg_oel65a-lv_root
# initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-
16.2.1.el6uek.x86_64)
root (hd0,0)
kernel /tboot.gz logging=vga,serial,memory numa=off
transparent_hugepage=never
module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro
root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us
rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet divider=10
module /initramfs-3.8.13-16.2.1.el6uek.x86_64.img
title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-
16.2.1.el6uek.x86_64.debug)
root (hd0,0)
kernel /tboot.gz logging=vga,serial,memory numa=off
transparent_hugepage=never
module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64.debug ro
root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us
rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet
module /initramfs-3.8.13-16.2.1.el6uek.x86_64.debug.img
title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-
431.el6.x86_64)
root (hd0,0)
kernel /tboot.gz logging=vga,serial,memory numa=off
transparent_hugepage=never
module /vmlinuz-2.6.32-431.el6.x86_64 ro
root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
rd_LVM_LV=vg_oel65a/lv_swap crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us
rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet
module /initramfs-2.6.32-431.el6.x86_64.img
title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-
16.2.1.el6uek.x86_64)
root (hd0,0)
kernel /tboot.gz logging=vga,serial,memory numa=off
transparent_hugepage=never
module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro
root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us
rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet
module /initramfs-3.8.13-16.2.1.el6uek.x86_64.img
title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-
16.2.1.el6uek.x86_64.debug)
root (hd0,0)
kernel /tboot.gz logging=vga,serial,memory numa=off
transparent_hugepage=never
module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64.debug ro
root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us
rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet
module /initramfs-3.8.13-16.2.1.el6uek.x86_64.debug.img
title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-
431.el6.x86_64)
root (hd0,0)
kernel /tboot.gz logging=vga,serial,memory numa=off
transparent_hugepage=never
module /vmlinuz-2.6.32-431.el6.x86_64 ro
root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
rd_LVM_LV=vg_oel65a/lv_swap crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us
rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet
module /initramfs-2.6.32-431.el6.x86_64.img
You have new mail in /var/spool/mail/root
[root@oel65a ~]#
2.2 Run the command yum install oracle-rdbms-server-12cR1-
preinstall. The output is in the Annex A. Note that you
need to temporary enable internet access by a) configure
any interface to obtain a DHCP address and b) modify the
corresponding adaptor to NAT. After successful installation
revert to the original configuration.
2.3 If you have not install all packages make sure that the
following packages are installed.
binutils-2.20.51.0.2-5.11.el6 (x86_64)
compat-libcap1-1.10-1 (x86_64)
compat-libstdc++-33-3.2.3-69.el6 (x86_64)
compat-libstdc++-33-3.2.3-69.el6.i686
gcc-4.4.4-13.el6 (x86_64)
gcc-c++-4.4.4-13.el6 (x86_64)
glibc-2.12-1.7.el6 (i686)
glibc-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6.i686
ksh
libgcc-4.4.4-13.el6 (i686)
libgcc-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6.i686
libstdc++-devel-4.4.4-13.el6 (x86_64)
libstdc++-devel-4.4.4-13.el6.i686
libaio-0.3.107-10.el6 (x86_64)
libaio-0.3.107-10.el6.i686
libaio-devel-0.3.107-10.el6 (x86_64)
libaio-devel-0.3.107-10.el6.i686
libXext-1.1 (x86_64)
libXext-1.1 (i686)
libXtst-1.0.99.2 (x86_64)
libXtst-1.0.99.2 (i686)
libX11-1.3 (x86_64)
libX11-1.3 (i686)
libXau-1.0.5 (x86_64)
libXau-1.0.5 (i686)
libxcb-1.5 (x86_64)
libxcb-1.5 (i686)
libXi-1.3 (x86_64)
libXi-1.3 (i686)
make-3.81-19.el6
sysstat-9.0.4-11.el6 (x86_64)
2.4 The oracle-rdbms-server-11gR2-preinstall takes care of the
oracle user. Similarly modify /etc/security/limits.conf to
address the grid user.
# oracle-rdbms-server-11gR2-preinstall setting for nofile soft limit is
1024
grid soft nofile 1024
# oracle-rdbms-server-11gR2-preinstall setting for nofile hard limit is
65536
grid hard nofile 65536
# oracle-rdbms-server-11gR2-preinstall setting for nproc soft limit is
2047
grid soft nproc 2047
# oracle-rdbms-server-11gR2-preinstall setting for nproc hard limit is
16384
grid hard nproc 16384
# oracle-rdbms-server-11gR2-preinstall setting for stack soft limit is
10240KB
grid soft stack 10240
# oracle-rdbms-server-11gR2-preinstall setting for stack hard limit is
32768KB
grid hard stack 32768
2.5 Add to /etc/pam.d/login a line session required
pam_limits.so
2.6 Enabling the Name Service Cache Daemon
chkconfig --level 35 nscd on
service nscd start
2.7 Configure NTP
You have two options for time synchronization: an operating
system configured network time protocol (NTP), or Oracle
Cluster Time Synchronization Service. Oracle Cluster Time
Synchronization Service is designed for organizations whose
cluster servers are unable to access NTP services. If you use
NTP, then the Oracle Cluster Time Synchronization daemon
(ctssd) starts up in observer mode. If you do not have NTP
daemons, then ctssd starts up in active mode and synchronizes
time among cluster members without contacting an external time
server. So there are two options:
For enabling the NTP make sure that /etc/sysconfig/ntpd
has the line modified to include -x.
OPTIONS=”-x -u ntp:ntp -p /var/run/ntpd.pid”
For disabling the NTP make sure that the NTP service is
stopped and disabled for auto-start and there is not
configuration file.
/sbin/service ntpd stop
chkconfig ntpd off
mv /etc/ntp.conf to /etc/ntp.conf.org
In the article NTP is enabled.
2.8 Set in /etc/ssh/sshd_config the following line LoginGraceTime 0
2.9 Create OS groups. The masked are create by oracle-rdbms-server-11gR2-preinstall
#groupadd -g 54321 oinstall
#groupadd -g 54322 dba
groupadd -g 54323 oper
groupadd -g 54324 backupdba
groupadd -g 54325 asmdba
groupadd -g 54326 dgdba
groupadd -g 54327 kmdba
groupadd -g 54328 asmadmin
groupadd -g 54329 asmoper
2.10 Create users.
useradd -u 54322 -g oinstall -G asmadmin,asmdba grid
##useradd -u 54321 -g oinstall -G dba,backupdba,dgdba,kmdba,asmdba
asmoper oracle
usermod -g oinstall -G dba,backupdba,dgdba,kmdba,asmdba,asmoper oracle
2.11 Create Directories
mkdir -p /u01/app/12.1.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/
chown oracle:oinstall /u01/app/oracle
2.12 List disks
[root@oel65a dev]# ls sd*
sda sda1 sda2
[root@oel65a dev]#
2.13 Create shared disks
From the host OS ( Windows in my case) run the following
commands.
VBoxManage createhd --filename e:\vb\asm1.vdi --size 10240 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm2.vdi --size 10240 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm3.vdi --size 10240 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm4.vdi --size 10240 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm5.vdi --size 10240 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm6.vdi --size 2048 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm7.vdi --size 2048 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm8.vdi --size 2048 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm9.vdi --size 2048 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm10.vdi --size 2048 --format VDI --
variant Fixed
VBoxManage createhd --filename e:\vb\asm11.vdi --size 2048 --format VDI --
variant Fixed
VBoxManage storageattach OEL65a --storagectl "SATA" --port 1 --device 0 --
type hdd --medium asm1.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 2 --device 0 --
type hdd --medium asm2.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 3 --device 0 --
type hdd --medium asm3.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 4 --device 0 --
type hdd --medium asm4.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 5 --device 0 --
type hdd --medium asm5.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 6 --device 0 --
type hdd --medium asm6.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 7 --device 0 --
type hdd --medium asm7.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 8 --device 0 --
type hdd --medium asm8.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 9 --device 0 --
type hdd --medium asm9.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 10 --device 0 --
type hdd --medium asm10.vdi --mtype shareable
VBoxManage storageattach OEL65a --storagectl "SATA" --port 11 --device 0 --
type hdd --medium asm11.vdi --mtype shareable
2.14 List and format the disks
Initially you will have
[root@oel65a dev]# ls -l sd*
brw-rw----. 1 root disk 8, 0 May 12 16:37 sda
brw-rw----. 1 root disk 8, 1 May 12 16:37 sda1
brw-rw----. 1 root disk 8, 2 May 12 16:37 sda2
brw-rw----. 1 root disk 8, 16 May 12 16:37 sdb
brw-rw----. 1 root disk 8, 32 May 12 16:37 sdc
brw-rw----. 1 root disk 8, 48 May 12 16:37 sdd
brw-rw----. 1 root disk 8, 64 May 12 16:37 sde
brw-rw----. 1 root disk 8, 80 May 12 16:37 sdf
brw-rw----. 1 root disk 8, 96 May 12 16:37 sdg
brw-rw----. 1 root disk 8, 112 May 12 16:37 sdh
brw-rw----. 1 root disk 8, 128 May 12 16:37 sdi
brw-rw----. 1 root disk 8, 144 May 12 16:37 sdj
brw-rw----. 1 root disk 8, 160 May 12 16:37 sdk
brw-rw----. 1 root disk 8, 176 May 12 16:37 sdl
[root@oel65a dev]#
For each disk from /dev/sdb to /dev/sdl perform the following
procedure.
[root@oel65a dev]# fdisk /dev/sdb
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305):
Using default value 1305
Command (m for help): p
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xeede9dba
Device Boot Start End Blocks Id System
/dev/sdb1 1 1305 10482381 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@oel65a dev]#
At the end you will have:
[root@oel65a dev]# ls -l /dev/sd*
brw-rw----. 1 root disk 8, 0 May 12 16:37 /dev/sda
brw-rw----. 1 root disk 8, 1 May 12 16:37 /dev/sda1
brw-rw----. 1 root disk 8, 2 May 12 16:37 /dev/sda2
brw-rw----. 1 root disk 8, 16 May 12 16:46 /dev/sdb
brw-rw----. 1 root disk 8, 17 May 12 16:46 /dev/sdb1
brw-rw----. 1 root disk 8, 32 May 12 16:46 /dev/sdc
brw-rw----. 1 root disk 8, 33 May 12 16:46 /dev/sdc1
brw-rw----. 1 root disk 8, 48 May 12 16:46 /dev/sdd
brw-rw----. 1 root disk 8, 49 May 12 16:46 /dev/sdd1
brw-rw----. 1 root disk 8, 64 May 12 16:47 /dev/sde
brw-rw----. 1 root disk 8, 65 May 12 16:47 /dev/sde1
brw-rw----. 1 root disk 8, 80 May 12 16:47 /dev/sdf
brw-rw----. 1 root disk 8, 81 May 12 16:47 /dev/sdf1
brw-rw----. 1 root disk 8, 96 May 12 16:47 /dev/sdg
brw-rw----. 1 root disk 8, 97 May 12 16:47 /dev/sdg1
brw-rw----. 1 root disk 8, 112 May 12 16:48 /dev/sdh
brw-rw----. 1 root disk 8, 113 May 12 16:48 /dev/sdh1
brw-rw----. 1 root disk 8, 128 May 12 16:48 /dev/sdi
brw-rw----. 1 root disk 8, 129 May 12 16:48 /dev/sdi1
brw-rw----. 1 root disk 8, 144 May 12 16:48 /dev/sdj
brw-rw----. 1 root disk 8, 145 May 12 16:48 /dev/sdj1
brw-rw----. 1 root disk 8, 160 May 12 16:49 /dev/sdk
brw-rw----. 1 root disk 8, 161 May 12 16:49 /dev/sdk1
brw-rw----. 1 root disk 8, 176 May 12 16:49 /dev/sdl
brw-rw----. 1 root disk 8, 177 May 12 16:49 /dev/sdl1
[root@oel65a dev]#
2.15 Configure ASMlib and ASmlib disks by runnin the following
commands in an ordered manner.
[root@oel65a dev]# /usr/sbin/oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
[root@oel65a dev]#
/usr/sbin/oracleasm init
[root@oel65a dev]# /usr/sbin/oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm
[root@oel65a dev]#
/usr/sbin/oracleasm createdisk DISK1 /dev/sdb1
/usr/sbin/oracleasm createdisk DISK2 /dev/sdc1
/usr/sbin/oracleasm createdisk DISK3 /dev/sdd1
/usr/sbin/oracleasm createdisk DISK4 /dev/sde1
/usr/sbin/oracleasm createdisk DISK5 /dev/sdf1
/usr/sbin/oracleasm createdisk DISK6 /dev/sdg1
/usr/sbin/oracleasm createdisk DISK7 /dev/sdh1
/usr/sbin/oracleasm createdisk DISK8 /dev/sdi1
/usr/sbin/oracleasm createdisk DISK9 /dev/sdj1
/usr/sbin/oracleasm createdisk DISK10 /dev/sdk1
/usr/sbin/oracleasm createdisk DISK11 /dev/sdl1
[root@oel65a dev]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@oel65a dev]# /usr/sbin/oracleasm listdisks
DISK1
DISK10
DISK11
DISK2
DISK3
DISK4
DISK5
DISK6
DISK7
DISK8
DISK9
[root@oel65a dev]#
2.16 Setting the Disk I/O Scheduler on Linux
Disk I/O schedulers reorder, delay, or merge requests for
disk I/O to achieve better throughput and lower latency.
Linux has multiple disk I/O schedulers available, including
Deadline, Noop, Anticipatory, and Completely Fair Queuing
(CFQ). For best performance for Oracle ASM, Oracle
recommends that you use the Deadline I/O Scheduler.
On each cluster node, enter the following command to ensure
that the Deadline disk I/O scheduler is configured for use:
# echo deadline > /sys/block/${ASM_DISK}/queue/scheduler
echo deadline > /sys/block/sdb/queue/scheduler
echo deadline > /sys/block/sdc/queue/scheduler
echo deadline > /sys/block/sdd/queue/scheduler
echo deadline > /sys/block/sde/queue/scheduler
echo deadline > /sys/block/sdf/queue/scheduler
echo deadline > /sys/block/sdg/queue/scheduler
echo deadline > /sys/block/sdh/queue/scheduler
echo deadline > /sys/block/sdi/queue/scheduler
echo deadline > /sys/block/sdj/queue/scheduler
echo deadline > /sys/block/sdk/queue/scheduler
echo deadline > /sys/block/sdl/queue/scheduler
2.17 Disable SElinux. Edit /etc/selinux/config and enable the
line
SELINUX=disabled
2.18 Disable iptables
chkconfig iptables off
2.19 Install the rpm running rpm -Uvh cvuqdisk-1.0.9-1.rpm
2.20 Set rp filtering by seting the rp_filter of the private
address to 2 (loose filtering), the public address to 1
(strict filtering), using the following entries in
/etc/sysctl.conf:
net.ipv4.conf.eth2.rp_filter = 2
net.ipv4.conf.eth2.rp_filter = 2
net.ipv4.conf.eth1.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 2
2.21 Shutdown the VM
2.22 This Concludes the OEL65A preparation for 12c installation.
3. Clone OEL61A to OEL61B.
3.1 Clone the OEL65a.vdi by issuing the following command.
E:\vb>VBoxManage clonehd OEL65a.vdi OEL65b.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone hard disk created in format 'VDI'. UUID: 5b4a256c-d6c9-4b32-8d02-
c5a1b10e3
1d5
E:\vb>
3.2 Create OEL65b VM and add the 4 NIC as in OEL65a
3.3 Attach the shared disks
VBoxManage storageattach OEL65b --storagectl "SATA" --port 1 --device 0 -
-type hdd --medium asm1.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 2 --device 0 -
-type hdd --medium asm2.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 3 --device 0 -
-type hdd --medium asm3.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 4 --device 0 -
-type hdd --medium asm4.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 5 --device 0 -
-type hdd --medium asm5.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 6 --device 0 -
-type hdd --medium asm6.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 7 --device 0 -
-type hdd --medium asm7.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 8 --device 0 -
-type hdd --medium asm8.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 9 --device 0 -
-type hdd --medium asm9.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 10 --device 0
--type hdd --medium asm10.vdi --mtype shareable
VBoxManage storageattach OEL65b --storagectl "SATA" --port 11 --device 0
--type hdd --medium asm11.vdi --mtype shareable
3.4 Startup the OEL65B VM
3.5 This concludes OEL65B clone
4. Change IP and hostname on OEL65B
4.1 Change the hostname in /etc/sysconfig/network
[root@oel65a ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=oel65b.gj.com
GATEWAY=192.168.2.1
NTPSERVERARGS=iburst
[root@oel65a ~]#
4.2 Change the IPs of the network interfaces
Issue the following command
#ifconfig –a | grep eth
You will see eth4,eth5,eth6,eth7
I the etc/sysconfig/network-scripts you will have
[root@oel65b network-scripts]# pwd
/etc/sysconfig/network-scripts
[root@oel65b network-scripts]# ls -l ifcfg-eth*
-rw-r--r-- 1 root root 294 May 13 17:09 ifcfg-eth0
-rw-r--r-- 1 root root 295 May 13 11:58 ifcfg-eth1
-rw-r--r-- 1 root root 294 May 13 11:58 ifcfg-eth2
-rw-r--r-- 1 root root 294 May 13 17:10 ifcfg-eth3
[root@oel65b network-scripts]#
Edit the ifcfg-eth0 and modify the following lines using
the HWADDR from the first line of #ifconfig –a | grep eth.
HWADDR=08:00:27:4F:66:B2
IPADDR=10.10.10.72
Edit the ifcfg-eth1 and modify the following lines using
the HWADDR from the second line of #ifconfig –a | grep eth
HWADDR=08:00:27:77:2B:DB
IPADDR=192.168.2.12
Edit the ifcfg-eth2 and modify the following lines using
the HWADDR from the third line of #ifconfig –a | grep eth
HWADDR=08:00:27:BC:10:F5
IPADDR=10.10.10.12
Edit the ifcfg-eth3 and modify the following lines using
the HWADDR from the fourth line of #ifconfig –a | grep eth
HWADDR=08:00:27:BC:10:F5
HWADDR=08:00:27:BC:10:F5
4.3 Reboot the VM
4.4 This concludes the network reconfiguration.
5. Install GI 12c on oel65a and oel65b.
5.1 Create .bash_proile for user grid and user oracle on both
oel65a and oel65b. Look at Annex A for the profiles.
5.2 Run ckuvfy to make sure that the prerequisites for GI
installations are met. You will find the output in the
Annex A.
./runcluvfy.sh stage -pre crsinst -n oel65a,oel65b
./runcluvfy.sh stage -post hwos -n oel65a,oel65b
5.3 runInstall from the grid 12c stage area and configure ssh.
5.4 Run OUI from the grid 12c stage area
Select Skip and press Next.
Select install and configure GI for a cluster.
Select Configure a standard Cluster
Select a Typical Installation
Enter the scan name and the public and virtual hostnames.
Select the interfaces and what they are used for.
Enter the software locations and ASM passwords.
Specify the pattern to use to discover the disks
Specify the 10g disk for a high redundancy disk group.
Note that the disks are as follows
Specify the inventory
Select an option to manually run the root scripts.
Wait for the checks to complete. Review and select Ignore All
and press Next.
The errors are as follows
PRVF-9802 : Attempt to get udev information from node "oel65a" failed
PRVF-9802 : Attempt to get udev information from node "oel65b" failed
PRVF-5636 : The DNS response time for an unreachable node exceeded
"15000" ms on following nodes: oel65a,oel65b
The only note ‘INS-20802 PRVF-9802 PRVF-5184 PRVF-5186 After
Successful Upgrade to 11gR2 Grid Infrastructure (Doc ID
974481.1)’ suggest that the disk should be managed by UDEV.
I wonder why ASMlib does not do the job. As I am running it
using Oracle VM Virtual Box I will ignore the error.
Note : ‘PRVF-5636 : The DNS response time for an unreachable
node exceeded "15000" ms on following nodes (Doc ID 1356975.1)’
addresses the PRVF-5636 problem. I will ignore it for the same
reason as stated above.
Review and press Install.
Wait until prompted to execute scripts as root.
Execute the scripts as root.
[root@oel65a ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@oel65a ~]#
[root@oel65b ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@oel65b ~]#
[root@oel65a ~]# /u01/app/12.1.0.1/grid/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0.1/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file:
/u01/app/12.1.0.1/grid/crs/install/crsconfig_params
2014/05/14 15:09:12 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2014/05/14 15:10:27 CLSRSC-330: Adding Clusterware entries to file 'oracle-
ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'oel65a'
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel65a'
CRS-2676: Start of 'ora.evmd' on 'oel65a' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel65a'
CRS-2676: Start of 'ora.gpnpd' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel65a'
CRS-2672: Attempting to start 'ora.gipcd' on 'oel65a'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel65a' succeeded
CRS-2676: Start of 'ora.gipcd' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel65a'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel65a'
CRS-2676: Start of 'ora.diskmon' on 'oel65a' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel65a' succeeded
ASM created and started successfully.
Disk Group DATA created successfully.
CRS-2672: Attempting to start 'ora.storage' on 'oel65a'
CRS-2676: Start of 'ora.storage' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel65a'
CRS-2676: Start of 'ora.crsd' on 'oel65a' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 9d8b11d713204f34bf32e2445c8c751d.
Successful addition of voting disk f791aee91e474f58bfae1efacb87c4ce.
Successful addition of voting disk 4f5bcd1ebae74f1dbf5a7edcb2c86bef.
Successful addition of voting disk d329f30451654f08bfc6b265f4007b98.
Successful addition of voting disk 0453894fdeda4f2ebf334a123938d781.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 9d8b11d713204f34bf32e2445c8c751d (/dev/oracleasm/disks/DISK1)
[DATA]
2. ONLINE f791aee91e474f58bfae1efacb87c4ce (/dev/oracleasm/disks/DISK2)
[DATA]
3. ONLINE 4f5bcd1ebae74f1dbf5a7edcb2c86bef (/dev/oracleasm/disks/DISK3)
[DATA]
4. ONLINE d329f30451654f08bfc6b265f4007b98 (/dev/oracleasm/disks/DISK4)
[DATA]
5. ONLINE 0453894fdeda4f2ebf334a123938d781 (/dev/oracleasm/disks/DISK5)
[DATA]
Located 5 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed
resources on 'oel65a'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel65a'
CRS-2677: Stop of 'ora.crsd' on 'oel65a' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel65a'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel65a'
CRS-2673: Attempting to stop 'ora.storage' on 'oel65a'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel65a'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel65a'
CRS-2677: Stop of 'ora.storage' on 'oel65a' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel65a'
CRS-2677: Stop of 'ora.mdnsd' on 'oel65a' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel65a' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel65a' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oel65a' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel65a' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel65a'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel65a' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel65a'
CRS-2677: Stop of 'ora.cssd' on 'oel65a' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel65a'
CRS-2677: Stop of 'ora.gipcd' on 'oel65a' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on
'oel65a' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel65a'
CRS-2672: Attempting to start 'ora.evmd' on 'oel65a'
CRS-2676: Start of 'ora.evmd' on 'oel65a' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel65a'
CRS-2676: Start of 'ora.gpnpd' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel65a'
CRS-2676: Start of 'ora.gipcd' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel65a'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel65a'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel65a'
CRS-2676: Start of 'ora.diskmon' on 'oel65a' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server
'oel65a'
CRS-2676: Start of 'ora.cssd' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel65a'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel65a'
CRS-2676: Start of 'ora.ctssd' on 'oel65a' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel65a'
CRS-2676: Start of 'ora.asm' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel65a'
CRS-2676: Start of 'ora.storage' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel65a'
CRS-2676: Start of 'ora.crsd' on 'oel65a' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: oel65a
CRS-6016: Resource auto-start has completed for server oel65a
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/05/14 15:17:53 CLSRSC-343: Successfully started Oracle clusterware stack
CRS-2672: Attempting to start 'ora.asm' on 'oel65a'
CRS-2676: Start of 'ora.asm' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'oel65a'
CRS-2676: Start of 'ora.DATA.dg' on 'oel65a' succeeded
2014/05/14 15:19:58 CLSRSC-325: Configure Oracle Grid Infrastructure for a
Cluster ... succeeded
You have new mail in /var/spool/mail/root
[root@oel65a ~]#
[root@oel65b grid]# pwd
/u01/app/12.1.0.1/grid
[root@oel65b grid]# /u01/app/12.1.0.1/grid/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0.1/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file:
/u01/app/12.1.0.1/grid/crs/install/crsconfig_params
2014/05/14 15:20:54 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2014/05/14 15:21:21 CLSRSC-330: Adding Clusterware entries to file 'oracle-
ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel65b'
CRS-2672: Attempting to start 'ora.evmd' on 'oel65b'
CRS-2676: Start of 'ora.mdnsd' on 'oel65b' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel65b'
CRS-2676: Start of 'ora.gpnpd' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel65b'
CRS-2676: Start of 'ora.gipcd' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel65b'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel65b'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel65b'
CRS-2676: Start of 'ora.diskmon' on 'oel65b' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server
'oel65b'
CRS-2676: Start of 'ora.cssd' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel65b'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel65b'
CRS-2676: Start of 'ora.ctssd' on 'oel65b' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel65b'
CRS-2676: Start of 'ora.asm' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel65b'
CRS-2676: Start of 'ora.storage' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel65b'
CRS-2676: Start of 'ora.crsd' on 'oel65b' succeeded
CRS-6017: Processing resource auto-start for servers: oel65b
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel65a'
CRS-2672: Attempting to start 'ora.ons' on 'oel65b'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel65a' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel65a'
CRS-2677: Stop of 'ora.scan1.vip' on 'oel65a' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel65b'
CRS-2676: Start of 'ora.scan1.vip' on 'oel65b' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel65b'
CRS-2676: Start of 'ora.ons' on 'oel65b' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel65b' succeeded
CRS-6016: Resource auto-start has completed for server oel65b
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/05/14 15:26:57 CLSRSC-343: Successfully started Oracle clusterware stack
2014/05/14 15:27:21 CLSRSC-325: Configure Oracle Grid Infrastructure for a
Cluster ... succeeded
[root@oel65b grid]#
The errors can be ignored. The errors are listed below
[root@oel65a logs]# cat installActions2014-05-14_02-32-31PM.log | grep ERROR
INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-7530 : Sufficient
physical memory is not available on node "oel65b" [Required physical memory =
4GB (4194304.0KB)]
INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-7530 : Sufficient
physical memory is not available on node "oel65a" [Required physical memory =
4GB (4194304.0KB)]
INFO: ERRORMSG(oel65b): PRVF-7530 : Sufficient physical memory is not
available on node "oel65b" [Required physical memory = 4GB (4194304.0KB)]
INFO: ERRORMSG(oel65a): PRVF-7530 : Sufficient physical memory is not
available on node "oel65a" [Required physical memory = 4GB (4194304.0KB)]
INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-9802 : Attempt to get
udev information from node "oel65a" failed
INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-9802 : Attempt to get
udev information from node "oel65b" failed
INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get
udev information from node "oel65b" failed
INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get
udev information from node "oel65a" failed
INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get
udev information from node "oel65b" failed
INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get
udev information from node "oel65a" failed
INFO: ERRORMSG(oel65b): PRVF-9802 : Attempt to get udev information
from node "oel65b" failed
INFO: ERRORMSG(oel65a): PRVF-9802 : Attempt to get udev information
from node "oel65a" failed
INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-5636 : The DNS
response time for an unreachable node exceeded "15000" ms on following nodes:
oel65a,oel65b
INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-5636 : The DNS
response time for an unreachable node exceeded "15000" ms on following nodes:
oel65a,oel65b
INFO: ERRORMSG(oel65b): PRVF-5636 : The DNS response time for an
unreachable node exceeded "15000" ms on following nodes: oel65a,oel65b
INFO: ERRORMSG(oel65a): PRVF-5636 : The DNS response time for an
unreachable node exceeded "15000" ms on following nodes: oel65a,oel65b
INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-5636 : The DNS
response time for an unreachable node exceeded "15000" ms on following nodes:
oel65a,oel65b
INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-5636 : The DNS
response time for an unreachable node exceeded "15000" ms on following nodes:
oel65a,oel65b
INFO: INFO: ERROR: [Result.addErrorDescription:607] Check for integrity of
file "/etc/resolv.conf" failed
INFO: INFO: ERROR: [Result.addErrorDescription:607] Check for integrity of
file "/etc/resolv.conf" failed
INFO: ERRORMSG(oel65b): PRVF-5636 : The DNS response time for an
unreachable node exceeded "15000" ms on following nodes: oel65a,oel65b
INFO: ERRORMSG(oel65b): Check for integrity of file
"/etc/resolv.conf" failed
INFO: ERRORMSG(oel65a): PRVF-5636 : The DNS response time for an
unreachable node exceeded "15000" ms on following nodes: oel65a,oel65b
INFO: ERRORMSG(oel65a): Check for integrity of file
"/etc/resolv.conf" failed
INFO: ERROR:
INFO: ERROR:
INFO: ERROR:
INFO: ERROR:
INFO: ERROR:
[root@oel65a logs]#
INFO: ERROR:
INFO: PRVG-1101 : SCAN name "scan" failed to resolve
INFO: ERROR:
INFO: PRVF-4657 : Name resolution setup check for "scan" (IP address:
192.168.2.51) failed
INFO: ERROR:
INFO: PRVF-4657 : Name resolution setup check for "scan" (IP address:
192.168.2.52) failed
INFO: ERROR:
INFO: PRVF-4657 : Name resolution setup check for "scan" (IP address:
192.168.2.53) failed
INFO: ERROR:
INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name
"scan"
INFO: Checking SCAN IP addresses...
5.5 Verify that GI is installed properly
[root@oel65a bin]# ./crsctl check cluster -all
**************************************************************
oel65a:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel65b:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@oel65a bin]#
[root@oel65a bin]# ./crsctl status resource -t
-------------------------------------------------------------------------
-------
Name Target State Server State
details
-------------------------------------------------------------------------
-------
Local Resources
-------------------------------------------------------------------------
-------
ora.DATA.dg
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.LISTENER.lsnr
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.asm
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b
Started,STABLE
ora.net1.network
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.ons
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
-------------------------------------------------------------------------
-------
Cluster Resources
-------------------------------------------------------------------------
-------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel65b STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oel65a STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oel65a STABLE
ora.cvu
1 ONLINE ONLINE oel65a STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.oel65a.vip
1 ONLINE ONLINE oel65a STABLE
ora.oel65b.vip
1 ONLINE ONLINE oel65b STABLE
ora.scan1.vip
1 ONLINE ONLINE oel65b STABLE
ora.scan2.vip
1 ONLINE ONLINE oel65a STABLE
ora.scan3.vip
1 ONLINE ONLINE oel65a STABLE
-------------------------------------------------------------------------
-------
[root@oel65a bin]#
5.6 This concludes the GI installation.
6. Install RAC RDBMS 12c on oel65a and oel65b
6.1 Login as oracle and run the OUI to install Oracle 12c
binaries only.
Skip security updates.
Skip software updates.
Select Install software only.
Select RAC database.
Select all nodes.
Select languages.
Select enterprise Edition.
Verify the locations.
Verify the OS groups.
Wait for the check to complete.
The errors are listed below. I can ignore them.
Single Client Access Name (SCAN) - This test verifies the Single Client
Access Name configuration. Error:
-
PRVG-1101 : SCAN name "scan" failed to resolve - Cause: An attempt to
resolve specified SCAN name to a list of IP addresses failed because SCAN
could not be resolved in DNS or GNS using 'nslookup'. - Action: Check
whether the specified SCAN name is correct. If SCAN name should be
resolved in DNS, check the configuration of SCAN name in DNS. If it
should be resolved in GNS make sure that GNS resource is online.
-
PRVF-4657 : Name resolution setup check for "scan" (IP address:
192.168.2.51) failed - Cause: Inconsistent IP address definitions found
for the SCAN name identified using DNS and configured name resolution
mechanism(s). - Action: Look up the SCAN name with nslookup, and make
sure the returned IP addresses are consistent with those defined in NIS
and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the
latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing
its cache and restarting it.
-
PRVF-4657 : Name resolution setup check for "scan" (IP address:
192.168.2.52) failed - Cause: Inconsistent IP address definitions found
for the SCAN name identified using DNS and configured name resolution
mechanism(s). - Action: Look up the SCAN name with nslookup, and make
sure the returned IP addresses are consistent with those defined in NIS
and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the
latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing
its cache and restarting it.
-
PRVF-4657 : Name resolution setup check for "scan" (IP address:
192.168.2.53) failed - Cause: Inconsistent IP address definitions found
for the SCAN name identified using DNS and configured name resolution
mechanism(s). - Action: Look up the SCAN name with nslookup, and make
sure the returned IP addresses are consistent with those defined in NIS
and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the
latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing
its cache and restarting it.
Check Failed on Nodes: [oel65b, oel65a]
Verification result of failed node: oel65b Details:
-
PRVF-4664 : Found inconsistent name resolution entries for SCAN name
"scan" - Cause: The nslookup utility and the configured name resolution
mechanism(s), as defined in /etc/nsswitch.conf, returned inconsistent IP
address information for the SCAN name identified. - Action: Check the
Name Service Cache Daemon (/usr/sbin/nscd), the Domain Name Server
(nslookup) and the /etc/hosts file to make sure the IP address for the
SCAN names are registered correctly.
Back to Top
Verification result of failed node: oel65a Details:
-
PRVF-4664 : Found inconsistent name resolution entries for SCAN name
"scan" - Cause: The nslookup utility and the configured name resolution
mechanism(s), as defined in /etc/nsswitch.conf, returned inconsistent IP
address information for the SCAN name identified. - Action: Check the
Name Service Cache Daemon (/usr/sbin/nscd), the Domain Name Server
(nslookup) and the /etc/hosts file to make sure the IP address for the
SCAN names are registered correctly.
Back to Top
Task resolv.conf Integrity - This task checks consistency of file
/etc/resolv.conf file across nodes
Check Failed on Nodes: [oel65b, oel65a]
Verification result of failed node: oel65b Details:
-
PRVF-5636 : The DNS response time for an unreachable node exceeded
"15000" ms on following nodes: oel65a,oel65b - Cause: The DNS response
time for an unreachable node exceeded the value specified on nodes
specified. - Action: Make sure that 'options timeout', 'options
attempts' and 'nameserver' entries in file resolv.conf are proper. On
HPUX these entries will be 'retrans', 'retry' and 'nameserver'. On
Solaris these will be 'options retrans', 'options retry' and
'nameserver'. Make sure that the DNS server responds back to name lookup
request within the specified time when looking up an unknown host name.
-
Check for integrity of file "/etc/resolv.conf" failed - Cause: Cause Of
Problem Not Available - Action: User Action Not Available
Back to Top
Verification result of failed node: oel65a Details:
-
PRVF-5636 : The DNS response time for an unreachable node exceeded
"15000" ms on following nodes: oel65a,oel65b - Cause: The DNS response
time for an unreachable node exceeded the value specified on nodes
specified. - Action: Make sure that 'options timeout', 'options
attempts' and 'nameserver' entries in file resolv.conf are proper. On
HPUX these entries will be 'retrans', 'retry' and 'nameserver'. On
Solaris these will be 'options retrans', 'options retry' and
'nameserver'. Make sure that the DNS server responds back to name lookup
request within the specified time when looking up an unknown host name.
-
Check for integrity of file "/etc/resolv.conf" failed - Cause: Cause Of
Problem Not Available - Action: User Action Not Available
Back to Top
Review and start the installation.
Wait until prompted to run scripts as root.
[root@oel65a bin]# /u01/app/oracle/product/12.1.0/db_1/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/12.1.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@oel65a bin]#
[root@oel65b grid]# /u01/app/oracle/product/12.1.0/db_1/root.sh
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/12.1.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@oel65b grid]#
Exit
6.2 This concludes the 12c binaries installation.
7. Create a Multitenant Container Database (DB) database RACDB oel65a and oel65b
7.1 Login as grid and start asmca to create FRA disk group
Click create and select disks for the FRA disk group. Press OK
to create the group.
Wait until confirmation.
Now you have the disk groups
7.2 Login as oracle and start dbca.
Select create a database.
Select Advanced Mode
Select options that interest you.
Select a CDB option.
Select all nodes.
Select the crosschecked options
Select a password.
Specify the storage.
Skip the Vault configurations.
Configure AMM
Configure character sets.
Select what to create.
Wait for the prerequisite checks to complete.
Review the components to be installed.
Wait for the creation to complete.
You can change the passwords. Note the URL
Login to the URL
Examine what is inside.
7.3 This concludes the CDB creation.
8. Verify database creation and create a service 8.1 Perform Verifications
[root@oel65a bin]# ./crsctl status resource -t
-------------------------------------------------------------------------
-------
Name Target State Server State
details
-------------------------------------------------------------------------
-------
Local Resources
-------------------------------------------------------------------------
-------
ora.DATA.dg
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.FRA.dg
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.LISTENER.lsnr
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.asm
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b
Started,STABLE
ora.net1.network
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.ons
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
-------------------------------------------------------------------------
-------
Cluster Resources
-------------------------------------------------------------------------
-------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel65b STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oel65a STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oel65a STABLE
ora.cvu
1 ONLINE ONLINE oel65a STABLE
ora.db12c.db
1 ONLINE ONLINE oel65a Open,STABLE
2 ONLINE ONLINE oel65b Open,STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.oel65a.vip
1 ONLINE ONLINE oel65a STABLE
ora.oel65b.vip
1 ONLINE ONLINE oel65b STABLE
ora.scan1.vip
1 ONLINE ONLINE oel65b STABLE
ora.scan2.vip
1 ONLINE ONLINE oel65a STABLE
ora.scan3.vip
1 ONLINE ONLINE oel65a STABLE
-------------------------------------------------------------------------
-------
[root@oel65a bin]#
[oracle@oel65b ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.1.0 Production on Wed May 14 21:06:19 2014
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit
Production
With the Partitioning, Real Application Clusters, Automatic Storage
Management, OLAP,
Advanced Analytics and Real Application Testing options
SQL> set linesize 400
SQL> select * from v$active_instances;
INST_NUMBER INST_NAME
CON_ID
----------- -------------------------------------------------------------
-------------------------------------------------------------------------
-------------------------------------------------------------------------
--------------------------------- ----------
1 oel65a.gj.com:DB12C1
0
2 oel65b.gj.com:DB12C2
0
SQL>
[root@oel65a bin]# ./crsctl check cluster -all
**************************************************************
oel65a:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
oel65b:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@oel65a bin]# ./crsctl status resource -t
-------------------------------------------------------------------------
-------
Name Target State Server State
details
-------------------------------------------------------------------------
-------
Local Resources
-------------------------------------------------------------------------
-------
ora.DATA.dg
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.FRA.dg
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.LISTENER.lsnr
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.asm
ONLINE ONLINE oel65a
Started,STABLE
ONLINE ONLINE oel65b
Started,STABLE
ora.net1.network
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
ora.ons
ONLINE ONLINE oel65a STABLE
ONLINE ONLINE oel65b STABLE
-------------------------------------------------------------------------
-------
Cluster Resources
-------------------------------------------------------------------------
-------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE oel65b STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE oel65a STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE oel65a STABLE
ora.cvu
1 ONLINE ONLINE oel65a STABLE
ora.db12c.db
1 ONLINE ONLINE oel65a Open,STABLE
2 ONLINE ONLINE oel65b Open,STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.oel65a.vip
1 ONLINE ONLINE oel65a STABLE
ora.oel65b.vip
1 ONLINE ONLINE oel65b STABLE
ora.scan1.vip
1 ONLINE ONLINE oel65b STABLE
ora.scan2.vip
1 ONLINE ONLINE oel65a STABLE
ora.scan3.vip
1 ONLINE ONLINE oel65a STABLE
-------------------------------------------------------------------------
-------
[root@oel65a bin]#
SQL> !srvctl config database -d db12c
Database unique name: DB12C
Database name: DB12C
Oracle home: /u01/app/oracle/product/12.1.0/db_1
Oracle user: oracle
Spfile: +DATA/DB12C/spfileDB12C.ora
Password file: +DATA/DB12C/orapwdb12c
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: DB12C
Database instances: DB12C1,DB12C2
Disk Groups: FRA,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
Database is administrator managed
SQL>
SQL> select dbms_xdb_config.gethttpsport from dual;
GETHTTPSPORT
------------
5500
SQL> SQL>
8.2 Check the PDBs
[oracle@oel65a ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.1.0 Production on Mon May 26 21:44:30 2014
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit
Production
With the Partitioning, Real Application Clusters, Automatic Storage
Management, OLAP,
Advanced Analytics and Real Application Testing options
SQL> set linesize 400
SQL> select * from cdb_pdbs;
PDB_ID PDB_NAME
DBID CON_UID GUID
STATUS CREATION_SCN CON_ID
---------- -------------------------------------------------------------
------------------------------------------------------------------- ----
------ ---------- -------------------------------- ------------- -------
----- ----------
3 PDB12C1
2284111037 2284111037
F95F7081C42A2554E0430B02A8C0FA31 NORMAL 1933353 1
2 PDB$SEED
4090173063 4090173063
F95E5546BA4A17E1E0430B02A8C01013 NORMAL 1720414 1
4 PDB12C2
1979843754 1979843754
F95F91A081152691E0430B02A8C0D522 NORMAL 1939488 1
5 PDB12C3
2029633601 2029633601
F95FADC38A482785E0430B02A8C039EC NORMAL 1944720 1
6 PDB12C4
2035084802 2035084802
F95FCC4A123D28DEE0430B02A8C03CBF NORMAL 1951902 1
7 PDB12C5
2084851122 2084851122
F95FE9ADF19D29F8E0430B02A8C08DB7 NORMAL 1958038 1
6 rows selected.
SQL> select * from v$pdbs;
CON_ID DBID CON_UID GUID NAME
OPEN_MODE RES OPEN_TIME CREATE_SCN TOTAL_SIZE
---------- ---------- ---------- -------------------------------- ------
------------------------ ---------- --- --------------------------------
------------------------------------------- ---------- ----------
2 4090173063 4090173063 F95E5546BA4A17E1E0430B02A8C01013 PDB$SEED
READ ONLY NO 26-MAY-14 02.13.03.875 PM
1720414 283115520
3 2284111037 2284111037 F95F7081C42A2554E0430B02A8C0FA31 PDB12C1
MOUNTED 1933353 0
4 1979843754 1979843754 F95F91A081152691E0430B02A8C0D522 PDB12C2
MOUNTED 1939488 0
5 2029633601 2029633601 F95FADC38A482785E0430B02A8C039EC PDB12C3
MOUNTED 1944720 0
6 2035084802 2035084802 F95FCC4A123D28DEE0430B02A8C03CBF PDB12C4
MOUNTED 1951902 0
7 2084851122 2084851122 F95FE9ADF19D29F8E0430B02A8C08DB7 PDB12C5
MOUNTED 1958038 0
6 rows selected.
SQL>
8.3 This concludes the CDB creation
Annex A
[root@oel65a ~]# yum install oracle-rdbms-server-12cR1-preinstall
Loaded plugins: aliases, changelog, downloadonly, kabi, presto, refresh-
packagekit, security, tmprepo, verify, versionlock
Loading support for kernel ABI
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-11.el6
will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=========================================================================
=========================================================================
========================
Package Arch
Version Repository
Size
=========================================================================
=========================================================================
========================
Installing:
oracle-rdbms-server-12cR1-preinstall x86_64
1.0-11.el6 public_ol6_latest
15 k
Transaction Summary
=========================================================================
=========================================================================
========================
Install 1 Package(s)
Total download size: 15 k
Installed size: 38 k
Is this ok [y/N]: Y
Downloading Packages:
Setting up and reading Presto delta metadata
Processing delta metadata
Package(s) data still to download: 15 k
oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64.rpm
| 15 kB 00:00
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID
ec551f03: NOKEY
Retrieving key from http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6
Importing GPG key 0xEC551F03:
Userid: "Oracle OSS group (Open Source Software group)
From : http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6
Is this ok [y/N]: Y
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64
1/1
Verifying : oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64
1/1
Installed:
oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-11.el6
Complete!
[root@oel65a ~]#
[grid@oel65a grid]$ ./runcluvfy.sh stage -pre crsinst -n oel65a,oel65b
[grid@oel65a grid]$ ./runcluvfy.sh stage -pre crsinst -n oel65a,oel65b
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "oel65a"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Node connectivity passed for subnet "10.10.10.0" with node(s)
oel65a,oel65b
TCP connectivity check passed for subnet "10.10.10.0"
Node connectivity passed for subnet "192.168.2.0" with node(s)
oel65a,oel65b
TCP connectivity check passed for subnet "192.168.2.0"
Node connectivity passed for subnet "192.168.122.0" with node(s)
oel65a,oel65b
ERROR:
PRVF-7617 : Node connectivity between "oel65a : 192.168.122.1" and
"oel65b : 192.168.122.1" failed
TCP connectivity check failed for subnet "192.168.122.0"
Interfaces found on subnet "192.168.2.0" that are likely candidates for
VIP are:
oel65a eth1:192.168.2.11
oel65b eth1:192.168.2.12
Interfaces found on subnet "10.10.10.0" that are likely candidates for a
private interconnect are:
oel65a eth0:10.10.10.71
oel65b eth0:10.10.10.72
Interfaces found on subnet "10.10.10.0" that are likely candidates for a
private interconnect are:
oel65a eth2:10.10.10.11
oel65b eth2:10.10.10.12
Interfaces found on subnet "10.10.10.0" that are likely candidates for a
private interconnect are:
oel65a eth3:10.10.10.81
oel65b eth3:10.10.10.82
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "10.10.10.0".
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.122.0".
Subnet mask consistency check passed.
ERROR:
PRVG-1172 : The IP address "192.168.122.1" is on multiple interfaces
"virbr0,virbr0" on nodes "oel65a,oel65b"
Node connectivity check failed
Checking multicast communication...
Checking subnet "10.10.10.0" for multicast communication with multicast
group "224.0.0.251"...
Check of subnet "10.10.10.0" for multicast communication with multicast
group "224.0.0.251" passed.
Check of multicast communication passed.
Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for
"oel65b:/usr,oel65b:/var,oel65b:/etc,oel65b:/sbin"
Free disk space check passed for
"oel65a:/usr,oel65a:/var,oel65a:/etc,oel65a:/sbin"
Free disk space check passed for "oel65b:/tmp"
Free disk space check passed for "oel65a:/tmp"
Check for multiple users with UID value 54322 passed
User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" failed
Check failed on nodes:
oel65b,oel65a
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "nfs-utils"
Checking availability of ports "6200,6100" required for component "Oracle
Notification Service (ONS)"
Port availability check passed for ports "6200,6100"
Check for multiple users with UID value 0 passed
Current group ID check passed
Starting check for consistency of primary group of root user
Check for consistency of root user's primary group passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
NTP Configuration file check passed
Checking daemon liveness...
Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
NTP common Time Server Check started...
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Clock time offset check passed
Clock synchronization check using Network Time Protocol(NTP) passed
Core file name pattern consistency check passed.
User "grid" is not part of "root" group. Check passed
Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes
"domain" and "search" entries do not coexist in any "/etc/resolv.conf"
file
All nodes have same "search" order defined in file "/etc/resolv.conf"
PRVF-5636 : The DNS response time for an unreachable node exceeded
"15000" ms on following nodes: oel65a,oel65b
Check for integrity of file "/etc/resolv.conf" failed
Time zone consistency check passed
Checking integrity of name service switch configuration file
"/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file
"/etc/nsswitch.conf" passed
Checking daemon "avahi-daemon" is not configured and running
Daemon not configured check passed for process "avahi-daemon"
Daemon not running check passed for process "avahi-daemon"
Starting check for /dev/shm mounted as temporary file system ...
Check for /dev/shm mounted as temporary file system passed
Pre-check for cluster services setup was unsuccessful on all the nodes.
[grid@oel65a grid]$
[grid@oel65a grid]$ ./runcluvfy.sh stage -post hwos -n oel65a,oel65b
Performing post-checks for hardware and operating system setup
Checking node reachability...
Node reachability check passed from node "oel65a"
Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Node connectivity passed for subnet "10.10.10.0" with node(s)
oel65a,oel65b
TCP connectivity check passed for subnet "10.10.10.0"
Node connectivity passed for subnet "192.168.2.0" with node(s)
oel65a,oel65b
TCP connectivity check passed for subnet "192.168.2.0"
Node connectivity passed for subnet "192.168.122.0" with node(s)
oel65a,oel65b
ERROR:
PRVF-7617 : Node connectivity between "oel65a : 192.168.122.1" and
"oel65b : 192.168.122.1" failed
TCP connectivity check failed for subnet "192.168.122.0"
Interfaces found on subnet "192.168.2.0" that are likely candidates for
VIP are:
oel65a eth1:192.168.2.11
oel65b eth1:192.168.2.12
Interfaces found on subnet "10.10.10.0" that are likely candidates for a
private interconnect are:
oel65a eth0:10.10.10.71
oel65b eth0:10.10.10.72
Interfaces found on subnet "10.10.10.0" that are likely candidates for a
private interconnect are:
oel65a eth2:10.10.10.11
oel65b eth2:10.10.10.12
Interfaces found on subnet "10.10.10.0" that are likely candidates for a
private interconnect are:
oel65a eth3:10.10.10.81
oel65b eth3:10.10.10.82
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "10.10.10.0".
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed for subnet "192.168.122.0".
Subnet mask consistency check passed.
ERROR:
PRVG-1172 : The IP address "192.168.122.1" is on multiple interfaces
"virbr0,virbr0" on nodes "oel65b,oel65a"
Node connectivity check failed
Checking multicast communication...
Checking subnet "10.10.10.0" for multicast communication with multicast
group "224.0.0.251"...
Check of subnet "10.10.10.0" for multicast communication with multicast
group "224.0.0.251" passed.
Check of multicast communication passed.
Check for multiple users with UID value 0 passed
Time zone consistency check passed
Checking shared storage accessibility...
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdb oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdb1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdc oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdc1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdd oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdd1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sde oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sde1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdf oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdf1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdg oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdg1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdh oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdh1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdi oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdi1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdj oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdj1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdk oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdk1 oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdl oel65a oel65b
Disk Sharing Nodes (2 in count)
------------------------------------ ------------------------
/dev/sdl1 oel65a oel65b
Shared storage check was successful on nodes "oel65a,oel65b"
Checking integrity of name service switch configuration file
"/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file
"/etc/nsswitch.conf" passed
Post-check for hardware and operating system setup was unsuccessful on
all the nodes.
[grid@oel65a grid]$
[root@oel65a ~]# su - grid
[grid@oel65a ~]$ cat .bash_profile
# .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
umask 022
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=/u01/app/12.1.0.1/grid
ORACLE_HOSTNAME=oel65a
ORACLE_SID=+ASM1
LD_LIBRARY_PATH=$ORACLE_HOME/lib
PATH=$PATH:$ORACLE_HOME/bin
export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH
ORACLE_HOSTNAME
TEMP=/tmp
TMPDIR=/tmp
export TEMP TMPDIR
ulimit -t unlimited
ulimit -f unlimited
ulimit -d unlimited
ulimit -s 32768
ulimit -v unlimited
ulimit -n 4096
if [ -t 0 ]; then
stty intr ^C
fi
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
[grid@oel65a ~]$
[root@oel65a ~]# su - oracle
[oracle@oel65a ~]$ cat .bash_profile
# .bash_profile
# .bash_profile
umask 022
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1
ORACLE_HOSTNAME=oel65a
ORACLE_SID=DB12C1
ORACLE_UNQNAME=DB12C
LD_LIBRARY_PATH=$ORACLE_HOME/lib
PATH=$PATH:$ORACLE_HOME/bin
export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH
ORACLE_HOSTNAME ORACLE_UNQNAME
TEMP=/tmp
TMPDIR=/tmp
export TEMP TMPDIR
ulimit -t unlimited
ulimit -f unlimited
ulimit -d unlimited
ulimit -s 32768
ulimit -v unlimited
ulimit -n 4096
if [ -t 0 ]; then
stty intr ^C
fi
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
[oracle@oel65a ~]$
[grid@oel65b ~]$ cat .bash_profile
# .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
umask 022
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=/u01/app/12.1.0.1/grid
ORACLE_HOSTNAME=oel65b
ORACLE_SID=+ASM1
LD_LIBRARY_PATH=$ORACLE_HOME/lib
PATH=$PATH:$ORACLE_HOME/bin
export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH
ORACLE_HOSTNAME
TEMP=/tmp
TMPDIR=/tmp
export TEMP TMPDIR
ulimit -t unlimited
ulimit -f unlimited
ulimit -d unlimited
ulimit -s 32768
ulimit -v unlimited
ulimit -n 4096
if [ -t 0 ]; then
stty intr ^C
fi
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
[grid@oel65b ~]$
[root@oel65b ~]# su - oracle
[oracle@oel65b ~]$ cat .bash_profile
# .bash_profile
# .bash_profile
umask 022
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1
ORACLE_HOSTNAME=oel65b
ORACLE_SID=DB12C2
ORACLE_UNQNAME=DB12C
LD_LIBRARY_PATH=$ORACLE_HOME/lib
PATH=$PATH:$ORACLE_HOME/bin
export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH
ORACLE_HOSTNAME ORACLE_UNQNAME
TEMP=/tmp
TMPDIR=/tmp
export TEMP TMPDIR
ulimit -t unlimited
ulimit -f unlimited
ulimit -d unlimited
ulimit -s 32768
ulimit -v unlimited
ulimit -n 4096
if [ -t 0 ]; then
stty intr ^C
fi
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
[oracle@oel65b ~]$
run OUI
00:26:18.768975 VM: Raising runtime error 'HostMemoryLow' (fFlags=0x2)
00:26:18.770070 AssertLogRel D:\tinderbox\win-
4.3\src\VBox\VMM\VMMR3\VMM.cpp(1756) int __cdecl
VMMR3EmtRendezvous(struct VM *,unsigned int,int (__cdecl *)(struct VM
*,struct VMCPU *,void *),void *): !pVCpu->vmm.s.fInRendezvous
00:26:18.791552 Console: VM runtime error: fatal=false,
errorID=HostMemoryLow message="Unable to allocate and lock memory. The
virtual machine will be paused. Please close applications to free up
memory or close the VM"
00:26:18.825633 Changing the VM state from 'RUNNING' to
'GURU_MEDITATION'.