ORACLE RAC

ORACLE RAC

Definition

Oracle Real Application Cluster (RAC) is a cluster system at the application level. It uses shared disk architecture that provides scalability for all kind of applications. Applications without any modifications can use the RAC database. (Somewhat)

ORACLE RAC

Goal

Since the requests in a RAC cluster are spread evenly across the RAC instances, and since all instances access the same shared storage, addition of server(s) require no architecture changes etc. And a failure of a single RAC node results only in the loss of scalability and not in the loss of data since a single database image is utilized.

ORACLE RAC

A requirement for Oracle Database 10g RAC cluster is a set of servers with shared disk access and interconnect connectivity. Since each instance in a RAC system must have access to the same database files, a shared storage is required that can be accessed from all RAC nodes concurrently.

Shared Disks

ORACLE RAC

The shared storage space can be used as raw devices, or by using a cluster file system or ASM. This article will address Oracle's Cluster File System OCFS and ASM. Note that Oracle 10g RAC provides it's own locking mechanisms and therefore it does not rely on other cluster software or on the operating system for handling locks.

Types of Shared Storage

ORACLE RAC

It is recommended to use newer Red Hat Enterprise Linux kernels since newer kernels might fix known database performance problems and other issues. However, you also need to make sure that the OCFS and ASM drivers are compatible with the kernel version!

Upgrading the Kernel or Parameters

ORACLE RAC

Each RAC node should have at least one static IP address for the public network and one static IP address for the private cluster interconnect.

Configuring the Network

ORACLE RAC

The private networks are critical components of a RAC cluster. The private networks should only be used by Oracle to carry Cluster Manager and Cache Fusion inter-node connection. A RAC database does not require a separate private network, but using a public network can degrade database performance (high latency, low bandwidth). Therefore the private network should have high-speed NICs (preferably one gigabit or more) and it should only be used by Oracle.

Private Networks

ORACLE RAC

Here is an example how the /etc/hosts file could look like: # Public hostnames for e.g. eth0 interfaces (public network)

216.88.5.1 rac1pub.puschitz.com rac1pub # RAC node 1216.88.5.2 rac2pub.puschitz.com rac2pub # RAC node 2216.88.5.3 rac3pub.puschitz.com rac3pub # RAC node 3# Private hostnames, private network for e.g. eth1 interfaces 192.168.2.1 rac1prv.puschitz.com rac1prv # RAC node 1192.168.2.2 rac2prv.puschitz.com rac2prv # RAC node 2192.168.2.3 rac3prv.puschitz.com rac3prv # RAC node 3

Network Example

ORACLE RAC

Configuring Shared Storage

fdisk -l

Disk /dev/sda: 255 heads, 63 sectors, 30515 cylindersUnits = cylinders of 16065 * 512 bytes

Device Boot Start End Blocks Id System/dev/sda1 1 6375 51207156 83 Linux/dev/sda2 6376 12750 51207187+ 83 Linux

ORACLE RAC

If you use OCFS, it is important that the UID of "oracle" and GID of "oinstall" are the same across all RAC nodes. Otherwise the Oracle files on the OCFS filesystems on some nodes could either be "unowned", or they could even be owned by another user account. In my setup the UID and GID of oracle:dba is 700:700.

su - rootgroupadd -g 700 dba # group of users to be granted with SYSDBA system privilegegroupadd -g 701 oinstall # group owner of Oracle filesuseradd -c "Oracle software owner" -u 700 -g oinstall -G dba oraclepasswd oracle

Creating Oracle User

ORACLE RAC

Since the Oracle Universal Installer (OUI) "runInstaller" is executed from the oracle account, some environment variables must be configured for the oracle account before OUI is started.

If you use bash which is the default shell on Red Hat Linux (to verify your shell run: echo $SHELL), execute the following commands: # Oracle Environmentexport ORACLE_BASE=/u01/app/oracleexport ORACLE_SID=orcl1 # Each RAC node must have a unique Oracle SID! E.g. orcl1, orcl2,...export LD_LIBRARY_PATH=$ORACLE_HOME/lib

Setting Oracle Accounts

ORACLE RAC

At the time of this writing, OCFS only supports Oracle Datafiles and a few other files. Therefore OCFS should not be used for Shared Oracle Home installs. See Installing and Configuring Oracle Cluster File Systems (OCFS) for more information.

Cluster File System

ORACLE RAC

If you want to comply with Oracle's Optimal Flexible Architecture (OFA), then you don't want to place the database files in the /u01 directory but in another directory like /u02. This is not a requirement but if you want to comply with OFA, then you might want to create the following directories as well: su - rootmkdir -p /u02/oradata/orclchown -R oracle.oinstall /u02

Creating Oracle Directories

ORACLE RAC

Note that it is important for the Redo Log files to be on the shared disks as well.

To partition the disks, you can use the fdisk utility: su - rootfdisk <device_name>

Here is an example how to create a new 50 GB partition on drive /dev/sda:

su - root# fdisk /dev/sda

Creating Partitions

ORACLE RAC

The Oracle Cluster File System (OCFS) was developed by Oracle to overcome the limits of Raw Devices and Partitions. It also eases administration of database files because it looks and feels just like a regular file system.

OCFS only supports Oracle Datafiles and a few other files:- Redo Log files- Archive log files - Control files- Database datafiles- Shared quorum disk file for the cluster manager - Shared init file (srv)

Installing and Configuring OCFS

ORACLE RAC

To create the OCFS filesystems, you can use the ocfstool: su - rootocfstool - Select "Task" - Select "Format"

Alternatively, you can execute the "mkfs.ocfs" command to create the OCFS filesystems:

su - rootmkfs.ocfs -F -b 128 -L /u02/oradata/orcl -m /u02/oradata/orcl \ -u `id -u oracle` -g `id -g oracle` -p 0775 <device_name>

Creating OCS File Systems

ORACLE RAC

Add lines to the /etc/fstab file similar to the following example: /dev/sda1 /u02/oradata/orcl ocfs _netdev 0 0

The "_netdev" option prevents the OCFS filesystem from being mounted until the network has first been enabled on the system. To make sure the ocfs.o kernel module is loaded and the OCFS file systems are mounted during the boot process, enter:

su - root# chkconfig --list ocfsocfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off

Having File System a Boot Time

ORACLE RAC

Oracle uses the Linux kernel module hangcheck-timer to monitor the system health of the cluster and to reset a RAC node in case of failures. The hangcheck-timer module uses a kernel-based timer to periodically check the system task scheduler. This timer resets the node when the system hangs or pauses. This module uses the Time Stamp Counter (TSC) CPU register which is a counter that is incremented at each clock signal.

The hangcheck-timer module comes now with the kernel:

find /lib/modules -name "hangcheck-timer.o"

Configue Hangcheck Timer in Kernel

ORACLE RAC

When you run the Oracle Installer on a RAC node, it will use ssh to copy Oracle software and data to other RAC nodes. Therefore, the oracle user on the RAC node where Oracle Installer is launched must be able to login to other RAC nodes without having to provide a password or passphrase.

The following procedure shows how ssh can be configured that no password is requested for oracle ssh logins.

To create an authentication key for oracle, enter the following command on all RAC node:

Remote Access for RAC Nodes

ORACLE RAC

Sizing Swap Space

Setting Shared Memory

Checking /tmp Space

Setting Semaphores

Setting File Handles

More System Setup

ORACLE RAC

 Cluster Ready Services (CRS) contains cluster and database configuration information for RAC, and it provides many system management features. CRS accepts registration of Oracle instances to the cluster and it sends ping messages to other RAC nodes. If the heartbeat fails, CRS will use shared disk to distinguish between a node failure and a network failure.

Once CRS is running on all RAC nodes, OUI will automatically recognice all nodes on the cluster. This means that you can run OUI on one RAC node to install the Oracle software on all other RAC nodes.

Installing Cluster Ready Services

ORACLE RAC

Make sure that the oracle user can ssh to all RAC nodes without ssh asking for a passphrase. This is very important because otherwise OUI won't be able to install the Oracle software on other RAC nodes. The following example shows how ssh-agent can do the authentication for you when the oracle account logs in to other RAC nodes using ssh.

Open a new terminal for the RAC node where you will execute runInstaller and use this terminal to login from your desktop using the following command:

$ ssh -X oracle@rac?pub

Starting The Installer

ORACLE RAC

Now configure ssh-agent to handle the authentication for the oracle account: oracle$ ssh-agent $SHELLoracle$ ssh-addEnter passphrase for /home/oracle/.ssh/id_dsa: Enter your passphraseIdentity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)oracle$

Configure ssh-agent

ORACLE RAC

In order to install the Cluster Ready Services (CRS) R1 (10.1.0.2) on all RAC nodes, OUI has to be launched on only one RAC node. To install CRS, insert the "Cluster Ready Services (CRS) R1 (10.1.0.2)" CD (downloadedd image name: "ship.crs.cpio.gz"), and mount it on e.g. rac1pub:

su - rootmount /mnt/cdromUse the oracle terminal that you prepared for ssh at Automating Authentication for oracle ssh Logins and execute runInstaller:

oracle$ /mnt/cdrom/runInstaller

Install Cluster Ready Services

ORACLE RAC

The following procedure shows the installation of the software for Oracle Database 10g Software R1 (10.1.0.2) with Real Application Clusters (RAC).

Note that Oracle Database 10g R1 (10.1) OUI will not be able to discover disks that are marked as Linux ASMLib. Therefore it is recommended to complete the software installation and then to use dbca to create the database, see http://otn.oracle.com/tech/linux/asmlib/install.html#10gr1 for more information.

Installing ORACLE with RAC

ORACLE RAC

The following steps show how to use dbca to create the database and its instances. Oracle recommends to use dbca to create the RAC database since the preconfigured databases are optimized for ASM, server parameter file, and automatic undo management. dbca also makes it much more easier to create new ASM disk groups etc.

Setting Up the Database with dbca

ORACLE RAC

Database File Locations: Select "Use Oracle-Managed Files" Database Area: +ORCL_DATA1 Click Next I selected TAF Policy "Basic". Click Next - Initialization Parameters: Change settings as needed. Click Next - Database Storage: Change settings as needed. Click Next - Creation Options: Check "Create Database" Click Finish

Now the database is being created.

DBCA concluded (creating database)

ORACLE RAC

RAC requires a CFS

RAC is complex

RAC provides transactions (OLTP)

Summary