Real Application Cluster (RAC) Kishore A

Real Application Real Application Cluster (RAC)Cluster (RAC)

Kishore AKishore A

Oracle10g - RACOracle10g - RAC What is all the hype about grid computing?What is all the hype about grid computing? Grid computing is intended to allow businesses to move Grid computing is intended to allow businesses to move

away from the idea of many individual servers, each of away from the idea of many individual servers, each of which is dedicated to a small number of applications. which is dedicated to a small number of applications. When configured in this manner, applications often either When configured in this manner, applications often either do not fully utilize the server’s available hardware do not fully utilize the server’s available hardware resource such as memory, CPU and disk or short of these resource such as memory, CPU and disk or short of these resources during peak usage.resources during peak usage.

Grid computing addresses these problems by providing an Grid computing addresses these problems by providing an adaptive software infrastructure that makes efficient use adaptive software infrastructure that makes efficient use of low-cost servers and modular storage, which balances of low-cost servers and modular storage, which balances work- loads more effectively and provides capacity on work- loads more effectively and provides capacity on demanddemand

By scaling out with small servers in small increments, you By scaling out with small servers in small increments, you get performance and reliability at low-cost. New unified get performance and reliability at low-cost. New unified management allows you to manage everything cheaply management allows you to manage everything cheaply and simply in the grid.and simply in the grid.

WHAT IS ENTERPRISE GRID COMPUTING?WHAT IS ENTERPRISE GRID COMPUTING?

Implement One from Many.Implement One from Many. Grid computing coordinates the use of clusters of Grid computing coordinates the use of clusters of

machines to create a single logical entity, such as a machines to create a single logical entity, such as a database or andatabase or an

application server. application server. By distributing work across many servers, grid By distributing work across many servers, grid

computing exhibits benefits of availability, scalability, computing exhibits benefits of availability, scalability, and performance using low-cost components. and performance using low-cost components.

Because a single logical entity is implemented across Because a single logical entity is implemented across many machines, companies can add or remove capacity many machines, companies can add or remove capacity in small increments, online. in small increments, online.

With the capability to add capacity on demand to a With the capability to add capacity on demand to a particular function, companies get more flexibility for particular function, companies get more flexibility for adapting to peak loads, thus achieving better hardware adapting to peak loads, thus achieving better hardware utilization and better business responsiveness.utilization and better business responsiveness.

Benefits of Enterprise Grid ComputingBenefits of Enterprise Grid Computing The primary benefit of grid computing to businesses The primary benefit of grid computing to businesses

is achieving high quality of service and flexibility at is achieving high quality of service and flexibility at lower cost. lower cost.

Enterprise grid computing lowers costs by:Enterprise grid computing lowers costs by:

Increasing hardware utilization and resource Increasing hardware utilization and resource sharingsharing

Enabling companies to scale out incrementally with Enabling companies to scale out incrementally with low- low-

cost componentscost components

Reducing management and administration Reducing management and administration requirementsrequirements

New Trends in HardwareNew Trends in Hardware Much of what makes grid computing possible today are the Much of what makes grid computing possible today are the

innovations in hardware. For example,innovations in hardware. For example,ProcessorProcessors. New low-cost, high volume Intel Itanium 2, Sun SPARC, s. New low-cost, high volume Intel Itanium 2, Sun SPARC,

and IBM PowerPC 64-bit processors now deliver performance equal to and IBM PowerPC 64-bit processors now deliver performance equal to or better than exotic processors used in high-end SMP servers.or better than exotic processors used in high-end SMP servers.

Blade servers. Blade servers. Blade server technology reduces the cost of hardwareBlade server technology reduces the cost of hardware and increases the density of servers, which further reduces and increases the density of servers, which further reduces

expensiveexpensive data center real estate requirements.data center real estate requirements.Networked storagNetworked storage. Disk storage costs continue to plummet evene. Disk storage costs continue to plummet even faster than processor costs. Network storage technologies such asfaster than processor costs. Network storage technologies such as Network Attached Storage (NAS) and Storage Area Networks (SANs) Network Attached Storage (NAS) and Storage Area Networks (SANs)

further reduce these costs by enabling sharing of storage across further reduce these costs by enabling sharing of storage across systems.systems.

Network interconnectNetwork interconnects. Gigabit Ethernet and Infiniband s. Gigabit Ethernet and Infiniband interconnect technologies are driving down the cost of connecting interconnect technologies are driving down the cost of connecting servers into clusters.servers into clusters.

Oracle Database 10Oracle Database 10gg

Oracle Database 10Oracle Database 10g g builds on the builds on the success of Oracle9success of Oracle9i i Database, and adds Database, and adds many new grid-specific capabilities. many new grid-specific capabilities.

Oracle Database 10Oracle Database 10g g is based on Real is based on Real Application Clusters, introduced in Application Clusters, introduced in Oracle9Oracle9ii. .

There are more than 500 production There are more than 500 production customers running Oracle’s clustering customers running Oracle’s clustering technology, helping to prove the validity technology, helping to prove the validity of Oracle’s grid infrastructure.of Oracle’s grid infrastructure.

Real Application ClustersReal Application Clusters Oracle Real Application Clusters enables a single Oracle Real Application Clusters enables a single

database to run across multiple clustered nodes in database to run across multiple clustered nodes in a grid, pooling the processing resources of several a grid, pooling the processing resources of several standard machines.standard machines.

In Oracle 10In Oracle 10gg, the database can immediately , the database can immediately begin balancing workload across a new node with begin balancing workload across a new node with new processing capacity as it gets re-provisioned new processing capacity as it gets re-provisioned from one database to another, and can relinquish from one database to another, and can relinquish a machine when it is no longer needed--a machine when it is no longer needed--

this is capacity on demand. Other databases this is capacity on demand. Other databases cannot grow and shrink while running and, cannot grow and shrink while running and, therefore, cannot utilize hardware as efficiently.therefore, cannot utilize hardware as efficiently.

Servers can be easily added and dropped to an Servers can be easily added and dropped to an Oracle cluster with no downtime.Oracle cluster with no downtime.

Service

RAC 10g Architecture RAC 10g Architecture public network

Node1

Operating System

Oracle Clusterware

instance 1

ASM

VIP1

ListenerNode 2

Operating System

Oracle Clusterware

instance 2

ASM

VIP2

Listener

Service Node n

Operating System

Oracle Clusterware

instance n

ASM

VIPn

Listener

Service

/…/

Redo / Archive logs all instances

shared storage

Database / Control files

OCR and Voting Disks

Managed by ASM

RAW Devices

Under the CoversUnder the Covers

Redo Log Files

Node nNode 2

Data Files and Control Files

Redo Log Files Redo Log Files

DictionaryCache

Log buffer

LCK0 LGWR DBW0

SMON PMON

LibraryCache

Global Resource Directory

LMS0

Instance 2

SGA

Instance n

Cluster Private High Speed Network

Buffer Cache

LMON LMD0 DIAG

Dictionary Cache

Log buffer

LCK0 LGWR DBW0

SMON PMON

Library Cache


LMS0

Buffer Cache

LMON LMD0 DIAG

Dictionary Cache

Log buffer

LCK0 LGWR DBW0

SMON PMON

Library Cache


LMS0

Buffer Cache

LMON LMD0 DIAG

Instance 1

Node 1

SGA SGA

Global Resource Global Resource DirectoryDirectory RAC Database System has two important services. They are Global Cache RAC Database System has two important services. They are Global Cache

Service (GCS) and Global Enqueue Service (GES). These are basically Service (GCS) and Global Enqueue Service (GES). These are basically collections of background processes. These two processes together cover and collections of background processes. These two processes together cover and manage the total Cache Fusion process, resource transfers, and resource manage the total Cache Fusion process, resource transfers, and resource escalations among the instances.escalations among the instances.

Global Resource DirectoryGlobal Resource Directory GES and GCS together maintain a Global Resource Directory (GRD) to record GES and GCS together maintain a Global Resource Directory (GRD) to record

the information about the resources and the enqueues. GRD remains in the the information about the resources and the enqueues. GRD remains in the memory and is stored on all the instances. Each instance manages a portion of memory and is stored on all the instances. Each instance manages a portion of the directory. This distributed nature is a key point for fault tolerance of the the directory. This distributed nature is a key point for fault tolerance of the RAC.RAC.

Global Resource Directory (GRD) is the internal database that records and Global Resource Directory (GRD) is the internal database that records and stores the current status of the data blocks. Whenever a block is transferred out stores the current status of the data blocks. Whenever a block is transferred out of a local cache to another instance’s cache the GRD is updated. The following of a local cache to another instance’s cache the GRD is updated. The following resources information is available in GRD.resources information is available in GRD.

* Data Block Identifiers (DBA)* Data Block Identifiers (DBA) * Location of most current version * Location of most current version * Modes of the data blocks: (N)Null, (S)Shared, (X)Exclusive * Modes of the data blocks: (N)Null, (S)Shared, (X)Exclusive * The Roles of the data blocks (local or global) held by each instance * The Roles of the data blocks (local or global) held by each instance * Buffer caches on multiple nodes in the cluster* Buffer caches on multiple nodes in the cluster GRD is akin to the previous version of Lock Directory in the functionality GRD is akin to the previous version of Lock Directory in the functionality

perspective but has been expanded with more components. It has accurate perspective but has been expanded with more components. It has accurate measure of inventory of resources and their status and location.measure of inventory of resources and their status and location.

Background Processes in a Background Processes in a RAC instanceRAC instance

Select name,description from Select name,description from v$bgprocess where paddr <> ’00’v$bgprocess where paddr <> ’00’

The one specific to a RAC instance The one specific to a RAC instance are the DIAG, LCK, LMON, LMNDn are the DIAG, LCK, LMON, LMNDn and LMSn process.and LMSn process.

DIAG : Diagnosability DIAG : Diagnosability DaemonDaemon

The diagnosability daemon is responsible The diagnosability daemon is responsible for capturing information on process for capturing information on process failures in a RAC environment, and writing failures in a RAC environment, and writing out trace information for failure analysis.out trace information for failure analysis.

The information produced by DIAG is most The information produced by DIAG is most useful when working in conjunction with useful when working in conjunction with Oracle Support to troubleshoot causes for Oracle Support to troubleshoot causes for a failure.a failure.

Only a single DIAG process is needed for Only a single DIAG process is needed for each instanceeach instance

LCK: Lock ProcessLCK: Lock Process

The lock process (LCK) manages The lock process (LCK) manages requests that are not cache-fusion requests that are not cache-fusion requests, such as row cache requests, such as row cache requests and library cache requestsrequests and library cache requests

Only a single LCK process is allowed Only a single LCK process is allowed for each instance. for each instance.

LCK maintains a list of lock elements LCK maintains a list of lock elements and uses this list to validate locks and uses this list to validate locks during instance recoveryduring instance recovery

LMD:Lock Manager LMD:Lock Manager Daemon ProcessDaemon Process

The global enqueue service daemon The global enqueue service daemon (LMD) is a lock agent process that (LMD) is a lock agent process that coordinates enqueue manager service coordinates enqueue manager service requests. The requests are for global requests. The requests are for global cache service enqueues that control cache service enqueues that control access to global enqueues and access to global enqueues and resources. resources.

The LMD process also handles deadlock The LMD process also handles deadlock detection and remote enqueue requests. detection and remote enqueue requests.

LMON: Lock Monitor LMON: Lock Monitor ProcessProcess

LMON is the global enqueue service monitor. LMON is the global enqueue service monitor. It is responsible for the reconfiguration of It is responsible for the reconfiguration of lock resources when an instance joins the lock resources when an instance joins the cluster or leaves the cluster, and also is cluster or leaves the cluster, and also is responsible for the dynamic lock remastering responsible for the dynamic lock remastering

LMON will generate a trace file whenever a LMON will generate a trace file whenever a reconfiguration occurs (as opposed to reconfiguration occurs (as opposed to remastering of a subset of locks).remastering of a subset of locks).

It is the responsibility of LMON to check for It is the responsibility of LMON to check for the death of instances clusterwide, and to the death of instances clusterwide, and to initiate reconfiguration as quickly as possibleinitiate reconfiguration as quickly as possible

LMS: Lock Manager Server LMS: Lock Manager Server ProcessProcess

The LMS process (or global cache service The LMS process (or global cache service process) is in charge of shipping the blocks process) is in charge of shipping the blocks between instances for cache-fusion requests. between instances for cache-fusion requests. In the event of a consistent-read request, the In the event of a consistent-read request, the LMS process will first roll the block back, LMS process will first roll the block back, creating the consistent read (CR) image of the creating the consistent read (CR) image of the block, and will then ship that version of the block, and will then ship that version of the block across the interconnect to the block across the interconnect to the foreground process making the request at the foreground process making the request at the remote instance.remote instance.

In addition, LMS must interact with the LMD In addition, LMS must interact with the LMD process to retrieve lock requests placed by process to retrieve lock requests placed by LMD. An instance may dynamically generate LMD. An instance may dynamically generate up to LMS processes, depending on the loadup to LMS processes, depending on the load

Server Control UtilityServer Control Utility To manage the RAC database and its instances, Oracle has provided To manage the RAC database and its instances, Oracle has provided

a new utility called the Server Control Utility (SRVCTL). This a new utility called the Server Control Utility (SRVCTL). This replaces the earlier utility ‘replaces the earlier utility ‘opsctl’opsctl’ which was used in the parallel which was used in the parallel server.server.

The Server Control Utility is a single point of control between the The Server Control Utility is a single point of control between the Oracle Intelligent agent and each node in the RAC system. The Oracle Intelligent agent and each node in the RAC system. The SRVCTL communicates with the global daemon service (GSD) and SRVCTL communicates with the global daemon service (GSD) and resides on each of the nodes. The SRVCTL gathers information from resides on each of the nodes. The SRVCTL gathers information from the database and instances and acts as an intermediary between the database and instances and acts as an intermediary between nodes and the Oracle Intelligent agent. nodes and the Oracle Intelligent agent.

When you use the SRVCTL to perform configuration operations on When you use the SRVCTL to perform configuration operations on your cluster, the SRVCTL stores configuration data in the Server your cluster, the SRVCTL stores configuration data in the Server Management (SRVM) configuration repository. The SRVM includes Management (SRVM) configuration repository. The SRVM includes all the components of Enterprise Manager such as the Intelligent all the components of Enterprise Manager such as the Intelligent Agent, the Server Control Utility (SRVCTL), and the Global Services Agent, the Server Control Utility (SRVCTL), and the Global Services Daemon. Thus, the SRVCTL is one of the SRVM Instance Daemon. Thus, the SRVCTL is one of the SRVM Instance Management Utilities. The SRVCTL uses SQL*Plus internally to Management Utilities. The SRVCTL uses SQL*Plus internally to perform stop and start activities on each node.perform stop and start activities on each node.

Server Control UtilityServer Control Utility For the SRVCTL to function, the Global Services Daemon (GSD) should be running For the SRVCTL to function, the Global Services Daemon (GSD) should be running

on the node. The SRVCTL performs mainly two types of administrative tasks: Cluster on the node. The SRVCTL performs mainly two types of administrative tasks: Cluster Database Tasks and Cluster Database Configuration Tasks.Database Tasks and Cluster Database Configuration Tasks.

SRVCTL Cluster Database tasks include:SRVCTL Cluster Database tasks include: · Starts and stops cluster databases. · Starts and stops cluster databases. · Starts and stops cluster database instances. · Starts and stops cluster database instances. · Starts and stops listeners associated with a cluster database instance. · Starts and stops listeners associated with a cluster database instance. · Obtains the status of a cluster database instance. · Obtains the status of a cluster database instance. · Obtains the status of listeners associated with a cluster database. · Obtains the status of listeners associated with a cluster database. SRVCTL Cluster Database Configuration tasks include:SRVCTL Cluster Database Configuration tasks include: · Adds and deletes cluster database configuration information. · Adds and deletes cluster database configuration information. · Adds an instance to, or deletes an instance from a cluster database.· Adds an instance to, or deletes an instance from a cluster database. · Renames an instance name within a cluster database configuration. · Renames an instance name within a cluster database configuration. · Moves instances in a cluster database configuration. · Moves instances in a cluster database configuration. · Sets and unsets the environment variable for an instance in a cluster · Sets and unsets the environment variable for an instance in a cluster

database database configuration. configuration. · Sets and unsets the environment variable for an entire cluster in a cluster · Sets and unsets the environment variable for an entire cluster in a cluster

database database configuration. configuration.

RAW Partitions, Cluster File RAW Partitions, Cluster File System and Automatic Storage System and Automatic Storage

Management (ASM)Management (ASM) Raw Partitions are a set of unformatted devices on a Raw Partitions are a set of unformatted devices on a

shared disk sub-system.A raw partition is a disk drive shared disk sub-system.A raw partition is a disk drive device that does not have a file system set up. The raw device that does not have a file system set up. The raw partition is portion of the physical disk that is partition is portion of the physical disk that is accessed at the lowest possible level. The actual accessed at the lowest possible level. The actual application that uses a raw device is responsible for application that uses a raw device is responsible for managing its own I/O to the raw device with no managing its own I/O to the raw device with no operating system buffering.operating system buffering.

Traditionally, they were required for Oracle Parallel Traditionally, they were required for Oracle Parallel Server (OPS) and they provided high performance by Server (OPS) and they provided high performance by bypassing the file system overhead. Raw partitions bypassing the file system overhead. Raw partitions were used in setting up databases for performance were used in setting up databases for performance gains and for the purpose of concurrent access by gains and for the purpose of concurrent access by multiple nodes in the cluster without system-level multiple nodes in the cluster without system-level buffering.buffering.

RAW Partitions, Cluster File RAW Partitions, Cluster File System and Automatic Storage System and Automatic Storage

Management (ASM)Management (ASM) Oracle 9i RAC and 10g now supports Oracle 9i RAC and 10g now supports

both the cluster file system and the raw both the cluster file system and the raw devices to store the shared data. In devices to store the shared data. In addition, 10g RAC supports shared addition, 10g RAC supports shared storage resources from ASM instance. storage resources from ASM instance. You will be able to create the data files You will be able to create the data files out of the disk resources located in the out of the disk resources located in the ASM instance. The ASM resources are ASM instance. The ASM resources are sharable and accessed by all the nodes sharable and accessed by all the nodes in the RAC system.in the RAC system.

RAW DevicesRAW Devices Raw Devices have been in use for very long time. They Raw Devices have been in use for very long time. They

were the primary storage structures for data files of the were the primary storage structures for data files of the Oracle Parallel Server. They remain in use even in the Oracle Parallel Server. They remain in use even in the RAC versions 9i and 10g. Raw Devices are difficult to RAC versions 9i and 10g. Raw Devices are difficult to manage and administer, but provide high performing manage and administer, but provide high performing shared storage structures. When you use the raw devices shared storage structures. When you use the raw devices for data files, redo log files and control files, you may have for data files, redo log files and control files, you may have to use the local file systems or some sort of network to use the local file systems or some sort of network attached file system for writing the archive log files, attached file system for writing the archive log files, handling the utl_file_dir files and files supporting the handling the utl_file_dir files and files supporting the external tables.external tables.

On Raw Devices On Local File SystemOn Raw Devices On Local File System Data files Archive log filesData files Archive log files Redo files Oracle Home filesRedo files Oracle Home files Control files CRS Home filesControl files CRS Home files Voting Disk Alert log, Trace filesVoting Disk Alert log, Trace files OCR file Files for external tablesOCR file Files for external tables utl_file_dir locationutl_file_dir location

RAW DevicesRAW Devices AdvantagesAdvantages Raw partitions have several advantages:Raw partitions have several advantages: They are not subject to any operating system locking.They are not subject to any operating system locking.

The operating system buffer or cache is bypassed, giving The operating system buffer or cache is bypassed, giving

performance gains and reduced memory consumption.performance gains and reduced memory consumption.

Multiple systems can be easily shared.Multiple systems can be easily shared.

The application or database system has full control to The application or database system has full control to manipulate the internals of access.manipulate the internals of access.

Historically, the support for asynchronous I/O on UNIX Historically, the support for asynchronous I/O on UNIX systems was generally limited to raw partitionssystems was generally limited to raw partitions

RAW DevicesRAW Devices Issues and DifficultiesIssues and Difficulties There are many administrative inconveniences and drawbacks such There are many administrative inconveniences and drawbacks such

as:as: The unit of allocation to the database is the entire raw partition. We The unit of allocation to the database is the entire raw partition. We

cannot use a raw partition for multiple tablespaces. A raw partition cannot use a raw partition for multiple tablespaces. A raw partition is not the same as a file system where we can create many files.is not the same as a file system where we can create many files.

Administrators have to create them with specific sizes. When the Administrators have to create them with specific sizes. When the databases grow in size, raw partitions cannot be extended. We need databases grow in size, raw partitions cannot be extended. We need to add extra partitions to support the growing tablespace. to add extra partitions to support the growing tablespace. Sometimes we may have limitations on the total number of raw Sometimes we may have limitations on the total number of raw partitions we can use in the system. Furthermore, there are no partitions we can use in the system. Furthermore, there are no database operations that can occur on an individual data file. There database operations that can occur on an individual data file. There is, therefore, no logical benefit from having a tablespace consisting is, therefore, no logical benefit from having a tablespace consisting of many data files except for those tablespaces that are larger than of many data files except for those tablespaces that are larger than the maximum Oracle can support in a single file. the maximum Oracle can support in a single file.

We cannot use the standard file manipulation commands on the raw We cannot use the standard file manipulation commands on the raw partitions, and thus, on the data files. We cannot use commands partitions, and thus, on the data files. We cannot use commands such as cpio or tar for backup purposes. Backup strategy will such as cpio or tar for backup purposes. Backup strategy will become more complicated become more complicated

RAW DevicesRAW Devices Raw partitions cannot be used for writing the archive logs.Raw partitions cannot be used for writing the archive logs.

Administrators need to keep track of the raw volumes with Administrators need to keep track of the raw volumes with

their cryptic naming conventions. However, by using the their cryptic naming conventions. However, by using the symbolic links, we can reduce the hassles associated with symbolic links, we can reduce the hassles associated with names. names.

For example, a cryptic name like /dev/rdsk/c8t4d5s4 or a For example, a cryptic name like /dev/rdsk/c8t4d5s4 or a name like /dev/sd/sd001 is an administrative challenge. To name like /dev/sd/sd001 is an administrative challenge. To alleviate this, administrators often rely on symbolic links to alleviate this, administrators often rely on symbolic links to provide logical names that make sense. This, however, provide logical names that make sense. This, however, substitutes one complexity for another.substitutes one complexity for another.

In a clustered environment like Linux clusters, it is not In a clustered environment like Linux clusters, it is not guaranteed that the physical devices will have the same guaranteed that the physical devices will have the same device names on different nodes or across reboots of a device names on different nodes or across reboots of a single node. To solve this problem, manual intervention is single node. To solve this problem, manual intervention is needed, which will increase administration overhead.needed, which will increase administration overhead.

Cluster File SystemCluster File System CFS offers a very good shared storage facility for CFS offers a very good shared storage facility for

building the RAC database. CFS provides a building the RAC database. CFS provides a shared file system, which is mounted on all the shared file system, which is mounted on all the cluster nodes simultaneously. When you cluster nodes simultaneously. When you implement the RAC database with the commercial implement the RAC database with the commercial CFS products such as the Veritas CFS or CFS products such as the Veritas CFS or PolyServe Matrix Server, you will able to store all PolyServe Matrix Server, you will able to store all kinds of database files including the shared kinds of database files including the shared Oracle Home and CRS Home. Oracle Home and CRS Home.

However, the capabilities of the CFS products are However, the capabilities of the CFS products are not the same. For example, Oracle CFS (OCFS), not the same. For example, Oracle CFS (OCFS), used in case of Linux RAC implementations, has used in case of Linux RAC implementations, has limitations. It is not a general purpose file system. limitations. It is not a general purpose file system. It cannot be used for shared Oracle Home. It cannot be used for shared Oracle Home.

Cluster File SystemCluster File System On Cluster File SystemOn Cluster File SystemData files Archive Log filesData files Archive Log filesRedo files Oracle Home FilesRedo files Oracle Home FilesControl files Alert log,Trace filesControl files Alert log,Trace filesVoting Disk Files for External TablesVoting Disk Files for External TablesOCR File utl_file_dir locationOCR File utl_file_dir location A cluster file system (CFS) is a file system that may be accessed (read and A cluster file system (CFS) is a file system that may be accessed (read and

write) by all the members in the cluster at the same time. This implies that write) by all the members in the cluster at the same time. This implies that all the members of the cluster have the same view. Some of the popular and all the members of the cluster have the same view. Some of the popular and widely used cluster file system products for Oracle RAC include HP Tru64 widely used cluster file system products for Oracle RAC include HP Tru64 CFS, Veritas CFS, IBM GPFS, Polyserve Matrix Server, and Oracle Cluster CFS, Veritas CFS, IBM GPFS, Polyserve Matrix Server, and Oracle Cluster File system. The cluster file system offers:File system. The cluster file system offers:

Simple management.Simple management.

The use of Oracle Managed Files with RAC.The use of Oracle Managed Files with RAC.

A Single Oracle Software Installation.A Single Oracle Software Installation.

Auto-extend Enabled on Oracle Data Files.Auto-extend Enabled on Oracle Data Files.

Uniform accessibility of Archive Logs.Uniform accessibility of Archive Logs.

ODM compliant File systems. ODM compliant File systems.

ASM – Automatic Storage ASM – Automatic Storage ManagementManagement

ASM is the new star on the block. ASM provides a vertical ASM is the new star on the block. ASM provides a vertical integration of the file system and volume manager for integration of the file system and volume manager for Oracle database files. ASM has the capability to spread Oracle database files. ASM has the capability to spread database files across all available storage for optimal database files across all available storage for optimal performance and resource utilization. It enables simple performance and resource utilization. It enables simple and non-intrusive resource allocation and provides and non-intrusive resource allocation and provides automatic rebalancingautomatic rebalancing

When you are using the ASM for building shared files, you When you are using the ASM for building shared files, you would get almost the same performance as that of raw would get almost the same performance as that of raw partitions. The ASM controlled disk devices will be part of partitions. The ASM controlled disk devices will be part of ASM instance, which can be shared by the RAC database ASM instance, which can be shared by the RAC database instance. It is similar to the situation where raw devices instance. It is similar to the situation where raw devices supporting the RAC database had to be shared by multiple supporting the RAC database had to be shared by multiple nodes. The shared devices need to be presented to nodes. The shared devices need to be presented to multiple nodes on the cluster and those devices will be multiple nodes on the cluster and those devices will be input to the ASM instance. There will be an ASM instance input to the ASM instance. There will be an ASM instance supporting each RAC instance on the respective nodesupporting each RAC instance on the respective node

ASM – Automatic Storage ASM – Automatic Storage ManagementManagement

From the ASM instance On local or CFSFrom the ASM instance On local or CFSData files Oracle Home Files Data files Oracle Home Files Redo files CRS Home Files Redo files CRS Home Files Control files Alert log, trace files Control files Alert log, trace files Archive log files Files for external Archive log files Files for external

tablestables--------------------------- util_file_dir location --------------------------- util_file_dir location Voting Disk and OCR fileVoting Disk and OCR fileAre located on raw partitionsAre located on raw partitions

ASM is for more Oracle specific data, redo log files ASM is for more Oracle specific data, redo log files and archived log files. and archived log files.

Automatic Storage ManagementAutomatic Storage Management Automatic Storage Management simplifies storage Automatic Storage Management simplifies storage

management for Oracle Databases.management for Oracle Databases. Instead of managing many database files, Oracle DBAs Instead of managing many database files, Oracle DBAs

manage only a small number of disk groups. A disk group is a manage only a small number of disk groups. A disk group is a set of disk devices that Oracle manages as a single, logical set of disk devices that Oracle manages as a single, logical unit. An administrator can define a particular disk group as unit. An administrator can define a particular disk group as the default disk group for a database, and Oracle the default disk group for a database, and Oracle automatically allocates storage for and creates or deletes the automatically allocates storage for and creates or deletes the files associated with the database object.files associated with the database object.

Automatic Storage Management also offers the benefits of Automatic Storage Management also offers the benefits of storage technologies such as RAID or Logical Volume storage technologies such as RAID or Logical Volume Managers (LVMs). Oracle can balance I/O from multiple Managers (LVMs). Oracle can balance I/O from multiple databases across all of the devices in a disk group, and it databases across all of the devices in a disk group, and it implements striping and mirroring to improve I/O implements striping and mirroring to improve I/O performance and data reliability. Because Automatic Storage performance and data reliability. Because Automatic Storage Management is written to work exclusively with Oracle, it Management is written to work exclusively with Oracle, it achieves better performance than generalized storage achieves better performance than generalized storage virtualization solutions.virtualization solutions.

Shared Disk StorageShared Disk Storage Oracle RAC relies on a shared disk architecture. The database files, Oracle RAC relies on a shared disk architecture. The database files,

online redo logs, and control files for the database must be online redo logs, and control files for the database must be accessible to each node in the cluster. The shared disks also store accessible to each node in the cluster. The shared disks also store the Oracle Cluster Registry and Voting Disk. There are a variety of the Oracle Cluster Registry and Voting Disk. There are a variety of ways to configure shared storage including direct attached disks ways to configure shared storage including direct attached disks (typically SCSI over copper or fiber), Storage Area Networks (typically SCSI over copper or fiber), Storage Area Networks (SAN), and Network Attached Storage (NAS).(SAN), and Network Attached Storage (NAS).

Private NetworkPrivate NetworkEach cluster node is connected to all other nodes via a private Each cluster node is connected to all other nodes via a private high-speed network, also known as the cluster interconnect or high-speed network, also known as the cluster interconnect or high-speed interconnect (HSI). This network is used by Oracle's high-speed interconnect (HSI). This network is used by Oracle's Cache Fusion technology to effectively combine the physical Cache Fusion technology to effectively combine the physical memory (RAM) in each host into a single cache. Oracle Cache memory (RAM) in each host into a single cache. Oracle Cache Fusion allows data stored in the cache of one Oracle instance to be Fusion allows data stored in the cache of one Oracle instance to be accessed by any other instance by transferring it across the private accessed by any other instance by transferring it across the private network. It also preserves data integrity and cache coherency by network. It also preserves data integrity and cache coherency by transmitting locking and other synchronization information across transmitting locking and other synchronization information across

cluster nodes.cluster nodes.

The private network is typically built with Gigabit The private network is typically built with Gigabit Ethernet, but for high-volume environments, many Ethernet, but for high-volume environments, many vendors offer proprietary low-latency, high-bandwidth vendors offer proprietary low-latency, high-bandwidth solutions specifically designed for Oracle RAC. Linux solutions specifically designed for Oracle RAC. Linux also offers a means of bonding multiple physical NICs also offers a means of bonding multiple physical NICs into a single virtual NIC to provide increased into a single virtual NIC to provide increased bandwidth and availability.bandwidth and availability.

Public NetworkPublic NetworkTo maintain high availability, each cluster node is To maintain high availability, each cluster node is assigned a virtual IP address (VIP). In the event of host assigned a virtual IP address (VIP). In the event of host failure, the failed node's IP address can be reassigned failure, the failed node's IP address can be reassigned to a surviving node to allow applications to continue to a surviving node to allow applications to continue accessing the database through the same IP address.accessing the database through the same IP address.

Why do we have a Why do we have a Virtual IP (VIP)Virtual IP (VIP) in 10 in 10gg? Why does ? Why does it just return a dead connection when its primary node it just return a dead connection when its primary node fails? fails?

It's all about availability of the application. When a node fails, the It's all about availability of the application. When a node fails, the VIP associated with it is supposed to be automatically failed over VIP associated with it is supposed to be automatically failed over to some other node. When this occurs, two things happen. to some other node. When this occurs, two things happen.

The new node re-arps the world indicating a new MAC address The new node re-arps the world indicating a new MAC address for the address. For directly connected clients, this usually causes for the address. For directly connected clients, this usually causes them to see errors on their connections to the old address. them to see errors on their connections to the old address.

Subsequent packets sent to the VIP go to the new node, which Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the will send error RST packets back to the clients. This results in the clients getting errors immediately. clients getting errors immediately.

This means that when the client issues SQL to the node that is This means that when the client issues SQL to the node that is now down, or traverses the address list while connecting, rather now down, or traverses the address list while connecting, rather than waiting on a very long TCP/IP time-out (~10 minutes), the than waiting on a very long TCP/IP time-out (~10 minutes), the client receives a TCP reset. In the case of SQL, this is ORA-3113. client receives a TCP reset. In the case of SQL, this is ORA-3113. In the case of connect, the next address in tnsnames is used. In the case of connect, the next address in tnsnames is used.

Without using VIPs, clients connected to a node that died will Without using VIPs, clients connected to a node that died will often wait a 10-minute TCP timeout period before getting an often wait a 10-minute TCP timeout period before getting an error. As a result, you don't really have a good HA solution error. As a result, you don't really have a good HA solution without using VIPs (Source - Metalink Note 220970.1) . without using VIPs (Source - Metalink Note 220970.1) .

The The Oracle CRSOracle CRS contains all the cluster and contains all the cluster and database configuration metadata along with several database configuration metadata along with several system management features for RAC. It allows the system management features for RAC. It allows the DBA to register and invite an Oracle instance (or DBA to register and invite an Oracle instance (or instances) to the cluster. During normal operation, instances) to the cluster. During normal operation, CRS will send messages (via a special ping CRS will send messages (via a special ping operation) to all nodes configured in the cluster—operation) to all nodes configured in the cluster—often called the "heartbeat." If the heartbeat fails for often called the "heartbeat." If the heartbeat fails for any of the nodes, it checks with the CRS any of the nodes, it checks with the CRS configuration files (on the shared disk) to distinguish configuration files (on the shared disk) to distinguish between a real node failure and a network failure. between a real node failure and a network failure.

CRS maintains two files: the Oracle Cluster Registry CRS maintains two files: the Oracle Cluster Registry (OCR) and the Voting Disk. The OCR and the Voting (OCR) and the Voting Disk. The OCR and the Voting Disk must reside on shared disks as either raw Disk must reside on shared disks as either raw partitions or files in a cluster filesystem. partitions or files in a cluster filesystem.

The The Voting DiskVoting Disk is used by the Oracle cluster manager in various is used by the Oracle cluster manager in various layers. The Cluster Manager and Node Monitor accepts registration layers. The Cluster Manager and Node Monitor accepts registration of Oracle instances to the cluster and it sends ping messages to of Oracle instances to the cluster and it sends ping messages to Cluster Managers (Node Monitor) on other RAC nodes. If this Cluster Managers (Node Monitor) on other RAC nodes. If this heartbeat fails, oracm uses a quorum file or a quorum partition on heartbeat fails, oracm uses a quorum file or a quorum partition on the shared disk to distinguish between a node failure and a network the shared disk to distinguish between a node failure and a network failure. So if a node stops sending ping messages, but continues failure. So if a node stops sending ping messages, but continues writing to the quorum file or partition, then the other Cluster writing to the quorum file or partition, then the other Cluster Managers can recognize it as a network failure. Hence the Managers can recognize it as a network failure. Hence the availability from the Voting Disk is critical for the operation of the availability from the Voting Disk is critical for the operation of the Oracle Cluster Manager.Oracle Cluster Manager.

The shared volumes created for the OCR and the voting disk should The shared volumes created for the OCR and the voting disk should be configured using RAID to protect against media failure. This be configured using RAID to protect against media failure. This requires the use of an external cluster volume manager, cluster file requires the use of an external cluster volume manager, cluster file system, or storage hardware that provides RAID protection. . system, or storage hardware that provides RAID protection. .

Oracle Cluster Registry (OCR)Oracle Cluster Registry (OCR) is used to store the cluster is used to store the cluster configuration information among other things. OCR needs to be configuration information among other things. OCR needs to be accessible from all nodes in the cluster. If OCR became inaccessible accessible from all nodes in the cluster. If OCR became inaccessible the CSS daemon would soon fail, and take down the node. PMON the CSS daemon would soon fail, and take down the node. PMON never needs to write to OCR. To confirm if OCR is accessible, try never needs to write to OCR. To confirm if OCR is accessible, try ocrcheck from your ORACLE_HOME and ORA_CRS_HOME.ocrcheck from your ORACLE_HOME and ORA_CRS_HOME.

Cache FusionCache Fusion One of the bigger differences between Oracle RAC and OPS One of the bigger differences between Oracle RAC and OPS

is the presence of Cache Fusion technology. In OPS, a is the presence of Cache Fusion technology. In OPS, a request for data between nodes required the data to be request for data between nodes required the data to be written to disk first, and then the requesting node could written to disk first, and then the requesting node could read that data. In RAC, data is passed along with locks. read that data. In RAC, data is passed along with locks.

Every time an instance wants to update a block, it has to Every time an instance wants to update a block, it has to obtain a lock on it to make sure no other instance in the obtain a lock on it to make sure no other instance in the cluster is updating the same cluster is updating the same

block. To resolve this problem, Oracle does a data block block. To resolve this problem, Oracle does a data block ping mechanism that allows it to get the status of the ping mechanism that allows it to get the status of the specific block before reading it from the disk. Cache Fusion specific block before reading it from the disk. Cache Fusion resolves data block read/read, read/write and write/write resolves data block read/read, read/write and write/write conflicts among ORACLE database nodes through high conflicts among ORACLE database nodes through high performance interconnect networks, bypassing much slower performance interconnect networks, bypassing much slower physical disk operations used in previous releases. Using physical disk operations used in previous releases. Using Oracle 9i RAC cache fusion feature, close to linear Oracle 9i RAC cache fusion feature, close to linear scalability of database performance can be achieved when scalability of database performance can be achieved when adding nodes to the cluster. ORACLE enables better adding nodes to the cluster. ORACLE enables better Database capacity planning and conserves capital Database capacity planning and conserves capital investments.investments.

Build Your Own Oracle RAC 10Build Your Own Oracle RAC 10gg Release 2 Cluster on Release 2 Cluster on Linux and FireWireLinux and FireWire

ContentsContents IntroductionIntroduction Oracle RAC 10Oracle RAC 10gg Overview Overview Shared-Storage OverviewShared-Storage Overview FireWire TechnologyFireWire Technology Hardware & CostsHardware & Costs Install the Linux Operating SystemInstall the Linux Operating System Network ConfigurationNetwork Configuration Obtain & Install FireWire ModulesObtain & Install FireWire Modules Create "oracle" User and DirectoriesCreate "oracle" User and Directories Create Partitions on the Shared FireWire Storage DeviceCreate Partitions on the Shared FireWire Storage Device


Configure the Linux Servers for OracleConfigure the Linux Servers for Oracle Configure the Configure the hangcheckhangcheck-timer Kernel Module-timer Kernel Module Configure RAC Nodes for Remote AccessConfigure RAC Nodes for Remote Access All Startup Commands for Each RAC NodeAll Startup Commands for Each RAC Node Check RPM Packages for Oracle 10Check RPM Packages for Oracle 10gg Release 2 Release 2 Install & Configure Oracle Cluster File System (OCFS2)Install & Configure Oracle Cluster File System (OCFS2) Install & Configure Automatic Storage Management (Install & Configure Automatic Storage Management (ASMLibASMLib

2.0) 2.0) Download Oracle 10Download Oracle 10gg RAC Software RAC Software Install Oracle 10Install Oracle 10gg ClusterwareClusterware Software Software Install Oracle 10Install Oracle 10gg Database Software Database Software Install Oracle10Install Oracle10gg Companion CD Software Companion CD Software Create TNS Listener ProcessCreate TNS Listener Process


Create the Oracle Cluster DatabaseCreate the Oracle Cluster Database Verify TNS Networking FilesVerify TNS Networking Files Create / Alter Create / Alter TablespacesTablespaces Verify the RAC Cluster & Database ConfigurationVerify the RAC Cluster & Database Configuration Starting / Stopping the ClusterStarting / Stopping the Cluster Transparent Application Failover - (TAF)Transparent Application Failover - (TAF) ConclusionConclusion AcknowledgementsAcknowledgements


DownloadDownload - Red Hat Enterprise Linux 4- Red Hat Enterprise Linux 4

- Oracle Cluster File System Release 2 - (1.2.3-1) - Single - Oracle Cluster File System Release 2 - (1.2.3-1) - Single Processor / SMP / Processor / SMP /

HugememHugemem - Oracle Cluster File System Releaase 2 Tools - (1.2.1-1) - Tools / - Oracle Cluster File System Releaase 2 Tools - (1.2.1-1) - Tools / ConsoleConsole - Oracle Database 10 - Oracle Database 10gg Release 2 EE, Clusterware, Companion CD Release 2 EE, Clusterware, Companion CD - (10.2.0.1.0)- (10.2.0.1.0) - Precompiled RHEL4 FireWire Modules - (2.6.9-22.EL) - Precompiled RHEL4 FireWire Modules - (2.6.9-22.EL) - ASMLib 2.0 Driver - (2.6.9-22.EL / 2.0.3-1) - Single Processor / - ASMLib 2.0 Driver - (2.6.9-22.EL / 2.0.3-1) - Single Processor / SMP / HugememSMP / Hugemem - ASMLib 2.0 Library and Tools - (2.0.3-1) - Driver Support Files / - ASMLib 2.0 Library and Tools - (2.0.3-1) - Driver Support Files / Userspace LibraryUserspace Library


IntroductionIntroduction One of the most efficient ways to become familiar with Oracle Real Application One of the most efficient ways to become familiar with Oracle Real Application

Clusters (RAC) 10Clusters (RAC) 10gg technology is to have access to an actual Oracle RAC 10 technology is to have access to an actual Oracle RAC 10gg cluster. There's no better way to understand its benefits—including fault tolerance, cluster. There's no better way to understand its benefits—including fault tolerance, security, load balancing, and scalability—than to experience them directly.security, load balancing, and scalability—than to experience them directly.

The Oracle Clusterware software will be installed to /u01/app/oracle/product/crs on The Oracle Clusterware software will be installed to /u01/app/oracle/product/crs on each of the nodes that make up the RAC cluster. However, the Clusterware each of the nodes that make up the RAC cluster. However, the Clusterware software requires that two of its files—the Oracle Cluster Registry (OCR) file and software requires that two of its files—the Oracle Cluster Registry (OCR) file and the Voting Disk file—be shared with all nodes in the cluster. These two files will be the Voting Disk file—be shared with all nodes in the cluster. These two files will be installed on shared storage using OCFS2. It is possible (but not recommended by installed on shared storage using OCFS2. It is possible (but not recommended by Oracle) to use RAW devices for these files; however, it is not possible to use ASM Oracle) to use RAW devices for these files; however, it is not possible to use ASM for these two Clusterware files. for these two Clusterware files.

The Oracle Database 10g Release 2 software will be installed into a separate The Oracle Database 10g Release 2 software will be installed into a separate Oracle Home, namely /u01/app/oracle/product/10.2.0/db_1, on each of the nodes Oracle Home, namely /u01/app/oracle/product/10.2.0/db_1, on each of the nodes that make up the RAC cluster. All the Oracle physical database files (data, online that make up the RAC cluster. All the Oracle physical database files (data, online redo logs, control files, archived redo logs), will be installed to different partitions of redo logs, control files, archived redo logs), will be installed to different partitions of the shared drive being managed by ASM. (The Oracle database files can just as the shared drive being managed by ASM. (The Oracle database files can just as easily be stored on OCFS2. Using ASM, however, makes the article that much easily be stored on OCFS2. Using ASM, however, makes the article that much more interesting!) more interesting!)

2. Oracle RAC 102. Oracle RAC 10gg Overview Overview

Oracle RAC, introduced with Oracle9Oracle RAC, introduced with Oracle9ii, is the successor to Oracle Parallel , is the successor to Oracle Parallel Server (OPS). RAC allows multiple instances to access the same Server (OPS). RAC allows multiple instances to access the same database (storage) simultaneously. It provides fault tolerance, load database (storage) simultaneously. It provides fault tolerance, load balancing, and performance benefits by allowing the system to scale out, balancing, and performance benefits by allowing the system to scale out, and at the same time—because all nodes access the same database—and at the same time—because all nodes access the same database—the failure of one instance will not cause the loss of access to the the failure of one instance will not cause the loss of access to the database. database.

At the heart of Oracle RAC is a shared disk subsystem. All nodes in the At the heart of Oracle RAC is a shared disk subsystem. All nodes in the cluster must be able to access all of the data, redo log files, control files cluster must be able to access all of the data, redo log files, control files and parameter files for all nodes in the cluster. The data disks must be and parameter files for all nodes in the cluster. The data disks must be globally available to allow all nodes to access the database. Each node globally available to allow all nodes to access the database. Each node has its own redo log and control files but the other nodes must be able to has its own redo log and control files but the other nodes must be able to access them in order to recover that node in the event of a system failure.access them in order to recover that node in the event of a system failure.

One of the bigger differences between Oracle RAC and OPS is the One of the bigger differences between Oracle RAC and OPS is the presence of Cache Fusion technology. In OPS, a request for data presence of Cache Fusion technology. In OPS, a request for data between nodes required the data to be written to disk first, and then the between nodes required the data to be written to disk first, and then the requesting node could read that data. In RAC, data is passed along with requesting node could read that data. In RAC, data is passed along with locks. locks.

3. Shared-Storage Overview3. Shared-Storage Overview

Fibre Channel is one of the most popular solutions for shared storage. As I Fibre Channel is one of the most popular solutions for shared storage. As I mentioned previously, Fibre Channel is a high-speed serial-transfer mentioned previously, Fibre Channel is a high-speed serial-transfer interface used to connect systems and storage devices in either point-to-interface used to connect systems and storage devices in either point-to-point or switched topologies. Protocols supported by Fibre Channel include point or switched topologies. Protocols supported by Fibre Channel include SCSI and IP. SCSI and IP.

Fibre Channel configurations can support as many as 127 nodes and have Fibre Channel configurations can support as many as 127 nodes and have a throughput of up to 2.12 gigabits per second. Fibre Channel, however, is a throughput of up to 2.12 gigabits per second. Fibre Channel, however, is very expensive; the switch alone can cost as much as US$1,000 and high-very expensive; the switch alone can cost as much as US$1,000 and high-end drives can reach prices of US$300. Overall, a typical Fibre Channel end drives can reach prices of US$300. Overall, a typical Fibre Channel setup (including cards for the servers) costs roughly US$5,000. setup (including cards for the servers) costs roughly US$5,000.

A less expensive alternative to Fibre Channel is SCSI. SCSI technology A less expensive alternative to Fibre Channel is SCSI. SCSI technology provides acceptable performance for shared storage, but for administrators provides acceptable performance for shared storage, but for administrators and developers who are used to GPL-based Linux prices, even SCSI can and developers who are used to GPL-based Linux prices, even SCSI can come in over budget at around US$1,000 to US$2,000 for a two-node come in over budget at around US$1,000 to US$2,000 for a two-node cluster. cluster.

Another popular solution is the Sun NFS (Network File System) found on a Another popular solution is the Sun NFS (Network File System) found on a NAS. It can be used for shared storage but only if you are using a network NAS. It can be used for shared storage but only if you are using a network appliance or something similar. Specifically, you need servers that appliance or something similar. Specifically, you need servers that guarantee direct I/O over NFS, TCP as the transport protocol, and guarantee direct I/O over NFS, TCP as the transport protocol, and read/write block sizes of 32K. read/write block sizes of 32K.

4. FireWire Technology 4. FireWire Technology

Developed by Apple Computer and Texas Instruments, Developed by Apple Computer and Texas Instruments, FireWire is a cross-platform implementation of a high-speed FireWire is a cross-platform implementation of a high-speed serial data bus. With its high bandwidth, long distances (up to serial data bus. With its high bandwidth, long distances (up to 100 meters in length) and high-powered bus, FireWire is 100 meters in length) and high-powered bus, FireWire is being used in applications such as digital video (DV), being used in applications such as digital video (DV), professional audio, hard drives, high-end digital still cameras professional audio, hard drives, high-end digital still cameras and home entertainment devices. Today, FireWire operates at and home entertainment devices. Today, FireWire operates at transfer rates of up to 800 megabits per second while next transfer rates of up to 800 megabits per second while next generation FireWire calls for speeds to a theoretical bit rate to generation FireWire calls for speeds to a theoretical bit rate to 1,600 Mbps and then up to a staggering 3,200 Mbps. That's 1,600 Mbps and then up to a staggering 3,200 Mbps. That's 3.2 gigabits per second. This speed will make FireWire 3.2 gigabits per second. This speed will make FireWire indispensable for transferring massive data files and for even indispensable for transferring massive data files and for even the most demanding video applications, such as working with the most demanding video applications, such as working with uncompressed high-definition (HD) video or multiple standard-uncompressed high-definition (HD) video or multiple standard-definition (SD) video streams. definition (SD) video streams.

Disk Interface Speed

Serial 115 kb/s - (.115 Mb/s)

Parallel (standard) 115 KB/s - (.115 MB/s)

USB 1.1 12 Mb/s - (1.5 MB/s)

Parallel (ECP/EPP) 3.0 MB/s

IDE 3.3 - 16.7 MB/s

ATA 3.3 - 66.6 MB/sec

SCSI-1 5 MB/s

SCSI-2 (Fast SCSI/Fast Narrow SCSI) 10 MB/s

Fast Wide SCSI (Wide SCSI) 20 MB/s

Ultra SCSI (SCSI-3/Fast-20/Ultra Narrow)

20 MB/s

Ultra IDE 33 MB/s

Wide Ultra SCSI (Fast Wide 20) 40 MB/s

Ultra2 SCSI 40 MB/s

IEEE1394(b) 100 - 400Mb/s - (12.5 - 50 MB/s)

USB 2.x 480 Mb/s - (60 MB/s)

Wide Ultra2 SCSI 80 MB/s

Ultra3 SCSI 80 MB/s

Wide Ultra3 SCSI 160 MB/s

FC-AL Fiber Channel 100 - 400 MB/s

1. Oracle Clusterware - /u01/app/oracle/product/crs/u01/app/oracle/product/crs2. Oracle 10g Software (Without database) – /u01/app/oracle/product/10.1.0/data_1 - (10.2.0.1)

1.Oracle Cluster Registry (OCR) File - /u02/oradata/orcl/OCRFile (OCFS2 )2.CRS Voting Disk - /u02/oradata/orcl/CSSFile (OCFS2 )3.Oracle Database files – ASM

5. Software Requirements5. Software Requirements

SoftwareSoftwareAt the software level, each node in a RAC cluster At the software level, each node in a RAC cluster needs: needs:

1.1. An An operatingoperating system system

2.2. Oracle Clusterware SoftwareOracle Clusterware Software

3.3. Oracle RAC software, and optionally Oracle RAC software, and optionally

An Oracle Automated Storage Management instance.An Oracle Automated Storage Management instance.

Oracle Automated Storage Management (ASM) Oracle Automated Storage Management (ASM)

ASM is a new feature in Oracle Database 10ASM is a new feature in Oracle Database 10gg that provides the services that provides the services of a filesystem, logical volume manager, and software RAID in a platform-of a filesystem, logical volume manager, and software RAID in a platform-independent manner. Oracle ASM can stripe and mirror your disks, allow independent manner. Oracle ASM can stripe and mirror your disks, allow disks to be added or removed while the database is under load, and disks to be added or removed while the database is under load, and automatically balance I/O to remove "hot spots." It also supports direct automatically balance I/O to remove "hot spots." It also supports direct and asynchronous I/O and implements the Oracle Data Manager API and asynchronous I/O and implements the Oracle Data Manager API (simplified I/O system call interface) introduced in Oracle9i. (simplified I/O system call interface) introduced in Oracle9i.

Oracle ASM is not a general-purpose filesystem and can be used only for Oracle ASM is not a general-purpose filesystem and can be used only for Oracle data files, redo logs, control files, and the RMAN Flash Recovery Oracle data files, redo logs, control files, and the RMAN Flash Recovery Area. Files in ASM can be created and named automatically by the Area. Files in ASM can be created and named automatically by the database (by use of the Oracle Managed Files feature) or manually by the database (by use of the Oracle Managed Files feature) or manually by the DBA. Because the files stored in ASM are not accessible to the operating DBA. Because the files stored in ASM are not accessible to the operating system, the only way to perform backup and recovery operations on system, the only way to perform backup and recovery operations on databases that use ASM files is through Recovery Manager (RMAN). databases that use ASM files is through Recovery Manager (RMAN).

ASM is implemented as a separate Oracle instance that must be up if ASM is implemented as a separate Oracle instance that must be up if other databases are to be able to access it. Memory requirements for other databases are to be able to access it. Memory requirements for ASM are light: only 64MB for most systems. In Oracle RAC environments, ASM are light: only 64MB for most systems. In Oracle RAC environments,

an ASM instance must be running on each cluster node.an ASM instance must be running on each cluster node.

6. Install the Linux Operating System6. Install the Linux Operating System

This article was designed to work with the Red Hat Enterprise Linux 4 (AS/ES) This article was designed to work with the Red Hat Enterprise Linux 4 (AS/ES) operating environment. operating environment. . . You will need three IP addresses for each server: one for You will need three IP addresses for each server: one for the private network, one for the public network, and one for the virtual IP address. the private network, one for the public network, and one for the virtual IP address. Use the operating system's network configuration tools to assign the private and Use the operating system's network configuration tools to assign the private and public network addresses. Do not assign the virtual IP address using the operating public network addresses. Do not assign the virtual IP address using the operating system's network configuration tools; this will be done by the Oracle Virtual IP system's network configuration tools; this will be done by the Oracle Virtual IP Configuration Assistant (VIPCA) during Oracle RAC software installation. Configuration Assistant (VIPCA) during Oracle RAC software installation.

Linux1 Linux1 eth0:eth0:

- Check off the option to [Configure using DHCP]- Check off the option to [Configure using DHCP]- Leave the [Activate on boot] checked- Leave the [Activate on boot] checked- IP Address: 192.168.1.100- IP Address: 192.168.1.100- Netmask: 255.255.255.0 - Netmask: 255.255.255.0

eth1:eth1:- Check off the option to [Configure using DHCP]- Check off the option to [Configure using DHCP]- Leave the [Activate on boot] checked- Leave the [Activate on boot] checked- IP Address: 192.168.2.100- IP Address: 192.168.2.100- Netmask: 255.255.255.0 - Netmask: 255.255.255.0

6. Install the Linux Operating System6. Install the Linux Operating System

Linux2 Linux2 eth0:eth0:

- Check off the option to [Configure using DHCP]- Check off the option to [Configure using DHCP]- Leave the [Activate on boot] checked- Leave the [Activate on boot] checked- IP Address: 192.168.1.101- IP Address: 192.168.1.101- Netmask: 255.255.255.0 - Netmask: 255.255.255.0

eth1:eth1:- Check off the option to [Configure using DHCP]- Check off the option to [Configure using DHCP]- Leave the [Activate on boot] checked- Leave the [Activate on boot] checked- IP Address: 192.168.2.101- IP Address: 192.168.2.101- Netmask: 255.255.255.0 - Netmask: 255.255.255.0

7. Configure Network Settings7. Configure Network Settings

Server 1 (linux1)

Device IP Address Subnet Purpose

eth0 192.168.1.100 255.255.255.0 Connects linux1 to the public network

eth1 192.168.2.100 255.255.255.0 Connects linux1 (interconnect) to linux2 (int-linux2)

/etc/hosts

127.0.0.1 localhost loopback

# Public Network - (eth0) 192.168.1.100 linux1 192.168.1.101 linux2

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv

# Public Virtual IP (VIP) addresses for - (eth0) 192.168.1.200 linux1-vip 192.168.1.201 linux2-vip


Server 2 (linux2)

Device IP Address Subnet Purpose

eth0 192.168.1.101 255.255.255.0 Connects linux2 to the public network

eth1 192.168.2.101 255.255.255.0 Connects linux2 (interconnect) to linux1 (int-linux1)

/etc/hosts

127.0.0.1 localhost loopback

# Public Network - (eth0) 192.168.1.100 linux1 192.168.1.101 linux2

# Private Interconnect - (eth1) 192.168.2.100 int-linux1 192.168.2.101 int-linux2

# Public Virtual IP (VIP) addresses for - (eth0) 192.168.1.200 vip-linux1 192.168.1.201 vip-linux2


Note that the virtual IP addresses only need to be Note that the virtual IP addresses only need to be defined in the /etc/hosts file for both nodes. The defined in the /etc/hosts file for both nodes. The public virtual IP addresses will be configured public virtual IP addresses will be configured automatically by Oracle when you run the Oracle automatically by Oracle when you run the Oracle Universal Installer, which starts Oracle's Virtual Universal Installer, which starts Oracle's Virtual Internet Protocol Configuration Assistant (VIPCA). All Internet Protocol Configuration Assistant (VIPCA). All virtual IP addresses will be activated when the srvctl virtual IP addresses will be activated when the srvctl start nodeapps -n <node_name> command is run. start nodeapps -n <node_name> command is run. This is the Host Name/IP Address that will be This is the Host Name/IP Address that will be configured in the client(s) tnsnames.ora file (more configured in the client(s) tnsnames.ora file (more details later). details later).


Adjusting Network SettingsAdjusting Network Settings Oracle now uses UDP as the default protocol on Linux for interprocess Oracle now uses UDP as the default protocol on Linux for interprocess

communication, such as cache fusion buffer transfers between the communication, such as cache fusion buffer transfers between the instances. instances. It is strongly suggested to adjust the default and maximum send buffer It is strongly suggested to adjust the default and maximum send buffer size (SO_SNDBUF socket option) to 256 KB, and the default and size (SO_SNDBUF socket option) to 256 KB, and the default and maximum receive buffer size (SO_RCVBUF socket option) to 256 KB. The maximum receive buffer size (SO_RCVBUF socket option) to 256 KB. The receive buffers are used by TCP and UDP to hold received data until is is receive buffers are used by TCP and UDP to hold received data until is is read by the application. The receive buffer cannot overflow because the read by the application. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This peer is not allowed to send data beyond the buffer size window. This means that datagrams will be discarded if they don't fit in the socket means that datagrams will be discarded if they don't fit in the socket receive buffer. This could cause the sender to overwhelm the receiver .receive buffer. This could cause the sender to overwhelm the receiver .

To make the change permanent, add the following lines to the To make the change permanent, add the following lines to the /etc/sysctl.conf file, which is used during the boot process: /etc/sysctl.conf file, which is used during the boot process: net.core.rmem_default=262144 net.core.rmem_default=262144 net.core.wmem_default=262144 net.core.wmem_default=262144

net.core.rmem_max=262144 net.core.rmem_max=262144 net.core.wmem_max=262144net.core.wmem_max=262144

8. Obtain and Install a Proper Linux Kernel 8. Obtain and Install a Proper Linux Kernel

http://oss.oracle.com/projects/firewire/http://oss.oracle.com/projects/firewire/dist/files/RedHat/RHEL4/i386/oracle-dist/files/RedHat/RHEL4/i386/oracle-firewire-modules-2.6.9-22.EL-1286-firewire-modules-2.6.9-22.EL-1286-1.i686.rpm1.i686.rpm

Install the supporting FireWire modules, as root:Install the supporting FireWire modules, as root: Install the supporting FireWire modules package by Install the supporting FireWire modules package by

running either of the following:running either of the following: # # rpm -ivh oracle-firewire-modules-2.6.9-22.EL-rpm -ivh oracle-firewire-modules-2.6.9-22.EL-

1286-1.i686.rpm1286-1.i686.rpm - - (for single processor)(for single processor)- OR -- OR -# # rpm -ivh oracle-firewire-modules-2.6.9-rpm -ivh oracle-firewire-modules-2.6.9-22.ELsmp-1286-1.i686.rpm22.ELsmp-1286-1.i686.rpm - - (for multiple (for multiple processors)processors) Add module options:Add module options:

Add the following lines to /etc/modprobe.conf: Add the following lines to /etc/modprobe.conf: options sbp2 exclusive_login=0options sbp2 exclusive_login=0


Connect FireWire drive to each machine and boot into the new Connect FireWire drive to each machine and boot into the new kernel:kernel:

After both machines are powered down, connect each of them to the back After both machines are powered down, connect each of them to the back of the FireWire drive. Power on the FireWire drive. Finally, power on each of the FireWire drive. Power on the FireWire drive. Finally, power on each Linux server and ensure to boot each machine into the new kernelLinux server and ensure to boot each machine into the new kernel

Check for SCSI Device:Check for SCSI Device: 01:04.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 01:04.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23

IEEE-1394a-2000 Controller (PHY/Link)IEEE-1394a-2000 Controller (PHY/Link) Second, let's check to see that the modules are loaded: Second, let's check to see that the modules are loaded: # lsmod |egrep "ohci1394|sbp2|ieee1394|sd_mod|scsi_mod" # lsmod |egrep "ohci1394|sbp2|ieee1394|sd_mod|scsi_mod"

sd_mod 13744 0 sd_mod 13744 0

sbp2 19724 0 sbp2 19724 0

scsi_mod 106664 3 [sg sd_mod sbp2] scsi_mod 106664 3 [sg sd_mod sbp2]

ohci1394 28008 0 (unused) ohci1394 28008 0 (unused)

ieee1394 62884 0 [sbp2 ohci1394] ieee1394 62884 0 [sbp2 ohci1394]


Third, let's make sure the disk was detected and an entry was made by the Third, let's make sure the disk was detected and an entry was made by the kernel: kernel:

# cat /proc/scsi/scsi # cat /proc/scsi/scsi Attached devices: Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: Maxtor Model: Vendor: Maxtor Model: OneTouch Rev: 0200 Type: Direct-Access ANSI SCSI revision: 06 OneTouch Rev: 0200 Type: Direct-Access ANSI SCSI revision: 06 Now Now

let's verify that the FireWire drive is accessible for multiple logins and shows a let's verify that the FireWire drive is accessible for multiple logins and shows a valid login:valid login:

# dmesg | grep sbp2 # dmesg | grep sbp2 ieee1394: sbp2: Query logins to SBP-2 device successfulieee1394: sbp2: Query logins to SBP-2 device successful ieee1394: sbp2: Maximum concurrent logins supported: 3 ieee1394: sbp2: Maximum concurrent logins supported: 3 ieee1394: sbp2: Number of active logins: 1 ieee1394: sbp2: Logged ieee1394: sbp2: Number of active logins: 1 ieee1394: sbp2: Logged

into SBP-2 device into SBP-2 device ieee1394: sbp2: Node[01:1023]: Max speed [S400] - Max payload ieee1394: sbp2: Node[01:1023]: Max speed [S400] - Max payload

[2048] [2048] # fdisk -l # fdisk -l Disk /dev/sda: 203.9 GB, 203927060480 bytes 255 heads, 63 Disk /dev/sda: 203.9 GB, 203927060480 bytes 255 heads, 63

sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = sectors/track, 24792 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes 8225280 bytes

9. Create "oracle" User and Directories (both nodes) 9. Create "oracle" User and Directories (both nodes)

Perform the following procedure on all nodes in the cluster!Perform the following procedure on all nodes in the cluster! I will be using the Oracle Cluster File System (OCFS) to store the files I will be using the Oracle Cluster File System (OCFS) to store the files

required to be shared for the Oracle Cluster Ready Services (CRS). When required to be shared for the Oracle Cluster Ready Services (CRS). When using OCFS, the UID of the UNIX user oracle and GID of the UNIX group using OCFS, the UID of the UNIX user oracle and GID of the UNIX group dba must be identical on all machines in the cluster. If either the UID or dba must be identical on all machines in the cluster. If either the UID or GID are different, the files on the OCFS file system will show up as GID are different, the files on the OCFS file system will show up as "unowned" or may even be owned by a different user. For this article, I will "unowned" or may even be owned by a different user. For this article, I will use 175 for the oracle UID and 115 for the dba GID. use 175 for the oracle UID and 115 for the dba GID.

Create Group and User for OracleCreate Group and User for Oracle Let's continue our example by creating the Unix dba group and oracle Let's continue our example by creating the Unix dba group and oracle

user account along with all appropriate directories. user account along with all appropriate directories. # mkdir -p /u01/app # groupadd -g 115 dba # useradd -u 175 -g # mkdir -p /u01/app # groupadd -g 115 dba # useradd -u 175 -g

115 -d /u01/app/oracle -s /bin/bash -c "Oracle Software Owner" -p 115 -d /u01/app/oracle -s /bin/bash -c "Oracle Software Owner" -p oracle oracle # chown -R oracle:dba /u01 # passwd oracle # su - oracle oracle # chown -R oracle:dba /u01 # passwd oracle # su - oracle oracle Note: When you are setting the Oracle environment variables for Note: When you are setting the Oracle environment variables for each RAC node, ensure to assign each RAC node a unique Oracle SID! each RAC node, ensure to assign each RAC node a unique Oracle SID! For this example, I used: For this example, I used:

linux1 : ORACLE_SID=orcl1 linux1 : ORACLE_SID=orcl1 linux2 : ORACLE_SID=orcl2 linux2 : ORACLE_SID=orcl2

9. Create "oracle" User and Directories (both nodes) 9. Create "oracle" User and Directories (both nodes)

Now, let's create the mount point for the Oracle Cluster File System Now, let's create the mount point for the Oracle Cluster File System (OCFS) that will be used to store files for the Oracle Cluster Ready (OCFS) that will be used to store files for the Oracle Cluster Ready Service (CRS). These commands will need to be run as the "root" user Service (CRS). These commands will need to be run as the "root" user account: account:

$ su – $ su – # mkdir -p /u02/oradata/orcl # mkdir -p /u02/oradata/orcl # chown -R oracle:dba /u02 # chown -R oracle:dba /u02 Oracle Cluster File System (OCFS) version 2Oracle Cluster File System (OCFS) version 2 OCFS version 1 is a great alternative to raw devices. Not only is it easier OCFS version 1 is a great alternative to raw devices. Not only is it easier

to administer and maintain, it overcomes the limit of 255 raw devices. to administer and maintain, it overcomes the limit of 255 raw devices. However, it is not a general-purpose cluster filesystem. It may only be However, it is not a general-purpose cluster filesystem. It may only be used to store the following types of files: used to store the following types of files:

Oracle data files Oracle data files Online redo logs Online redo logs Archived redo logs Archived redo logs Control files Control files Spfiles Spfiles CRS shared files (Oracle Cluster Registry and CRS voting disk). CRS shared files (Oracle Cluster Registry and CRS voting disk).

10. Creating Partitions on the Shared FireWire Storage 10. Creating Partitions on the Shared FireWire Storage Device Device

Create the following partitions on only Create the following partitions on only one nodeone node in the cluster! in the cluster! The next step is to create the required partitions on the FireWire The next step is to create the required partitions on the FireWire

(shared) drive. As I mentioned previously, we will use OCFS to store the (shared) drive. As I mentioned previously, we will use OCFS to store the two files to be shared for CRS. We will then use ASM for all physical two files to be shared for CRS. We will then use ASM for all physical database files (data/index files, online redo log files, control files, database files (data/index files, online redo log files, control files, SPFILE, and archived redo log files). SPFILE, and archived redo log files).

The following table lists the individual partitions that will be created on The following table lists the individual partitions that will be created on the FireWire (shared) drive and what files will be contained on them.the FireWire (shared) drive and what files will be contained on them.

Reboot All Nodes in RAC Cluster # fdisk -l /dev/sda

11. Configure the Linux Servers11. Configure the Linux Servers

Several of the commands within this section will need to be performed on every Several of the commands within this section will need to be performed on every node within the cluster every time the machine is booted. This section provides node within the cluster every time the machine is booted. This section provides very detailed information about setting shared memory, semaphores, and file very detailed information about setting shared memory, semaphores, and file handle limits.handle limits.

Setting SHMMAXSetting SHMMAX /etc/sysctl.conf /etc/sysctl.conf echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf Setting Semaphore Kernel ParametersSetting Semaphore Kernel Parameters echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf Setting File HandlesSetting File Handles echo "fs.file-max=65536" >> /etc/sysctl.confecho "fs.file-max=65536" >> /etc/sysctl.conf # ulimit # ulimit unlimitedunlimited

12. Configure the hangcheck-timer Kernel Module 12. Configure the hangcheck-timer Kernel Module

Perform the following configuration procedures on Perform the following configuration procedures on all nodesall nodes in the cluster! in the cluster! Oracle 9.0.1 and 9.2.0.1 used a userspace watchdog daemon called watchdogd to monitor the health Oracle 9.0.1 and 9.2.0.1 used a userspace watchdog daemon called watchdogd to monitor the health

of the cluster and to restart a RAC node in case of a failure. Starting with Oracle 9.2.0.2, the of the cluster and to restart a RAC node in case of a failure. Starting with Oracle 9.2.0.2, the watchdog daemon was deprecated by a Linux kernel module named hangcheck-timer that watchdog daemon was deprecated by a Linux kernel module named hangcheck-timer that addresses availability and reliability problems much better. The hang-check timer is loaded into the addresses availability and reliability problems much better. The hang-check timer is loaded into the Linux kernel and checks if the system hangs. It will set a timer and check the timer after a certain Linux kernel and checks if the system hangs. It will set a timer and check the timer after a certain amount of time. There is a configurable threshold to hang-check that, if exceeded will reboot the amount of time. There is a configurable threshold to hang-check that, if exceeded will reboot the machine. Although the hangcheck-timer module is not required for Oracle CRS, it is highly machine. Although the hangcheck-timer module is not required for Oracle CRS, it is highly recommended by Oracle.recommended by Oracle.

The hangcheck-timer.o ModuleThe hangcheck-timer.o Module The hangcheck-timer module uses a kernel-based timer that periodically checks the system task The hangcheck-timer module uses a kernel-based timer that periodically checks the system task

scheduler to catch delays in order to determine the health of the system. If the system hangs or scheduler to catch delays in order to determine the health of the system. If the system hangs or pauses, the timer resets the node. The hangcheck-timer module uses the Time Stamp Counter pauses, the timer resets the node. The hangcheck-timer module uses the Time Stamp Counter (TSC) CPU register, which is incremented at each clock signal. The TCS offers much more (TSC) CPU register, which is incremented at each clock signal. The TCS offers much more accurate time measurements because this register is updated by the hardware automatically. accurate time measurements because this register is updated by the hardware automatically.

Configuring Hangcheck Kernel Module ParametersConfiguring Hangcheck Kernel Module Parameters # su – # su – # echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> # echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >>

/etc/modules.conf/etc/modules.conf

13. Configure RAC Nodes for Remote Access 13. Configure RAC Nodes for Remote Access

Perform the following configuration procedures on Perform the following configuration procedures on all nodesall nodes in the cluster! in the cluster! When running the Oracle Universal Installer on a RAC node, it will use the rsh (or ssh) When running the Oracle Universal Installer on a RAC node, it will use the rsh (or ssh)

command to copy the Oracle software to all other nodes within the RAC cluster. The command to copy the Oracle software to all other nodes within the RAC cluster. The oracle UNIX account on the node running the Oracle Installer (runInstaller) must be oracle UNIX account on the node running the Oracle Installer (runInstaller) must be trusted by all other nodes in your RAC cluster. Therefore you should be able to run trusted by all other nodes in your RAC cluster. Therefore you should be able to run r*r* commands like rsh, rcp, and rlogin on the Linux server you will be running the Oracle commands like rsh, rcp, and rlogin on the Linux server you will be running the Oracle installer from, against all other Linux servers in the cluster without a password. The rsh installer from, against all other Linux servers in the cluster without a password. The rsh daemon validates users using the /etc/hosts.equiv file or the .rhosts file found in the daemon validates users using the /etc/hosts.equiv file or the .rhosts file found in the user's (oracle's) home directory. (The use of rcp and rsh are not required for normal user's (oracle's) home directory. (The use of rcp and rsh are not required for normal RAC operation. However rcp and rsh should be enabled for RAC and patchset RAC operation. However rcp and rsh should be enabled for RAC and patchset installation.) installation.)

Oracle added support in 10Oracle added support in 10gg for using the Secure Shell (SSH) tool suite for setting up user for using the Secure Shell (SSH) tool suite for setting up user equivalence. This article, however, uses the older method of rcp for copying the Oracle equivalence. This article, however, uses the older method of rcp for copying the Oracle software to the other nodes in the cluster. When using the SSH tool suite, the scp (as software to the other nodes in the cluster. When using the SSH tool suite, the scp (as opposed to the rcp) command would be used to copy the software in a very secure opposed to the rcp) command would be used to copy the software in a very secure manner. manner.

First, let's make sure that we have the rsh RPMs installed on each node in the RAC cluster: First, let's make sure that we have the rsh RPMs installed on each node in the RAC cluster: # rpm -q rsh rsh-server rsh-0.17-17 rsh-server-0.17-17# rpm -q rsh rsh-server rsh-0.17-17 rsh-server-0.17-17

13. Configure RAC Nodes for Remote Access 13. Configure RAC Nodes for Remote Access To enable the "rsh" service, the "disable" attribute in the /etc/xinetd.d/rsh file must be set to "no" and To enable the "rsh" service, the "disable" attribute in the /etc/xinetd.d/rsh file must be set to "no" and xinetdxinetd

must be reloaded. Do that by running the following commands on must be reloaded. Do that by running the following commands on all nodesall nodes in the cluster: in the cluster: # su – # su – # chkconfig rsh on # chkconfig rsh on # chkconfig rlogin on # chkconfig rlogin on # service xinetd reload # service xinetd reload Reloading configuration: [ OK ] Reloading configuration: [ OK ] To allow the "oracle" UNIX user account to be trusted among the RAC nodes, create the /etc/hosts.equiv file To allow the "oracle" UNIX user account to be trusted among the RAC nodes, create the /etc/hosts.equiv file

on on all nodesall nodes in the cluster: in the cluster: # su – # su – # touch /etc/hosts.equiv # touch /etc/hosts.equiv # chmod 600 /etc/hosts.equiv # chmod 600 /etc/hosts.equiv # chown root.root /etc/hosts.equiv # chown root.root /etc/hosts.equiv Now add all RAC nodes to the /etc/hosts.equiv file similar to the following example for Now add all RAC nodes to the /etc/hosts.equiv file similar to the following example for all nodesall nodes in the cluster: in the cluster: # cat /etc/hosts.equiv # cat /etc/hosts.equiv +linux1 oracle +linux1 oracle +linux2 oracle +linux2 oracle +int-linux1 oracle +int-linux1 oracle +int-linux2 oracle +int-linux2 oracle

14. All Startup Commands for Each RAC Node14. All Startup Commands for Each RAC Node

Verify that the following startup commands are included on Verify that the following startup commands are included on all nodesall nodes in the cluster! in the cluster! Up to this point, we have examined in great detail the parameters and resources that Up to this point, we have examined in great detail the parameters and resources that

need to be configured on all nodes for the Oracle RAC 10need to be configured on all nodes for the Oracle RAC 10gg configuration. In this configuration. In this section we will take a "deep breath" and recap those parameters, commands, and section we will take a "deep breath" and recap those parameters, commands, and entries (in previous sections of this document) that you must include in the startup entries (in previous sections of this document) that you must include in the startup scripts for each Linux node in the RAC cluster. scripts for each Linux node in the RAC cluster.

/etc/modules.conf/etc/modules.conf /etc/sysctl.conf/etc/sysctl.conf /etc/hosts/etc/hosts /etc/hosts.equiv/etc/hosts.equiv /etc/grub.conf/etc/grub.conf /etc/rc.local/etc/rc.local

15. Check RPM Packages for Oracle 1015. Check RPM Packages for Oracle 10gg Release 1 Release 1Perform the following checks on Perform the following checks on all nodesall nodes in the cluster! in the cluster! make-3.79.1 make-3.79.1 gcc-3.2.3-34 gcc-3.2.3-34 glibc-2.3.2-95.20 glibc-2.3.2-95.20 glibc-devel-2.3.2-95.20 glibc-devel-2.3.2-95.20 glibc-headers-2.3.2-95.20 glibc-headers-2.3.2-95.20 glibc-kernheaders-2.4-8.34 glibc-kernheaders-2.4-8.34 cpp-3.2.3-34 cpp-3.2.3-34 compat-db-4.0.14-5 compat-db-4.0.14-5 compat-gcc-7.3-2.96.128 compat-gcc-7.3-2.96.128 compat-gcc-c++-7.3-2.96.128 compat-gcc-c++-7.3-2.96.128 compat-libstdc++-7.3-2.96.128 compat-libstdc++-7.3-2.96.128 compat-libstdc++-compat-libstdc++-devel-7.3-2.96.128 devel-7.3-2.96.128 openmotif-2.2.2-16 openmotif-2.2.2-16 setarch-1.3-1 setarch-1.3-1

init 6init 6

16. Install and Configure OCFS Release 216. Install and Configure OCFS Release 2 Most of the configuration procedures in this section should be performed on Most of the configuration procedures in this section should be performed on all nodesall nodes in the cluster! in the cluster!

Creating the OCFS2 filesystem, however, should be executed on only Creating the OCFS2 filesystem, however, should be executed on only one nodeone node in the cluster. in the cluster. It is now time to install OCFS2. OCFS2 is a cluster filesystem that allows all nodes in a cluster to It is now time to install OCFS2. OCFS2 is a cluster filesystem that allows all nodes in a cluster to

concurrently access a device via the standard filesystem interface. This allows for easy management concurrently access a device via the standard filesystem interface. This allows for easy management of applications that need to run across a cluster. of applications that need to run across a cluster.

OCFS Release 1 was released in 2002 to enable Oracle RAC users to run the clustered database OCFS Release 1 was released in 2002 to enable Oracle RAC users to run the clustered database without having to deal with RAW devices. The filesystem was designed to store database related files, without having to deal with RAW devices. The filesystem was designed to store database related files, such as data files, control files, redo logs, archive logs, etc. OCFS Release 2 (OCFS2), in contrast, has such as data files, control files, redo logs, archive logs, etc. OCFS Release 2 (OCFS2), in contrast, has been designed as a general-purpose cluster filesystem. With it, one can store not only database been designed as a general-purpose cluster filesystem. With it, one can store not only database related files on a shared disk, but also store Oracle binaries and configuration files (shared Oracle related files on a shared disk, but also store Oracle binaries and configuration files (shared Oracle

Home) making management of RAC even easier.Home) making management of RAC even easier. Downloading OCFSDownloading OCFS (Available in the Red Hat 4 CD’s) (Available in the Red Hat 4 CD’s) ocfs2-2.6.9-22.EL-1.2.3-1.i686.rpm - ocfs2-2.6.9-22.EL-1.2.3-1.i686.rpm - (for single processor)(for single processor) oror

ocfs2-2.6.9-22.ELsmp-1.2.3-1.i686.rpm - ocfs2-2.6.9-22.ELsmp-1.2.3-1.i686.rpm - (for multiple processors)(for multiple processors)

Installing OCFSInstalling OCFS We will be installing the OCFS files onto two single-processor machines. The installation process is simply We will be installing the OCFS files onto two single-processor machines. The installation process is simply

a matter of running the following command a matter of running the following command on all nodeson all nodes in the cluster as the root user account: in the cluster as the root user account:

16. Install and Configure OCFS Release 216. Install and Configure OCFS Release 2

$ su – $ su –

# # rpm -Uvh ocfs2-2.6.9-22.EL-1.2.3-rpm -Uvh ocfs2-2.6.9-22.EL-1.2.3-1.i686.rpm \1.i686.rpm \ocfs2console-1.2.1-1.i386.rpm \ocfs2console-1.2.1-1.i386.rpm \ocfs2-tools-1.2.1-1.i386.rpmocfs2-tools-1.2.1-1.i386.rpm

17. Install and Configure Automatic Storage 17. Install and Configure Automatic Storage Management and Disks Management and Disks

Most of the installation and configuration procedures should be performed on Most of the installation and configuration procedures should be performed on all nodesall nodes. Creating the ASM disks, however, will only need to be . Creating the ASM disks, however, will only need to be performed on a performed on a single nodesingle node within the cluster. within the cluster.

In this section, we will configure Automatic Storage Management (ASM) to be In this section, we will configure Automatic Storage Management (ASM) to be used as the filesystem/volume manager for all Oracle physical database used as the filesystem/volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs). files (data, online redo logs, control files, archived redo logs).

ASM was introduced in Oracle Database 10ASM was introduced in Oracle Database 10gg and relieves the DBA from and relieves the DBA from having to manage individual files and drives. ASM is built into the having to manage individual files and drives. ASM is built into the Oracle kernel and provides the DBA with a way to manage thousands of Oracle kernel and provides the DBA with a way to manage thousands of disk drives 24x7 for single as well as clustered instances. All the files disk drives 24x7 for single as well as clustered instances. All the files and directories to be used for Oracle will be contained in a and directories to be used for Oracle will be contained in a disk groupdisk group. . ASM automatically performs load balancing in parallel across all ASM automatically performs load balancing in parallel across all available disk drives to prevent hot spots and maximize performance, available disk drives to prevent hot spots and maximize performance, even with rapidly changing data usage patterns. even with rapidly changing data usage patterns.


Downloading the ASMLib PackagesDownloading the ASMLib Packages

Installing ASMLib PackagesInstalling ASMLib Packages Edit the file /etc/sysconfig/rawdevices as follows: # raw device Edit the file /etc/sysconfig/rawdevices as follows: # raw device

bindingsbindings# format: <rawdev> <major> <minor># format: <rawdev> <major> <minor># <rawdev> <blockdev># <rawdev> <blockdev># example: /dev/raw/raw1 /dev/sda1# example: /dev/raw/raw1 /dev/sda1# /dev/raw/raw2 8 5# /dev/raw/raw2 8 5/dev/raw/raw2 /dev/sda2/dev/raw/raw2 /dev/sda2/dev/raw/raw3 /dev/sda3/dev/raw/raw3 /dev/sda3/dev/raw/raw4 /dev/sda4/dev/raw/raw4 /dev/sda4The raw device bindings will be created on each reboot. The raw device bindings will be created on each reboot.

You would then want to change ownership of all raw devices to the You would then want to change ownership of all raw devices to the "oracle" user account: # "oracle" user account: # chown oracle:dba /dev/raw/raw2; chmod chown oracle:dba /dev/raw/raw2; chmod 660 /dev/raw/raw2660 /dev/raw/raw2# # chown oracle:dba /dev/raw/raw3; chmod 660 /dev/raw/raw3chown oracle:dba /dev/raw/raw3; chmod 660 /dev/raw/raw3# # chown oracle:dba /dev/raw/raw4; chmod 660 /dev/raw/raw4chown oracle:dba /dev/raw/raw4; chmod 660 /dev/raw/raw4

The last step is to reboot the server to bind the devices or simply The last step is to reboot the server to bind the devices or simply restart the restart the rawdevicesrawdevices service: # service: # service rawdevices restartservice rawdevices restart


Creating ASM Disks for OracleCreating ASM Disks for Oracle Install ASMLib 2.0 PackagesInstall ASMLib 2.0 Packages This installation needs to be performed on This installation needs to be performed on all nodesall nodes as the root user account: as the root user account: $ $ su -su -

# # rpm -Uvh oracleasm-2.6.9-22.EL-2.0.3-1.i686.rpm \rpm -Uvh oracleasm-2.6.9-22.EL-2.0.3-1.i686.rpm \oracleasmlib-2.0.2-1.i386.rpm \oracleasmlib-2.0.2-1.i386.rpm \oracleasm-support-2.0.3-1.i386.rpmoracleasm-support-2.0.3-1.i386.rpmPreparing... Preparing... ########################################### [100%]########################################### [100%]1:oracleasm-support 1:oracleasm-support ########################################### [ 33%]########################################### [ 33%]2:oracleasm-2.6.9-22.EL 2:oracleasm-2.6.9-22.EL ########################################### [ 67%]########################################### [ 67%]3:oracleasmlib 3:oracleasmlib ########################################### [100%]########################################### [100%]


$ su – $ su –

# /etc/init.d/oracleasm createdisk VOL1 /dev/sda2 # /etc/init.d/oracleasm createdisk VOL1 /dev/sda2

Marking disk "/dev/sda2" as an ASM disk [ OK ] Marking disk "/dev/sda2" as an ASM disk [ OK ]





If you do receive a failure, try listing all ASM disks using: If you do receive a failure, try listing all ASM disks using:

# /etc/init.d/oracleasm listdisks # /etc/init.d/oracleasm listdisks

VOL1 VOL1

VOL2 VOL2

VOL3 VOL3


On On all other nodesall other nodes in the cluster, you must perform a scandisk in the cluster, you must perform a scandisk to recognize the new volumes: to recognize the new volumes: # /etc/init.d/oracleasm # /etc/init.d/oracleasm scandisks Scanning system for ASM disks [ OK ] scandisks Scanning system for ASM disks [ OK ] We We can now test that the ASM disks were successfully created can now test that the ASM disks were successfully created by using the following command on by using the following command on all nodesall nodes as the root as the root user account: user account:

# /etc/init.d/oracleasm listdisks # /etc/init.d/oracleasm listdisks

VOL1 VOL1

VOL2 VOL2

VOL3VOL3

18. Download Oracle RAC 1018. Download Oracle RAC 10gg Release 2 Software Release 2 Software The following download procedures only need to be The following download procedures only need to be

performed on performed on one nodeone node in the cluster! in the cluster! The next logical step is to install Oracle Clusterware The next logical step is to install Oracle Clusterware

Release 2 (10.2.0.1.0), Oracle Database 10Release 2 (10.2.0.1.0), Oracle Database 10gg Release 2 Release 2 (10.2.0.1.0), and finally the Oracle Database 10(10.2.0.1.0), and finally the Oracle Database 10gg Companion CD Release 2 (10.2.0.1.0) for Linux x86 Companion CD Release 2 (10.2.0.1.0) for Linux x86 software. However, you must first download and software. However, you must first download and extract the required Oracle software packages from extract the required Oracle software packages from OTN. OTN.

You will be downloading and extracting the required You will be downloading and extracting the required software from Oracle to only one of the Linux nodes in software from Oracle to only one of the Linux nodes in the cluster—namely, linux1. You will perform all the cluster—namely, linux1. You will perform all installs from this machine. The Oracle installer will installs from this machine. The Oracle installer will copy the required software packages to all other copy the required software packages to all other nodes in the RAC configuration we set up in Section nodes in the RAC configuration we set up in Section 13. 13.

Login to one of the nodes in the Linux RAC cluster as Login to one of the nodes in the Linux RAC cluster as the oracle user account. In this example, you will be the oracle user account. In this example, you will be downloading the required Oracle software to linux1 downloading the required Oracle software to linux1 and saving them to /u01/app/oracle/orainstall. and saving them to /u01/app/oracle/orainstall.

19. 19. Install Oracle 10Install Oracle 10gg Clusterware Clusterware

SoftwareSoftware Perform the following installation procedures on only Perform the following installation procedures on only one nodeone node in the in the

cluster! The Oracle Clusterware software will be installed to all other nodes cluster! The Oracle Clusterware software will be installed to all other nodes in the cluster by the Oracle Universal Installer.in the cluster by the Oracle Universal Installer.

You are now ready to install the "cluster" part of the environment - the You are now ready to install the "cluster" part of the environment - the Oracle Clusterware. In the previous section, you downloaded and extracted Oracle Clusterware. In the previous section, you downloaded and extracted the install files for Oracle Clusterware to linux1 in the directory the install files for Oracle Clusterware to linux1 in the directory /u01/app/oracle/orainstall/clusterware. This is the only node from which you /u01/app/oracle/orainstall/clusterware. This is the only node from which you need to perform the install. need to perform the install.

During the installation of Oracle Clusterware, you will be asked for the nodes During the installation of Oracle Clusterware, you will be asked for the nodes involved and to configure in the RAC cluster. Once the actual installation involved and to configure in the RAC cluster. Once the actual installation starts, it will copy the required software to all nodes using the remote access starts, it will copy the required software to all nodes using the remote access we configured in the section Section 13 ("Configure RAC Nodes for Remote we configured in the section Section 13 ("Configure RAC Nodes for Remote Access"). Access").

So, what exactly is the Oracle Clusterware responsible for? It contains all of So, what exactly is the Oracle Clusterware responsible for? It contains all of the cluster and database configuration metadata along with several system the cluster and database configuration metadata along with several system management features for RAC. It allows the DBA to register and invite an management features for RAC. It allows the DBA to register and invite an Oracle instance (or instances) to the cluster. During normal operation, Oracle instance (or instances) to the cluster. During normal operation, Oracle Clusterware will send messages (via a special ping operation) to all Oracle Clusterware will send messages (via a special ping operation) to all nodes configured in the cluster, often called the "heartbeat." If the heartbeat nodes configured in the cluster, often called the "heartbeat." If the heartbeat fails for any of the nodes, it checks with the Oracle Clusterware fails for any of the nodes, it checks with the Oracle Clusterware configuration files (on the shared disk) to distinguish between a real node configuration files (on the shared disk) to distinguish between a real node failure and a network failure. failure and a network failure.

After installing Oracle Clusterware, the Oracle Universal Installer (OUI) used After installing Oracle Clusterware, the Oracle Universal Installer (OUI) used to install the Oracle 10to install the Oracle 10gg database software (next section) will automatically database software (next section) will automatically recognize these nodes. Like the Oracle Clusterware install you will be recognize these nodes. Like the Oracle Clusterware install you will be performing in this section, the Oracle Database 10performing in this section, the Oracle Database 10gg software only needs to software only needs to be run from one node. The OUI will copy the software packages to all nodes be run from one node. The OUI will copy the software packages to all nodes configured in the RAC cluster. configured in the RAC cluster.

20. Install Oracle Database 1020. Install Oracle Database 10gg Release 2 Software Release 2 Software

Perform the following installation procedures on only Perform the following installation procedures on only one nodeone node in the cluster! The Oracle database software will be installed in the cluster! The Oracle database software will be installed to all other nodes in the cluster by the Oracle Universal to all other nodes in the cluster by the Oracle Universal Installer.Installer.

After successfully installing the Oracle Clusterware software, the next step is to install the Oracle After successfully installing the Oracle Clusterware software, the next step is to install the Oracle

Database 10Database 10gg Release 2 (10.2.0.1.0) with RAC. Release 2 (10.2.0.1.0) with RAC. ..

Installing Oracle Database 10Installing Oracle Database 10gg Software Software Install the Oracle Install the Oracle Database 10Database 10gg software with the following: software with the following:

$ cd ~oracle $ cd ~oracle

$ /u01/app/oracle/orainstall/db/Disk1/runInstaller -$ /u01/app/oracle/orainstall/db/Disk1/runInstaller -ignoreSysPrereqs ignoreSysPrereqs

21. Create the TNS Listener Process 21. Create the TNS Listener Process

Perform the following configuration procedures on only Perform the following configuration procedures on only one nodeone node in the in the cluster! The Network Configuration Assistant will setup the TNS listener cluster! The Network Configuration Assistant will setup the TNS listener in a clustered configuration on all nodes in the cluster.in a clustered configuration on all nodes in the cluster.

The DBCA requires the Oracle TNS Listener process to be configured and The DBCA requires the Oracle TNS Listener process to be configured and running on all nodes in the RAC cluster before it can create the running on all nodes in the RAC cluster before it can create the clustered database. clustered database.

The process of creating the TNS listener only needs to be performed on one The process of creating the TNS listener only needs to be performed on one node in the cluster. All changes will be made and replicated to all nodes node in the cluster. All changes will be made and replicated to all nodes in the cluster. On one of the nodes (I will be using linux1) bring up the in the cluster. On one of the nodes (I will be using linux1) bring up the Network Configuration Assistant (NETCA) and run through the process Network Configuration Assistant (NETCA) and run through the process of creating a new TNS listener process and also configure the node for of creating a new TNS listener process and also configure the node for local access. local access.

To start the NETCA, run the following GUI utility as the oracle user account: To start the NETCA, run the following GUI utility as the oracle user account:

$ netca & $ netca &

21. Create the TNS Listener Process 21. Create the TNS Listener Process

The Oracle TNS listener process should now be running on all The Oracle TNS listener process should now be running on all nodes in the RAC cluster: nodes in the RAC cluster:

$ hostname linux1 $ ps -ef | grep lsnr | grep -v 'grep' | $ hostname linux1 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' grep -v 'ocfs' | awk '{print $9}'

LISTENER_LINUX1 LISTENER_LINUX1

===================== =====================

$ hostname linux2 $ ps -ef | grep lsnr | grep -v 'grep' | $ hostname linux2 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' grep -v 'ocfs' | awk '{print $9}'

LISTENER_LINUX2 LISTENER_LINUX2

22. Create the Oracle Cluster Database 22. Create the Oracle Cluster Database

The database creation process should only be performed from The database creation process should only be performed from one nodeone node in the cluster! in the cluster!

We will use the DBCA to create the clustered database. We will use the DBCA to create the clustered database.

Creating the Clustered DatabaseCreating the Clustered Database

To start the database creation process, run the following: To start the database creation process, run the following:

# xhost + # xhost +

access control disabled, clients can connect from any access control disabled, clients can connect from any host host

# su - oracle # su - oracle

$ dbca & $ dbca &

22. Create the Oracle Cluster Database 22. Create the Oracle Cluster Database

Creating the orcltest ServiceCreating the orcltest Service During the creation of the Oracle clustered database, we added a serviceDuring the creation of the Oracle clustered database, we added a servicenamed orcltest that will be used to connect to the database with TAF enabled.named orcltest that will be used to connect to the database with TAF enabled.During several of my installs, the service was added to the tnsnames.ora, butDuring several of my installs, the service was added to the tnsnames.ora, butwas never updated as a service for each Oracle instance. was never updated as a service for each Oracle instance. Use the following to verify the orcltest service was successfully added: Use the following to verify the orcltest service was successfully added: SQL> show parameter service SQL> show parameter service NAME TYPE VALUE NAME TYPE VALUE -------------------- ----------- -------------------------------- -------------------- ----------- -------------------------------- service_names string orcl.idevelopment.info, orcltest service_names string orcl.idevelopment.info, orcltest If the only service defined was for orcl.idevelopment.info, then you will need If the only service defined was for orcl.idevelopment.info, then you will need to manually add the service to both instances: to manually add the service to both instances: SQL> SQL> show parameter service show parameter service NAME TYPE VALUE NAME TYPE VALUE -------------------- ----------- -------------------------- -------------------- ----------- -------------------------- service_names string orcl.idevelopment.info service_names string orcl.idevelopment.info SQL> alter system set service_names = 2 'orcl.idevelopment.info, SQL> alter system set service_names = 2 'orcl.idevelopment.info,

orcltest.idevelopment.info' scope=both; orcltest.idevelopment.info' scope=both;

23. Verify the TNS Networking Files 23. Verify the TNS Networking Files

Ensure that the TNS networking files are configured on Ensure that the TNS networking files are configured on all nodesall nodes in the in the cluster! cluster!

Connecting to Clustered Database From an External ClientConnecting to Clustered Database From an External Client

This is an optional step, but I like to perform it in order to verifyThis is an optional step, but I like to perform it in order to verify

my TNS files are configured correctly. Use another machine (i.e.my TNS files are configured correctly. Use another machine (i.e.

a Windows machine connected to the network) that has Oraclea Windows machine connected to the network) that has Oracle

installed (either 9installed (either 9ii or 10 or 10gg) and add the TNS entries (in the) and add the TNS entries (in the

tnsnames.ora) from either of the nodes in the cluster that were tnsnames.ora) from either of the nodes in the cluster that were

created for the clustered database. created for the clustered database.

Then try to connect to the clustered database using all available service Then try to connect to the clustered database using all available service names defined in the tnsnames.ora file: names defined in the tnsnames.ora file:

C:\> sqlplus system/manager@orcl2 C:\> sqlplus system/manager@orcl2

C:\> sqlplus system/manager@orcl1 C:\> sqlplus system/manager@orcl1

C:\> sqlplus system/manager@orcltest C:\> sqlplus system/manager@orcltest

C:\> sqlplus system/manager@orcl C:\> sqlplus system/manager@orcl

24. Creating/Altering Tablespaces 24. Creating/Altering Tablespaces

When creating the clustered database, we left all tablespaces set to their When creating the clustered database, we left all tablespaces set to their default size. If you are using a large drive for the shared storage, you may default size. If you are using a large drive for the shared storage, you may want to make a sizable testing database. want to make a sizable testing database. Below are several optional SQL commands for modifying and creating all Below are several optional SQL commands for modifying and creating all tablespaces for the test database. Please keep in mind that the database file tablespaces for the test database. Please keep in mind that the database file names (OMF files) used in this example may differ from what Oracle creates names (OMF files) used in this example may differ from what Oracle creates for your environment. for your environment. $ sqlplus "/ as sysdba" $ sqlplus "/ as sysdba" SQL> create user scott identified by tiger default tablespace users; SQL> create user scott identified by tiger default tablespace users; SQL> grant dba, resource, connect to scott; SQL> grant dba, resource, connect to scott; SQL> alter database datafile '+ORCL_DATA1/orcl/datafile/users.264.1' SQL> alter database datafile '+ORCL_DATA1/orcl/datafile/users.264.1'

resize 1024m; resize 1024m; SQL> alter tablespace users add datafile '+ORCL_DATA1' size 1024m SQL> alter tablespace users add datafile '+ORCL_DATA1' size 1024m

autoextend off; autoextend off; SQL> create tablespace indx datafile '+ORCL_DATA1' size 1024m 2 SQL> create tablespace indx datafile '+ORCL_DATA1' size 1024m 2

autoextend on next 50m maxsize unlimited 3 extent autoextend on next 50m maxsize unlimited 3 extent management local autoallocate 4 segment space management management local autoallocate 4 segment space management auto; auto;

;;

24. Creating/Altering Tablespaces 24. Creating/Altering Tablespaces

SQL> create tablespace indx datafile '+ORCL_DATA1' size SQL> create tablespace indx datafile '+ORCL_DATA1' size 1024m 2 autoextend on next 50m maxsize unlimited 3 1024m 2 autoextend on next 50m maxsize unlimited 3 extent management local autoallocate 4 segment space extent management local autoallocate 4 segment space management auto; management auto;

SQL> alter database datafile SQL> alter database datafile '+ORCL_DATA1/orcl/datafile/system.259.1' resize 800m; '+ORCL_DATA1/orcl/datafile/system.259.1' resize 800m; SQL> alter database datafile SQL> alter database datafile '+ORCL_DATA1/orcl/datafile/sysaux.261.1' resize 500m; '+ORCL_DATA1/orcl/datafile/sysaux.261.1' resize 500m;

SQL> alter tablespace undotbs1 add datafile '+ORCL_DATA1' SQL> alter tablespace undotbs1 add datafile '+ORCL_DATA1' size 1024m 2 autoextend on next 50m maxsize 2048m; size 1024m 2 autoextend on next 50m maxsize 2048m;

SQL> alter tablespace undotbs2 add datafile '+ORCL_DATA1' SQL> alter tablespace undotbs2 add datafile '+ORCL_DATA1' size 1024m 2 autoextend on next 50m maxsize 2048m; size 1024m 2 autoextend on next 50m maxsize 2048m;

SQL> alter database tempfile SQL> alter database tempfile '+ORCL_DATA1/orcl/tempfile/temp.262.1' resize 1024m;'+ORCL_DATA1/orcl/tempfile/temp.262.1' resize 1024m;

25. Verify the RAC Cluster/Database Configuration 25. Verify the RAC Cluster/Database Configuration The following RAC verification checks should be performed on The following RAC verification checks should be performed on

all nodesall nodes in the cluster! For this guide, we will perform in the cluster! For this guide, we will perform these checks only from linux1these checks only from linux1

Status of all instances and servicesStatus of all instances and services

$ srvctl status database -d orcl $ srvctl status database -d orcl

Instance orcl1 is running on node linux1 Instance orcl1 is running on node linux1


Status of a single instanceStatus of a single instance

$ srvctl status instance -d orcl -i orcl2 $ srvctl status instance -d orcl -i orcl2


Status of a named service globally across the databaseStatus of a named service globally across the database

$ srvctl status service -d orcl -s orcltest $ srvctl status service -d orcl -s orcltest

Service orcltest is running on instance(s) orcl2, orcl1Service orcltest is running on instance(s) orcl2, orcl1

25. Verify the RAC Cluster/Database Configuration 25. Verify the RAC Cluster/Database Configuration Status of node applications on a particular nodeStatus of node applications on a particular node

$ srvctl status nodeapps -n linux1 $ srvctl status nodeapps -n linux1

VIP is running on node: linux1 VIP is running on node: linux1

GSD is running on node: linux1 GSD is running on node: linux1

Listener is running on node: linux1 Listener is running on node: linux1

ONS daemon is running on node: linux1 ONS daemon is running on node: linux1

Status of an ASM instanceStatus of an ASM instance

$ srvctl status asm -n linux1 $ srvctl status asm -n linux1

ASM instance +ASM1 is running on node linux1. ASM instance +ASM1 is running on node linux1.

List all configured databasesList all configured databases

$ srvctl config database orcl $ srvctl config database orcl

Display configuration for our RAC databaseDisplay configuration for our RAC database

$ srvctl config database -d orcl $ srvctl config database -d orcl

linux1 orcl1 /u01/app/oracle/product/10.1.0/db_1 linux1 orcl1 /u01/app/oracle/product/10.1.0/db_1

linux2 orcl2 /u01/app/oracle/product/10.1.0/db_1 linux2 orcl2 /u01/app/oracle/product/10.1.0/db_1

25. Verify the RAC Cluster/Database Configuration 25. Verify the RAC Cluster/Database Configuration Display all services for the specified cluster databaseDisplay all services for the specified cluster database

$ srvctl config service -d orcl $ srvctl config service -d orcl

orcltest PREF: orcl2 orcl1 AVAIL: orcltest PREF: orcl2 orcl1 AVAIL:

Display the configuration for node applications - (VIP, GSD, Display the configuration for node applications - (VIP, GSD, ONS, Listener)ONS, Listener)

$ srvctl config nodeapps -n linux1 -a -g -s -l $ srvctl config nodeapps -n linux1 -a -g -s -l

VIP exists.: VIP exists.: /vip-linux1/192.168.1.200/255.255.255.0/eth0:eth1/vip-linux1/192.168.1.200/255.255.255.0/eth0:eth1

GSD exists. GSD exists.

ONS daemon exists. ONS daemon exists.

Listener exists. Listener exists.

Display the configuration for the ASM instance(s)Display the configuration for the ASM instance(s)

$ srvctl config asm -n linux1 +ASM1 $ srvctl config asm -n linux1 +ASM1 /u01/app/oracle/product/10.1.0/db_1 /u01/app/oracle/product/10.1.0/db_1

25. Verify the RAC Cluster/Database Configuration 25. Verify the RAC Cluster/Database Configuration All running instances in the clusterAll running instances in the cluster

SELECT inst_id , instance_number inst_no , SELECT inst_id , instance_number inst_no , instance_name inst_name , parallel , status , instance_name inst_name , parallel , status , database_status db_status , active_state state , database_status db_status , active_state state , host_name host FROM gv$instance ORDER BY host_name host FROM gv$instance ORDER BY inst_id; inst_id;

INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST STATE HOST

-------- -------- ---------- --- ------- ------------ --------- ------- -------- -------- ---------- --- ------- ------------ --------- -------

1 1 orcl1 YES OPEN ACTIVE NORMAL linux1 1 1 orcl1 YES OPEN ACTIVE NORMAL linux1

2 2 orcl2 YES OPEN ACTIVE NORMAL linux2 2 2 orcl2 YES OPEN ACTIVE NORMAL linux2

25. Verify the RAC Cluster/Database Configuration 25. Verify the RAC Cluster/Database Configuration All data files which are in the disk groupAll data files which are in the disk group

select name from v$datafile union select name from v$datafile union select member from v$logfile union select member from v$logfile union select name from v$controlfile union select name from v$controlfile union select name from v$tempfile; select name from v$tempfile; All ASM disk that belong to the 'ORCL_DATA1' disk groupAll ASM disk that belong to the 'ORCL_DATA1' disk group SELECT path FROM v$asm_disk WHERE group_number SELECT path FROM v$asm_disk WHERE group_number

IN (select group_number from v$asm_diskgroup IN (select group_number from v$asm_diskgroup where name = 'ORCL_DATA1'); where name = 'ORCL_DATA1');

PATH PATH ---------------------------------- ---------------------------------- ORCL:VOL1 ORCL:VOL1 ORCL:VOL2 ORCL:VOL2 ORCL:VOL3 ORCL:VOL3

26. Starting & Stopping the Cluster 26. Starting & Stopping the Cluster

At this point, we've installed and configured Oracle RAC 10At this point, we've installed and configured Oracle RAC 10gg entirely and entirely and have a fully functional clustered database. have a fully functional clustered database.

After all the work done up to this point, you may well ask, "OK, so how do I After all the work done up to this point, you may well ask, "OK, so how do I start and stop services?" If you have followed the instructions in this start and stop services?" If you have followed the instructions in this guide, all services—including CRS, all Oracle instances, Enterprise guide, all services—including CRS, all Oracle instances, Enterprise Manager Database Console, and so on—should start automatically on Manager Database Console, and so on—should start automatically on each reboot of the Linux nodes. each reboot of the Linux nodes.

Stopping the Oracle RAC 10Stopping the Oracle RAC 10gg Environment Environment

The first step is to stop the Oracle instance. When the instance (and related The first step is to stop the Oracle instance. When the instance (and related

services) is down, then bring down the ASM instance. Finally, shut down the services) is down, then bring down the ASM instance. Finally, shut down the

node applications (Virtual IP, GSD, TNS Listener, and ONS). node applications (Virtual IP, GSD, TNS Listener, and ONS).

$ export ORACLE_SID=orcl1 $ export ORACLE_SID=orcl1

$ emctl stop dbconsole $ emctl stop dbconsole

$ srvctl stop instance -d orcl -i orcl1 $ srvctl stop instance -d orcl -i orcl1

$ srvctl stop asm -n linux1$ srvctl stop asm -n linux1

$ srvctl stop nodeapps -n linux1 $ srvctl stop nodeapps -n linux1

26. Starting & Stopping the Cluster 26. Starting & Stopping the Cluster

Starting the Oracle RAC 10Starting the Oracle RAC 10gg Environment Environment

The first step is to start the node applications (Virtual IP, GSD, TNS Listener, The first step is to start the node applications (Virtual IP, GSD, TNS Listener,

and ONS). When the node applications are successfully started, then bring and ONS). When the node applications are successfully started, then bring

up the ASM instance. Finally, bring up the Oracle instance (and related up the ASM instance. Finally, bring up the Oracle instance (and related

services) and the Enterprise Manager Database console. services) and the Enterprise Manager Database console.

$ export ORACLE_SID=orcl1 $ export ORACLE_SID=orcl1

$ srvctl start nodeapps -n linux1 $ srvctl start nodeapps -n linux1

$ srvctl start asm -n linux1 $ srvctl start asm -n linux1

$ srvctl start instance -d orcl -i orcl1 $ srvctl start instance -d orcl -i orcl1

$ emctl start dbconsole $ emctl start dbconsole

Start/Stop All Instances with SRVCTLStart/Stop All Instances with SRVCTL

Start/stop all the instances and their enabled services. I have included this Start/stop all the instances and their enabled services. I have included this step just for fun as a way to bring down all instances! step just for fun as a way to bring down all instances!

$ srvctl start database -d orcl $ srvctl start database -d orcl

$ srvctl stop database -d orcl $ srvctl stop database -d orcl

27. Managing Transparent Application Failover 27. Managing Transparent Application Failover

It is not uncommon for businesses to demand 99.99% (or even 99.999%) It is not uncommon for businesses to demand 99.99% (or even 99.999%) availability for their enterprise applications. Think about what it would take availability for their enterprise applications. Think about what it would take to ensure a downtime of no more than .5 hours or even no downtime to ensure a downtime of no more than .5 hours or even no downtime during the year. To answer many of these high-availability requirements, during the year. To answer many of these high-availability requirements, businesses are investing in mechanisms that provide for automatic businesses are investing in mechanisms that provide for automatic failover when one participating system fails. When considering the failover when one participating system fails. When considering the availability of the Oracle database, Oracle RAC 10availability of the Oracle database, Oracle RAC 10gg provides a superior provides a superior solution with its advanced failover mechanisms. Oracle RAC 10solution with its advanced failover mechanisms. Oracle RAC 10gg includes includes the required components that all work within a clustered configuration the required components that all work within a clustered configuration responsible for providing continuous availability; when one of the responsible for providing continuous availability; when one of the participating systems fail within the cluster, the users are automatically participating systems fail within the cluster, the users are automatically migrated to the other available systems. migrated to the other available systems.

A major component of Oracle RAC 10A major component of Oracle RAC 10gg that is responsible for failover that is responsible for failover processing is the Transparent Application Failover (TAF) option. All processing is the Transparent Application Failover (TAF) option. All database connections (and processes) that lose connections are database connections (and processes) that lose connections are reconnected to another node within the cluster. The failover is completely reconnected to another node within the cluster. The failover is completely transparent to the user. transparent to the user.

This final section provides a short demonstration on how TAF works in Oracle This final section provides a short demonstration on how TAF works in Oracle RAC 10RAC 10gg. Please note that a complete discussion of failover in Oracle . Please note that a complete discussion of failover in Oracle RAC 10RAC 10gg would require an article in itself; my intention here is to present would require an article in itself; my intention here is to present only a brief overview. only a brief overview.


Setup the tnsnames.ora FileSetup the tnsnames.ora File Before demonstrating TAF, we need to verify that a valid entry exists in the Before demonstrating TAF, we need to verify that a valid entry exists in the

tnsnames.ora file on a non-RAC client machine (if you have a Windows tnsnames.ora file on a non-RAC client machine (if you have a Windows machine lying around). Ensure that you have the Oracle RDBMS software machine lying around). Ensure that you have the Oracle RDBMS software installed. (Actually, you only need a client install of the Oracle software.) installed. (Actually, you only need a client install of the Oracle software.)

During the creation of the clustered database in this guide, we created a new During the creation of the clustered database in this guide, we created a new service that will be used for testing TAF named ORCLTEST. It provides all service that will be used for testing TAF named ORCLTEST. It provides all the necessary configuration parameters for load balancing and failover. the necessary configuration parameters for load balancing and failover. You can copy the contents of this entry to the %ORACLE_HOME%\You can copy the contents of this entry to the %ORACLE_HOME%\network\admin\tnsnames.ora file on the client machine (my Windows network\admin\tnsnames.ora file on the client machine (my Windows laptop is being used in this example) in order to connect to the new Oracle laptop is being used in this example) in order to connect to the new Oracle clustered database: clustered database:

... ORCLTEST = ... ORCLTEST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-

linux1)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = linux1)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-linux2)(PORT = 1521)) (LOAD_BALANCE = yes) vip-linux2)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcltest.idevelopment.info) (FAILOVER_MODE = (TYPE = SELECT) orcltest.idevelopment.info) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) ) ) ) ... (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) ) ) ) ...


SQL Query to Check the Session's Failover InformationSQL Query to Check the Session's Failover Information

The following SQL query can be used to check a session's The following SQL query can be used to check a session's failover type, failover method, and if a failover has occurred. failover type, failover method, and if a failover has occurred. We will be using this query throughout this example. We will be using this query throughout this example.

COLUMN instance_name FORMAT a13 COLUMN instance_name FORMAT a13

COLUMN host_name FORMAT a9 COLUMN host_name FORMAT a9

COLUMN failover_method FORMAT a15 COLUMN failover_method FORMAT a15

COLUMN failed_over FORMAT a11 COLUMN failed_over FORMAT a11

SELECT instance_name , host_name , NULL AS SELECT instance_name , host_name , NULL AS failover_type , NULL AS failover_method , NULL AS failover_type , NULL AS failover_method , NULL AS failed_over FROM v$instance failed_over FROM v$instance

UNION SELECT NULL , NULL , failover_type , UNION SELECT NULL , NULL , failover_type , failover_method , failed_over FROM v$session failover_method , failed_over FROM v$session WHERE username = 'SYSTEM'; WHERE username = 'SYSTEM';


TAF DemoTAF Demo From a Windows machine (or other non-RAC client machine), From a Windows machine (or other non-RAC client machine), login to the clustered database using the orcltest service as the login to the clustered database using the orcltest service as the SYSTEM user: SYSTEM user: C:\> sqlplus system/manager@orcltest C:\> sqlplus system/manager@orcltest COLUMN instance_name FORMAT a13 COLUMN instance_name FORMAT a13 COLUMN host_name FORMAT a9 COLUMN host_name FORMAT a9 COLUMN failover_method FORMAT a15 COLUMN failover_method FORMAT a15 COLUMN failed_over FORMAT a11 COLUMN failed_over FORMAT a11 SELECT instance_name , host_name , NULL AS failover_type , NULL SELECT instance_name , host_name , NULL AS failover_type , NULL

AS failover_method , NULL AS failed_over FROM v$instance AS failover_method , NULL AS failed_over FROM v$instance UNION UNION SELECT NULL , NULL , failover_type , failover_method , failed_over SELECT NULL , NULL , failover_type , failover_method , failed_over

FROM v$session WHERE username = 'SYSTEM'; FROM v$session WHERE username = 'SYSTEM'; INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD

FAILED_OVER FAILED_OVER ------------- --------- ------------- --------------- ----------- ------------- --------- ------------- --------------- ----------- orcl1 linux1 SELECT BASIC NO orcl1 linux1 SELECT BASIC NO


DO NOT logout of the above SQL*Plus session!DO NOT logout of the above SQL*Plus session!

Now that we have run the query (above), we should now shutdown the Now that we have run the query (above), we should now shutdown the instance orcl1 on linux1 using the abort option. To perform this instance orcl1 on linux1 using the abort option. To perform this operation, we can use the srvctl command-line utility as follows: operation, we can use the srvctl command-line utility as follows:

# su - oracle # su - oracle




$ srvctl stop instance -d orcl -i orcl1 -o abort $ srvctl stop instance -d orcl -i orcl1 -o abort


Instance orcl1 is not running on node linux1 Instance orcl1 is not running on node linux1


Now let's go back to our SQL session and rerun the SQL statement in the Now let's go back to our SQL session and rerun the SQL statement in the buffer: buffer:


COLUMN instance_name FORMAT a13 COLUMN instance_name FORMAT a13

COLUMN host_name FORMAT a9 COLUMN host_name FORMAT a9

COLUMN failover_method FORMAT a15COLUMN failover_method FORMAT a15

COLUMN failed_over FORMAT a11 COLUMN failed_over FORMAT a11

SELECT instance_name , host_name , NULL AS failover_type , NULL SELECT instance_name , host_name , NULL AS failover_type , NULL AS failover_method , NULL AS failed_over FROM v$instance AS failover_method , NULL AS failed_over FROM v$instance

UNION UNION

SELECT NULL , NULL , failover_type , failover_method , failed_over SELECT NULL , NULL , failover_type , failover_method , failed_over

FROM v$session WHERE username = 'SYSTEM'; FROM v$session WHERE username = 'SYSTEM';

INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER FAILED_OVER

------------- --------- ------------- --------------- ----------- ------------- --------- ------------- --------------- -----------

orcl2 linux2 SELECT BASIC YES orcl2 linux2 SELECT BASIC YES

SQL> exit SQL> exit

From the above demonstration, we can see that the above session has now From the above demonstration, we can see that the above session has now been failed over to instance orcl2 on linux2. been failed over to instance orcl2 on linux2.

Additional InformationAdditional Information

Tnsnames.ora example.Tnsnames.ora example. A typical tnsnames.ora file configured to use TAF would similar to:A typical tnsnames.ora file configured to use TAF would similar to:

ORCLTEST =ORCLTEST =(DESCRIPTION =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT (ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))= 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT (ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))= 1521))(LOAD_BALANCE = yes)(LOAD_BALANCE = yes)(CONNECT_DATA =(CONNECT_DATA =(SERVER = DEDICATED)(SERVER = DEDICATED)(SERVICE_NAME = orcltest.idevelopment.info)(SERVICE_NAME = orcltest.idevelopment.info)(FAILOVER_MODE =(FAILOVER_MODE =(TYPE = SELECT)(TYPE = SELECT)(METHOD = BASIC)(METHOD = BASIC)(RETRIES = 180)(RETRIES = 180)(DELAY = 5)(DELAY = 5)))))))) )

CRS Trouble ShootingCRS Trouble Shooting

CRS and 10g Real Application CRS and 10g Real Application Clusters Clusters

Doc ID: Note:259301.1 Doc ID: Note:259301.1 ..

Contact InformationContact Information

Kishore AKishore A

[email protected]@yahoo.com

Real Application Cluster (RAC) Kishore A

Documents

Transcript of Real Application Cluster (RAC) Kishore A