- WordPress.com · Oracle Clusterware 11g Release 2 ... Oracle Grid Infrastructure ... –...

<Insert Picture Here>

Oracle Clusterware 11g Release 2 – A First Glimpse Under the HoodMarkus MichalewiczProduct Manager Oracle Clusterware

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Agenda

• Introduction

• Oracle Clusterware Files in Oracle ASM and easier Management

• Server Pools and easier management of Oracle RAC Databases

• Tips and tricks when upgrading from previous versions

• More Information

Oracle Grid InfrastructureThe universal grid foundation

DB Datafiles

OCR &Voting Files

Binaries

FileSystem

Siebel PSFT

Ora

cle

Grid

Infr

astr

uctu

re

RACOne FR

EE

RACDB2

RACDB1

• Standardize infrastructure software– Eliminates the need for 3rd-party solutions– Combines Oracle Automatic Storage

Management (ASM) & Oracle Clusterware– Typically used by System Administrators

• Includes:

– Oracle ASM– ASM Cluster File System (ACFS)– ACFS Snapshots– Oracle Clusterware– Cluster Health Manager

New Concepts in Oracle Clusterware• Oracle Clusterware 11g Release 2 major development goals:

– Easier Installation – Easier Management

• An introduction to the new concepts was presented in October:

• This presentation focuses on the following concepts in detail:– Oracle Clusterware Files in Oracle ASM and easier Management– Server Pools and easier management of Oracle RAC Databases– Tips and tricks when upgrading from previous versions


Oracle Clusterware Files in Oracle ASM and easier Management

OCR / Voting Files stored in Oracle ASM

Create an ASM Disk Group

• The OCR is managed like a datafile in ASM (new type)– It adheres completely to the redundancy settings for the DG

The OCR Managed in Oracle ASM

Voting Files Managed in Oracle ASM

• Unlike the OCR, Voting Files are– Stored on distinguished ASM disks

• Oracle ASM auto creates 1/3/5 Voting Files – Based on Ext/Normal/High redundancy

and on Failure Groups in the Disk Group– Per default there is one failure group per disk– ASM will enforce the required number of disks– New failure group type: Quorum Failgroup

[GRID]> crsctl query css votedisk 1. 2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA] 2. 2 aafab95f9ef84f03bf6e26adc2a3b0e8 (/dev/sde5) [DATA] 3. 2 28dd4128f4a74f73bf8653dabd88c737 (/dev/sdd6) [DATA]Located 3 voting disk(s).

What if... I have 2 Storage Arrays • Configuring environments with 2 or more storage

arrays will probably require post installation steps.

• Solution: Quorum Failgroup

• Note: “<another disk or file>” can be placed on e.g. NFS

• Configuration flow– Create any ASM diskgroup during installation– After the installation either create a new disk group

or modify the existing one using a Quorum failgroup

• More information on http://otn.oracle.com/clusterwareUsing standard NFS to support a third voting disk on an Extended Distance cluster configuration on Linux, AIX, HP-UX, or Solaris

SQL> CREATE DISKGROUP TEST NORMAL REDUNDANCY FAILGROUP fg1 DISK ‘<a disk in SAN1>' FAILGROUP fg2 DISK '<a disk in SAN2>' QUORUM FAILGROUP fg3 DISK '<another disk or file>' ATTRIBUTE 'compatible.asm' = '11.2.0.0.0';

Oracle Clusterware Startup

• The Voting Files are backed up into the OCR– Any configuration change in the cluster (e.g. node

addition) triggers a new backup of the Voting Files. – A single, failed Voting Disks can be restored by

ASM automatically within a Disk Group – Note: Do not use DD to back up the Voting Disks anymore!

• The OCR is backed up automatically every 4 hours – Manual Backups can be taken as required– Automatic backup is stored in the Grid Infrastructure

home on the node where the backup is performed

• ONLY IF all Voting Disks are corrupted or failed AND (all copies of) the OCR are also corrupted or unavailable THEN manual intervention would be required

Backup of Clusterware FilesFully automatic

• The cluster does not rely on ASM to access the Voting Files• If just the ASM instance fails, there is no impact

• If an ASM instance fails on any node, the OCR becomes unavailable on this particular node, but the node remains operational.

• If an ASM instance fails on a particular node, the (RAC) databases instances cannot access their data on this node anymore, but the cluster remains operational.

• If the CRSD process running on the node affected by the ASM instance failure is the OCR writer, AND the majority of the OCR locations is stored in ASM, AND an IO is attempted on the OCR during the time the ASM instance is down on this node, THEN CRSD stops and becomes inoperable.

• Under no circumstances will the failure of one ASM instance on one node affect the whole cluster.

Clusterware Files in Oracle ASMWhat happens, if an ASM instance fails?


Server Pools and easierManagement of Oracle RAC Databases

New Grid Concept: Server Pools Foundation for a Dynamic Cluster Partitioning

Siebel PSFT

Ora

cle

Grid

Infr

astr

uctu

re

RACOne

Ora

cle

RAC

DB

sFR

EE

RACDB2

RACDB1

• Logical division of a cluster into pools of servers.

• Hosts applications (which could be databases or applications)

Why Use Server Pools?• Easy allocation of resources to workload

• Easy management of Oracle RAC – Just define instance requirements

(# of nodes – no fixed assignment)

• Facilitates Consolidation of Applications and Databases on Clusters

Siebel PSFT

Ora

cle

Grid

Infr

astr

uctu

re

RACOne

Ora

cle

RAC

DB

sFR

EE

RACDB2

RACDB1

• A Server Pool is defined by 4 attributes:

– Server Pool Name– Min – specifies the “minimum” number of servers

that should run in the server pool – Max – states the “maximum” number of servers

that can run in the server pool. – Imp – “importance” specifies the relative

importance between server pools. This parameter is of relevance at the time of the server assignment to server pools or when servers need to be re-shuffled in the cluster due to failures.

• Server Pools hosting Oracle (RAC) Databases are managed by SRVCTL – Use CRSCTL if Server Pool hosts anything else

than a database

Enable Policy-based Cluster Management Define Server Pools using the appropriate Definition

1. Administrator Managed– Specifically define where the database should run

with a list of servers names (“traditional way”)– Define where services should run within the database

2. Policy Managed– Define resource requirements for expected workload – Enough instances are started to support expected workload – Goal: remove hard coding of a service to a specific instance

Server Pools and Database Management Two Management Styles for Oracle RAC Databases

Database Server Pools

• Special server pool used for Oracle databases• Non-overlapping: server can only be in one pool at a time

• Automatically created by DBCA – asks for cardinality (number of instances)– Provide: max=cardinality, min (default 0), importance (default 0)

• Managed by SRVCTL or Enterprise Manager– Not allowed to modify with CRSCTL– Cluster resource prefixed by ora.*

srvctl add serverpool –g <name> –u <max>

Cluster Managed Services with Server Pools

• One to one mapping– IE a service can only run in one server pool

• Services are – uniform (run on all instances in the pool)

OR – singleton (runs on only one instance in the pool)

FREE Server Pool

• A special server pool, used for management of spare capacity

• Any unassigned server will go to FREE

• Always exists but may be of size 0

• Default Importance is 0, but is editable. • Min and Max are defined automatically.

Server Assignment

• Servers are assigned in the following order:1. Generic server pool 2. User assigned server pool3. Free

• Importance is used to determine the order: 1. Fill all server pools in order of importance

until they meet their minimum2. Fill all server pools in order of importance

until they meet their maximum3. By Default any left over go into FREE

– One can change the importance of free, if required

Example – Server Allocation to Server Pool

• Cluster with 6 nodes– ProdDB server pool for production database (min 0, max 3, Imp 2)

ProdDB Server Pool FREE

Example – Server Allocation to Server Pool

• Cluster with 6 nodes– ProdDB server pool for production database (min 0, max 3, Imp 2)– EMAIL server pool for Email database (min 0, max 3, Imp 2)

ProdDB Server Pool EMAIL Server Pool

Example – Server Allocation to Server PoolsServices managed in Server Pools

• Cluster with 6 nodes– ProdDB server pool for production database (min 0, max 3, Imp 2)– Service OLTP: uniform, service BATCH: singleton – EMAIL server pool for Email database (min 0, max 3, Imp 2)– Service EMAIL: uniform


Example – Server Allocation to Server PoolsServices managed in Server Pools

• Cluster with 6 nodes– ProdDB server pool for production database (min 0, max 3, Imp 2)– BATCH server pool for production database (min 1, max 1, Imp 2)– Service OLTP: uniform, service BATCH: singleton (isolated)– EMAIL server pool for Email database (min 0, max 3, Imp 2)– Service EMAIL: uniform

ProdDB Server Pool EMAIL Server PoolBATCH

GENERIC Server Pool

• Used to model databases in “The traditional WAY!”

• “The traditional WAY!” “Administrator Managed” – Parent of all server pools for “Administrator Managed” databases

• Always exists but may be of size 0

• Used for upgrade from 10g or 11g Release 1

• Use SRVCTL of the respective version of the database home to manage the databases in the Generic Pool.

• Servers in Generic are named (hosting member in cluster resource)

Example – Mixed Environments

• Cluster with 6 nodes– ProdDB server pool for production database (min 0, max 3, Imp 2)– EMAIL Oracle RAC database runs on Node 4, Node 5, Node 6

ProdDB Server Pool EMAIL



ProdDB Server Pool EMAILGENERIC




Node 4




Node 4 Node 5 Node 6

• If a server leaves the cluster, Oracle Clusterware may move servers from one server pool to another

– BUT ONLY if a server pool falls below its minimum

• It chooses the server to move from as follows:1. The server pool of least importance2. IF server pools are of the same importance,

THEN then the Server Pool that has more than its defined minimum servers Is chosen

• Note:– Oracle Clusterware will only move servers

if you have non-default values for minimum or importance

Server Relocation due to Cluster Reconfiguration

Example – Cluster Reconfiguration




Tips and tricks when Upgrading from previous versions

Upgrading to Oracle Grid Infrastructure 11g Release 2

• With Oracle Grid Infrastructure 11g Release 2, – ASM and Oracle Clusterware are installed into a single home

directory, which is referred to as the “Grid Infrastructure home”. – However, Oracle Clusterware and ASM remain separate products.

• General rule: The grid infrastructure version must be greater than or equal to the version of the resources it manages – for example: Oracle ASM, Oracle RAC Databases

• All 11.2 upgrades are out of place

• Remember: Also when upgrading, SCAN is required (3 more IPs)clusterSCANname.example.com IN A 133.22.67.194IN A 133.22.67.193IN A 133.22.67.192

General Upgrade Considerations

• For the Oracle Home and Oracle Base– ORACLE_BASE for Oracle Grid Infrastructure should be

different than the ORACLE_BASE for the Oracle Databases– Each installation user should have it’s own Oracle Base– The 11g Rel. 2 Oracle Grid Infrastructure home must be owned

by the same user as the pre-11.2 Oracle Clusterware home. – rootupgrade.sh will change the ownership of the Oracle Grid

Infrastructure home directory and its parents to “root”

• Do not set ORACLE_HOME, ORACLE_BASE environment variables when upgrading

• For Rolling Upgrades (min:10.1.0.5 or 10.2.0.3): – Leave Oracle Clusterware running– Shutdown NodeApps, Oracle ASM, and the Oracle

Database Instance(s) on the node you are upgrading

Before you start … • … the first place to go is the documentation:• http://www.oracle.com/technology/documentation/database.html

• Check the Installing and Upgrading Section in • Oracle Grid Infrastructure Installation Guide• Real Application Clusters Installation Guide for Linux and UNIX • Clusterware Administration and Deployment Guide• Real Application Clusters Administration and Deployment Guide • Storage Administrator's Guide • 11g Upgrade Guide• New Features Guide

• Check My Oracle Support for Upgrade Issues• Check Upgrade Companion Note: 785351.1• Check Certification Matrix

– http://www.oracle.com/technology/support/metalink/index.html

http://www.oracle.com/technology/documentation/database.html

http://www.oracle.com/technology/support/metalink/index.html



• Oracle Database 10g and Oracle Database 11g are supported with Oracle Clusterware 11g Release 2

• You must upgrade the Oracle Clusterware and Oracle ASM software to 11.2.0.1 at the same time– Oracle ASM can be upgraded after Oracle Clusterware

• Apply DBCA patch for Bug 8288940

• “Pin” the nodes crsctl pin css -n nodename

Version Compatibility

Post-Upgrade Validation – Part 1• Clusterware checks

– $ crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.2.0.1.0]

• $ crsctl check cluster -all ************************************************************** rat-rm2-ipfix006: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** rat-rm2-ipfix007: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** rat-rm2-ipfix008: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online **************************************************************

Post-Upgrade Validation – Part 2

• Clusterware Processes– $ ps -ef|grep -v grep |grep d.bin

oracle 9824 1 0 Jul14 ? 00:00:00 /u01/app/grid11gR2/bin/oclskd.bin root 22161 1 0 Jul13 ? 00:00:15 /u01/app/grid11gR2/bin/ohasd.bin reboot oracle 24161 1 0 Jul13 ? 00:00:00 /u01/app/grid11gR2/bin/mdnsd.bin oracle 24172 1 0 Jul13 ? 00:00:00 /u01/app/grid11gR2/bin/gipcd.bin oracle 24183 1 0 Jul13 ? 00:00:03 /u01/app/grid11gR2/bin/gpnpd.bin oracle 24257 1 0 Jul13 ? 00:01:26 /u01/app/grid11gR2/bin/ocssd.bin root 24309 1 0 Jul13 ? 00:00:06 /u01/app/grid11gR2/bin/octssd.bin root 24323 1 0 Jul13 ? 00:01:03 /u01/app/grid11gR2/bin/crsd.bin reboot root 24346 1 0 Jul13 ? 00:00:00 /u01/app/grid11gR2/bin/oclskd.bin oracle 24374 1 0 Jul13 ? 00:00:03 /u01/app/grid11gR2/bin/evmd.bin

• Where did OPROCD go? – OPROCD, OCLSOMON, OCLSVMON

no longer exist with Oracle Clusterware 11g Release 2– Hangcheck timer is not required on Linux anymore– Code has been rewritten and functionality has been internalized


More Information

• http://otn.oracle.com/clusterware– Oracle Clusterware 11g Release 2 Technical Overview– Using standard NFS to support a third voting disk on an Extended

Distance cluster configuration on Linux, AIX, HP-UX, or Solaris

• http://otn.oracle.com/rac– Oracle Real Application Clusters 11g Release 2 Overview of SCAN– Oracle Real Application Clusters 11g Release 2 Technical Overview

• Metalink Note: 45715.1RAC Assurance Support Team: 11gR2 RAC Starter Kit and Best Practices

• http://www.oracle.com/technology/documentation/database.html

More Information

http://otn.oracle.com/clusterware

http://otn.oracle.com/rac

http://www.oracle.com/technology/documentation/database.html


Questionsand

Answers

- WordPress.com · Oracle Clusterware 11g Release 2 ... Oracle Grid Infrastructure ... –...

Documents

Transcript of - WordPress.com · Oracle Clusterware 11g Release 2 ... Oracle Grid Infrastructure ... –...