Diagnostics and Verification of Oracle Real Application Clusters 10g

Jack Cai

Principal Product Manager

Oracle Corporation

40248

Agenda

Real Application Clusters 10g is for Everyone

Benefits of RAC Diagnostics and Verification

Diagnostics Framework Cluster Verification Framework Q&A

Oracle RAC 10g for Everyone Foundation of Oracle grid computing Much easier to use and manage

– Full stack reduces complication Clusterware Cluster filesystem Automatic storage manager

– Integral to Enterprise Manager– Workload management with policy based

service– Cluster diagnostics and verification

Diagnostics & Verification Benefits• Improve problem resolution time

Reduce need to reproduce problems Eliminate problems related to incorrect

cluster configuration Improve quality of services through a

more systematic approach

Diagnostics Framework

Trace instrumentation of Database Server Clusterization of oradebug, hang analyzer Trace loader Navigator Equally useful on single instance database For database server internal diagnostics ONLY

– Not for diagnosis of application problems/tuning– Use them ONLY under Oracle support supervisions

Diagnostics Framework Goals Generate enough diagnostic data for

first pass analysis Minimal overhead to database server Easy to use

– Single system view of a cluster– Tool for data gathering, conversion and

repository– Tool for display, navigate and analysis

DIAG Process

Instance 1

SGA

Diagnostics Architecture (Cluster View)

Off-line Tools

On-line Tools

Trace Files

OradebugX$VIEWs

Trace Navigation ToolTrace Loader

DIAG Process

Instance 2

SGA

DIAG Process

Instance 3

SGA

Diagnostics Architecture (Instance)

Off-line Tools

On-line Tools

X$ Views

Trace Buffer Pi

Trace Buffer Pj

Process Pi

Process Pj

DIAG Process

SGA

Trace Files

Instance

Trace Navigational ToolTrace Navigational Tool

Trace LoaderTrace Loader

OradebugOradebug

Flexible Trace Instrumentation Focus on issues encountered by customers

– Instrument trace on events Oracle support handles most frequently

– Foundation laid in 9i, more component instrumentation in 10g

Trace control– Trace is event based, events can be anything within a

program– Trace is controllable on a per process basis– 1000 event id (10000-10999), 256 opcodes, 256 levels– Trace is generated to SGA buffer rather than directly to

files– X$Trace, X$Trace_events can be queried during run time

“Always-on” minimal tracing Optional trace archiving

Trace Control Example

Dynamic control– ALTER TRACING <cmd>

ON OFF ENABLE <event-string> DISABLE <event-spec> FLUSH <proc-spec>

– Examples alter tracing enable <event>12345:<level>5:<pid>32 alter tracing disable 12345:5:BGS alter tracing flush 32

oradebug Is Cluster Aware Single system image of the cluster Commands are clusterized DIAG based implementation

– Damen process– Minimal overhead

HANGANALYZE analyzes the entire cluster

ExamplesSQL> oradebug -g <inst_list>/ALL <cmd>SQL> oradebug setinst “3 4”

Offline Trace Loader

• Offline trace loader converts binary trace files into text format, and loads files into Oracle databases for archiving

• Better data analysis through power of database queries

• Build a repository for problem tracking

• Facilitate trace data exchange in cross-platform environments

Offline Trace Loader Extracts trace data from archived trace

files Converts binary trace data to text Loads trace data into database as trace

repository Simple command line interface

– trcldr [option,...]– trcldr mode=append ifile=f1.trw userid=scott/tiger

Raw Trace Files(in binary or text format)

Converted Trace Files(in text format)

Target Database

Data Conversion Data Loading

TraceRepository

TraceFile Trace

LoaderTrace

Loader

TraceFile

Trace Navigation Tool

Navigate traces with links and highlights across cluster

– send/recc, malloc/free– User Defined Protocol Navigation

Coloring of traces based upon attributes Well-defined queries against keywords Cluster wide time-ordered event

diagram

Navigation Tool Protocol Trace

Trace Navigation

Navigation Tool Event Diagram

Summary of Diagnostics

Diagnostics facilities allow support to resolve issues more quickly

– First pass failure diagnosis

Allows developers to find bugs faster Tools are not meant for end users End users should only use the tools

under guidance from Oracle support personnel

Cluster Verification Goal

Eliminate errors through: – Pre/post validation of installation steps

and/or configuration changes– Ongoing cluster validation

Flexible– API invoked by other programs

OUI, OEM, DBCA, ASM– Command line interface– Can be run anytime

To be released post Oracle Database 10g

Cluster Verification Framework Verification categories

– Stage verification– Component verification

Diagnostic mode API for vendor verification tools

– Utilizing vendor platforms’ verification capabilities

Stage Verification RAC deployment process is divided into stages

– Stages are steps of installation/configuration changes

– All RAC deployment stages are verified– Verification performs pre-stage and post-stage

checks

Example stages– Cluster filesystem setup– Clusterware setup– Database installation– Cluster node addition– Database configuration

Component Verification

RAC is divided into key components– Each component can be individually

verified

Examples– Node connectivity– Shared storage accessibility– Cluster integrity– System requirements– CFS integrity

Diagnostic Mode

Attempt to pinpoint root cause of a verification failure in diagnostic mode

Applicable to stage and component verification

Specific testings are performed when a verification fails

Only triggered by verification failure

How Verification is done

Command line tool: cluvfy– cluvfy stage –post HWOS (verifies

hardware and OS stage)– cluvfy comp sys –p RDBMS (verifies

system components for RDBMS installation)

APIs by other relevant tools

Summary• Diagnostics speed up problem resolutions • Cluster verification eliminates incorrect

configuration• Ultimately reducing unscheduled system

down time and improving system availability

Diagnostics tools should ONLY be used under support guidance

End Users can use cluvfy to perform verifications

Next Steps…. Recommended sessions

– 40131: Oracle Real Application Clusters 10g:  The Fourth Generation

– 40132: Oracle Real Application Clusters 10g: Manage your cluster Without it Managing You

Recommended demos and hands-on labs– Visit DEMOgrounds under Real Application Clusters

See Your Business in Our Software– Visit the DEMOgrounds for more information.

Relevant web sites to visit for more information– http://otn.oracle.com/products/database/clustering/

content.html

Reminder – please complete the OracleWorld online session survey

Thank you.

AQ&Q U E S T I O N SQ U E S T I O N SA N S W E R SA N S W E R S