Real Life RAC Performance Tuning Arup Nanda Lead DBA Starwood Hotels.

Real Life RAC Performance Real Life RAC Performance TuningTuning

Arup NandaArup Nanda

Lead DBALead DBA

Starwood HotelsStarwood Hotels

© Arup Nanda© Arup Nanda

Who am IWho am I

Oracle DBA for 13 years and countingOracle DBA for 13 years and countingWorking on OPS from 1999Working on OPS from 1999Speak at conferences, write papers, booksSpeak at conferences, write papers, books


Why This SessionWhy This Session

I get emails like this:I get emails like this:We are facing performance issues in RAC. We are facing performance issues in RAC.

What should I do next?What should I do next?Real Life AdviceReal Life Advice

Common Issues (with Wait Events)Common Issues (with Wait Events)Dispelling MythsDispelling MythsFormulate a Plan of AttackFormulate a Plan of AttackReal Life Case StudyReal Life Case Study

proligence.com/downloads.htmlproligence.com/downloads.html


Our RAC ImplementationOur RAC Implementation

Oracle 10g RAC in March 2004Oracle 10g RAC in March 2004 Itanium Platform running HP/UXItanium Platform running HP/UXOracle 10.1.0.2Oracle 10.1.0.2

Result: Failed Result: Failed Second Attempt: Dec 2004 – 10.1.0.3Second Attempt: Dec 2004 – 10.1.0.3Result: Failed Again Result: Failed Again Third Attempt: March 2005 – 10.1.0.4Third Attempt: March 2005 – 10.1.0.4Result: Success! Result: Success!


ChallengesChallengesTechnologyTechnology

Lone rangerLone rangerA lot of “mystery” and disconnected “facts”!A lot of “mystery” and disconnected “facts”!

PeoplePeopleBuilding a team that could not only deliver; but Building a team that could not only deliver; but

also sustain the delivered partsalso sustain the delivered partsEach day we learned something newEach day we learned something new

In today’s session: real performance In today’s session: real performance issues we faced and how we resolved issues we faced and how we resolved them, along with wait events.them, along with wait events.


Why “RAC” Performance?Why “RAC” Performance?

All tuning concepts in single instance All tuning concepts in single instance applied to RAC as wellapplied to RAC as well

RAC has other complexitiesRAC has other complexitiesMore than 1 buffer cacheMore than 1 buffer cacheMultiple caches – library cache, row cacheMultiple caches – library cache, row cache InterconnectInterconnectPingingPingingGlobal LockingGlobal Locking


Why RAC Perf TuningWhy RAC Perf Tuning

We want to make sure we identify the right We want to make sure we identify the right problem and go after it problem and go after it

……. not just . not just aa problem problem


Switch

VIP

Service

Listener

Instance

ASM

Clusterware

Op Sys

VIP

Service

Listener

Instance

ASM

Clusterware

Op Sys

Public Interface

Cache

Cache Fusion

OCR Voting

SwitchInterconnect

Storage

Node1 Node2

Lock Manager


Areas of Concern in RACAreas of Concern in RAC

More than 1 buffer cacheMore than 1 buffer cacheMultiple caches – library cache, row cacheMultiple caches – library cache, row cache InterconnectInterconnectGlobal LockingGlobal Locking


Cache IssuesCache Issues

Two Caches, requires synchronizationTwo Caches, requires synchronizationWhat that means:What that means:

A changed block in one instance, when A changed block in one instance, when requested by another, should be sent across requested by another, should be sent across via a “bridge”via a “bridge”

This bridge is the InterconnectThis bridge is the Interconnect


Interconnect PerformanceInterconnect Performance

Interconnect must be on a private LANInterconnect must be on a private LANPort aggregation to increase throughputPort aggregation to increase throughput

APA on HPUXAPA on HPUX If using Gigabit over Ethernet, use Jumbo If using Gigabit over Ethernet, use Jumbo

FramesFrames


Checking Interconnect UsedChecking Interconnect Used

Identify the interconnect usedIdentify the interconnect used$ oifcfg getif$ oifcfg getiflan902 172.17.1.0 global lan902 172.17.1.0 global cluster_interconnectcluster_interconnect

lan901 10.28.188.0 global publiclan901 10.28.188.0 global public

Is lan902 the bonded interface? If not, Is lan902 the bonded interface? If not, then set itthen set it

$ oifcfg setif …$ oifcfg setif …


Pop QuizPop Quiz

If I have a very fast interconnect, I can If I have a very fast interconnect, I can perform the same work in multiple node perform the same work in multiple node RAC as a single server with faster CPUs. RAC as a single server with faster CPUs. True/False?True/False?

Since cache fusion is now write-write, a Since cache fusion is now write-write, a fast interconnect will compensate for a fast interconnect will compensate for a slower IO subsystem. True/False?slower IO subsystem. True/False?


Cache Coherence TimesCache Coherence Times

The time is a sum of time for:The time is a sum of time for:Finding the block in the cacheFinding the block in the cache Identifying the masterIdentifying the masterGet the block in the interconnectGet the block in the interconnectTransfer speed of the interconnectTransfer speed of the interconnectLatency of the interconnectLatency of the interconnectReceive the block by the remote instanceReceive the block by the remote instanceCreate the consistent image for the userCreate the consistent image for the user

CPU

CPU

Interconnect


So it all boils down to:So it all boils down to: Block Access CostBlock Access Cost

more blocks -> more the timemore blocks -> more the time Parallel QueryParallel Query

Lock Management CostLock Management Cost More coordination -> more timeMore coordination -> more time Implicit Cache Checks – Sequence NumbersImplicit Cache Checks – Sequence Numbers

Interconnect CostInterconnect Cost LatencyLatency SpeedSpeed more data to transfer -> more the timemore data to transfer -> more the time


Hard LessonsHard Lessons

In RAC, problem symptoms may not In RAC, problem symptoms may not indicate the correct problem!indicate the correct problem!

Example:Example:When the CPU is too busy to receive or send When the CPU is too busy to receive or send

packets via UDP, the packets fails and the packets via UDP, the packets fails and the Clusterware thinks the node is down and Clusterware thinks the node is down and evicts it.evicts it.


OS TroubleshootingOS Troubleshooting

OS utilities to troubleshoot CPU issuesOS utilities to troubleshoot CPU issues toptopglanceglance

OS Utilities to troubleshoot process OS Utilities to troubleshoot process issues:issues: trusstrussstracestracedbxdbxpstackpstack


Reducing LatencyReducing Latency A factor of technologyA factor of technology TCP is the most latentTCP is the most latent UDP is better (over Ethernet)UDP is better (over Ethernet) Proprietary protocols are usually betterProprietary protocols are usually better

HyperFabric by HPHyperFabric by HP Reliable Datagram (RDP) Reliable Datagram (RDP) Direct Memory ChannelDirect Memory Channel

InfinibandInfiniband UDP over InfinibandUDP over Infiniband RDP over InfinibandRDP over Infiniband


Start with AWRStart with AWR


gc current|cr grant 2-waygc current|cr grant 2-way

Instance 1

Instance 2

Session

Database

LMS LGWR

Log BufferLog Buffer

LMS

gc current block requestgc current grant 2-way


gc current|cr block 2-waygc current|cr block 2-way

Instance 1

Instance 2

Session

Database

LMS LGWR

Log BufferLog Buffer

LMS

gc current block request log file syncgc current block 2-way


gc current|cr block 3-waygc current|cr block 3-way

Instance 1

Instance 2

Instance 3

Session

Requestor

Master

Holder

gc current block 3-way


RAC related StatsRAC related Stats


RAC Stats contd.RAC Stats contd.


Other GC Block WaitsOther GC Block Waits

gc current/cr block lostgc current/cr block lostLost blocks due to Interconnect or CPULost blocks due to Interconnect or CPU

gc curent/cr block busygc curent/cr block busyThe consistent read request was delayed, The consistent read request was delayed,

most likely an I/O bottleneckmost likely an I/O bottleneckgc current/cr block congestedgc current/cr block congested

Long run queues and/or paging due to Long run queues and/or paging due to memory deficiency.memory deficiency.


Hung or Slow?Hung or Slow?

Check Check V$SESSIONV$SESSION for for WAIT_TIMEWAIT_TIME If 0, then it’s not waiting; it’s hungIf 0, then it’s not waiting; it’s hung

When hung:When hung:Take a systemstate dump from all nodesTake a systemstate dump from all nodesWait some timeWait some timeTake another systemstate dumpTake another systemstate dumpCheck change in values. If unchanged, then Check change in values. If unchanged, then

system is hungsystem is hung


Chart a PlanChart a Plan

Rule out the obviousRule out the obviousStart with AWR ReportStart with AWR ReportStart with Top-5 WaitsStart with Top-5 WaitsSee if they have any significant waitsSee if they have any significant waits… … especially RAC relatedespecially RAC relatedGo on to RAC StatisticsGo on to RAC StatisticsBase your solution based on the wait Base your solution based on the wait

eventevent


Rule out the obviousRule out the obvious

Is interconnect private?Is interconnect private? Is interconnect on UDP?Is interconnect on UDP?Do you see high CPU?Do you see high CPU?Do you see a lot of IO bottleneck?Do you see a lot of IO bottleneck?How about memory?How about memory?Are the apps spread over evenly?Are the apps spread over evenly?Do you see lost blocks?Do you see lost blocks?


Make Simple FixesMake Simple Fixes

Strongly consider RAID 0+1Strongly consider RAID 0+1Highest possible number of I/O pathsHighest possible number of I/O pathsUse fastest interconnect possibleUse fastest interconnect possibleUse private collision free domain for I/CUse private collision free domain for I/CCache and NOORDER sequencesCache and NOORDER sequences


Enterprise ManagerEnterprise Manager


Buffer BusyBuffer Busy

CauseCause Instance wants to bring something from disk Instance wants to bring something from disk

to the buffer cacheto the buffer cacheDelay, due to space not availableDelay, due to space not availableDelay, b’coz the source buffer is not readyDelay, b’coz the source buffer is not readyDelay, I/O is slowDelay, I/O is slowDelay, b’coz redo log is being flushedDelay, b’coz redo log is being flushed

In summaryIn summaryLog buffer flush -> gc buffer busyLog buffer flush -> gc buffer busy


Parallel QueryParallel Query

One major issue in RAC is parallel query One major issue in RAC is parallel query that goes across many nodesthat goes across many nodes

Instance 1 Instance 2QC

Slave Slave Slave Slave

ViaInterconnect


Restricting PQRestricting PQ

Define Instance GroupsDefine Instance Groups

Specify in init.oraSpecify in init.oraprodb1.instance_groups='pqgroup1'prodb1.instance_groups='pqgroup1'

prodb2.instance_groups='pqgroup2' prodb2.instance_groups='pqgroup2' Specify Instance Groups in SessionSpecify Instance Groups in Session

SQL> alter session set SQL> alter session set parallel_instance_group = parallel_instance_group = 'pqgroup1'; 'pqgroup1';


Forcing PQ on both NodesForcing PQ on both Nodes

Define a common Instance GroupDefine a common Instance Groupprodb1.instance_groups='pqgroup1‘prodb1.instance_groups='pqgroup1‘

prodb1.instance_groups=‘pq2nodes'prodb1.instance_groups=‘pq2nodes'

prodb2.instance_groups='pqgroup2'prodb2.instance_groups='pqgroup2'

prodb2.instance_groups='pq2nodes' prodb2.instance_groups='pq2nodes' Specify Instance Groups in SessionSpecify Instance Groups in Session

SQL> alter session set SQL> alter session set parallel_instance_group = parallel_instance_group = 'pq2nodes'; 'pq2nodes';


Vital Cache Fusion ViewsVital Cache Fusion Views

gv$cache_transfer:gv$cache_transfer: Monitor blocks Monitor blocks transferred by objecttransferred by object

gv$class_cache_transfer:gv$class_cache_transfer: Monitor Monitor block transfer by classblock transfer by class

gv$file_cache_transfer:gv$file_cache_transfer: Monitor Monitor the blocks transferred per file the blocks transferred per file

gv$temp_cache_transfer:gv$temp_cache_transfer: Monitor Monitor the transfer of temporary tablespace the transfer of temporary tablespace blocksblocks


““Hot” TablesHot” Tables Tables, e.g. Rate PlansTables, e.g. Rate Plans

SmallSmall Compact blocksCompact blocks High updatesHigh updates High readsHigh reads

SymptomsSymptoms gc buffer busy waitsgc buffer busy waits

SolutionSolution Less rows per blockLess rows per block High PCTFREE, INITRANS, High PCTFREE, INITRANS, ALTER TABLE … MINIMIZE ALTER TABLE … MINIMIZE

RECORDS_PER_BLOCKRECORDS_PER_BLOCK


Hot SequencesHot Sequences

Symptoms:Symptoms:High waits on Sequence Number latchHigh waits on Sequence Number latchHigh waits on SEQ$ tableHigh waits on SEQ$ table

Solution:Solution: Increase the cacheIncrease the cacheMake it NOORDERMake it NOORDER

Especially AUDSESS$ sequence in SYS, Especially AUDSESS$ sequence in SYS, used in Auditingused in Auditing


Read Only? Say So.Read Only? Say So.

Reading table data from other instances Reading table data from other instances create “gc *” contentionscreate “gc *” contentions

Suggestion:Suggestion:Move Read Only tables to a single tablespaceMove Read Only tables to a single tablespaceMake this tablespace Read OnlyMake this tablespace Read Only

SQL> alter tablespace ROD read SQL> alter tablespace ROD read only;only;


PartitioningPartitioning

Partitioning creates several segments for Partitioning creates several segments for the same table (or index) the same table (or index)

=> more resources => more resources => less contention=> less contention


Monotonically Increasing IndexMonotonically Increasing IndexProblem:Problem:

““Reservation ID”, a sequence generated keyReservation ID”, a sequence generated key Index is heavy on one sideIndex is heavy on one side

SymptomsSymptomsBuffer busy waitsBuffer busy waits Index block spiltsIndex block spilts

Solutions:Solutions:Reverse key indexesReverse key indexesHash partitioned index (even if the table is not Hash partitioned index (even if the table is not

partitioned) 10gR2partitioned) 10gR2


Library CacheLibrary Cache

In RAC, Library Cache is globalIn RAC, Library Cache is globalSo, parsing cost is worse than non-RACSo, parsing cost is worse than non-RACSolutions:Solutions:

Minimize table alters, drops, creates, Minimize table alters, drops, creates, truncatestruncates

Use PL/SQL stored programs, not unnamed Use PL/SQL stored programs, not unnamed blocksblocks


Log FilesLog Files In 10g R2, the log files are in a single location:In 10g R2, the log files are in a single location: $CRS_HOME/log/<Host>/…$CRS_HOME/log/<Host>/…

racgracg crsdcrsd cssdcssd evmdevmd clientclient cssd/oclsmoncssd/oclsmon

$ORACLE_HOME/racg/dump$ORACLE_HOME/racg/dump


Case StudyCase Study


DiagnosisDiagnosis

ifconfig -aifconfig -a shows no congestion or shows no congestion or dropped packetsdropped packets

Top shows 1% idle time on node 2Top shows 1% idle time on node 2Top processesTop processes

LMS and LMDLMS and LMDAnd, several Netbackup processesAnd, several Netbackup processes


Further DiagnosisFurther Diagnosis SQL:SQL:

select * from v$instance_cache_transferselect * from v$instance_cache_transferwhere class = 'data block'where class = 'data block'and instance = 1;and instance = 1;

Output:Output: INSTANCE CLASS CR_BLOCK CR_BUSYINSTANCE CLASS CR_BLOCK CR_BUSY---------- ------------------ ---------- -------------------- ------------------ ---------- ----------CR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTEDCR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTED------------ ------------- ------------ ----------------------------- ------------- ------------ ----------------- 1 data block 162478682 50971491 data block 162478682 5097149 477721 347917908 2950144 16320267477721 347917908 2950144 16320267

After sometime:After sometime: INSTANCE CLASS CR_BLOCK CR_BUSYINSTANCE CLASS CR_BLOCK CR_BUSY---------- ------------------ ---------- -------------------- ------------------ ---------- ----------CR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTEDCR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTED------------ ------------- ------------ ----------------------------- ------------- ------------ ----------------- 1 data block 162480580 50971851 data block 162480580 5097185 477722 347923719 2950376 16320269477722 347923719 2950376 16320269

See increases


Diagnosis:Diagnosis:CPU starvation by LMS/D processes caused CPU starvation by LMS/D processes caused

GC waits.GC waits.Solution:Solution:

Killed the Netbackup processesKilled the Netbackup processesLMD and LMS got the CPULMD and LMS got the CPU


Increasing Interconnect SpeedIncreasing Interconnect Speed Faster HardwareFaster Hardware

Gigabit Ethernet; not FastGigabit Ethernet; not Fast Infiniband, even if IP over IBInfiniband, even if IP over IB

NIC settingsNIC settings Duplex ModeDuplex Mode Highest Top Bit RateHighest Top Bit Rate

TCP SettingsTCP Settings Flow Control SettingsFlow Control Settings Network Interrupts for CPUNetwork Interrupts for CPU Socket Receive BufferSocket Receive Buffer

LAN PlanningLAN Planning Private LANsPrivate LANs Collision DomainsCollision Domains


High Speed InterconnectsHigh Speed Interconnects

Oracle will support RDS over InfinibandOracle will support RDS over Infinibandhttp://oss.oracle.com/projects/rds/http://oss.oracle.com/projects/rds/On 10 Gig Ethernet as wellOn 10 Gig Ethernet as well


In summary: PlanningIn summary: Planning

Adequate CPU, Network, MemoryAdequate CPU, Network, MemorySequences – cache, noorderSequences – cache, noorderTablespaces read onlyTablespaces read onlyUn-compact small hot tablesUn-compact small hot tablesKeep undo and redo on fastest disksKeep undo and redo on fastest disksAvoid full table scans of large tablesAvoid full table scans of large tablesAvoid DDLs and unnamed PL/SQL blocksAvoid DDLs and unnamed PL/SQL blocks


In summary: DiagnosisIn summary: Diagnosis

Start with AWRStart with AWR Identify symptoms and assign causesIdentify symptoms and assign causesDon’t get fooled by “gc” waits as Don’t get fooled by “gc” waits as

interconnect issuesinterconnect issuesFind the correlation between “dropped” Find the correlation between “dropped”

packets in network, CPU issues from sar packets in network, CPU issues from sar and “gc buffer lost” in sysstat reports.and “gc buffer lost” in sysstat reports.


Thank You!Thank You!Download from:Download from:

proligence.com/downloads.htmlproligence.com/downloads.html

Real Life RAC Performance Tuning Arup Nanda Lead DBA Starwood Hotels.

Documents

Transcript of Real Life RAC Performance Tuning Arup Nanda Lead DBA Starwood Hotels.