Realworld Rac PerfRealworld Rac Perf

52
Real Life RAC Real Life RAC Performance Tuning Performance Tuning Arup Nanda Arup Nanda Lead DBA Lead DBA Starwood Hotels Starwood Hotels

Transcript of Realworld Rac PerfRealworld Rac Perf

Page 1: Realworld Rac PerfRealworld Rac Perf

Real Life RAC Performance Real Life RAC Performance TuningTuning

Arup NandaArup NandaLead DBALead DBA

Starwood HotelsStarwood Hotels

Page 2: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Who am IWho am IOracle DBA for 13 years and countingOracle DBA for 13 years and countingWorking on OPS from 1999Working on OPS from 1999Speak at conferences, write papers, booksSpeak at conferences, write papers, books

Page 3: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Why This SessionWhy This Session I get emails like this:I get emails like this:

We are facing performance issues in RAC. We are facing performance issues in RAC. What should I do next?What should I do next?

Real Life AdviceReal Life AdviceCommon Issues (with Wait Events)Common Issues (with Wait Events)Dispelling MythsDispelling MythsFormulate a Plan of AttackFormulate a Plan of AttackReal Life Case StudyReal Life Case Study

proligence.com/downloads.htmlproligence.com/downloads.html

Page 4: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Our RAC ImplementationOur RAC ImplementationOracle 10g RAC in March 2004Oracle 10g RAC in March 2004

Itanium Platform running HP/UXItanium Platform running HP/UXOracle 10.1.0.2Oracle 10.1.0.2

Result: Failed Result: Failed Second Attempt: Dec 2004 – 10.1.0.3Second Attempt: Dec 2004 – 10.1.0.3Result: Failed Again Result: Failed Again Third Attempt: March 2005 – 10.1.0.4Third Attempt: March 2005 – 10.1.0.4Result: Success! Result: Success!

Page 5: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

ChallengesChallengesTechnologyTechnology

Lone rangerLone rangerA lot of “mystery” and disconnected “facts”!A lot of “mystery” and disconnected “facts”!

PeoplePeopleBuilding a team that could not only deliver; but Building a team that could not only deliver; but

also sustain the delivered partsalso sustain the delivered partsEach day we learned something newEach day we learned something new

In today’s session: real performance In today’s session: real performance issues we faced and how we resolved issues we faced and how we resolved them, along with wait events.them, along with wait events.

Page 6: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Why “RAC” Performance?Why “RAC” Performance?All tuning concepts in single instance All tuning concepts in single instance

applied to RAC as wellapplied to RAC as wellRAC has other complexitiesRAC has other complexities

More than 1 buffer cacheMore than 1 buffer cacheMultiple caches – library cache, row cacheMultiple caches – library cache, row cache InterconnectInterconnectPingingPingingGlobal LockingGlobal Locking

Page 7: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Why RAC Perf TuningWhy RAC Perf TuningWe want to make sure we identify the right We want to make sure we identify the right

problem and go after it problem and go after it ……. not just . not just aa problem problem

Page 8: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Switch

VIP

Service

Listener

Instance

ASM

Clusterware

Op Sys

VIP

Service

Listener

Instance

ASM

Clusterware

Op Sys

Public Interface

Cache

Cache Fusion

OCR Voting

SwitchInterconnect

Storage

Node1 Node2

Lock Manager

Page 9: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Areas of Concern in RACAreas of Concern in RACMore than 1 buffer cacheMore than 1 buffer cacheMultiple caches – library cache, row cacheMultiple caches – library cache, row cache InterconnectInterconnectGlobal LockingGlobal Locking

Page 10: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Cache IssuesCache IssuesTwo Caches, requires synchronizationTwo Caches, requires synchronizationWhat that means:What that means:

A changed block in one instance, when A changed block in one instance, when requested by another, should be sent across requested by another, should be sent across via a “bridge”via a “bridge”

This bridge is the InterconnectThis bridge is the Interconnect

Page 11: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Interconnect PerformanceInterconnect Performance Interconnect must be on a private LANInterconnect must be on a private LANPort aggregation to increase throughputPort aggregation to increase throughput

APA on HPUXAPA on HPUX If using Gigabit over Ethernet, use Jumbo If using Gigabit over Ethernet, use Jumbo

FramesFrames

Page 12: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Checking Interconnect UsedChecking Interconnect Used Identify the interconnect usedIdentify the interconnect used$ oifcfg getif$ oifcfg getiflan902 172.17.1.0 global lan902 172.17.1.0 global cluster_interconnectcluster_interconnect

lan901 10.28.188.0 global publiclan901 10.28.188.0 global public

Is lan902 the bonded interface? If not, Is lan902 the bonded interface? If not, then set itthen set it

$ oifcfg setif …$ oifcfg setif …

Page 13: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Pop QuizPop Quiz If I have a very fast interconnect, I can If I have a very fast interconnect, I can

perform the same work in multiple node perform the same work in multiple node RAC as a single server with faster CPUs. RAC as a single server with faster CPUs. True/False?True/False?

Since cache fusion is now write-write, a Since cache fusion is now write-write, a fast interconnect will compensate for a fast interconnect will compensate for a slower IO subsystem. True/False?slower IO subsystem. True/False?

Page 14: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Cache Coherence TimesCache Coherence TimesThe time is a sum of time for:The time is a sum of time for:

Finding the block in the cacheFinding the block in the cache Identifying the masterIdentifying the masterGet the block in the interconnectGet the block in the interconnectTransfer speed of the interconnectTransfer speed of the interconnectLatency of the interconnectLatency of the interconnectReceive the block by the remote instanceReceive the block by the remote instanceCreate the consistent image for the userCreate the consistent image for the user

CPU

CPU

Interconnect

Page 15: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

So it all boils down to:So it all boils down to: Block Access CostBlock Access Cost

more blocks -> more the timemore blocks -> more the time Parallel QueryParallel Query

Lock Management CostLock Management Cost More coordination -> more timeMore coordination -> more time Implicit Cache Checks – Sequence NumbersImplicit Cache Checks – Sequence Numbers

Interconnect CostInterconnect Cost LatencyLatency SpeedSpeed more data to transfer -> more the timemore data to transfer -> more the time

Page 16: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Hard LessonsHard Lessons In RAC, problem symptoms may not In RAC, problem symptoms may not

indicate the correct problem!indicate the correct problem!Example:Example:

When the CPU is too busy to receive or send When the CPU is too busy to receive or send packets via UDP, the packets fails and the packets via UDP, the packets fails and the Clusterware thinks the node is down and Clusterware thinks the node is down and evicts it.evicts it.

Page 17: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

OS TroubleshootingOS TroubleshootingOS utilities to troubleshoot CPU issuesOS utilities to troubleshoot CPU issues

toptopglanceglance

OS Utilities to troubleshoot process OS Utilities to troubleshoot process issues:issues: trusstrussstracestracedbxdbxpstackpstack

Page 18: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Reducing LatencyReducing Latency A factor of technologyA factor of technology TCP is the most latentTCP is the most latent UDP is better (over Ethernet)UDP is better (over Ethernet) Proprietary protocols are usually betterProprietary protocols are usually better

HyperFabric by HPHyperFabric by HP Reliable Datagram (RDP) Reliable Datagram (RDP) Direct Memory ChannelDirect Memory Channel

InfinibandInfiniband UDP over InfinibandUDP over Infiniband RDP over InfinibandRDP over Infiniband

Page 19: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Start with AWRStart with AWR

Page 20: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

gc current|cr grant 2-waygc current|cr grant 2-way

Instance 1

Instance 2

Session

Database

LMS LGWR

Log Buffer

LMS

gc current block requestgc current grant 2-way

Page 21: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

gc current|cr block 2-waygc current|cr block 2-way

Instance 1

Instance 2

Session

Database

LMS LGWR

Log Buffer

LMS

gc current block request log file syncgc current block 2-way

Page 22: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

gc current|cr block 3-waygc current|cr block 3-way

Instance 1

Instance 2

Instance 3

Session

Requestor

Master

Holder

gc current block 3-way

Page 23: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

RAC related StatsRAC related Stats

Page 24: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

RAC Stats contd.RAC Stats contd.

Page 25: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Page 26: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Other GC Block WaitsOther GC Block Waitsgc current/cr block lostgc current/cr block lost

Lost blocks due to Interconnect or CPULost blocks due to Interconnect or CPUgc curent/cr block busygc curent/cr block busy

The consistent read request was delayed, The consistent read request was delayed, most likely an I/O bottleneckmost likely an I/O bottleneck

gc current/cr block congestedgc current/cr block congestedLong run queues and/or paging due to Long run queues and/or paging due to

memory deficiency.memory deficiency.

Page 27: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Hung or Slow?Hung or Slow?Check Check V$SESSIONV$SESSION for for WAIT_TIMEWAIT_TIME

If 0, then it’s not waiting; it’s hungIf 0, then it’s not waiting; it’s hungWhen hung:When hung:

Take a systemstate dump from all nodesTake a systemstate dump from all nodesWait some timeWait some timeTake another systemstate dumpTake another systemstate dumpCheck change in values. If unchanged, then Check change in values. If unchanged, then

system is hungsystem is hung

Page 28: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Chart a PlanChart a PlanRule out the obviousRule out the obviousStart with AWR ReportStart with AWR ReportStart with Top-5 WaitsStart with Top-5 WaitsSee if they have any significant waitsSee if they have any significant waits… … especially RAC relatedespecially RAC relatedGo on to RAC StatisticsGo on to RAC StatisticsBase your solution based on the wait Base your solution based on the wait

eventevent

Page 29: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Rule out the obviousRule out the obvious Is interconnect private?Is interconnect private? Is interconnect on UDP?Is interconnect on UDP?Do you see high CPU?Do you see high CPU?Do you see a lot of IO bottleneck?Do you see a lot of IO bottleneck?How about memory?How about memory?Are the apps spread over evenly?Are the apps spread over evenly?Do you see lost blocks?Do you see lost blocks?

Page 30: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Make Simple FixesMake Simple FixesStrongly consider RAID 0+1Strongly consider RAID 0+1Highest possible number of I/O pathsHighest possible number of I/O pathsUse fastest interconnect possibleUse fastest interconnect possibleUse private collision free domain for I/CUse private collision free domain for I/CCache and NOORDER sequencesCache and NOORDER sequences

Page 31: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Enterprise ManagerEnterprise Manager

Page 32: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Buffer BusyBuffer BusyCauseCause

Instance wants to bring something from disk Instance wants to bring something from disk to the buffer cacheto the buffer cache

Delay, due to space not availableDelay, due to space not availableDelay, b’coz the source buffer is not readyDelay, b’coz the source buffer is not readyDelay, I/O is slowDelay, I/O is slowDelay, b’coz redo log is being flushedDelay, b’coz redo log is being flushed

In summaryIn summaryLog buffer flush -> gc buffer busyLog buffer flush -> gc buffer busy

Page 33: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Parallel QueryParallel QueryOne major issue in RAC is parallel query One major issue in RAC is parallel query

that goes across many nodesthat goes across many nodes

Instance 1 Instance 2QC

Slave Slave Slave Slave

ViaInterconnect

Page 34: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Restricting PQRestricting PQDefine Instance GroupsDefine Instance Groups

Specify in init.oraSpecify in init.oraprodb1.instance_groups='pqgroup1'prodb1.instance_groups='pqgroup1'prodb2.instance_groups='pqgroup2' prodb2.instance_groups='pqgroup2'

Specify Instance Groups in SessionSpecify Instance Groups in SessionSQL> alter session set SQL> alter session set parallel_instance_group = parallel_instance_group = 'pqgroup1'; 'pqgroup1';

Page 35: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Forcing PQ on both NodesForcing PQ on both NodesDefine a common Instance GroupDefine a common Instance Group

prodb1.instance_groups='pqgroup1‘prodb1.instance_groups='pqgroup1‘prodb1.instance_groups=‘pq2nodes'prodb1.instance_groups=‘pq2nodes'prodb2.instance_groups='pqgroup2'prodb2.instance_groups='pqgroup2'prodb2.instance_groups='pq2nodes' prodb2.instance_groups='pq2nodes'

Specify Instance Groups in SessionSpecify Instance Groups in SessionSQL> alter session set SQL> alter session set parallel_instance_group = parallel_instance_group = 'pq2nodes'; 'pq2nodes';

Page 36: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Vital Cache Fusion ViewsVital Cache Fusion Viewsgv$cache_transfer:gv$cache_transfer: Monitor blocks Monitor blocks

transferred by objecttransferred by objectgv$class_cache_transfer:gv$class_cache_transfer: Monitor Monitor

block transfer by classblock transfer by classgv$file_cache_transfer:gv$file_cache_transfer: Monitor Monitor

the blocks transferred per file the blocks transferred per file gv$temp_cache_transfer:gv$temp_cache_transfer: Monitor Monitor

the transfer of temporary tablespace the transfer of temporary tablespace blocksblocks

Page 37: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

““Hot” TablesHot” Tables Tables, e.g. Rate PlansTables, e.g. Rate Plans

SmallSmall Compact blocksCompact blocks High updatesHigh updates High readsHigh reads

SymptomsSymptoms gc buffer busy waitsgc buffer busy waits

SolutionSolution Less rows per blockLess rows per block High PCTFREE, INITRANS, High PCTFREE, INITRANS, ALTER TABLE … MINIMIZE ALTER TABLE … MINIMIZE

RECORDS_PER_BLOCKRECORDS_PER_BLOCK

Page 38: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Hot SequencesHot SequencesSymptoms:Symptoms:

High waits on Sequence Number latchHigh waits on Sequence Number latchHigh waits on SEQ$ tableHigh waits on SEQ$ table

Solution:Solution: Increase the cacheIncrease the cacheMake it NOORDERMake it NOORDER

Especially AUDSESS$ sequence in SYS, Especially AUDSESS$ sequence in SYS, used in Auditingused in Auditing

Page 39: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Read Only? Say So.Read Only? Say So.Reading table data from other instances Reading table data from other instances

create “gc *” contentionscreate “gc *” contentionsSuggestion:Suggestion:

Move Read Only tables to a single tablespaceMove Read Only tables to a single tablespaceMake this tablespace Read OnlyMake this tablespace Read Only

SQL> alter tablespace ROD read SQL> alter tablespace ROD read only;only;

Page 40: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

PartitioningPartitioningPartitioning creates several segments for Partitioning creates several segments for

the same table (or index) the same table (or index) => more resources => more resources => less contention=> less contention

Page 41: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Monotonically Increasing IndexMonotonically Increasing IndexProblem:Problem:

““Reservation ID”, a sequence generated keyReservation ID”, a sequence generated key Index is heavy on one sideIndex is heavy on one side

SymptomsSymptomsBuffer busy waitsBuffer busy waits Index block spiltsIndex block spilts

Solutions:Solutions:Reverse key indexesReverse key indexesHash partitioned index (even if the table is not Hash partitioned index (even if the table is not

partitioned) 10gR2partitioned) 10gR2

Page 42: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Library CacheLibrary Cache In RAC, Library Cache is globalIn RAC, Library Cache is globalSo, parsing cost is worse than non-RACSo, parsing cost is worse than non-RACSolutions:Solutions:

Minimize table alters, drops, creates, Minimize table alters, drops, creates, truncatestruncates

Use PL/SQL stored programs, not unnamed Use PL/SQL stored programs, not unnamed blocksblocks

Page 43: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Log FilesLog Files In 10g R2, the log files are in a single location:In 10g R2, the log files are in a single location: $CRS_HOME/log/<Host>/…$CRS_HOME/log/<Host>/…

racgracg crsdcrsd cssdcssd evmdevmd clientclient cssd/oclsmoncssd/oclsmon

$ORACLE_HOME/racg/dump$ORACLE_HOME/racg/dump

Page 44: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Case StudyCase Study

Page 45: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

DiagnosisDiagnosisifconfig -aifconfig -a shows no congestion or shows no congestion or

dropped packetsdropped packetsTop shows 1% idle time on node 2Top shows 1% idle time on node 2Top processesTop processes

LMS and LMDLMS and LMDAnd, several Netbackup processesAnd, several Netbackup processes

Page 46: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Further DiagnosisFurther Diagnosis SQL:SQL:

select * from v$instance_cache_transferselect * from v$instance_cache_transferwhere class = 'data block'where class = 'data block'and instance = 1;and instance = 1;

Output:Output: INSTANCE CLASS CR_BLOCK CR_BUSYINSTANCE CLASS CR_BLOCK CR_BUSY---------- ------------------ ---------- -------------------- ------------------ ---------- ----------CR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTEDCR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTED------------ ------------- ------------ ----------------------------- ------------- ------------ ----------------- 1 data block 162478682 50971491 data block 162478682 5097149 477721 347917908 2950144 16320267477721 347917908 2950144 16320267

After sometime:After sometime: INSTANCE CLASS CR_BLOCK CR_BUSYINSTANCE CLASS CR_BLOCK CR_BUSY---------- ------------------ ---------- -------------------- ------------------ ---------- ----------CR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTEDCR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTED------------ ------------- ------------ ----------------------------- ------------- ------------ ----------------- 1 data block 162480580 50971851 data block 162480580 5097185 477722 347923719 2950376 16320269477722 347923719 2950376 16320269

See increases

Page 47: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Diagnosis:Diagnosis:CPU starvation by LMS/D processes caused CPU starvation by LMS/D processes caused

GC waits.GC waits.Solution:Solution:

Killed the Netbackup processesKilled the Netbackup processesLMD and LMS got the CPULMD and LMS got the CPU

Page 48: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Increasing Interconnect SpeedIncreasing Interconnect Speed Faster HardwareFaster Hardware

Gigabit Ethernet; not FastGigabit Ethernet; not Fast Infiniband, even if IP over IBInfiniband, even if IP over IB

NIC settingsNIC settings Duplex ModeDuplex Mode Highest Top Bit RateHighest Top Bit Rate

TCP SettingsTCP Settings Flow Control SettingsFlow Control Settings Network Interrupts for CPUNetwork Interrupts for CPU Socket Receive BufferSocket Receive Buffer

LAN PlanningLAN Planning Private LANsPrivate LANs Collision DomainsCollision Domains

Page 49: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

High Speed InterconnectsHigh Speed InterconnectsOracle will support RDS over InfinibandOracle will support RDS over Infinibandhttp://oss.oracle.com/projects/rds/http://oss.oracle.com/projects/rds/On 10 Gig Ethernet as wellOn 10 Gig Ethernet as well

Page 50: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

In summary: PlanningIn summary: PlanningAdequate CPU, Network, MemoryAdequate CPU, Network, MemorySequences – cache, noorderSequences – cache, noorderTablespaces read onlyTablespaces read onlyUn-compact small hot tablesUn-compact small hot tablesKeep undo and redo on fastest disksKeep undo and redo on fastest disksAvoid full table scans of large tablesAvoid full table scans of large tablesAvoid DDLs and unnamed PL/SQL blocksAvoid DDLs and unnamed PL/SQL blocks

Page 51: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

In summary: DiagnosisIn summary: DiagnosisStart with AWRStart with AWR Identify symptoms and assign causesIdentify symptoms and assign causesDon’t get fooled by “gc” waits as Don’t get fooled by “gc” waits as

interconnect issuesinterconnect issuesFind the correlation between “dropped” Find the correlation between “dropped”

packets in network, CPU issues from sar packets in network, CPU issues from sar and “gc buffer lost” in sysstat reports.and “gc buffer lost” in sysstat reports.

Page 52: Realworld Rac PerfRealworld Rac Perf

© Arup Nanda© Arup Nanda

Thank You!Thank You!Download from:Download from:

proligence.com/downloads.htmlproligence.com/downloads.html