SkillBuilders Presents Kyle Hailey ( ... buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free...

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.SKILLBUILDERS

SkillBuilders PresentsKyle Hailey (www.perfvision.com)Oracle Performance Tuning

Average Session Load(ASL)

The Golden Metric ?

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

2

Who is Kyle Hailey1990 Oracle

90 Oracle Unix support92 Ported v6 onto Dec Ultrix93 Oracle France94 Gold Support Specialist95 Performance Expert at Digital Europe98 Kernel Development California

2000 Dot.Com2001 Quest 2002 Oracle OEM 10g2005 Embarcadero2006 Independent

When not being being a Geek writing for the Oaktable etc- photographer- Just Married and new home owner in Portland- and wonder how I missed the dot.com millions


3

In this SessionThe Holy Grail of Performance : ASL

1. ASL : stethoscope for Database HealthTap is a metric that taps directly into the heart of your DB

Heart stopped - Hung?Beating very slow - Idle ?Beating fast - Heavily Loaded ?

2. ASL ComponentsCPUWaitsTime series

3. Maximum CPU as YardstickTo measure ASL against

OEM 10g ties it all together


4

Goal of this Presentation

Simplify the data and empower the DBA

My goal is to cut out a lot of the unnecessary information and extraneous data andhone in on the essential

When to actHow hard to act


5

What’s the Database Doing?

Your manager calls, it’s 2am, do you know where your database is?

BlockedRunning okIdle

I want a quick and easy way to see what the database is doing


6

Checking the Database

How do *you* check the Database Health?Routine Exams?Statspack?

1300 lines of dataWhich linesOnce and hour

Automated Alerts?What do you set alerts on?What if no alerts fire ? Are you ok?Do alerts really tell you what’s happening?


7

What’s the Database Doing?!

Whip out the stethoscope

ASL


8Welcome to …The Cult of ASL

Once you’ve been initiated …

there is no going back

For those of you who are already members, welcome back


9

Average Session Load (ASL)

The Golden Metric Powerful MultidimensionalIndispensable

Average load of sessions on the database measured in number of sessions


10

Core of 10g Performance Monitoring


11

Performance Page


12

Average Session Load (ASL)

aka

Average Active Sessions (AAS?)OEM 10g Graphs

Session LoadI often refer to it this way

Centi-seconds per second (or secs/sec)In the dark ages, before OEM 10g, waits were often measured as Centi-secs per sec


13

ASL – Average Session LoadAverage

over 15 secs in OEM 10gRepresents Load Active Sessions Only

Active sessions put load the databaseInactive Sessions don’t put load

Except for memory usage


14

ASL : sources

1. Manuallyv$system_eventv$sysstat

2. Statspack some calculations 3. OEM 10g4. ASHRPT


15

ASL Calculation

ASL = DB TIME / Elapsed Time

DB Time (DBT) = sum over all sessions of time spent in a call. A call could be

executing SQLbackground work

DBWR writing blocksLGWR writing redo


16

DB Time

DB Time (DBT) = sum active session time

DB TIME (10g) =

DB TIME =

select value from v$sysstat where name = ‘DB time’;select value from v$sysstat where name = ‘DB time’;‘DB time’

Select sum(time_waited) from v$system_event where event not in ( ... idle events …);+Select value from v$sysstat where name = ‘CPU used by this session’;

Select sum(time_waited) from v$system_event where event not in ( ... idle events …);+Select value from v$sysstat where name = ‘CPU used by this session’;


17

1. ASL ManuallyData

10g, easy DBTime = V$sysstat ‘DB time’ 9i ( 7 to 11 actually), DBTime =

V$system_eventSum wait timesNon-idle waits

V$sysstat CPU used by this session

ASL= DBTime/elapsed timeProduces

Session time / elapsed timeSession cent-secs/secSession secs/ secAvg Session Load (ASL)


18

2. Statspack: Calculating ASL

Look forTop 5 Timed EventsElapsed Time


19

Statspack: Cheat Sheet

InstallConnect as SYSDBA@?/rdbms/admin/spcreate.sql

RunExec statspack.snap;

Generate Reports@?/rdbms/admin/spreport.sql


20

Statspack Output

Trusty statspack reportElapased TimeCheck “Top 5 Timed Events”

Start at line 52 of Start at line 52 of about 1300!about 1300!


21Statspack Elapsed Time

STATSPACK report for

DB Name DB Id Instance Inst Num Release RAC Host------- ----------- -------- -------- ---------- --- -------LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr

Snap Id Snap Time Sessions Curs/Sess--------- ------------------ -------- ---------

Begin Snap: 1 03-Apr-06 12:34:06 18 5.6End Snap: 2 03-Apr-06 12:34:36 18 4.8Elapsed: 1.00 (mins)

STATSPACK report for

DB Name DB Id Instance Inst Num Release RAC Host------- ----------- -------- -------- ---------- --- -------LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr

Snap Id Snap Time Sessions Curs/Sess--------- ------------------ -------- ---------

Begin Snap: 1 03-Apr-06 12:34:06 18 5.6End Snap: 2 03-Apr-06 12:34:36 18 4.8Elapsed: 1.00 (mins)


22

Statspack: Top 5 Timed Events

Look at Top 5 Timed Events

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time--------------------- --------- --------- -------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51

Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time------------------------------------------ ------------------ ------------------ --------------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51


23

Example

CPU + WAITSCPU = 32WAITS = 250 + 15 + 8 + 5

= 278 secsElapsed Time

= 60 secs(32 + 278) secs / 60 secs= 5.1 Average Session Load

Top 5 Timed Events

Event Time (s) ----------------- -----buffer busy waits 250CPU time 32free buffer waits 15write complete waits 8log buffer space 5

Top 5 Timed EventsTop 5 Timed Events

Event Time (s) Event Time (s) ---------------------------------- ----------buffer busy waits 250buffer busy waits 250CPU time 32CPU time 32free buffer waits 15free buffer waits 15write complete waits 8write complete waits 8log buffer spacelog buffer space 55


24

3. ASL in OEM DB Home Page


25

4. ASHRPT

?@/rdbms/admin/ashrpt.sql

Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)Avg Active

Event Class % Activity Sessions--------------------------- ----------- ---------- ----------CPU + Wait for CPU CPU 67.98 8.42enq: TX - row lock co Application 23.98 2.97buffer busy waits Concurrency 4.66 0.58latch: cache buffers chai Concurrency 2.26 0.28

Top User EventsTop User Events DB/Inst: TESTDB/DB/Inst: TESTDB/testdbtestdb (Apr 21 12:00 to 12:05)(Apr 21 12:00 to 12:05)

AvgAvg ActiveActive

EventEvent ClassClass % Activity% Activity SessionsSessions

------------------------------------------------------ ---------------------- -------------------- --------------------

CPU + Wait for CPUCPU + Wait for CPU CPUCPU 67.9867.98 8.428.42

enqenq: TX : TX -- row lock corow lock co ApplicationApplication 23.9823.98 2.972.97

buffer busy waitsbuffer busy waits ConcurrencyConcurrency 4.664.66 0.580.58

latch: cache buffers latch: cache buffers chaichai ConcurrencyConcurrency 2.262.26 0.280.28


26

Got ASL, Now What ?

Need one more item: CPU Count

# of CPUs available on SystemShared with other applications ? difficultOn dual & quad cores, lower the CPU countRepresents max active sessions doing work

A value above CPU represents some bottleneck


27

CPU count, OEM DB Home Page


28

CPU Count, OEM Perf Page

CPU CountCPU Count


29

Available CPU – init.ora

Statspack 10g shows # of CPUSStatspack 9i # of CPUs missing:

SQLPLUS> show parameters cpu_count

NAME VALUE------------------ ----------cpu_count 2

SQLPLUS> show parameters cpu_count

NAME VALUE------------------ ----------cpu_count 2


30

4. ASHRPTASH Report For TESTDB/testdbDB Name DB Id Instance Inst Num Release RAC Host------------ ----------- ------------ -------- ----------- --- ------------TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aCPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size---- ------------------ ------------------ ------------------ ------------------

2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)Analysis Begin Time: 21-Apr-06 12:00:01

Analysis End Time: 21-Apr-06 12:05:01Elapsed Time: 5.0 (mins)Sample Count: 3,716

Average Session Load: 12.39Avg. Active Session per CPU: 6.19

Report Target: None specifiedTop User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)

Avg ActiveEvent Event Class % Activity Sessions----------------------------------- --------------- ---------- ----------CPU + Wait for CPU CPU 67.98 8.42enq: TX - row lock contention Application 23.98 2.97buffer busy waits Concurrency 4.66 0.58latch: cache buffers chains Concurrency 2.26 0.28

ASH Report For TESTDB/ASH Report For TESTDB/testdbtestdb

DB NameDB Name DB IdDB Id InstanceInstance Inst Num ReleaseInst Num Release RAC HostRAC Host

------------------------ ---------------------- ------------------------ ---------------- ---------------------- ------ ------------------------

TESTDBTESTDB 2371570538 2371570538 testdbtestdb 1 10.2.0.1.01 10.2.0.1.0 NONO sdbe604asdbe604a

CPUsCPUs SGA SizeSGA Size Buffer CacheBuffer Cache Shared PoolShared Pool ASH Buffer SizeASH Buffer Size

-------- ------------------------------------ ------------------------------------ ------------------------------------ ------------------------------------

22 1,000M (100%)1,000M (100%) 468M (46.8%)468M (46.8%) 112M (11.2%)112M (11.2%) 4.0M (0.4%)4.0M (0.4%)

Analysis Begin Time:Analysis Begin Time: 2121--AprApr--06 12:00:0106 12:00:01

Analysis End Time:Analysis End Time: 2121--AprApr--06 12:05:0106 12:05:01

Elapsed Time:Elapsed Time: 5.0 (5.0 (minsmins))

Sample Count:Sample Count: 3,7163,716

Average Session Load:Average Session Load: 12.3912.39

Avg. Active Session per CPU:Avg. Active Session per CPU: 6.196.19

Report Target:Report Target: None specifiedNone specified

Top User EventsTop User Events DB/Inst: TESTDB/DB/Inst: TESTDB/testdbtestdb (Apr 21 12:00 to 12:05)(Apr 21 12:00 to 12:05)

AvgAvg ActiveActive

EventEvent EventEvent ClassClass % Activity% Activity SessionsSessions

---------------------------------------------------------------------- ------------------------------ -------------------- --------------------

CPU + Wait for CPUCPU + Wait for CPU CPUCPU 67.9867.98 8.428.42

enqenq: TX : TX -- row lock contentionrow lock contention ApplicationApplication 23.9823.98 2.972.97

buffer busy waitsbuffer busy waits ConcurrencyConcurrency 4.664.66 0.580.58

latch: cache buffers chainslatch: cache buffers chains ConcurrencyConcurrency 2.262.26 0.280.28


31

ASL FormulasUse CPU count as yardstick:

< 1 Database is not blockedASL ~= 0

Database basically idleASL < # of CPUs

CPU availableDatabase is probably not blocked

• ASL > # of CPUsCould have performance problemsASL >> # of CPUS

There is a bottleneck


32

Available CPU vs ASL

ASL = 5.1ASL = 5.1

# of CPU = 2# of CPU = 2

ASL far above available CPU => problem


33

Going Farther with ASL

ASL can tell you a lotBut it’s components tell you much moreTo go farther need the components of ASL

CPU How many CPUs (max CPU available)

WaitWhich waits

Value over time


34

EM DB Home Page


35OEM 10g Perf PagesDB Home DB Home

PerformancePerformance

ASL Point in TimeASL Point in Time

ASL over TimeASL over Time


36

Performance Page


37

Performance Page


38

ASL – Performance Page


39

OEM 10g - ASL

ASL is the top of the curveASL is the top of the curve

33 1.91.9 22


40

OEM 10g - CPU

CPU UsageCPU Usage


41

OEM 10g - Waits

WaitsWaits


42

OEM 10g – CPU vs Wait

WaitsWaitsCPUCPU


43

OEM 10g – Max CPU

Maximum Possible CPUMaximum Possible CPU


44

OEM 10g Zoom-In

CPU + WAIT

Available CPU


45

OEM 10g – get to work !

Looks Looks OK OK

But …But …

Relax Relax Get to Work!Get to Work!


46

Alternative ASL Calculations

ASL = DB TIME / Elapsed Time

But there is another wayBut there is another way ……


47

ASL based on ASHASH - Active Session History

v$active_session_historyFormula

Count active sessions over 15 secondsDivided by # of samples in 15 seconds= Average Session Load

A statistical approximation, but surprisingly closeASH data source empowers drilldown into the causes of bottlenecks


48

2. ASL from ASH

776655443322

22

776655443322

ASHASH

11

t0 t0 tntn

11

t0t0--nn

active sessions(ti)nnΣΣ00

ASL =ASL = // nn


49

Two Sources comparison1. v$system_event & v$sysstat

IndirectAccurateLags (especially CPU)Limits analysis

2. v$active_session_historyDirectReal timeApproximationAllows drilldowns




Top ActivityTop Activity

11

Same Chart butSame Chart but

calculated differently calculated differently

22


51

1. ASL – Performance Page

v$v$sysstatsysstat

DB Home DB Home


Top ActivityTop Activityv$system_eventv$system_event

ASL = ∆DBT / ASL = ∆DBT / ∆Time∆Time


52

2. ASL – Top Activity Page

v$active_session_historyv$active_session_history

active sessions(ti)nnΣΣ00

ASL =ASL =

DB Home DB Home




53

Samples VS Counters

CountersCounters

SamplesSamples

Slight LagsSlight Lags

v$system_eventv$system_event

v$active_session_historyv$active_session_history


54The Power ASH gives ASLDB Home DB Home



ASHASH


55

ASH in OEM

Top Activity gives Top Activity gives more informationmore information


56

Top Activity : Based on ASH

missingmissing

Thanks Thanks To To ASHASH


57

Top Activity : ASH Dimensions


58ASL – %Session Time Issue

Shown in % DB TimeShown in % DB TimeMissing % Session TimeMissing % Session Time


59

Top Activity: ASH Sessions

Many Users Active Many Users Active

On Performance Page, no way to tell how many usersOn Performance Page, no way to tell how many usersBut Top Activity Page fixes thatBut Top Activity Page fixes that


60

Top Activity: ASH Sessions

Two Users ActiveTwo Users Active





SQLSQLSessionSession


62OEM 10g Perf PagesTop ActivityTop Activity

SQLSQLSessionSession


63Session : ASH Activity


64SQL : ASH Activity


65

Getting the Most out of ASLNeed to know the System’s ProfileWhat your application is like?

Data WarehouseOLTPTypical load

Once you get to know it you can see anomaliesIs ASL near 0 when it should be higherIs that Data Warehouse query running normal

Do you know what it looks like?Is there an unusual bottleneck


66

Knowing your DB Profile


67

When to tune?

General rules of ThumbWaits >> CPUCPU > Max CPU


68

Waits > CPU


69

CPU > Max CPU


70

Idle Database

Value of proving the database is Idle It’s the Databases Fault

How many times do you hear that?Database Idle

No load on databaseDatabase “performance” is fine

Under utilized

Problem lies elsewhereSaved me time and stress many times


71

Idle Database – Perf Page


72

Idle Database – Top Activity


73

Harnessing ASLStatspack

lackingOEM 10g

Best tool10g (or 11) onlyNeed licences

Third Party ToolsSome good productsGenerally as expensive as OEM

Ashmon – “OEM-lite”FreeAny version with S-ASHLimited analysis


74

Statspack

Free and any versionNeeds numerical massagingLacking detailed SQL and Session info


75

OEM 10gClearest, easiest, most powerful

Overview Load chart - ASL

Breakdown –Top SQL and SessionOr Service, Module, Action, Objects, Files

Drill downsSQL Session

PlusADDM – Automatic Database Diagnostics MonitorSQL Advisor – SQL tuning advice

DRAWBACK – 10g (or 11) and need license





SessionSession SQLSQL


77

Alternatives

Third Party toolsQuest – Performance AnalyzerSymantec – I3Confio – Ignite


78

ASHMON a free “OEM-lite”

http://perfvision.com/ashmon.phpWindows application Monitors Oracle 10g or 11Show “Top Activity” page from OEM

Note: Can be used on any version of Oracle with performance repository available on http://perfvision.com/sash.php


79CPU > Max CPU


80

Radical Change in Profile


81

Going Further – Wait Analysis

If ASL identifies Wait BottlenecksUse OEM 10g

Automatic Diagnostic Database Monitor (ADDM)Oracle Wait Interface

by R Shee and K Deshpande Wait Interface Class

Detail explanation, analysis and solutions to the top 30 wait events (98% of all waits encountered)


82

Where Can You Learn More

Oracle 10g Advanced Performance Tuning Class

With Kyle Hailey and SkillBuilders2 Day ClassOffsite or onsiteRegistration

www.skillbuilders.comGary Belke at 1-888-803-5607


83

Oracle 10g Advanced Performance Tuning Class

Learn new 10g performance and diagnostic toolsUnderstand the 10g Enterprise Manager Performance page ASL, AWR, ADDMAdvanced level of detail

Collection, Aggregation, Display, Analyze

Top 30 wait eventsASHConquer your database woes!


84

In summary

ASL is simple and PowerfulASL’s components are even more powerful

CPUWAITValue over Time

Use # of CPUs as a yardstick Know your application load profile to see anomaliesMonitor ASL with OEM 10g, Ashmon or third party tools

SkillBuilders Presents Kyle Hailey ( ... buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free...

Documents

Transcript of SkillBuilders Presents Kyle Hailey ( ... buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free...