05 WAITS Intro
-
Upload
harshchauhan -
Category
Documents
-
view
230 -
download
0
Transcript of 05 WAITS Intro
-
8/8/2019 05 WAITS Intro
1/41
Part II : Waits Eventsand the
Geeks who love them
Kyle Hailey
http://perfvision.com
-
8/8/2019 05 WAITS Intro
2/41
Copyright 2006 Kyle Hailey
Wait Events
Wait EventsWait Events
-
8/8/2019 05 WAITS Intro
3/41
-
8/8/2019 05 WAITS Intro
4/41
Copyright 2006 Kyle Hailey
In this Section:
Introduction to WaitsTuning Methodology
Plan of Action
Statspacks, AWR or OEM forCollectionData
Based on Waits
Using Waits to Solve Bottlenecks
-
8/8/2019 05 WAITS Intro
5/41
Copyright 2006 Kyle Hailey
Database is Hung!
Everybody blames the database
Yet 9 out of 10 dbas agree its not the
databaseHow do you prove it to management?
On the off chance its the database, what dowe do?
-
8/8/2019 05 WAITS Intro
6/41
Copyright 2006 Kyle Hailey
Database:
Guilty until proven innocent
*$%@!!*$%@!!
-
8/8/2019 05 WAITS Intro
7/41
Copyright 2006 Kyle Hailey
Oracle Instrumentation
RedoRedo LibLibCacheCache
BufferBufferCacheCache
IOIO
LocksLocks
NetworkNetwork
CPUCPU
-
8/8/2019 05 WAITS Intro
8/41
Copyright 2006 Kyle Hailey
Waits
I/O
Library Cache
Locks
Redo
Buffer Cache
SQL*Net
Wait Areas
Well discuss Waits in these logical database areasWell discuss Waits in these logical database areas
-
8/8/2019 05 WAITS Intro
9/41
Copyright 2006 Kyle HaileyCopyright 2006 Kyle Hailey
Wait Tree
WaitsWaits
IOIO
Buffer CacheBuffer Cache
Library CacheLibrary Cache
LockLock
RedoRedo
SQL NetSQL Net
Buffer BusyBuffer Busy
RollbackRollback
Free listsFree lists
IO ReadIO ReadCache LatchesCache Latches
Library CacheLibrary Cache
Shared PoolShared Pool
TX Row LockTX Row Lock
TX ITL LockTX ITL Lock
HW LockHW Lock
Write IOWrite IO
Read IORead IO
Log BufferLog Buffer
Log File SyncLog File Sync
Log FileLog File
-
8/8/2019 05 WAITS Intro
10/41
Copyright 2006 Kyle Hailey
Waits
Introduced in v7Revolutionized tuning
Changed from Ratio Guesswork to empiricalmeasure of time lost to bottlenecks
10g added the crucial addition ASHNot only identifies bottlenecks but
Who (session, service, package, procedure)
Where (CPU, Wait)
When (time)
What (SQL statement)
-
8/8/2019 05 WAITS Intro
11/41
Copyright 2006 Kyle Hailey
Tuning Methodology
1. Machine Run queue (CPU) Check other applications
reduce CPU usage or add CPUs
Paging
Reduce memory usage or add memory
2. Oracle Waits + CPU > Available CPU
Tune waits
CPU 100% Tune SQL
Else low waits, available CPU then Its the application
We are going toWe are going toconcentrate hereconcentrate hereon WAITSon WAITS
-
8/8/2019 05 WAITS Intro
12/41
Copyright 2006 Kyle Hailey
Dependable Tuning Strategy
Determine AAS :1. Run Statspack or AWR Report
Top 5 Timed Events ~50 lines down from top
Need AvailableC
PU Elapsed Time
CPU_COUNT
2. ASH Report : ashrpt.sql
3. OEM 10gPerformance Page does everything
If there is a wait bottleneck tune the wait
-
8/8/2019 05 WAITS Intro
13/41
Copyright 2006 Kyle Hailey
Tuning Methodology Graphics
Relax, its theRelax, its theapplicationapplication
Get to Work!Get to Work!
-
8/8/2019 05 WAITS Intro
14/41
Copyright 2006 Kyle Hailey
Waits beyond OEM
OEM identifies Wait problems
Provides solutions with ADDM sometimes
But What do you do when ADDM isnt sufficient?
What do you do if you dont have OEM 10g?Waits
Need to know about waits
How they work
How to analyze them
-
8/8/2019 05 WAITS Intro
15/41
Copyright 2006 Kyle Hailey
v$active_session_history
When ADDM fails or we dont have ADDM we cancollect the necessary information from
v$active_session_history Session (user, service, client, package, procedure, etc)
SQL statement For IO waits, buffer busy waits and enqueue TX waits
CURRENT_OBJ# ,CURRENT_FILE# ,CURRENT_BLOCK#
Blocking_Session
P1
P2 P3
-
8/8/2019 05 WAITS Intro
16/41
Copyright 2006 Kyle Hailey
What are P1,P2,P3 ?
Each Wait has a 3 parameters P1,P2,P3Give detailed information
Meaning different for each wait
Meaning definitions in V$event_name
col parameter1 for a10col parameter1 for a10
col parameter2 for a10col parameter2 for a10col parameter3 for a10col parameter3 for a10
select parameter1 ,parameter2 , parameter3select parameter1 ,parameter2 , parameter3from v$event_namefrom v$event_namewhere name = '&1';where name = '&1';
-
8/8/2019 05 WAITS Intro
17/41
Copyright 2006 Kyle Hailey
Wait Arguments Example
NAME PARAMETER1 PARAMETER2 PARAMETER
------------------------------ ----------- --------------- ---------------
latch: cache buffers chains address number tries
free buffer waits file# block# set-id# buffer busy waits file# block# class#
latch: redo copy address number tries
log buffer space
switch logfile command
log file sync buffer#
db file sequential read file# block# blocks
enq: TM - contention name|mode object # table/partition
undo segment extension segment#
enq: TX - row lock contention name|mode usn
-
8/8/2019 05 WAITS Intro
18/41
Copyright 2006 Kyle Hailey
Wait Analysis requires p1,p2,p3
Of the top 30 wait events 8 can be solvedwithout ASH
The rest needSQL
P1,P2,P3
Statspack , AWR fail
free buffer waitslog buffer spacelog file switch (archiving needed)
log file switch (checkpoint incomplete)log file switch completionlog file syncswitch logfile commandwrite complete waits
Example hard waitsExample hard waitsBuffer busy waitBuffer busy waitRow cache lockRow cache lockLatch freeLatch freerow lock contentionrow lock contentionLatch: cache buffers chainsLatch: cache buffers chains
-
8/8/2019 05 WAITS Intro
19/41
Wait Analysis
SQL Most often the tuning answer lies in looking at what the application is
doing, and changing it
Parameters Find extended wait information
Parameter1, Parameter2, Parameter3
Defined in v$event_name
Guess Work Sometimes the wait events that are found are not in the
documentation and it takes some educated guesswork to figure outthe problem
-
8/8/2019 05 WAITS Intro
20/41
Copyright 2006 Kyle Hailey
Difficult Waits
These 4 waits have multiple causes
Latches p2 = latch # (p1= address, p3= tries)
Locks p1 = lock type and mode ( p2 = id1, p3= id2)
Buffer Busy p3 = block class#, p1= file, p2=block
(in 9i p3 was the bbw type)
Row Cache Lock p1 = cache id (p2 = mode, p3=request)
-
8/8/2019 05 WAITS Intro
21/41
Copyright 2006 Kyle Hailey
Example ASH Query
Select ash.p1,Select ash.p1,
ash.p2,ash.p2,CURRENT_OBJ#||' '||CURRENT_OBJ#||' '||o.object_nameo.object_name objnobjn,,o.object_typeo.object_type otypeotype,,CURRENT_FILE#CURRENT_FILE# filenfilen,,CURRENT_BLOCK#CURRENT_BLOCK# blocknblockn,,ash.SQL_IDash.SQL_ID,,w.classw.class ||' '||||' '||to_charto_char(ash.p3)(ash.p3) block_typeblock_type
fromfrom v$active_sv$active_session_historyession_history ash,ash,( select( select rownumrownum class#, class fromclass#, class from v$waitstatv$waitstat ) w,) w,all_objectsall_objects oo
wherewhere event='buffer busy waits'event='buffer busy waits'andand w.classw.class#(+)=ash.p3#(+)=ash.p3andand o.object_ido.object_id (+)=(+)= ash.CURRENT_OBJash.CURRENT_OBJ##andand ash.session_stateash.session_state='WAITING'='WAITING'andand ash.sample_timeash.sample_time >> sysdatesysdate -- &1/(60*24)&1/(60*24)
Order byOrder by sample_timesample_timeP1 P2 OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPEP1 P2 OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPE---- ------------ ------------------------------------------ ---------- ---------- ------------ -------------------------- --------------
1 112796 66053 BBW_INDEX_VAL_I INDEX 1 112796 6avm49ys4k7t6 data block 11 112796 66053 BBW_INDEX_VAL_I INDEX 1 112796 6avm49ys4k7t6 data block 1
1 112401 66053 BBW_INDEX_VAL_I INDEX 1 112401 5wqps1quuxqr4 data block 11 112401 66053 BBW_INDEX_VAL_I INDEX 1 112401 5wqps1quuxqr4 data block 1
1 112796 66053 BBW_INDEX_VAL_I INDEX 1 112796 5wqps1quuxqr4 data block 11 112796 66053 BBW_INDEX_VAL_I INDEX 1 112796 5wqps1quuxqr4 data block 1
1 113523 66053 BBW_INDEX_VAL_I INDEX 1 113523 5wqps1quuxqr4 data block 11 113523 66053 BBW_INDEX_VAL_I INDEX 1 113523 5wqps1quuxqr4 data block 1
-
8/8/2019 05 WAITS Intro
22/41
Copyright 2006 Kyle Hailey
Waits we will Ignore
One thing that makes waits difficult is knowing whichones to look at and which ones to ignore.
Background
Idle
Resource Manager
Parallel Query
RAC
Good stuff, but not covered in this seminar
-
8/8/2019 05 WAITS Intro
23/41
Copyright 2006 Kyle Hailey
Background & Foreground
Background ProcessesDBWR
LGWR
PMON
SMON
Etc
Foreground ProcessesSQL*Plus
Pro*C
SQL*FormsOracle applications
Only interested in Foreground waits
-
8/8/2019 05 WAITS Intro
24/41
Copyright 2006 Kyle Hailey
Background Waits
ASH
Avoid Background waits in ASH with
V$session_wait joined to v$session
Select from v$active_session_historywhere SESSION_TYPE='FOREGROUND'
select
from v$session s,
v$session_wait wwhere w.sid=s.sid
and s.type='USER'
-
8/8/2019 05 WAITS Intro
25/41
Copyright 2006 Kyle Hailey
Idle Waits
Filtered Out of ASH by default 10g
where wait_class != Idle
Create a list
9i
Create a list with
Documentation List created from 10g
Stats$idle_events from statspack
Select name from v$event_name wherewait_class=Idle;
SQL*Net message from clientSQL*Net message from client
-
8/8/2019 05 WAITS Intro
26/41
Copyright 2006 Kyle Hailey
PQO and Resource Manager
Resource manager throttles user
Creates wait
Obfuscates problems
Parallel Query Wait events are unusable
Save waits are both idle and waits
Parallel Query Waits start with PX or KX PX Deq: Par Recov Reply
PX Deq: Parse Reply
select name from v$event_name wherewait_class='Scheduler';
-
8/8/2019 05 WAITS Intro
27/41
Copyright 2006 Kyle Hailey
RAC Waits
RAC
waits are certainly interesting but will be coveredoutside of this presentation.
You are on your own Check documentation
If you are not using RAC then no worries 10g
9i RAC and OPS waits usually contain the word global
Select event from v$event_name wherewait_class=Cluster;
-
8/8/2019 05 WAITS Intro
28/41
Copyright 2006 Kyle Hailey
Latches
Protect areas of memory from concurrent use Light weight locks
Bit in memory
Atomic processor call
Fast and cheapGone if memory is lost
Often used in cache coherency managementChanges to a data block
Exclusive GenerallySharing reading has been introduced for some latches
-
8/8/2019 05 WAITS Intro
29/41
Copyright 2006 Kyle Hailey
Finding Latches
latch free Covers many latches, find the problem latch by
1. select name from v$latchname where latch# = p1;OR
2. Find highest sleeps in Statspack latch section In 10g, important latches have a wait event
latch: cache buffers chains
latch: shared pool
latch: library cache Latches avoid:Latches avoid: deallocatingdeallocating while someone is accessingwhile someone is accessing reading while someone is modifyingreading while someone is modifying modifingmodifing while someone is modifyingwhile someone is modifying modifingmodifing while someone is readingwhile someone is reading
-
8/8/2019 05 WAITS Intro
30/41
Latches and Spin Count
Latch contention is a symptom of a problem
Spin Count
Band aid
Doesnt solve the problem
If there is free CPU Can increase spin count
Underscore parameter in 10g (not recommended to change)
Fix the problem
Instead of using spin countFind the problem and fix it
Copyright 2006 Kyle Hailey
-
8/8/2019 05 WAITS Intro
31/41
Copyright 2006 Kyle Hailey
Enqueues aka Locks
Enqueue wait covers all locks pre 10
Protect data against concurrent changes
Lock info written into data structures
Block headers
Data blocksWritten in cache structures
Shareable in compatible modes
-
8/8/2019 05 WAITS Intro
32/41
Copyright 2006 Kyle Hailey
Locks 10g
10g breaks all Enqueues out enq: HW - contention Configuration
enq: TM - contention Application
enq: TX - allocate ITL entry Configuration
enq: TX - index contention Concurrency
enq: TX - row lock contention Application enq: UL - contention Application
-
8/8/2019 05 WAITS Intro
33/41
Copyright 2006 Kyle Hailey
Row Cache Lock
Need p1 to see the cache type
SQL> select cache#, parameter from v$rowcache;
CACHE# PARAMETER---------- --------------------------------
1 dc_free_extents4 dc_used_extents2 dc_segments0 dc_tablespaces5 dc_tablespace_quotas6 dc_files7 dc_users3 dc_rollback_segments8 dc_objects17 dc_global_oids12 dc_constraints
-
8/8/2019 05 WAITS Intro
34/41
Copyright 2006 Kyle Hailey
Additional Support
AWR Tables on disk for 7 days by default
DBA_HIST_ACTIVE_SESS_HISTORY
1 in 10 ASH samples
DBA_HIST_SEG_STAT
Good for ITL and buffer busy wait
DBA_HIST_SYSTEM_EVENT
Important for getting avg wait times
DBA_HIST_SQLSTAT sql execution deltas
DBA_HIST_SYSMETRIC_SUMMARY Statistics avg, max, min
Metric Tables in memory deltas
V$EVENTMETRIC
-
8/8/2019 05 WAITS Intro
35/41
Copyright 2006 Kyle Hailey
All Events over 7 days
select count(*), event fromselect count(*), event from
( select event from DBA_HIST_ACTIVE_SESS_HISTORY( select event from DBA_HIST_ACTIVE_SESS_HISTORY
wherewhere sample_timesample_time < ( select min(< ( select min(sample_timesample_time) from) from
v$active_session_historyv$active_session_history))union allunion all
select event fromselect event from v$active_session_historyv$active_session_history
))
group by eventgroup by eventorder by eventorder by event
//
-
8/8/2019 05 WAITS Intro
36/41
Copyright 2006 Kyle Hailey
Average Wait Times Historic
select
btime,
(time_ms_end-time_ms_beg)/nullif(count_end-count_beg,0) avg_ms
from (
select
to_char(s.BEGIN_INTERVAL_TIME,'DD-MON-YY HH24:MI') btime,
total_waits count_end,
time_waited_micro/1000 time_ms_end,
LagLag ((e.time_waited_micro/1000)OVER( PARTITION BY e.event_name ORDER BY s.snap_id) time_ms_beg,
LagLag ((e.total_waits)
OVER( PARTITION BY e.event_name ORDER BY s.snap_id) count_beg
from
DBA_HIST_SYSTEM_EVENT e,
DBA_HIST_SNAPSHOT s
where
s.snap_id=e.snap_idand e.event_name= '&1'
order by begin_interval_time
)
order by btime;
BTIME AVG_MSBTIME AVG_MS
---------------------------------------- ------------------------0808--JANJAN--08 01:00 1.01708 01:00 1.017
0808--JANJAN--08 02:00 .72008 02:00 .720
0808--JANJAN--08 03:00 .62108 03:00 .621
0808--JANJAN--08 04:00 1.74708 04:00 1.747
0808--JANJAN--08 05:00 1.04608 05:00 1.046
0808--JANJAN--08 06:00 1.44408 06:00 1.444
-
8/8/2019 05 WAITS Intro
37/41
Copyright 2006 Kyle Hailey
Avg Wait times now
selectselect
en.name,en.name,
(time_waited)/nullif(wait_count,0) avg_ms,(time_waited)/nullif(wait_count,0) avg_ms,
wait_countwait_count
fromfrom
v$eventmetric e,v$eventmetric e,
v$event_name env$event_name en
wherewhere
e.event# = en.event#e.event# = en.event#
and en.name like '%&1%;and en.name like '%&1%;
NAME AVG_MS WAIT_COUNTNAME AVG_MS WAIT_COUNT-------------------------------------------------- -------------------- --------------------db file sequential read .658863707 6420db file sequential read .658863707 6420
db file scattered read .549427419 186db file scattered read .549427419 186
db file parallel write .089073438 64db file parallel write .089073438 64
-
8/8/2019 05 WAITS Intro
38/41
Object Translation
Current fields in v$active_session_history CURRENT_OBJ#
CURRENT_FILE#
CURRENT_BLOCK#
Called ROW_WAIT_% in v$session
Only apply to Buffer Busy Waits
IO Waits
Enqueue TX
Ignore these fields for other wait events
-
8/8/2019 05 WAITS Intro
39/41
Copyright 2006 Kyle Hailey
Wait interface Weaknesses
Logons EM 10g shows these on perf page Time model helps
V$SYS_TIME_MODEL
connection management call elapsed time
Ive had problems
Paging/Memory issues CPU starvation
Null Events
Bugs read external table reports CPU http://blog.tanelpoder.com/
-
8/8/2019 05 WAITS Intro
40/41
Copyright 2006 Kyle Hailey
Dependable Tuning Strategy
Run Statspack/AWR reportTop 5 Timed Events
~50 lines down from top
Need Available CPU Elapsed Time
CPU_
COUNT
OEM 10gPerformance Page does everything !
OEM doesnt solve the problem
Query v$active_session_history directly
-
8/8/2019 05 WAITS Intro
41/41
Copyright 2006 Kyle Hailey
Summary
Waits make Tuning Easy Check Machine Health
Tune Waits
Tune CPU
Tune SQL
Change Application Architecture
Use OEM10g
Statspack/AWR,
S/ASH Ignore Background, Idle, Resmgr, PQO
Use ASH if OEM fails
See http://perfvision.com for more info