Tracing Parallel Execution (UKOUG 2006)
-
Upload
doug-burns -
Category
Technology
-
view
3.334 -
download
2
description
Transcript of Tracing Parallel Execution (UKOUG 2006)
Tracing PX SlavesTracing PX Slaves
Doug BurnsDoug Burns
[email protected]@yahoo.com
http://oracledoug.comhttp://oracledoug.com
AgendaAgenda
IntroductionIntroduction Test EnvironmentTest Environment Tracing Parallel Tracing Parallel
ExecutionExecution Trace OutputTrace Output ConsolidationConsolidation Timing IssuesTiming Issues ConclusionConclusion
IntroductionIntroduction
Who (or what) am I ?Who (or what) am I ? ScottishScottish Predominantly a DBAPredominantly a DBA Training and ConsultancyTraining and Consultancy
Current AssignmentCurrent Assignment BSkyBBSkyB Very Cool Projects and Very Cool Projects and
HardwareHardware Less Cool Release Less Cool Release
ManagementManagement http://oracledoug.comhttp://oracledoug.com
BlogBlog
IntroductionIntroduction
Previous Related PapersPrevious Related Papers Suck It Dry - Tuning Parallel ExecutionSuck It Dry - Tuning Parallel Execution How Many Slaves? – PX and the Magic of 2How Many Slaves? – PX and the Magic of 2 Available at Available at http://oracledoug.comhttp://oracledoug.com
ObjectivesObjectives Answer tracing questions that cropped upAnswer tracing questions that cropped up Show the detailShow the detail
Why Parallel Execution?Why Parallel Execution? Increasing Volumes of DataIncreasing Volumes of Data Increasing User ExpectationsIncreasing User Expectations More Powerful HardwareMore Powerful Hardware
Parallel Execution (PX) splits a Parallel Execution (PX) splits a single large task into multiple single large task into multiple smaller tasks which are handled smaller tasks which are handled by separate processes running by separate processes running concurrently.concurrently. Full Table ScansFull Table Scans SortsSorts Index Creation, Direct Path inserts etc …Index Creation, Direct Path inserts etc …
IntroductionIntroduction
Test EnvironmentTest Environment
ExampleExample Small TableSmall Table
Easier to wade through trace filesEasier to wade through trace files Quick executionQuick execution Not representative of real applicationsNot representative of real applications Force PX using HintsForce PX using Hints
TEST_TABTEST_TAB 2,048,000 rows2,048,000 rows Average row length – 109 bytesAverage row length – 109 bytes 8Kb Blocks/ 32M Extents / 256 Mb Segment8Kb Blocks/ 32M Extents / 256 Mb Segment No IndexesNo Indexes
Test EnvironmentTest Environment
Oracle 9i Release 2Oracle 9i Release 2 The most common database at workThe most common database at work Tracing similar to earlier versionsTracing similar to earlier versions
Oracle 10g Release 2Oracle 10g Release 2 Principles are the samePrinciples are the same Improved tracing functionalityImproved tracing functionality
Single CPU / Single HDDSingle CPU / Single HDD TryingTrying to introduce wait events to introduce wait events
Test EnvironmentTest Environment
The QueryThe Query
SELECT /*+ parallel(tt1, 2) */ SELECT /*+ parallel(tt1, 2) */
MOD(tt1.pk_id, 2), COUNT(*)MOD(tt1.pk_id, 2), COUNT(*)
FROM test_tab tt1FROM test_tab tt1
GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)
ORDER BY MOD(tt1.pk_id,2);ORDER BY MOD(tt1.pk_id,2);
Degree of Parallelism (DOP) = 2Degree of Parallelism (DOP) = 2 Two Sets of PX Slaves RequestedTwo Sets of PX Slaves Requested
One set for Full Table ScanOne set for Full Table Scan One set for Grouping OperationOne set for Grouping Operation
Tracing Parallel Tracing Parallel ExecutionExecution
Enable TracingEnable Tracing 10g10galter session set tracefile_identifier=‘doug_test';alter session set tracefile_identifier=‘doug_test';exec dbms_session.set_identifier(‘doug_test');exec dbms_session.set_identifier(‘doug_test');exec dbms_monitor.client_id_trace_enable(exec dbms_monitor.client_id_trace_enable(
client_id => ‘doug_test');client_id => ‘doug_test'); 9i9ialter session set tracefile_identifier=‘doug_test';alter session set tracefile_identifier=‘doug_test';alter session set events '10046 trace name context alter session set events '10046 trace name context
forever, level 8'; forever, level 8';
tracefile_identifier not as useful for PXtracefile_identifier not as useful for PX Only applies to the Query Co-ordinator (QC) processOnly applies to the Query Co-ordinator (QC) process Makes sense as the PX slave doesn’t ‘belong’ to the Makes sense as the PX slave doesn’t ‘belong’ to the
sessionsession
Tracing Parallel Tracing Parallel ExecutionExecution
Run QueryRun Query Disable TracingDisable Tracing
10g10gexec dbms_monitor.client_id_trace_disable(exec dbms_monitor.client_id_trace_disable(
client_id => ‘doug_test');client_id => ‘doug_test'); 9i9ialter session set events '10046 trace name alter session set events '10046 trace name context off'; context off';
Several Trace files producedSeveral Trace files produced Remember – this is for the Remember – this is for the lowestlowest DOP of 2 DOP of 2
Tracing Parallel Tracing Parallel ExecutionExecution
USER PROCESS
SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)FROM test_tab tt1GROUP BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2);
Slave 0
Sorting/Grouping 0
Slave 1
Sorting/Grouping 1
Slave 2
Reading first half of blocks
Slave 3
Reading second half of blocks
TEST_TAB
Query Coordinator
· Allocate sort ranges· Allocate FTS block
ranges· Collate data
Tracing Parallel Tracing Parallel ExecutionExecution
user_dump_destuser_dump_desttest1020_ora_17239_test1020_ora_17239_doug_testdoug_test.trc.trc
background_dump_destbackground_dump_desttest1020_test1020_p000p000_17219.trc_17219.trc
test1020_test1020_p001p001_17221.trc_17221.trc
test1020_test1020_p002p002_17223.trc_17223.trc
test1020_test1020_p003p003_17225.trc_17225.trc
No mention of tracefile_identifierNo mention of tracefile_identifier Slave numbers could be differentSlave numbers could be different
Tracing Parallel Tracing Parallel ExecutionExecution
IssuesIssues Difficult to get a consolidated view of the user Difficult to get a consolidated view of the user
actionaction
PX Slaves are shared background processesPX Slaves are shared background processes How to ensure that you’re looking at the correct slaves?How to ensure that you’re looking at the correct slaves? How to ensure that you don’t include unrelated activity?How to ensure that you don’t include unrelated activity?
9i - With a little difficulty and rm!9i - With a little difficulty and rm! 10g – dbms_monitor and trcsess10g – dbms_monitor and trcsess
Trace Output – 9iTrace Output – 9i Query Co-ordinator (QC)Query Co-ordinator (QC)
Unix process pid: 5790, image: Unix process pid: 5790, image: [email protected] (TNS V1-V3)[email protected] (TNS V1-V3)
*** SESSION ID:(8.283) 2006-09-12 10:54:33.245*** SESSION ID:(8.283) 2006-09-12 10:54:33.245APPNAME mod='SQL*Plus' mh=3669949024 act='' APPNAME mod='SQL*Plus' mh=3669949024 act=''
ah=4029777240ah=4029777240
Trace Output – 9iTrace Output – 9i Followed by tons of recursive callsFollowed by tons of recursive calls
Enabling Tracing, CBO work, etc.Enabling Tracing, CBO work, etc.
PARSING IN CURSOR #1 len=130 dep=0 uid=23 oct=3 lid=23 PARSING IN CURSOR #1 len=130 dep=0 uid=23 oct=3 lid=23 tim=1130912963021298 hv=3017045374 ad='959ffae8'tim=1130912963021298 hv=3017045374 ad='959ffae8'
SELECT /*+ parallel(tt1, 2) */SELECT /*+ parallel(tt1, 2) */MOD(tt1.pk_id, 2), COUNT(*)MOD(tt1.pk_id, 2), COUNT(*)
FROM test_tab1 tt1FROM test_tab1 tt1GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)END OF STMTEND OF STMT
PARSE PARSE #1:c=10998,e=12643,p=0,cr=7,cu=3,mis=1,r=0,dep=0,og=4#1:c=10998,e=12643,p=0,cr=7,cu=3,mis=1,r=0,dep=0,og=4,tim=1130912963021287,tim=1130912963021287
WAIT #1: nam='enqueue' ela= 2998428 p1=1413677062 WAIT #1: nam='enqueue' ela= 2998428 p1=1413677062 p2=65544 p3=0p2=65544 p3=0
Trace Output – 9i (QC Trace Output – 9i (QC Contd)Contd)
Now to set up the slave setsNow to set up the slave setsWAIT #1: nam='rdbms ipc reply' ela= 122 p1=5 p2=21474836 WAIT #1: nam='rdbms ipc reply' ela= 122 p1=5 p2=21474836
p3=0p3=0WAIT #1: nam=WAIT #1: nam='process startup'process startup' ela= 13 p1=80 p2=1 p3=0' ela= 13 p1=80 p2=1 p3=0WAIT #1: nam='process startup' ela= 34572 p1=80 p2=1 p3=1WAIT #1: nam='process startup' ela= 34572 p1=80 p2=1 p3=1WAIT #1: nam='WAIT #1: nam='PX Deq: Join ACKPX Deq: Join ACK' ela= 1983 p1=268500992 ' ela= 1983 p1=268500992
p2=1 p3=0p2=1 p3=0WAIT #1: nam='PX Deq: Join ACK' ela= 1479 p1=268500993 WAIT #1: nam='PX Deq: Join ACK' ela= 1479 p1=268500993
p2=1 p3=0p2=1 p3=0WAIT #1: nam='process startup' ela= 19432 p1=80 p2=2 p3=0WAIT #1: nam='process startup' ela= 19432 p1=80 p2=2 p3=0WAIT #1: nam='process startup' ela= 26803 p1=80 p2=3 p3=0WAIT #1: nam='process startup' ela= 26803 p1=80 p2=3 p3=0WAIT #1: nam='PX Deq: Join ACK' ela= 2930 p1=268500994 WAIT #1: nam='PX Deq: Join ACK' ela= 2930 p1=268500994
p2=1 p3=0p2=1 p3=0WAIT #1: nam='PX Deq: Join ACK' ela= 1557 p1=268500995 WAIT #1: nam='PX Deq: Join ACK' ela= 1557 p1=268500995
p2=1 p3=0p2=1 p3=0
Trace Output – 9i (QC Trace Output – 9i (QC Contd)Contd)
Set up queues and wait for slaves to parse Set up queues and wait for slaves to parse statementsstatements
WAIT #1: nam='WAIT #1: nam='PX qref latchPX qref latch' ela= 4 p1=1 p2=1 p3=-' ela= 4 p1=1 p2=1 p3=-18215416161821541616
WAIT #1: nam='PX qref latch' ela= 10385 p1=1 p2=2 p3=-WAIT #1: nam='PX qref latch' ela= 10385 p1=1 p2=2 p3=-18215416161821541616
WAIT #1: nam='PX qref latch' ela= 1182 p1=1 p2=3 p3=-WAIT #1: nam='PX qref latch' ela= 1182 p1=1 p2=3 p3=-18215416161821541616
WAIT #1: nam='PX qref latch' ela= 29903 p1=1 p2=4 p3=-WAIT #1: nam='PX qref latch' ela= 29903 p1=1 p2=4 p3=-18215416161821541616
WAIT #1: nam='WAIT #1: nam='PX Deq: Parse ReplyPX Deq: Parse Reply' ela= 3277 p1=200 ' ela= 3277 p1=200 p2=1 p3=0p2=1 p3=0
WAIT #1: nam='PX Deq: Parse Reply' ela= 1992474 p1=200 WAIT #1: nam='PX Deq: Parse Reply' ela= 1992474 p1=200 p2=1 p3=0p2=1 p3=0
WAIT #1: nam='PX Deq: Parse Reply' ela= 960495 p1=200 WAIT #1: nam='PX Deq: Parse Reply' ela= 960495 p1=200 p2=2 p3=0p2=2 p3=0
Trace Output – 9i (QC Trace Output – 9i (QC Contd)Contd)
Start waiting for responses and take out Start waiting for responses and take out PX qref latch when communicatingPX qref latch when communicating
WAIT #1: nam='WAIT #1: nam='PX Deq: Execute ReplyPX Deq: Execute Reply' ela= 210184 p1=200 ' ela= 210184 p1=200 p2=1 p3=0p2=1 p3=0
WAIT #1: nam='PX Deq: Execute Reply' ela= 42955 p1=200 WAIT #1: nam='PX Deq: Execute Reply' ela= 42955 p1=200 p2=1 p3=0p2=1 p3=0
WAIT #1: nam='PX Deq: Execute Reply' ela= 1636370 WAIT #1: nam='PX Deq: Execute Reply' ela= 1636370 p1=200 p2=1 p3=0p1=200 p2=1 p3=0
WAIT #1: nam='WAIT #1: nam='PX qref latchPX qref latch' ela= 11128 p1=1 p2=1 p3=-' ela= 11128 p1=1 p2=1 p3=-18215430641821543064
WAIT #1: nam='PX Deq: Execute Reply' ela= 501343 p1=200 WAIT #1: nam='PX Deq: Execute Reply' ela= 501343 p1=200 p2=1 p3=0p2=1 p3=0
WAIT #1: nam='PX qref latchWAIT #1: nam='PX qref latch'' ela= 11011 p1=1 p2=1 p3=- ela= 11011 p1=1 p2=1 p3=-18215416161821541616
WAIT #1: nam='PX Deq: Execute Reply' ela= 1424051 WAIT #1: nam='PX Deq: Execute Reply' ela= 1424051 p1=200 p2=1 p3=0p1=200 p2=1 p3=0
Trace Output – 9i (QC Trace Output – 9i (QC Contd)Contd)
Some wrap-up towards the endSome wrap-up towards the end
WAIT #1: nam='PX Deq: Signal ACK' ela= 2 p1=0 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 2 p1=0 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=10 p2=2 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=10 p2=2 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 98523 p1=10 p2=3 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 98523 p1=10 p2=3 p3=0FETCH FETCH
#1:c=1000,e=98953,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=11#1:c=1000,e=98953,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=113091303363795330913033637953
WAIT #1: nam='PX Deq: Signal ACK' ela= 2351 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 2351 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 200 p1=200 p2=2 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 200 p1=200 p2=2 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 1031 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 1031 p1=200 p2=1 p3=0
Trace Output – 9iTrace Output – 9i Slaves 0 and 1 – Sort/GroupSlaves 0 and 1 – Sort/Group
Instance name: TEST92Instance name: TEST92Redo thread mounted by this instance: 1Redo thread mounted by this instance: 1Oracle process number: 9Oracle process number: 9Unix process pid: 5794, image: [email protected] Unix process pid: 5794, image: [email protected]
(P000)(P000)
*** SESSION ID:(13.34) 2006-09-12 10:55:19.741*** SESSION ID:(13.34) 2006-09-12 10:55:19.741<<snipped><<snipped>PARSING IN CURSOR #1 len=98 dep=0 uid=23 oct=3 lid=23 PARSING IN CURSOR #1 len=98 dep=0 uid=23 oct=3 lid=23
tim=1130913007567950 hv=4220356981 ad='959d6c34'tim=1130913007567950 hv=4220356981 ad='959d6c34'SELECT /*+ CIV_GB */ A1.C0,COUNT(SYS_OP_CSR(A1.C1,0)) SELECT /*+ CIV_GB */ A1.C0,COUNT(SYS_OP_CSR(A1.C1,0))
FROM :Q3000 A1 GROUP BY A1.C0 ORDER BY A1.C0FROM :Q3000 A1 GROUP BY A1.C0 ORDER BY A1.C0END OF STMTEND OF STMTPARSE PARSE
#1:c=11998,e=14410,p=0,cr=2,cu=0,mis=1,r=0,dep=0,og=4,tim=113#1:c=11998,e=14410,p=0,cr=2,cu=0,mis=1,r=0,dep=0,og=4,tim=11309130075679390913007567939
Trace Output – 9i (Sort Trace Output – 9i (Sort Contd)Contd)
Cycle round waiting for data to come Cycle round waiting for data to come back from other slave setback from other slave set
WAIT #1: nam='PX Deq: Execution Msg' ela= 3 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 3 p1=268566527 p2=1 p3=0p2=1 p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 1992813 WAIT #1: nam='PX Deq: Execution Msg' ela= 1992813 p1=268566527 p2=2 p3=0p1=268566527 p2=2 p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 1992862 WAIT #1: nam='PX Deq: Execution Msg' ela= 1992862 p1=268566527 p2=3 p3=0p1=268566527 p2=3 p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 1993857 WAIT #1: nam='PX Deq: Execution Msg' ela= 1993857 p1=268566527 p2=4 p3=0p1=268566527 p2=4 p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 1993834 WAIT #1: nam='PX Deq: Execution Msg' ela= 1993834 p1=268566527 p2=5 p3=0p1=268566527 p2=5 p3=0
Trace Output – 9i (Sort Trace Output – 9i (Sort Contd)Contd)
EXEC EXEC #1:c=0,e=248,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=11309130334#1:c=0,e=248,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=113091303343553635536WAIT #1: nam='PX Deq: Table Q Sample' ela= 98589 p1=10 p2=1 WAIT #1: nam='PX Deq: Table Q Sample' ela= 98589 p1=10 p2=1
p3=0p3=0WAIT #1: nam='PX Deq: Table Q Sample' ela= 2042 p1=200 p2=2 WAIT #1: nam='PX Deq: Table Q Sample' ela= 2042 p1=200 p2=2
p3=0p3=0FETCH FETCH
#1:c=1000,e=101033,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1#1:c=1000,e=101033,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913033536644130913033536644
WAIT #1: nam=WAIT #1: nam='PX Deq Credit: send blkd'PX Deq Credit: send blkd' ela= 511 p1=268566527 ' ela= 511 p1=268566527 p2=1 p3=0p2=1 p3=0
FETCH FETCH #1:c=0,e=33,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=11309130#1:c=0,e=33,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=113091303353770033537700
WAIT #1: nam='PX Deq: Execution Msg' ela= 1543 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 1543 p1=268566527 p2=1 p3=0p2=1 p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 98633 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 98633 p1=268566527 p2=1 p3=0p2=1 p3=0
Trace Output – 9iTrace Output – 9i
Slaves 2 and 3 – FTSSlaves 2 and 3 – FTS
PARSING IN CURSOR #1 len=208 dep=0 uid=23 oct=3 lid=23 PARSING IN CURSOR #1 len=208 dep=0 uid=23 oct=3 lid=23 tim=1130913007579832 hv=721990200 ad='959d0330'tim=1130913007579832 hv=721990200 ad='959d0330'
SELECT /*+ PIV_GB */ MOD(A1.C0,2) SELECT /*+ PIV_GB */ MOD(A1.C0,2) C0,SYS_OP_MSR(COUNT(*)) C1 FROM (SELECT /*+ NO_EXPAND C0,SYS_OP_MSR(COUNT(*)) C1 FROM (SELECT /*+ NO_EXPAND ROWID(A2) */ A2."PK_ID" C0 FROM ROWID(A2) */ A2."PK_ID" C0 FROM "TESTUSER"."TEST_TAB1" PX_GRANULE(0, BLOCK_RANGE, "TESTUSER"."TEST_TAB1" PX_GRANULE(0, BLOCK_RANGE, DYNAMIC) A2) A1 GROUP BY MOD(A1.C0,2)DYNAMIC) A2) A1 GROUP BY MOD(A1.C0,2)
END OF STMTEND OF STMT
PARSE PARSE #1:c=9999,e=9226,p=0,cr=1,cu=0,mis=1,r=0,dep=0,og=4,t#1:c=9999,e=9226,p=0,cr=1,cu=0,mis=1,r=0,dep=0,og=4,tim=1130913007579821im=1130913007579821
Trace Output – 9i FTS Trace Output – 9i FTS (Contd)(Contd)
Go out and read data (with Direct Path Go out and read data (with Direct Path Reads)Reads)
WAIT #1: nam='PX Deq: Execution Msg' ela= 1992624 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 1992624 p1=268566527 p2=1 p3=0p2=1 p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 994085 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 994085 p1=268566527 p2=2 p3=0p2=2 p3=0
EXEC EXEC #1:c=0,e=180,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913010#1:c=0,e=180,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913010567272567272
WAIT #1: nam=WAIT #1: nam='direct path read'direct path read' ela= 50 p1=5 p2=10 p3=128' ela= 50 p1=5 p2=10 p3=128WAIT #1: nam='direct path read' ela= 14 p1=5 p2=138 p3=128WAIT #1: nam='direct path read' ela= 14 p1=5 p2=138 p3=128WAIT #1: nam='direct path read' ela= 17 p1=5 p2=266 p3=128WAIT #1: nam='direct path read' ela= 17 p1=5 p2=266 p3=128
<snipped><snipped>
WAIT #1: nam='direct path read' ela= 32 p1=5 p2=522 p3=128WAIT #1: nam='direct path read' ela= 32 p1=5 p2=522 p3=128WAIT #1: nam='PX Deq: Execution Msg' ela= 11253 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 11253 p1=268566527
p2=1 p3=0p2=1 p3=0
Trace Output – 9i FTS Trace Output – 9i FTS (Contd)(Contd)
Wrap-upWrap-up
WAIT #1: nam='PX Deq: Execution Msg' ela= 10387 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 10387 p1=268566527 p2=1 p3=0p2=1 p3=0
FETCH FETCH #1:c=11410266,e=22810016,p=16263,cr=16486,cu=0,mis=0,r=0,dep=#1:c=11410266,e=22810016,p=16263,cr=16486,cu=0,mis=0,r=0,dep=0,og=4,tim=11309130333773630,og=4,tim=1130913033377363
WAIT #1: nam='PX Deq: Execution Msg' ela= 59137 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 59137 p1=268566527 p2=1 p3=0p2=1 p3=0
FETCH FETCH #1:c=0,e=86,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=1130913033#1:c=0,e=86,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=1130913033436946436946
WAIT #1: nam='PX Deq: Execution Msg' ela= 20 p1=268566527 p2=1 WAIT #1: nam='PX Deq: Execution Msg' ela= 20 p1=268566527 p2=1 p3=0p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 389 p1=268566527 p2=2 WAIT #1: nam='PX Deq: Execution Msg' ela= 389 p1=268566527 p2=2 p3=0p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 101813 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 101813 p1=268566527 p2=3 p3=0p2=3 p3=0
WAIT #1: nam='PX Deq: Execution Msg' ela= 100783 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 100783 p1=268566527 p2=1 p3=0p2=1 p3=0
Trace Output – 10gTrace Output – 10g
Query Co-ordinator (QC)Query Co-ordinator (QC)
Unix process pid: Unix process pid: 1723917239, image: [email protected] (TNS , image: [email protected] (TNS V1-V3)V1-V3)
*** ACTION NAME:() 2006-08-20 10:59:33.605*** ACTION NAME:() 2006-08-20 10:59:33.605
*** MODULE NAME:(SQL*Plus) 2006-08-20 10:59:33.605*** MODULE NAME:(SQL*Plus) 2006-08-20 10:59:33.605
*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.605*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.605
*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.605*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.605
*** SESSION ID:(*** SESSION ID:(1095.51095.5) 2006-08-20 10:59:33.605) 2006-08-20 10:59:33.605
Followed by tons of recursive callsFollowed by tons of recursive calls Enabling Tracing, CBO work, etc.Enabling Tracing, CBO work, etc.
Trace Output – 10g (QC Trace Output – 10g (QC Contd)Contd)
Essentially the sameEssentially the same
PARSING IN CURSOR #4 len=129 dep=0 uid=27 oct=3 lid=27 PARSING IN CURSOR #4 len=129 dep=0 uid=27 oct=3 lid=27 tim=1128972630780520 hv=3411843401 ad='77e67d54'tim=1128972630780520 hv=3411843401 ad='77e67d54'
SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)
FROM test_tab1 tt1FROM test_tab1 tt1
GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)
ORDER BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)
END OF STMTEND OF STMT
PARSE PARSE #4:c=267960,e=270534,p=20,cr=447,cu=3,mis=1,r=0,dep=0,og=1,tim=#4:c=267960,e=270534,p=20,cr=447,cu=3,mis=1,r=0,dep=0,og=1,tim=11289726307805081128972630780508
Trace Output – 10g (QC Trace Output – 10g (QC Contd)Contd)
Note the different timed event formatNote the different timed event format
WAIT #4: nam='PX Deq: Execute Reply' ela= 31 WAIT #4: nam='PX Deq: Execute Reply' ela= 31 sleeptime/senderid=200 passes=1 p3=0 obj#=10907 sleeptime/senderid=200 passes=1 p3=0 obj#=10907 tim=1129017680317433tim=1129017680317433
WAIT #4: nam='PX Deq: Execute Reply' ela= 282420 WAIT #4: nam='PX Deq: Execute Reply' ela= 282420 sleeptime/senderid=200 passes=2 p3=0 obj#=10907 sleeptime/senderid=200 passes=2 p3=0 obj#=10907 tim=1129017680599950tim=1129017680599950
WAIT #4: nam='PX qref latch' ela= 57346 function=1 WAIT #4: nam='PX qref latch' ela= 57346 function=1 sleeptime=2042549608 qref=0 obj#=10907 tim=1129017680657526sleeptime=2042549608 qref=0 obj#=10907 tim=1129017680657526
WAIT #4: nam='PX Deq: Execute Reply' ela= 916249 WAIT #4: nam='PX Deq: Execute Reply' ela= 916249 sleeptime/senderid=200 passes=2 p3=0 obj#=10907 sleeptime/senderid=200 passes=2 p3=0 obj#=10907 tim=1129017681574125tim=1129017681574125
WAIT #4: nam='PX qref latch' ela= 1594 function=1 WAIT #4: nam='PX qref latch' ela= 1594 function=1 sleeptime=2042549608 qref=0 obj#=10907 tim=1129017681575903sleeptime=2042549608 qref=0 obj#=10907 tim=1129017681575903
Trace Output – 10gTrace Output – 10g
Slaves 0 and 1 – Sort / GroupSlaves 0 and 1 – Sort / Group
Instance name: TEST1020Instance name: TEST1020
Redo thread mounted by this instance: 1Redo thread mounted by this instance: 1
Oracle process number: 12Oracle process number: 12
Unix process pid: Unix process pid: 1721917219, image: [email protected] , image: [email protected] ((P000P000))
*** 2006-08-20 10:59:33.938*** 2006-08-20 10:59:33.938
*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.938*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.938
*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.938*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.938
*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938
WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1128972630798951tim=1128972630798951
Trace Output – 10g (Sort Trace Output – 10g (Sort contd)contd)
Change to a shared cursor modelChange to a shared cursor modelPARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27 PARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27
tim=1129017679619468 hv=3411843401 ad='77e66bd4'tim=1129017679619468 hv=3411843401 ad='77e66bd4'SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)FROM test_tab1 tt1FROM test_tab1 tt1GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)END OF STMTEND OF STMTPARSE PARSE
#1:c=1000,e=731,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,tim=1129#1:c=1000,e=731,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,tim=1129017679619454017679619454
WAIT #1: nam='PX qref latch' ela= 52088 function=1 WAIT #1: nam='PX qref latch' ela= 52088 function=1 sleeptime=2042551468 qref=0 obj#=-1 tim=1129017679672279sleeptime=2042551468 qref=0 obj#=-1 tim=1129017679672279
WAIT #1: nam='PX Deq: Execution Msg' ela= 1954088 WAIT #1: nam='PX Deq: Execution Msg' ela= 1954088 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1129017681626630tim=1129017681626630
WAIT #1: nam='PX Deq: Execution Msg' ela= 1954772 WAIT #1: nam='PX Deq: Execution Msg' ela= 1954772 sleeptime/senderid=268566527 passes=2 p3=0 obj#=-1 sleeptime/senderid=268566527 passes=2 p3=0 obj#=-1 tim=1129017683581534tim=1129017683581534
Trace Output – 10gTrace Output – 10g
Slaves 2 and 3 - FTSSlaves 2 and 3 - FTS
Instance name: TEST1020Instance name: TEST1020
Redo thread mounted by this instance: 1Redo thread mounted by this instance: 1
Oracle process number: 12Oracle process number: 12
Unix process pid: Unix process pid: 1721917219, image: [email protected] , image: [email protected] ((P000P000))
*** 2006-08-20 10:59:33.938*** 2006-08-20 10:59:33.938
*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.938*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.938
*** CLIENT ID:(*** CLIENT ID:(doug_testdoug_test) 2006-08-20 10:59:33.938) 2006-08-20 10:59:33.938
*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938
WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1128972630798951tim=1128972630798951
Trace Output – 10g (FTS Trace Output – 10g (FTS Contd)Contd)
PARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27 PARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27 tim=1129017679653431 hv=3411843401 ad='77e66bd4'tim=1129017679653431 hv=3411843401 ad='77e66bd4'
SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)FROM test_tab1 tt1FROM test_tab1 tt1GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)END OF STMTEND OF STMTPARSE PARSE
#1:c=1000,e=649,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,tim=1129#1:c=1000,e=649,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,tim=1129017679653417017679653417
WAIT #1: nam='PX qref latch' ela= 18705 function=1 WAIT #1: nam='PX qref latch' ela= 18705 function=1 sleeptime=2042549980 qref=0 obj#=-1 tim=1129017679672803sleeptime=2042549980 qref=0 obj#=-1 tim=1129017679672803
WAIT #1: nam='PX Deq: Execution Msg' ela= 500 WAIT #1: nam='PX Deq: Execution Msg' ela= 500 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1129017679673500tim=1129017679673500
WAIT #1: nam='direct path read' ela= 46 file number=6 first WAIT #1: nam='direct path read' ela= 46 file number=6 first dba=10 block cnt=118 obj#=10907 tim=1129017679688488dba=10 block cnt=118 obj#=10907 tim=1129017679688488
WAIT #1: nam='PX qref latch' ela= 11838 function=1 WAIT #1: nam='PX qref latch' ela= 11838 function=1 sleeptime=2042549980 qref=0 obj#=10907 tim=1129017679700708sleeptime=2042549980 qref=0 obj#=10907 tim=1129017679700708
ConsolidationConsolidation trcsesstrcsess utility (10g) utility (10g)
Consolidates all trace files associated with a Consolidates all trace files associated with a specific specific session identifiersession identifier
Consolidated trace file looks like any standard Consolidated trace file looks like any standard trace filetrace file
Can be processed using Can be processed using tkproftkprof to produce to produce statement-level summary informationstatement-level summary information
trcsess output=output_file trcsess output=output_file clientid=session_identifier clientid=session_identifier /oracle/admin/TEST1020/udump/*.trc /oracle/admin/TEST1020/udump/*.trc /oracle/admin/TEST1020/bdump/*.trc/oracle/admin/TEST1020/bdump/*.trc
tkprof trcsess_output_file output_file tkprof trcsess_output_file output_file sort=prsela,fchela,exeelasort=prsela,fchela,exeela
Output is in cursor order, not strict time orderOutput is in cursor order, not strict time order
Timing IssuesTiming Issues Example TimingsExample Timings
Wall Clock elapsed timeWall Clock elapsed time 13.5613.56 seconds seconds
call count cpu elapsed call count cpu elapsed ------- ------ -------- ---------- ------- ------ -------- ---------- Parse Parse 55 0.01 0.02 0.01 0.02Execute Execute 55 12.98 51.57 12.98 51.57Fetch Fetch 22 0.01 12.81 0.01 12.81------- ------ -------- ----------------- ------ -------- ----------total 12 13.01 total 12 13.01 64.4164.41
Picking up the wrong trace files?Picking up the wrong trace files? trcsess bug?trcsess bug?
Timing IssuesTiming Issues
Wait Time also ‘too high’Wait Time also ‘too high’Event waited on Times Max. Wait Total WaitedEvent waited on Times Max. Wait Total Waited---------------------------------------- Waited ---------- ---------------------------------------------------- Waited ---------- ------------PX qref latch 31 0.09 0.76PX qref latch 31 0.09 0.76PX Deq: Execution Msg 49 1.95 PX Deq: Execution Msg 49 1.95 27.7627.76direct path read 281 0.00 0.00direct path read 281 0.00 0.00db file sequential read 2 0.00 0.00db file sequential read 2 0.00 0.00PX Deq: Parse Reply 2 0.00 0.00PX Deq: Parse Reply 2 0.00 0.00SQL*Net message to client 2 0.00 0.00SQL*Net message to client 2 0.00 0.00PX Deq: Execute Reply 39 0.92 PX Deq: Execute Reply 39 0.92 12.1512.15PX Deq: Table Q qref 1 0.00 0.00PX Deq: Table Q qref 1 0.00 0.00SQL*Net message from client 2 0.00 0.00SQL*Net message from client 2 0.00 0.00PX Deq: Signal ACK 6 0.00 0.01PX Deq: Signal ACK 6 0.00 0.01enq: PS - contention 1 0.00 0.00enq: PS - contention 1 0.00 0.00PX Deq: Table Q Sample 2 0.00 0.00PX Deq: Table Q Sample 2 0.00 0.00PX Deq Credit: send blkd 3 0.00 0.00PX Deq Credit: send blkd 3 0.00 0.00
Timing IssuesTiming Issues
This is a This is a consolidatedconsolidated trace file trace file Multiple concurrent processesMultiple concurrent processes Some running on multiple CPUsSome running on multiple CPUs
There is more CPU time available than wall clock timeThere is more CPU time available than wall clock time Some just waitingSome just waiting
Systems have an infinite capacity for waitingSystems have an infinite capacity for waiting Will vary, depending on the actual DOP Will vary, depending on the actual DOP
acquiredacquired The DOP acquired could varyThe DOP acquired could vary How many slaves are available in the pool?How many slaves are available in the pool? parallel_adaptive_multi_userparallel_adaptive_multi_user
Timing IssuesTiming Issues
Exactly the same job, but DOP 16Exactly the same job, but DOP 16 Wall Clock elapsed time = Wall Clock elapsed time = 16.6016.60 seconds seconds
Slightly higher than Slightly higher than 13.5613.56 seconds seconds
call count cpu elapsed call count cpu elapsed ------- ------ -------- ---------- ------- ------ -------- ---------- Parse Parse 3333 0.03 0.16 0.03 0.16Execute Execute 3333 13.42 430.24 13.42 430.24Fetch 2 0.02 12.28Fetch 2 0.02 12.28------- ------ -------- ----------------- ------ -------- ----------total 68 13.48 total 68 13.48 442.69442.69
MuchMuch higher elapsed time higher elapsed time Couple of seconds additional wait timeCouple of seconds additional wait time
ConclusionConclusion Tracing PX is a little more tricky that tracing Tracing PX is a little more tricky that tracing
serial processesserial processes A bit like trying to trace multiple users, or an A bit like trying to trace multiple users, or an
entire server?entire server? 10g dbms_monitor package and trcsess 10g dbms_monitor package and trcsess
utility helputility help Consolidated timings need to be interpreted Consolidated timings need to be interpreted
differentlydifferently More papers available at More papers available at http://oracledoug.comhttp://oracledoug.com Next presentation – How Many Slaves – Hall Next presentation – How Many Slaves – Hall
7b at 16:007b at 16:00
Tracing PX SlavesTracing PX Slaves
Doug BurnsDoug Burns
[email protected]@yahoo.com
http://oracledoug.comhttp://oracledoug.com