Moscone South Room 301 San Francisco, CA October 13, 2009

Oracle Open World 2009Measure, Interpret, and Analyze Oracle I/O Performance DataPresenter: John Moffett of BIAS CorporationMoscone South Room 301 San Francisco, CA October 13, 2009

Grid Architecture

AgendaExamine Four (4) Main Oracle Read OperationsCost Based Optimizer (CBO) and I/O Cost ComputationsHow to Tap Into Oracle AWR and STATSPACK I/O Tables to Create I/O ProfilesASM Striping ConfigurationsQuick Dive into Todays Disk Drive TechnologyCase StudiesTakeaways

Grid Architecture

Introduction to BIASDirector of Architecture/Infrastructure for BIASStarted Working With Oracle Version 5 Beta in 198513 Years of Oracle Performance Tuning8 Years with Oracle5 Years with BIASMy [email protected]

Grid Architecture

Four Primary Oracle Read OperationsSingle Block ReadMulti-Block ReadDirect Path ReadParallel Read

Grid Architecture

Single Block ReadIndex Range Scans and Index Full ScansRow Fetch Based on Index LookupRequestor Supplies File#, Start Block#On-Line Transaction Processing (OLTP) Workhorse

Grid Architecture

Multi-Block ReadFull Table Scan (Up to High Water Mark)Index Fast Full Scan (Up To High Water Mark)Requestor Supplies File#, Start Block#, and Num of Contiguous BlocksBlock Count Cant ExceedDB_FILE_MULTIBLOCK_READ_COUNT (MBRC)Max I/O Size Supported by OSOracle Reads Around Blocks in the Buffer CacheBatch Job Workhorse

select count(*) from emp;

Grid Architecture

Multi-Block ReadSet By DBA in Oracle 7, 8, 9, 10Set By Oracle in 11gFormula is (Max IO Size)/(Oracle Block Size)DB_FILE_MULTIBLOCK_READ_COUNT (MBRC)If Max IO Size is 1 MByte (Operating System and Storage I/O Stripe Size)

Grid Architecture

Setting MBRC High Encourages CBO to Choose Multi-Block Reads Instead of Single Block ReadsMBRC TruthsSetting MBRC Higher Than 8 For RAC (9i,10g) Might Trigger Excessive gc cr multi block request WaitsProper Setting Maximizes Large I/O Request ProcessingGoing to Read All Up to the HWMBest Practice is to Minimize Number of I/O RequestsMulti-Block ReadMBRC Myths

Grid Architecture

Direct Path ReadSort I/O or Hash Join I/O (TEMP Tablespace)Parallel Query (PX)Requestor SuppliesFile#, Start Block#, num blocksOracle Reads Directly into PGA CursorBypasses Buffer CacheIntense I/O Workhorse for Batch and BI/DW

INSERT /*+ APPEND */ into MYTABLE SELECT * FROM EMP;

Grid Architecture

Direct Path ReadExecution Plan----------------------------------------------------------Plan hash value: 293734533-----------------------------------------------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | TQ |IN-OUT| PQ Distrib |-----------------------------------------------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 1 | 20 | 77864 (3)| 00:18:11 | | | | | || 1 | PX COORDINATOR | | | | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 1 | 20 | 77864 (3)| 00:18:11 | | | Q1,01 | P->S | QC (RAND) ||* 3 | HASH JOIN | | 1 | 20 | 77864 (3)| 00:18:11 | | | Q1,01 | PCWP | || 4 | PX RECEIVE | | 1890K| 9230K| 91 (4)| 00:00:02 | | | Q1,01 | PCWP | || 5 | PX SEND BROADCAST | :TQ10000 | 1890K| 9230K| 91 (4)| 00:00:02 | | | Q1,00 | P->P | BROADCAST || 6 | PX BLOCK ITERATOR | | 1890K| 9230K| 91 (4)| 00:00:02 | | | Q1,00 | PCWC | || 7 | TABLE ACCESS FULL| CLM_SDI_UOW_AGNS | 1890K| 9230K| 91 (4)| 00:00:02 | | | Q1,00 | PCWP | || 8 | PX BLOCK ITERATOR | | 482M| 6897M| 77207 (2)| 00:18:01 | 29 | 29 | Q1,01 | PCWC | || 9 | TABLE ACCESS FULL | CLM_CLAIM_ENTITY_ASSOC | 482M| 6897M| 77207 (2)| 00:18:01 | 29 | 29 | Q1,01 | PCWP | |-----------------------------------------------------------------------------------------------------------------------------------------------Predicate Information (identified by operation id):--------------------------------------------------- 3 - access("CEA"."UOW_AGN"="AGNS"."U_AGN")Partition: 29Equi-Join Method: Hash JoinExecution Plan: Parallel Query and Partition

Grid Architecture

Direct Path ReadSTATSPACK report for DB Name DB Id Instance Inst Num Release Cluster Host ------------ ----------- -------- ---- ----------- ----------- -------T2BL85A 3434773797 T2BL85A 1 9.2.0.6.0 NO dc1-beta CPU ElapsedBuffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value ----------- ---------- ------------- ------ -------- -------- ---------- 856,070 16 53,504.4 15.9 77.69 2201.99 1061718588Module: oracle@whltlg2 (TNS V1-V3) SELECT /*+ Q607000 NO_EXPAND ROWID(A1) */ A1.ROWID,A1."BAN",A1." ACCOUNT_TYPE",A1."BILL_CYCLE",A1."BAN_STATUS",A1."STATUS_LAST_DA TE",A1."START_SERVICE_DATE",A1."BL_CUR_BILL_SEQ_NO",A1."BL_LAST_ PROD_DATE",A1."BL_LAST_CYC_RUN_YEAR",A1."BL_LAST_CYC_RUN_MNTH",A 1."BL_LAST_CYC_CODE",A1."BL_COMPLT_STATUS",A1."BL_SPECIAL_CYCLE"

Statistic Total per Second per Trans --------------------------------- ------------------ -------------- ------------- physical reads 4,614,109 5,121.1 3,841.9 physical reads direct 1,123,055 1,246.5 935.1 physical writes 113,268 125.7 94.3 physical writes directImpact on Oracle Read Statistics (8i and 9i)

Grid Architecture

Direct Path ReadTablespace IO Stats for DB: T2BL85A Instance: T2BL85A Snaps: 71727 -71728->ordered by IOs (Reads + Writes) desc

Av Av Av Av Buffer Av BufTablespace Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)-------------------- ------- ------- ------- ------- -------- -------- ------- ------TLGP03M 291,562 324 1.9 8.6 24 0 0 0.0 TBS01M 28,592 32 49.9 29.3 1,077 1 0 0.0TLGP05M 27,737 31 5.9 10.3 2 0 0 0.0TBS01L 19,828 22 11.1 28.7 0 0 0 0.0TBS01S 7,838 9 13.9 31.9 800 1 0 0.0Impact on Oracle Read Statistics (8i and 9i)PX Query Coordinator Wait Times Included in the Read Wait Statistics

Grid Architecture

Parallel ReadProprietary Oracle AlgorithmCommon Operation for RecoveryAlso Used For Index Scans and Pre-Fetch OperationsMore Prevalent in 11gIssue Parallel Reads Against Multiple FilesBlocks are Not Contiguous (Unlike Multi-block)Wait Event Reports #Files, #Blocks, #RequestsWaits for Last Request to Mark I/O as CompletedChosen at Runtime - Ever Mysterious

Grid Architecture

Cost Based OptimizerDB_FILE_MULTIBLOCK_READ_COUNT: 32ORACLE BLOCK SIZE: 16K

Read Operation Selection in Execution Plan (Default)

Grid Architecture

Cost Based OptimizerDBMS_STATS.GATHER_SYSTEM_STATS( ) STARTSTOPSYS.AUX_STATS$Override Default CBO System Statistics

Grid Architecture

Timed Statistic ViewsCommonly Used Views for Performance Analysis

Grid Architecture

Important StatsPack/AWR TablesDerived from Timed Statistic V$ Views

Grid Architecture

PERFSTAT.STATS$FILESTATXSI/O InterpretationAll Times in Centi-Seconds (One-Hundredth of a Second)

Grid Architecture

DBA_HIST_FILESTATXS (AWR)I/O InterpretationAll Times in Centi-Seconds (One-Hundredth of a Second)

Grid Architecture

Timed StatisticsRead Wait Events

Grid Architecture

SQL to Capture StatsPack/AWR DataQuery to Capture I/O Data

Grid Architecture

SQL to Capture StatsPack/AWR DataWait Event Query

Grid Architecture

I/O AnalysisPlot Data Using EXCEL

Grid Architecture

Case Study 1 EMC Clariion CX-500SUN V890 6 Dual-Core CPUs and 32 GB of RAMClock Speed is 2100 MHzOperating System is Solaris 10 (64-bit)Oracle 10.2.0.3 Enterprise Single InstanceOracle Block Size 8KOracle Buffer Cache 2.7GOracle SGA 3 GBASMRaid-5Travel and Hospitality IndustryData Warehouse and Reporting

Grid Architecture

Case Study 1 EMC Clariion CX-500Transactions Per Second

Grid Architecture

Case Study 1 EMC Clariion CX-500Single Block Read Performance (Avg Read Time 8.9 ms)

Grid Architecture

Case Study 1 EMC Clariion CX-500Multi Block Read Performance (Avg Read Time 2.13 ms)

Grid Architecture

Case Study 1 EMC Clariion CX-500System Event Statistics

Grid Architecture

Case Study 1 EMC Clariion CX-500Read Waits (Single Block and Multi-Block)

Grid Architecture

Case Study 1 EMC Clariion CX-500Redo Write Time Analysis (ms/block)Redo Write Time Suspect for Oracle Versions < 9.2.0.8Use Log File Sync Waits for Sanity CheckGood Metric to Check forSan Fabric WaitsNAS Network Waits

Grid Architecture

Case Study 1 EMC Clariion CX-500Redo Write Time Analysis for CX-340 (OLTP Database)

Grid Architecture

Case Study 1 EMC Clariion CX-500ADDM Analysis

Grid Architecture

Case Study 1 EMC Clariion CX-500Server CPU Activity

Grid Architecture

Case Study 1 EMC Clariion CX-500No Obvious I/O BottlenecksIf Business Users Were Complaining, Initiate Method R Analysis of Poor Performing Business Functions10046 Trace of Poorly Performing Application ProcessesSee Book by Cary Millsap and Jeffrey Holt Optimizing Oracle PerformanceFor the Functions Identified Above, Isolate the Hot SQLTune the SQL for Better PerformanceBetter Execution PlanBetter AlgorithmNext Steps

Grid Architecture

Automatic Storage Management (ASM)Juan Loaizas 2000 OOW Paper on S.A.M.E

Grid Architecture

Automatic Storage Management (ASM)Oracles Logical Volume ManagerSpecial ASM Instance (in its own $ORACLE_HOME)Uses RAW Disks11gR2 Adds Cluster File System SupportCan Support any Oracle Database (RAC, Non-RAC)Users Typically Create Two Logical VolumesDATA (Schema Objects, Redo, Control)FLASH (Archives, RMAN bkups, FlashBack Logs, Control)Other Common Logical Volumes IncludeREDO (Redo Logs in own volume group)TEMP (If Running Big Sorts and/or Big Hash Joins)Configuration Overview

Grid Architecture

Automatic Storage Management (ASM)If We Present These Four Disks to ASM

Which Configuration is Best?Disk Configuration (S.A.M.E)

Grid Architecture

Automatic Storage Management (ASM)Very important Planning ConsiderationASM Always Stripe Across All Members of Each Disk GroupStriping Option 1 Coarse StripingSet at 1 MByte (Largest single I/O operation that Oracle currently performs up through and including version 11g)Recommended for OLTPStriping Option 2 - Fine Grain StripingFine grain stripe size that is 128KAppropriate for redo logsCan Be Appropriate for DSS or DW TablespacesDisk Configuration (Striping)

Grid Architecture

Automatic Storage Management (ASM)Which is Correct If using Coarse Striping (1 MB)?Disk Configuration (Stripe Width and Stripe Depth)Stripe Depth: 256K Stripe Width: 1 MBStripe Depth: 1M Stripe Width: 4 MB

Grid Architecture

Automatic Storage Management (ASM)Writing the StripeStep 1: Preparation

WAIT #4: nam='Data file init write' ela= 534 count=1 intr=256 timeout=4294967295 obj#=-1 tim=14093356333941WAIT #4: nam='Data file init write' ela= 471 count=1 intr=256 timeout=4294967295 obj#=-1 tim=14093356334587WAIT #4: nam='Data file init write' ela= 524 count=1 intr=256 timeout=4294967295 obj#=-1 tim=14093356335266WAIT #4: nam='Data file init write' ela= 468 count=1 intr=256 timeout=4294967295 obj#=-1 tim=14093356335882

Grid Architecture

Automatic Storage Management (ASM)Writing the StripeStep 2: Writing the Stripe

Above Repeated 250 timesOne for each of the 4 Mbyte wide ASM Stripes used to Create the 1 GByte tablespace.Step 3: Two Final Write(s)

WAIT #4: nam='Data file init write' ela= 17234 count=1 intr=32 timeout=2147483647 obj#=-1 tim=14093356353180

WAIT #4: nam='Data file init write' ela= 79 count=1 intr=256 timeout=4294967295 obj#=-1 tim=14093361349183WAIT #4: nam='Data file init write' ela= 313 count=4294967295 intr=32 timeout=2147483647 obj#=-1 tim=14093361349516

Grid Architecture

Automatic Storage Management (ASM)ASM is Going to StripeStriping on Storage Side and ASM Side Perfectly okay Creates Plaid StripeI/O performance tests from SAN Vendors and Others Confirm that Plaid Stripe Gives Best Overall I/O PerformanceHow Much Benefit Depends on the I/O Mix If Its Potential to Boost I/O is Modest, is Plaid Striping worth doing?ASM and Not Plaid: Very Safe ConfigurationASM and Plaid Not Done Properly: Adds Risk to I/O PerformanceDisk Configuration (Plaid Stripe)

Grid Architecture

Automatic Storage Management (ASM)Present Complete Raw Disks To Oracle

Raid 1 Mirroring is used on Storage SystemASM Stripes Go Across Different DisksGood Performance Low RiskDisk Configuration 1 (No Stripe on I/O Subsystem)

Grid Architecture

Automatic Storage Management (ASM)Configuration Mismatch

ASM Stripes Not Evenly Distributed Across All DisksStripe Across Same Physical DiskLess Than Optimal DesignDisk Configuration 2 (Plaid Stripe Configuration Error)Raw Disks In I/O Subsystem Carved into LUNsASM Plaid Stripe

Grid Architecture

Automatic Storage Management (ASM)Optimum DesignCreate two hardware disks groups from the four disksCarve 2 LUNs on each disk group as follows Disk Configuration 3 (Plaid Stripe Applied Correctly)ASM Stripes Across Different DisksOptimal DesignASM Creates Two Disk GroupsDATA1 (A1 and A2)DATA2 (B1 and B2)

Grid Architecture

Modern Disk Drive TechnologyDisk Drive ComponentsPlatterRead/Write HeadsActuatorSpindle

Grid Architecture

Modern Disk Drive TechnologySpecs for Seagate Cheetah 15K.5-Internal-3.5in-UCapacity: 36.7 300 GB36.7 GB Per Platter SurfaceCylinders: 73,340Density: 1/2 MB/Cylinder/SurfaceDiscs: Up to 4Heads: Up to 8Buffer Size: 16 MBAvg Seek (R/W): 3.4/4 msTrack-to-Track (R/W): 0.2/0.4 msOnly One Set of Heads Active at any One timeStripe Downward from One Platter to Another without Moving HeadsDrives Read Entire Cylinder and return requested sectorsDisk Buffer Cache Supplements I/O Storage System Read Cache

Grid Architecture

Modern Disk Drive TechnologyOracle SpecificsSingle Block ReadDisk Hardware Reads Entire Cylinder Containing BlockJust the Requested Oracle Block is return through the I/O Subsystem Read CacheRandom I/O Performance is Enhanced by the 16MB Disk Buffer Cache

Multi-Block and Direct Path Reads Fetches Consecutive Blocks up to MBRC Setting (1 MB Max)Reading Down a Cylinder Maximizes Throughput Since Heads Dont Move

Grid Architecture

Modern Disk Drive TechnologyRaid 5 Has Historically Had a Lot of Bad Press from Oracle VeteransOak TableBAARFTodays Modern Storage Systems Successfully Mitigate Raid 5 Performance IssuesRaid 1+0 Is Still Recommended for Heavy OLTP SystemsUse Raid 5 for Everything ElseConsider Solid State To Handle Heavy Redo Log ActivityRaid 1+0 or Raid 5?

Grid Architecture

Case Study 2 EMC Symmetrix DMX-4IBM 9117-570 with 12 CPUs and 24 GB of RAMTier 1 Storage (Raid 1+0)CPUs are PowerPC_POWER5 (SMT_Enabled)Clock Speed is 1.9 GHzOperating System is AIX 5.3 (64-bit)Oracle 9.2.0.6Oracle Block Size 16KOracle Buffer Cache 2.048GOracle SGA 3.12GVeritas File SystemInsurance Claim NetworkClaims Processing Database

Grid Architecture

Case Study 2 EMC Symmetrix DMX-4Transactions Per Second (Day and Night Shifts)

Grid Architecture

Case Study 2 EMC Symmetrix DMX-4Single Block Reads (Avg Read Time is 3.72 ms)

Grid Architecture

Case Study 2 EMC Symmetrix DMX-4Multi-Block Reads (Avg Read Time is 10 ms)

Grid Architecture

Case Study 2 EMC Symmetrix DMX-4Average Redo Write Time

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4IBM pSeries 660 with 4 CPUs and 12 GB of RAMCPUs are PowerPCClock Speed is 600 MHzOperating System is AIX 5.2 (64-bit)Oracle 9.2.0.6Oracle Block Size 4KOracle Buffer Cache 3.6GOracle SGA 5.1GRaid 5Insurance ProviderCustomer Facing Claims Processing Database

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4Transactions Per Second (Avg 10/sec)

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4Single Block Reads (Avg Read Time is 8.45 ms)

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4Multi-Block Reads (Avg Read Time is 19.59 ms)

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4System Statistics

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4Average Redo Write Time

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4Log File Sync Waits

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4DB File Sequential & Scattered Read Waits Per Second

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4DB File Parallel Reads

Grid Architecture

Case Study 3 EMC Symmetrix DMX-4AIX I/O Request Size in Oracle Blocks (4K Block Size)Should See a Value > 1AIX File System Mis-Match with EMC Storage Layout

Grid Architecture

Case Study 4 SUN Hitachi 9970SUN E12K 8 CPUs & 8 GB of RAMOperating System is Solaris 8 (64-bit)Oracle 9.2.0.6 Enterprise Single InstanceOracle Block Size 8KOracle Cache Buffer 3.5GOracle SGA 4GStandard Veritas Cooked File SystemRaid 5Sales and Settlement Solutions for Airline Travel Industry Ticket Processing Engine

Grid Architecture

Case Study 4 SUN Hitachi 9970Converting from IBM Green Screen to Oracle, Web/JavaLoad Process on Oracle Taking Too LongStep 6 (Load Taxes) Never Finishing V$LONGOPS (2 Hrs, 1:40 Hrs, 2 Hrs, 3 Hrs, 4 Hrs)Background Info ProblemRedo (Raid 5) Triggering Cache Cramming on HitachiSwitch to Solid StateAdd More Cache to HitachiI/O Pipe Too Small on Solaris 8Configure Disks Using S.A.M.E (1 MB Stripe)Configure Kernel to Align with Hitachi StripeOracleTurn-Off Tablespace CompressionSolution

Grid Architecture

Case Study 4 SUN Hitachi 9970Transactions Per Second

Grid Architecture

Case Study 4 SUN Hitachi 9970Single Block Read Performance (Avg Read Time 1.79 ms)

Grid Architecture

Case Study 4 SUN Hitachi 9970Multi Block Read Performance (Avg Read Time 5.04 ms)

Grid Architecture

Case Study 4 SUN Hitachi 9970System Event Statistics

Grid Architecture

Case Study 4 SUN Hitachi 9970Read Waits (Single Block and Multi-Block)

Grid Architecture

Case Study 4 SUN Hitachi 9970Redo Write Time Analysis (ms/block)Redo Write Time Suspect for Oracle Versions < 9.2.0.8Use Log File Sync Waits for Sanity CheckGood Check forSan Fabric WaitsNAS Network Waits

Grid Architecture

Case Study 4 SUN Hitachi 9970Log File Sync Wait Analysis (3 Years Later)

Grid Architecture

Case Study 5 NetAppsHP Blade with (4) AMD Dual Core CPUs & 32 GB of RAMClock Speed is xxx MHzOperating System is Linux (64-bit)Oracle 11.1.0.6 Enterprise Four Node RACOracle Block Size 8KOracle Cache Buffer 7.4GOracle SGA 9.6GClustered File SystemRaid 5Tax Preparation and Electronic Filing Tax Processing Engine

Grid Architecture

Case Study 5 NetAppsTransactions Per Second

Grid Architecture

Case Study 5 NetAppsSingle Block Read Performance (Avg Read Time 8.31 ms)

Grid Architecture

Case Study 5 NetAppsMulti Block Read Performance (Avg Read Time 7.62 ms)

Grid Architecture

Case Study 6 EMC Symmetrix DMX-4SUN E25K with 42 Dual Core CPUs and 32 GB RAMCPUs are US-IV+Clock Speed is 1500 MHzOperating System is Solaris 10 (64-bit)Oracle 10.2.0.2Oracle Block Size 32KOracle Buffer Cache 50GOracle SGA 65GRaid 5Government AgencyIncident Tracking Database

Grid Architecture

Case Study 6 EMC Symmetrix DMX-4Multi Block Read Performance (Avg Read Time 991 ms)

Grid Architecture

TakeawaysReviewed Primary Oracle Read I/O RequestsSingle Block, Multi-Block, Direct Path Read, Parallel ReadShould How To Capture I/O Data by QueryingPERFSTAT.STATS$FILESTATXS (StatsPack)DBA_HIST_FILESTATXS (AWR)Presented Some Key I/O Metrics to Capture and ExamineReviewed ASM Configuration IssuesPresented a High Level Overview of Todays Disk TechnologyPresented 6 Case StudiesDiscussed Acceptable I/O Ranges for Real World ApplicationsDescribed Problems and How to Fix Them

Grid Architecture

To Ask [email protected]

Grid Architecture

Moscone South Room 301 San Francisco, CA October 13, 2009

Documents

Transcript of Moscone South Room 301 San Francisco, CA October 13, 2009