Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach...

25
Washington Area Informix/DB2 User’s Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland [email protected] Bluepoint Consulting Inc.

Transcript of Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach...

Page 1: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Washington Area Informix/DB2 User’s Group July 26, 2006

DB2 Performance Tuning: A Practical Approach

By Jim Cleveland

[email protected]

Bluepoint Consulting Inc.

Page 2: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

IntroductionIntroduction

What is our aim?What is our aim?

– Performance tuningPerformance tuning is:is: Analyzing the system’s configuration Analyzing the system’s configuration Observing performance characteristics Observing performance characteristics Formulating recommendations intended to enhance performanceFormulating recommendations intended to enhance performance (Rinse, shampoo, repeat)(Rinse, shampoo, repeat)

– Metrics driven modelMetrics driven model involves: involves: Gathering system performance data (at all levels)Gathering system performance data (at all levels) Focusing on a particular quantifiable performance measurementFocusing on a particular quantifiable performance measurement Adjusting performance parameters or modifying architectural Adjusting performance parameters or modifying architectural

characteristicscharacteristics Gauging the effect of those changes on overall performance and Gauging the effect of those changes on overall performance and

selected measurementselected measurement

Page 3: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Look familiar?Look familiar?

Page 4: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

The Diagnostic Tools – DB2 The Diagnostic Tools – DB2

DB2 Snapshot MonitorsDB2 Snapshot Monitors

– Internal counters set at global or session levels using monitor switchesInternal counters set at global or session levels using monitor switches

– Monitors collect cumulative statistics either from the initiation of a CLP Monitors collect cumulative statistics either from the initiation of a CLP session (switches and stats are local to session), since time of last reboot session (switches and stats are local to session), since time of last reboot (global switches set and stats available to all users), or last “reset (global switches set and stats available to all users), or last “reset monitors” command at either level monitors” command at either level

– Useful for collecting point-in-time information regarding performance Useful for collecting point-in-time information regarding performance metrics with respect to overall db/dbm behavior. Also able to provide metrics with respect to overall db/dbm behavior. Also able to provide more specific information on:more specific information on:

LockingLocking – at given moment lists all locks & types held by all applications – at given moment lists all locks & types held by all applications BufferpoolsBufferpools – cumulative stats for memory, physical & logical I/O’s, – cumulative stats for memory, physical & logical I/O’s,

synch/asych synch/asych SortsSorts – provides detailed statistics regard sortheap usage, overflows, # active – provides detailed statistics regard sortheap usage, overflows, # active

etc.etc. TablespacesTablespaces – detailed I/O stats and other activity for a given tablespace – detailed I/O stats and other activity for a given tablespace UOWUOW – displays status of application Unit of Work at point-in-time – displays status of application Unit of Work at point-in-time Dynamic SQLDynamic SQL – shows contents of package cache and related stats at point-in- – shows contents of package cache and related stats at point-in-

timetime

Page 5: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

The Diagnostic Tools – DB2The Diagnostic Tools – DB2

DB2 Event MonitorsDB2 Event Monitors

Event Monitors are the DB2 facility used to collect information regarding a specific event or Event Monitors are the DB2 facility used to collect information regarding a specific event or chainchain

of events within the context of database operations.of events within the context of database operations.

– They are event driven versus continuously collected as with snapshotsThey are event driven versus continuously collected as with snapshots

– Must be explicitly created and activated via DB2 commands or API’sMust be explicitly created and activated via DB2 commands or API’s

– Are the best way to effectively diagnose and resolve deadlock issues Are the best way to effectively diagnose and resolve deadlock issues from the database perspective, since deadlocks require tracing a chain of from the database perspective, since deadlocks require tracing a chain of events through timeevents through time

– Output may be directed to a DB2 table and results then analyzed using Output may be directed to a DB2 table and results then analyzed using SQL. For example, output from statement Event Monitor can be used to: SQL. For example, output from statement Event Monitor can be used to:

identify and rank most frequently executed or most time consumingidentify and rank most frequently executed or most time consuming SQL SQL track the sequence of statements leading up to a track the sequence of statements leading up to a lock timeout or deadlocklock timeout or deadlock track track connectionsconnections to the database to the database breakdown overall SQL statement execution time into sub-operations such as breakdown overall SQL statement execution time into sub-operations such as

sort timesort time and use or system and use or system CPU timeCPU time

Page 6: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

The Diagnostic Tools - UnixThe Diagnostic Tools - Unix

Concurrent Monitoring of OS Resource Usage is Imperative.Concurrent Monitoring of OS Resource Usage is Imperative.

When collecting DB2 snapshot/event data it will be useful (necessaryWhen collecting DB2 snapshot/event data it will be useful (necessaryeven?) to have, at a minimum, time stamped output from:even?) to have, at a minimum, time stamped output from:

– iostatiostat [interval in sec. #iterations][interval in sec. #iterations] provides detailed information, broken down by provides detailed information, broken down by hdisk for % of time disk busy, read/write transfer rates and totals, xacts per sec. hdisk for % of time disk busy, read/write transfer rates and totals, xacts per sec. Particularly useful for identifying heavily used (‘hot’) disks which may be the cause Particularly useful for identifying heavily used (‘hot’) disks which may be the cause I/O bottlenecks (i.e. – hdisks with continuous activity > 40% deserve attention) I/O bottlenecks (i.e. – hdisks with continuous activity > 40% deserve attention)

– vmstatvmstat [interval in sec. #iterations] [interval in sec. #iterations] provides information regarding CPU activity (% provides information regarding CPU activity (% for system, user, and time waiting for I/O), available/free memory and paging for system, user, and time waiting for I/O), available/free memory and paging activity. Useful for identifying shortage of CPU resources - overall activity > 70% activity. Useful for identifying shortage of CPU resources - overall activity > 70% indicates there may be inadequate ‘headroom’ for handling peak loads indicates there may be inadequate ‘headroom’ for handling peak loads DB2 DB2 processes may back up in wait queue.processes may back up in wait queue.

– Other useful OS performance monitoring/config commands: Other useful OS performance monitoring/config commands: filemonfilemon - drill down to retrieve detailed I/O statistics on individual devices - drill down to retrieve detailed I/O statistics on individual devices netstatnetstat - identify latency in transactions due to network data transfer time - identify latency in transactions due to network data transfer time psps - provides capability to identify DB2 (or other) processes - provides capability to identify DB2 (or other) processes sarsar - system activity report, combines metrics from iostat, vmstat, and others- system activity report, combines metrics from iostat, vmstat, and others lspslsps – describes paging space – describes paging space lscfg/smitlscfg/smit –– system hardware inventory command and configuration utilitiysystem hardware inventory command and configuration utilitiy

Page 7: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

The Diagnostic Tools – Unix/AIXThe Diagnostic Tools – Unix/AIX

In the best of all possible worlds...get topas installed (or In the best of all possible worlds...get topas installed (or nmon):nmon):

Page 8: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Resort to bribery….Resort to bribery….

Page 9: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

The DB2 Process ModelThe DB2 Process Model

To understand the context in which performance issues arise we need to understand To understand the context in which performance issues arise we need to understand how the various DB2 components function together as a whole. That is, how the DB2 how the various DB2 components function together as a whole. That is, how the DB2 process model works –process model works –

Page 10: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Identifying the Problem – Step IIdentifying the Problem – Step I

Collect Preliminary Configuration and Performance Information Collect Preliminary Configuration and Performance Information

– Do a cursory configuration and architecture review (beforehand, if possible)Do a cursory configuration and architecture review (beforehand, if possible)

– Capturing db and dbm level snapshot stats over a ‘representative’ period of time Capturing db and dbm level snapshot stats over a ‘representative’ period of time provides a good jumping off/drill-down point for non-specific performance issues provides a good jumping off/drill-down point for non-specific performance issues (i.e. – focus is not on a single transaction or process). (i.e. – focus is not on a single transaction or process).

– Simple shell script run from Unix prompt or scheduled using crontab will sufficeSimple shell script run from Unix prompt or scheduled using crontab will sufficeto collect data over specified period (sub # of iterations for i1, interval len for i2): to collect data over specified period (sub # of iterations for i1, interval len for i2):

For example:For example:

#!/bin/ksh#!/bin/kshdate; x=o;date; x=o;db2 update monitor switches using lock sort on bufferpool on uow on table ondb2 update monitor switches using lock sort on bufferpool on uow on table onstatement on;statement on;db2 reset monitor alldb2 reset monitor alldo while x < i1; # determines number of iterations, total monitoring perioddo while x < i1; # determines number of iterations, total monitoring perioddatedatedb2 get snapshot for dbm;db2 get snapshot for dbm;db2 get snapshot for db on xxxxxx;db2 get snapshot for db on xxxxxx;sleep i2 # interval in seconds for loop, longer interval for longer monitoring periodsleep i2 # interval in seconds for loop, longer interval for longer monitoring periodx=x+1x=x+1enddoenddo

As mentioned cross time indexed OS level stats should be collected concurrently – same script using As mentioned cross time indexed OS level stats should be collected concurrently – same script using iostat/vmstat maybe.iostat/vmstat maybe.

Page 11: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Identifying the Problem – Step IIIdentifying the Problem – Step II

The question at this stage is: on what tasks and in what proportion isThe question at this stage is: on what tasks and in what proportion is

DB2 spending its time?DB2 spending its time?

To identify common performance issues, breakdown as follows:To identify common performance issues, breakdown as follows:

– LockingLocking - time waiting on resolution of lock & deadlock issues - time waiting on resolution of lock & deadlock issues– SortsSorts - time spent performing sorts, hash joins, (other?) - time spent performing sorts, hash joins, (other?)– Logging ActivityLogging Activity - time for log writes, reads and overhead - time for log writes, reads and overhead – System CPUSystem CPU - time executing system calls - time executing system calls– User CPUUser CPU - time spent on SQL prep, ovhd for locking and sorting activity - time spent on SQL prep, ovhd for locking and sorting activity– General I/OGeneral I/O - no single cause, likely a confluence of factors for e.g. - no single cause, likely a confluence of factors for e.g.

ts design, container mapping, bp configuration, db/dbm params etc.ts design, container mapping, bp configuration, db/dbm params etc.

(Specific calculations for above available in Admin Guide: Performance) (Specific calculations for above available in Admin Guide: Performance)

Page 12: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

The Tuning Process – a RoadmapThe Tuning Process – a Roadmap

Collect db/dbm & OS stats

Sort, Logging,Lock or

Prefetch?

Memory, CPU,or I/O problems?

Perform I/O diagnosis & tuning

Remaining lock/sort? list db2

apps., examine OS processes,

finally: stack trace

Perform memory accounting, adjust

paging params,release

overcommitted memory

Wait on IO > 10%, or

Disk Busy > 40%

Demand Paging

Tune for Sort, Logging, P-fetch

orLock problem

SQL analysis and/or application design consult

Excessive DB2 or OS System /User CPU

Yes

No

Page 13: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Sort IssuesSort Issues

SQL containing GROUP BY, ORDER BY, or DISTINCT will result in a sortSQL containing GROUP BY, ORDER BY, or DISTINCT will result in a sort

if no index is available to order the result set. Also in prep for mergeif no index is available to order the result set. Also in prep for merge

join. To identify sort problems, look for the following:join. To identify sort problems, look for the following:

– Sort overflows are nominally < 3% (Total Overflows/Total Sorts * 100)Sort overflows are nominally < 3% (Total Overflows/Total Sorts * 100)– More than minimal post-threshold sorts or rejected piped sortsMore than minimal post-threshold sorts or rejected piped sorts– Sort heap high water mark > SHEAPTHRESH (post-threshold sorts)Sort heap high water mark > SHEAPTHRESH (post-threshold sorts)– The # of active sorts at a given time is greater than x2 ( or The # of active sorts at a given time is greater than x2 ( or 33 σσ) baseline) baseline– Any large or significant small hash join overflows, any hash loopsAny large or significant small hash join overflows, any hash loops– Rows selected >> rows will generally causes ^ sorts Rows selected >> rows will generally causes ^ sorts need for indexes need for indexes– > 5 sorts per statement (Total sorts/Total Statements) > 5 sorts per statement (Total sorts/Total Statements) Problem SQL Problem SQL

Page 14: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Adjusting Sort ParametersAdjusting Sort Parameters

Increasing Sort Heap/Threshold (remember this is treating the symptom!):Increasing Sort Heap/Threshold (remember this is treating the symptom!):

– may allow for larger in memory sorts preventing disk spillsmay allow for larger in memory sorts preventing disk spills– fewer sort passes/merge joins and may also cut down overall exec timefewer sort passes/merge joins and may also cut down overall exec time– same benefits for hash joins, dynamic bit index AND’ing, & star joinssame benefits for hash joins, dynamic bit index AND’ing, & star joins

Caveats: large sort memory may cause excessive paging, in memory sort ratherCaveats: large sort memory may cause excessive paging, in memory sort ratherthan index scan, and sort space may come at the expense of bufferpool space than index scan, and sort space may come at the expense of bufferpool space

Rule Of Thumb:Rule Of Thumb:

– Calculate average sort space = (sort_heap allocated/avg. # active sorts) Calculate average sort space = (sort_heap allocated/avg. # active sorts) use as baseline sortheap then adjust to minimize overflows, pipe use as baseline sortheap then adjust to minimize overflows, pipe rejects, hash overflowsrejects, hash overflows

– Sheapthresh = max(avg # appls connected, avg. active sorts) * Sheapthresh = max(avg # appls connected, avg. active sorts) * max(sortheap among db’s in instance) adjust to minimize post-thresh max(sortheap among db’s in instance) adjust to minimize post-thresh sorts, hash overflows etc.sorts, hash overflows etc.

Page 15: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Logging Issues Logging Issues

Transaction logging and overhead can represent a large part of I/O timeTransaction logging and overhead can represent a large part of I/O time

in OLTP/mixed environments. To identify logging problems look for:in OLTP/mixed environments. To identify logging problems look for:

– Hot (> 40%) hdisks where logs located (use db2 snaps for raw devices)Hot (> 40%) hdisks where logs located (use db2 snaps for raw devices)– Snapshots showing frequent use of secondary logs, large volume ofSnapshots showing frequent use of secondary logs, large volume of

writes/reads to logs, log high watermark approached, ^ opens/closeswrites/reads to logs, log high watermark approached, ^ opens/closes– Logging in Non-OLTP environment? Check db2diag for ROLLBACK’sLogging in Non-OLTP environment? Check db2diag for ROLLBACK’s

Potential adjustments/remediesPotential adjustments/remedies::

– Increase size & # of primary logs, SOFTMAX, MINCOMMIT, LOGBUFSIZEIncrease size & # of primary logs, SOFTMAX, MINCOMMIT, LOGBUFSIZE– Relocate logs on faster devices? Relocate logs on faster devices? – Design options: compound SQL, group > 5 statements into Stored Proc.Design options: compound SQL, group > 5 statements into Stored Proc.

Page 16: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Prefetch Wait Time – Refining the CausePrefetch Wait Time – Refining the Cause

To determine why prefetch is not happening effectively examine To determine why prefetch is not happening effectively examine (for both tablespaces and individual tables):(for both tablespaces and individual tables):

Asynch Read PctAsynch Read Pct: Asynch Reads/Buffer Pool Physical Reads: Asynch Reads/Buffer Pool Physical Reads Asynch Read RatioAsynch Read Ratio: Asynch Pages Read/Logical Reads : Asynch Pages Read/Logical Reads Asynch Pages Per Req (APPR)Asynch Pages Per Req (APPR): Asynch Pages Read/Asynch : Asynch Pages Read/Asynch

RequestsRequests Asynch Read TimeAsynch Read Time: Asynch Read time in ms/Overall Read : Asynch Read time in ms/Overall Read

Time msTime ms Number of unread prefetch pagesNumber of unread prefetch pages Is wait caused by an underlying I/O bandwidth limitation?Is wait caused by an underlying I/O bandwidth limitation? Bufferpool configurationBufferpool configuration

Purpose of calculations is to determine:Purpose of calculations is to determine:

Is prefetch taking place on correct tables/tablespaces? Is prefetch taking place on correct tables/tablespaces? In large enough chunks? In large enough chunks? In proper proportion to overall logical and synchronous I/O? In proper proportion to overall logical and synchronous I/O? Without being overwritten by other pfetchers?Without being overwritten by other pfetchers?

Page 17: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Excessive Prefetch Time – Possible RemediesExcessive Prefetch Time – Possible Remedies

Check db/dbm configuration and architecture:Check db/dbm configuration and architecture:

– DB2_PARALLEL_IO=* regvar, SEQDETECT=Y, # of IO_SERVERSDB2_PARALLEL_IO=* regvar, SEQDETECT=Y, # of IO_SERVERS– Physical parameters - page/extent/prefetch sized/RAID strip sizePhysical parameters - page/extent/prefetch sized/RAID strip size

Pfetch size = n * [# devices in array] * extent size where n Pfetch size = n * [# devices in array] * extent size where n ЄЄ 1,2,3… 1,2,3…

Incremental improvement may be achieved by:Incremental improvement may be achieved by:

– Implementing Big-Block I/OImplementing Big-Block I/O– Adjust the number and size of prefetch queues Adjust the number and size of prefetch queues – bp’s only need to be sized to provide for effective pfetchbp’s only need to be sized to provide for effective pfetch– Need separate bp’s for tables w/high ARR’s or contention will occurNeed separate bp’s for tables w/high ARR’s or contention will occur– Table/jfs reorganization - Pre-fetch advantage is offset by fragmented Table/jfs reorganization - Pre-fetch advantage is offset by fragmented

DB2/OS data check contiguity at OS level and w/DB2 REORGCHKDB2/OS data check contiguity at OS level and w/DB2 REORGCHK– I/O bandwidth limiting factor? Adequate parallelism with existingI/O bandwidth limiting factor? Adequate parallelism with existing devices? devices? Solution: map to more containers or more/faster devices. Solution: map to more containers or more/faster devices.

Page 18: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Detection & Classifying of Locking IssuesDetection & Classifying of Locking Issues

To detect and diagnose locking problems, look for:To detect and diagnose locking problems, look for:

Snapshots show significant Snapshots show significant Time Spent Waiting on Time Spent Waiting on LocksLocks (compared to overall execution time) (compared to overall execution time)

Snapshots show Snapshots show lock escalationslock escalations

LocklistLocklist High Water Mark High Water Mark > 50% of total space> 50% of total space

db2diag.log shows repeated db2diag.log shows repeated SQL0911NSQL0911N (rc 1 or 2) (rc 1 or 2) meaning Lock timeouts or deadlocks detected and meaning Lock timeouts or deadlocks detected and resolvedresolved

Look for ROLLBACK’s in diaglog as well Look for ROLLBACK’s in diaglog as well deadlocks deadlocks

Page 19: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Locking Problem MitigationLocking Problem Mitigation

Locking is fundamentally an application logic issue. That said, recourse Locking is fundamentally an application logic issue. That said, recourse on db side includes:on db side includes:

– Insure LOCKTIMEOUT changed from default -1 to e.g. 10000Insure LOCKTIMEOUT changed from default -1 to e.g. 10000– Set LOCKTABLE for heavily updated tables Set LOCKTABLE for heavily updated tables – Provide sufficient LOCKLIST space to prevent lock escalationProvide sufficient LOCKLIST space to prevent lock escalation– Tune db parameters MAXLOCKS, LOCKTIMEOUT & DLCHKTIMETune db parameters MAXLOCKS, LOCKTIMEOUT & DLCHKTIME– Set registry variables LOCK_TO_RB , KEEPTABLELOCK, Set registry variables LOCK_TO_RB , KEEPTABLELOCK,

MAX_NON_TABLE_LOCKS, DB2_EVALUNCOMMITTED, DB2_SKIPDELETED and MAX_NON_TABLE_LOCKS, DB2_EVALUNCOMMITTED, DB2_SKIPDELETED and DB2_SKIPINSERTEDDB2_SKIPINSERTED

– Type II indexes ^ concurrency if migrated from v7, Use REORG to convert Type II indexes ^ concurrency if migrated from v7, Use REORG to convert

Application development principles:Application development principles:

– Frequent of COMMIT’s (use event monitor or db2diag.log to track)Frequent of COMMIT’s (use event monitor or db2diag.log to track)– Specifying FOR FETCH cursors whenever possibleSpecifying FOR FETCH cursors whenever possible– Lowest isolation level for entire application or single statement Lowest isolation level for entire application or single statement

Page 20: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

System or User CPUSystem or User CPU

User CPU is often a result of waiting on locks or sorts. If that’s still theUser CPU is often a result of waiting on locks or sorts. If that’s still the

case after after previous steps, application design consult is required.case after after previous steps, application design consult is required.

Barring that, the need is to identify processes or particular SQL Barring that, the need is to identify processes or particular SQL

statement consuming CPU cycles: statement consuming CPU cycles:

– Use ps -elf to identify processes at OS levelUse ps -elf to identify processes at OS level– List applications show detail for connections within DB2List applications show detail for connections within DB2– Event monitors to isolate apps/SQL using extensive CPU resources Event monitors to isolate apps/SQL using extensive CPU resources

Stack trace, explained in the DB2 Problem Determination Guide’s, is Stack trace, explained in the DB2 Problem Determination Guide’s, is

the ultimate means of identifying process (and time) flow within DB2. the ultimate means of identifying process (and time) flow within DB2.

Page 21: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Paging Issues – Paging Issues –

AIX VMM allows system to assign more memory to a process than AIX VMM allows system to assign more memory to a process than

physically exists. Properly tuned DB2 system on dedicated server physically exists. Properly tuned DB2 system on dedicated server should should

incur NO paging whatsoever. Page faults/sec > 50 or > 10-20 pi/po incur NO paging whatsoever. Page faults/sec > 50 or > 10-20 pi/po problem.problem.

Paging means memory is overcommitted – common sources in 32-bit envirnoments:Paging means memory is overcommitted – common sources in 32-bit envirnoments:- Over allocation of bufferpools or sortheapOver allocation of bufferpools or sortheap- Database memory allocated based on avg. # applicationsDatabase memory allocated based on avg. # applications- Private memory allocated by large number of connectionsPrivate memory allocated by large number of connections

Remedies:Remedies:- Identify and release overcommitted memory (use db2mtrk/visualizer)Identify and release overcommitted memory (use db2mtrk/visualizer)- AIX minperm and maxperm parametersAIX minperm and maxperm parameters- Paging space on disk x2 physical memoryPaging space on disk x2 physical memory- Pin bufferpools in memory using DB2_PINNED_BPPin bufferpools in memory using DB2_PINNED_BP- Release agent private memory using DB2 regvars MEMDISCLAIM/MAXFREERelease agent private memory using DB2 regvars MEMDISCLAIM/MAXFREE- Remember that Intra-parallelism and FCM take up 2 segments (i.e. – less for Remember that Intra-parallelism and FCM take up 2 segments (i.e. – less for

BP’s)BP’s)

Page 22: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Whatever is left is I/O, right?Whatever is left is I/O, right?

Sorta-kinda-maybe. I/O efficiency can be addressed within context ofSorta-kinda-maybe. I/O efficiency can be addressed within context ofexisting configuration But, later we’ll need consider performance drivers existing configuration But, later we’ll need consider performance drivers inherent in design.inherent in design.

Checklist of common opportunities for improving I/O performance: Checklist of common opportunities for improving I/O performance:

◊ ◊ Parallel I/OParallel I/O – Registry variables, num_ioservers, adequate # containers (hdisks) – Registry variables, num_ioservers, adequate # containers (hdisks)◊ ◊ Turn off Turn off Memory Mapped I/OMemory Mapped I/O (MMAP_R/W) but beware of i-node latching (MMAP_R/W) but beware of i-node latching◊ ◊ Consider Consider DMSDMS RAW DEVICERAW DEVICE containers. DMS file are worst of both worldscontainers. DMS file are worst of both worlds◊ ◊ Examine behavior of Examine behavior of HASH JOIN’sHASH JOIN’s – may degrade performance wo/resources – may degrade performance wo/resources◊ ◊ RUNSTATSRUNSTATS and and REORG’sREORG’s – automate! Don’t leave to discretion of overtaxed DBA! – automate! Don’t leave to discretion of overtaxed DBA!

◊ ◊ OLTPOLTP specifics – consider optimization 2/3, AVG_APPLS = 1, watch logging closely, specifics – consider optimization 2/3, AVG_APPLS = 1, watch logging closely, CHNGPGS_THRESH 20 – 30%, smaller pages CHNGPGS_THRESH 20 – 30%, smaller pages more efficient I/O, MINFREE and more efficient I/O, MINFREE and PCTFREE to avoid block splitting/overflowsPCTFREE to avoid block splitting/overflows◊ ◊ DSSDSS specifics – consider INTRAPARALLEL, optimization > 5? AA=connections, specifics – consider INTRAPARALLEL, optimization > 5? AA=connections, watch sorts closesly, 32 K pages? (but remember only 255 rows per watch sorts closesly, 32 K pages? (but remember only 255 rows per page), DLCHKTIME ^, page), DLCHKTIME ^, large result sets - rqioblock to 64 Klarge result sets - rqioblock to 64 K

Page 23: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

Laundry List of I/O issues (continued)Laundry List of I/O issues (continued)

ESSESS: stripe size is 64 K, stripe ts containers across LUN’s, keep in : stripe size is 64 K, stripe ts containers across LUN’s, keep in mind mind

LUN’s can be on same rank – (same array), do not duplicate RAID at LUN’s can be on same rank – (same array), do not duplicate RAID at OSOS

FAStTFAStT: stripe size configurable from 8 – 256 K, multiple RAID configs : stripe size configurable from 8 – 256 K, multiple RAID configs

Less prevalent, but no less debilitating architectural/configuration Less prevalent, but no less debilitating architectural/configuration issues:issues:

– Page size correct for row size and access method?Page size correct for row size and access method?– Unless there is a distinct advantage, (synch/asych I/O) fewer bufferpoolsUnless there is a distinct advantage, (synch/asych I/O) fewer bufferpools– Benchmark index/data separation – not always advantageousBenchmark index/data separation – not always advantageous– Pay attention to victim page steals – adjust # of cleaner/thresholdPay attention to victim page steals – adjust # of cleaner/threshold– CPU Speed -1, let optimizer decideCPU Speed -1, let optimizer decide– File open/close overhead File open/close overhead MAXFILEOPEN/ulimits settings MAXFILEOPEN/ulimits settings

Page 24: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

AIX Rec’s - The good, the bad & the ugly AIX Rec’s - The good, the bad & the ugly

General settings, rules, suggestions, incantations:General settings, rules, suggestions, incantations:

– Use AIX 5.2 with latest maintenance release if possibleUse AIX 5.2 with latest maintenance release if possible– Use 64 bit DB2 with 64 bit kernel, 32 bit / 32 bit, recompile & rebindUse 64 bit DB2 with 64 bit kernel, 32 bit / 32 bit, recompile & rebind SP’s etc. when migration to 64 bit takes place, point to 64 bit librariesSP’s etc. when migration to 64 bit takes place, point to 64 bit libraries– Steady state load of hardware config (CPU, Memory, I/O) should use noSteady state load of hardware config (CPU, Memory, I/O) should use no more than 70% of available resources to allow for peak loads/unseen overheadmore than 70% of available resources to allow for peak loads/unseen overhead– Number of licensed users and maxuproc set to max DB2 connectionsNumber of licensed users and maxuproc set to max DB2 connections– Maxperm/minperm settings along with DB2 reg vars for memory disclaimMaxperm/minperm settings along with DB2 reg vars for memory disclaim– Maxmemory/minmemory to regulate AIX Maxmemory/minmemory to regulate AIX – Ulimits – maxprocs increased, unlimited for other paramsUlimits – maxprocs increased, unlimited for other params– Make sure asynchronous I/O (aio) is enabled w/adequate # of agentsMake sure asynchronous I/O (aio) is enabled w/adequate # of agents– Paging space at least x2 physical memoryPaging space at least x2 physical memory– If pinning memory using regvar, vmtune tuning parameter ?If pinning memory using regvar, vmtune tuning parameter ?– Disabling MMAP reads/writes in DB2 means jfs cache bybassed…can provide Disabling MMAP reads/writes in DB2 means jfs cache bybassed…can provide

significant I/O improvement, but i-node latches significant I/O improvement, but i-node latches 3 containers 3 containers– Make sure all processors in an SMP environment are enabled! Ask about WLM Make sure all processors in an SMP environment are enabled! Ask about WLM

and DLPAR settings.and DLPAR settings.– Check in smit for LP/PP ratio – is there disk mirroring on top of RAID-5?Check in smit for LP/PP ratio – is there disk mirroring on top of RAID-5?– db2oscnfg – makes rec’s for OS kernel settings in HP-UX and Solarisdb2oscnfg – makes rec’s for OS kernel settings in HP-UX and Solaris

Page 25: Washington Area Informix/DB2 Users Group July 26, 2006 DB2 Performance Tuning: A Practical Approach By Jim Cleveland jimcleve@aol.com Bluepoint Consulting.

SQL Analysis & TipsSQL Analysis & Tips

The Golden Rule for Join PerformanceThe Golden Rule for Join Performance::

– Columns used to join tables or that are used in a group by, order by or distinct Columns used to join tables or that are used in a group by, order by or distinct clause should be indexed to improve performanceclause should be indexed to improve performance

Other useful SQL rules:Other useful SQL rules:

– Optimizer is all about ‘SARGABLE’ predicates (stage 1, indexable, Optimizer is all about ‘SARGABLE’ predicates (stage 1, indexable, – For join with N tables, there must be N-1 relationships defined to avoid a Cartesian For join with N tables, there must be N-1 relationships defined to avoid a Cartesian

product -- Corollary: establishing higher selectivity early in plan.product -- Corollary: establishing higher selectivity early in plan.– All SQL optimization efforts marginalized with outdated statistics or fragged data All SQL optimization efforts marginalized with outdated statistics or fragged data – Use optimize/fetch only rows clause if possibleUse optimize/fetch only rows clause if possible– Test alternate optimization classes for problem SQL (i.e. – 5, 7 & 9)Test alternate optimization classes for problem SQL (i.e. – 5, 7 & 9)– Use singleton select instead of cursor – use routines to identify node in EEE, make Use singleton select instead of cursor – use routines to identify node in EEE, make

cursors READ ONLY where possible cursors READ ONLY where possible non-deletable non-deletable no exclusive locks, BLOCKING no exclusive locks, BLOCKING ALL option alsoALL option also

– Use UR or lowest possible isolation level when possible to avoid waiting on locksUse UR or lowest possible isolation level when possible to avoid waiting on locks– Consider Global Temporary tables – scanning workfile faster sometimes than subqueryConsider Global Temporary tables – scanning workfile faster sometimes than subquery– Run logic as function instead of procedure Run logic as function instead of procedure