1

DB2 Performance Tuning Fundamentals

Berni SchieferIBM Toronto Lab

Session: D01Monday, Oct 24, 2005 @ 11:15 AM

Platform: DB2 UDB for Linux, Unix, Windows

2

2

Agenda

• Basic assumptions• Choosing/sizing a configuration• Building a database• Tuning a database• Performance problem origins

3

3

Before You Start …

4

4

Basic Assumptions

• You are not an expert• You don’t necessarily want to become an expert

• Just to do your job well

• You don’t want to know all the ways• One good way will do

• 80% of optimal performance is sufficient

Data skew is the situation where data is not evenly distributed across the nodes. In this situation, nodes with a heavy data concentration will be a bottleneck.

Data availability at the processor level is an issue if a node goes down. In this case, the data on the node is lost until the administrator recovers it. On a Shared Disk system, no data is lost if the system goes down.

5

5

Performance “Truisms”

• There is always a bottleneck!• Remember the 5 fundamental bottleneck areas:

1. Application2. CPU3. Memory4. Disk5. Network

Balance is key!

6

6

Choosing Your Configuration

7

7

• Ideally one should understand:• The application• Load process requirements• Number of concurrent users/jobs• Largest tables' sizes• Typical query scenarios • Size of answer sets being generated• Response time objectives for loads and queries• Availability requirements• …

Sizing a Configuration

Data skew is the situation where data is not evenly distributed across the nodes. In this situation, nodes with a heavy data concentration will be a bottleneck.

Data availability at the processor level is an issue if a node goes down. In this case, the data on the node is lost until the administrator recovers it. On a Shared Disk system, no data is lost if the system goes down.

8

8

Sizing “Rules of Thumb”

• Platform choice• DPF (database partitioning facility)

• GB/node• See BCU documentation

• Memory• Disk

• Space• Spindles

9

9

Platform Selection

• DB2 is highly optimized for all major platforms• Linux, AIX, Windows, Solaris, HP-UX

• Much more than a performance question• Integration with other systems• Skills / Ease of Use• $$$

• Often more than 1 “good” choice

10

10

Selecting DB2 with and without DPF

• Differences becoming smaller• Function and manageability

• Consider starting with 1-node DPF• Equivalent performance• Facilitates future growth

• DPF is less common for• OLTP,ERP,CRM

• DPF most common for • Business Intelligence

11

11

Memory! How Much Do I Need?• Highly dependent on many factors

• Depends on number of users (connections)• Depends on the query workload• Depends on whether or not other software is sharing the machines

being measured

• These days a 64-bit processor is a sound assumption • But the OS and DB2 may still be 32-bit

• Though we strongly prefer 64-bit throughout

4GB / processor is a good starting point, 8GB is even better

12

12

Disk! How Many GB Do I Need?

• More than you think!• Don’t forget about

• Working storage• Tempspace• Indexes, MQT’s etc.

• But big drives tend to give lots of space• 146GB drives now common

Raw data x 4 (unmirrored)Raw data x 5 (RAID5)Raw data x 8 (RAID10)

13

13

Disk! How Many Spindles Do I Need?• Need to define a balanced system

• Don't want too many small disks (rarely a concern now)• Don't want too few large disks

• Causes I/O bottleneck

• Different kinds of requirements• Latency• Throughput

Configure at least 10 disks for each CPU. More is better. Additional drives for logs.

14

14

Building Your Database

15

15

Physical Database Design

• Create 1 database for each instance• Issue “create database” simply

• Use default tablespace definition• If feasible use the AUTOMATIC STORAGE option

• New with 8.2.2• E.g.. CREATE DATABASE DB2 AUTOMATIC STORAGE YES ON X:

• Augment with “CREATE TABLESPACE” later

Do not simply expand the default tablespace during createdb

-Allows for ease of tuning by instance-Ability to bring one down for maintenance, config changes, etc. We are supporting different fp levels.

16

16

Selecting a Page Size

• Use no more than 2 page sizes• For example, 4K and 16K or 8K and 32K

• OLTP• Smaller page sizes are better (e.g. 4K, 8K)

• DSS• Larger pages sizes often beneficial (e.g. 8K, 16K, 32K)

With 8.2.2 you can choose any page size as the base page size for the DB

For ease of tuning.

17

17

Tablespace Design

• Ensure temp tablespaces exist• 1 for each page size

• Keep number of tablespaces small• 1 for look up tables in single node nodegroup• 1 for fact table (largest table) • 1 for all others

Spread each tablespace over many/all disks

18

18

Consider Schema optimizations

• Decide on how to structure your data• Consider distributing your data across nodes

• Using DPF hash-partitioning• Consider organizing your data

• Using MDC (multi dimensional clustering)

• Auxiliary data structures• Do the right indexes exist ?

• Clustered, clustering, include columns for unique indexes• Would Materialized query tables (MQT) help?

• Consider using the Design Advisor

19

19

Choosing DMS vs. SMS

• Hard to make a general recommendation• DMS raw can be fastest• DMS file is now the strategic choice

• Performance near RAW• Especially when bypassing filesystem caching

• Flexibility near SMS (especially with 8.2.2.)• SMS is still the best choice for TEMP tablespace

(unless there are very large temps)• Can use a combination

With 8.2.2, direct I/O (concurrent I/O)applies to temps (SMS and DMS file)

DMS TS can grow by adding more containers (not necessarily of the same size). The data is automatically rebalancedIn DMS, indexes and long data can be separated from "short" data for administrative and performance reasonsIn SMS space is allocated as data is added -- no wasted space caused by pre-allocationBoth DMS and SMS exhibit good performance in Decision Support type of workloadsDMS-raw has the potential for higher performance than SMS or DMS-file, particularly in heavy OLTP environment (lots of inserts/updates, CPU cycles are precious)SMS table spaces are easy to administer as long as they are not large (not close to file or file system limits)DMS table spaces offer a greater flexibility of administration and tuning for performance potential

20

20

Tablespace Tips for RAID Systems• In general, spread “everything over everything”• For DSS workloads

• Choose an EXTENTSIZE that is a multiple of the RAID stripe size (ideally, raid_stripe_size*num_disks_in_stripe=extent_size)

• May want to set DB2_PARALLEL_IO if only 1 container in the tablespace• See registry documentation for details

• For OLTP workloads:• Same layout as DSS, • If you are getting undesired prefetch, consider turning off prefetching on the

tablespaces if number of “prefetched pages read” is low. • Eg. DB2 “alter tablespace <tbname> prefetchsize 0”

A not so noticeable change in FP9 are changes to the DB2_PARALLEL_IOregistry variable. As with prior behavior, when db2_parallel_io=* is specified, the degree of parallelism for a table space is the ratio between the prefetch size and the extent size for the table space. For example, with a table space with a prefetch size of 128 and an extent size of 32, there will be 4 extent-sized prefetch requests issued. However, with FP9 and DB2_PARALLEL_IO set to * and table space prefetchsize set to AUTOMATIC, DB2 will now consider the number of disks per container and factor this into the equation (for table spaces created with a prefetchsize set to AUTOMATIC, the default of 6 disks per container will be used). For example, with DB2_PARALLEL_IO set to * and for table spaces with prefetch size set to AUTOMATIC and residing on RAID devices, DB2 will calculate the degree of I/O parallelism as follows:Prefetch size = (number of containers) * (number of disks per container) * (extent size)Using our previous example, DB2 will plug-in the number of disks per container into the formula and the prefetch size will be 768. The degree of parallelism will now be 24 (4 * 6 * 32) = 768, 768 / 32 (extent size) = 24.The number of disks per container can be overridden by specifying a : after the table space container id in the DB2_PARALLEL_IO setting as follows: DB2_PARALLEL_IO=*,1:4. Using this example, all table spaces in the database will use 6 as the default except for the table space with id 1, which will use 4 disks per container.

(Write-up from Phil Gunning’s DB2 Zone -http://www.gunningts.com/db2zone.htm)

21

21

Tablespace Tips for RAID Systems (2)

• Typically the RAID controllers/Storage Managers have caches which can selectively enable read/write cache.• For DSS:

• Consider enabling read cache along with read ahead functionality this may help prefetching

• Consider enabling write cache for temps • For OLTP:

• Consider disabling both read/write cache if the amount of data is large and the access is truly random.

• Consider enabling read cache if data set is small and memory is limited (some caches may be 2GB or more in size)

It pays to understand what happens under the coversin a RAID subysystem.

22

22

Table Design

• OK to have multiple tables in a tablespace• Once defined, use ALTER table to select options

• APPEND MODE - use for tables where inserts are at end of table (ALTER TABLE ... APPEND ON)

• This also enables concurrent append points for high concurrent INSERT activity• LOCKSIZE - use to select table level locking (ALTER TABLE ... LOCKSIZE

TABLE)• PCTFREE - use to reserve space during load/reorg (ALTER TABLE ...PCTFREE 10)

• Add constraints after index creation

When loading a table usefastparse, totalfreespace

23

23

Index Design

• In general, every table should have at least 1 index• Choose appropriate options

• PCTFREE - should be 0 for read-only table• PAGE SPLIT HIGH/LOW – for ascending inserts especially• CLUSTER - define a clustering index• INCLUDE columns - extra cols in unique index for index-only access• ALLOW REVERSE SCANS - make indexes bi-directional

• CREATE INDEX ... ON ... (CUSTKEY) INCLUDE (C_ADDRESS)• COLLECT STATISTICS …

NEW in 8.2.2 you can COLLECT STATISTICSon indexes on global temporary tables

- In the create index command, create index on DGTT collect statistics

24

24

Design Advisor

• When to use indexes/MDC/MQT/ partitioning• The decision may be complex, but DB2 takes care of it

automatically. The design advisor recommends any combination of the features above !

• Available• Through the control center• Command Line tool

Design advisor is significantly enhanced in v8.2

25

25

Tuning Your Database

26

26

Tuning “Knobs”

performance smartguideconfiguration parametersregistry settings

The "magic" behind optimal performance ...

27

27

Operating System Tuning

• Ensure current with maintenance• Paging space

• Place on 2 or more separate disks • Should be 2x real memory

• Very controversial with OS people

• vmtune minperm/maxperm options• Worth tuning/understanding if using SMS (AIX)

• New options in more recent maintenance levels (e.g strict_max_client)

Use vmstat/iostat regularly!(on Linux they work better on 2.6 kernels)

28

28

Configuration Advisor

• Suggests a good starter set of configuration parameters• Database and Database Manager

• Available• Through the control center• An API call• Command Line Processor (CLP)

• DB2 AUTOCONFIGURE … APPLY DB AND DBM

An ideal place to start!

29

29

Database Manager Config Parms• INTRA_PARALLEL

• In general leave as NO• ONLY set to YES if low CPU utilization and purely complex SQL environment

• NUM_INITAGENTS / NUM_POOLAGENTS• set to average number of expected concurrent users, with INITAGENTS a bit lower

• SHEAPTHRES• depends on number of sort-intensive concurrent users• usually 10x-100x the SORTHEAP (change in conjunction with SORTHEAP)

• RQRIOBLK and ASLHEAPSZ• Increase for large answer sets

The snapshot monitor is effective at monitoring many of these

30

30

Database Configuration Parameters• DFT_DEGREE

• typically set this to -1 (ANY)• DFT_QUERYOPT

• Default 5, often 2 or 7 are good alternatives• APPLHEAPSZ, PCKCACHESZ

• Increase• SEQDETECT

• In general, leave this at the default of YES

SET CURRENT DEGREE and the DEGREE bind option will determine the desired parallelism at compile time

31

31

For a complete picture, combine DB2 monitoring and OS monitoring

BUFFERPOOLs• single most important tuning parameter• NEVER leave as default• normally, use only a few BUFFERPOOLs

• 2 for each page size (1 temp, 1 all else) • Ensure you check SYSBUFFERPOOLS not buffpage

• SOFTMAX• Controls dirty page writes• Workload dependent and affects performance/recovery time

• lower value = shorter recovery time = potentially more writes• higher value = longer recovery time = less writes = potentially better perf

32

32

SORTHEAP

• Critical to distinguish between OLTP and DSS• OLTP

• Make this small, especially with high concurrency• Remember! This influences optimizer plan

• DSS• Applies to both private and shared (SMP) sorts• Sort / hash join / table in memory

Watch key values in monitor output

33

33

Use DB2 and OS tools to watch logging activity

Logging• Database logs need to be on their own disks• Must be durable i.e.. mirrored, with battery-backed write cache• LOGFILSIZ

• Increase beyond default, e.G. 5000 4K pages or more• LOGPRIMARY

• Allocate all logs as primary logs• Use LOGSECOND for "emergency" space only

• LOGBUFSZ• Increase to 512 or greater (only useful if “Log pages read” is high)

• MINCOMMIT• Keep this at default of 1, very very rarely higher than 2-3

34

34

Use the advice in the Administration guide to help you configure your database

I/O Configuration Parameters

• NUM_IOSERVERS• one for each disk (or per raid array) with database data, limit to 2 x #CPUs

• NUM_IOCLEANERS• one for each CPU

• CHNGPGS_THRESH• lowering from default may help

• e.g. 40

• Worth considering DB2_USE_ALTERNATE_PAGE_CLEANING

DB2_USE_ALTERNATE_PAGE_CLEANING: Specifies whether DB2 uses the alternate method of page cleaning algorithms instead of the default method of page cleaning. When this variable is set to "ON," DB2 uses a proactive method of page cleaning, writing changed pages to disk, keeping ahead of LSN_GAP, and proactively finding victims. Doing this allows the page cleaners to better utilize available disk I/O bandwidth. When this variable is set to "ON," the chngpgs_thresh database configuration parameter is no longer relevant because it does not control page cleaner activity.

35

35

I/O Configuration Parameters (2)• File system caching may impact the performance of

SMS/DMS file tablespaces• If you are on a 64bit DB2 and have more than 3GB of memory, let the

bufferpools do the caching and turn off the file system cache• If you are on 32 bit DB2 and have more than 3GB of memory,

consider enabling the filesystem cache for tablespaces which caches often-used rows. (i.e.. Use the file system cache as an “extension” of db2 bufferpool caching)

• Remember LOBS are not cached in the bufferpool, so cache them inthe file system cache if possible

New in v8.2 you can control the use of the file system cache on a per tablespace ex.OFF:ALTER TABLESPACE <tablespace name> ... NO FILE SYSTEM CACHINGON:ALTER TABLESPACE <tablespace name> ... FILE SYSTEM CACHING

36

36

Check README files in each fixpack for new features

Registry Settings

• Typically used to enable• brand new features (not yet in main documentation)• unusual/rare features

• New Features• DB2LINUXAIO=true (8.2.2)

• Rare and unusual Features (use with caution)• DB2_SKIPINSERTED (8.2.2)• DB2_SKIPDELETED• DB2_EVALUNCOMMTTED

DB2_SKIPINSERTED registry variable You can use the DB2_SKIPINSERTED registry variable to skip uncommitted inserted rows for Cursor Stability (CS) and Read Stability (RS) isolation levels. 9 The registry variables DB2_SKIPDELETED and DB2_EVALUNCOMMITTEDare used 9 to skip uncommitted deletions and uncommitted updates. _that_do_no_qualify_the_predicate_.“ Predicate being the predicate that was pushed e.g. into the table scan. Otherwise, CS and RS 9 isolation levels require the processing of committed data only. 9 If you decide that you can skip any row that is locked because it is an 9 uncommitted inserted row, you can now turn the DB2_SKIPINSERTED registry variable 9 on to allow you to skip those rows. Having this registry variable on produces 9 greater concurrency and would therefore be the preferred choice for most applications. There are cases where skipping uncommitted inserts may not be preferred. 9 For example: When two applications use a table to pass data between them 9 When an application does not use UPDATE statements, but instead deletes 9 the old data and then inserts the new data.

37

37

Performance Problem Origins

382

38

Top 10 Reasons for Poor Performance

• What they are• Based on real life customer problems

• How to diagnose• Using DB2 and OS tools

• How to fix• Change in H/W• Change in S/W• Tuning DB2

39

39

#1: Too few disks• Remember to balance I/O and CPU• Databases are I/O intensive

• Random I/O and sequential I/O• Can be detected by:

• High I/O wait, high disk utilization• System works well at low utilization, keels over under load

• You can compute • Random I/O’s (io’s/txn * #txn’s/sec = io’s/sec)• Seq. I/O ( MB/sec )

• Can be fixed by:• Adding/reconfiguring disk

• This can be especially important with large SAN environments

40

40

#2: Poor Bufferpool Configuration

• Remember to exploit your real memory well• Beware of defaults

• DB configuration parameter “Buffpage” only applies if BUFFPOOL size is –1 in the catalog

• Can be detected by:• Memory underutilized• Disks busier than expected• Monitoring Table Level I/O

• Can be fixed by:• Reviewing assignment of tablespaces to bufferpools

• Sometimes you may have unused bufferpools• Increasing bufferpool sizes

41

41

#3: Logs poorly configured• Logs are pure sequential write

• Funnel for all transactions• Both logical and physical characteristics matter• Can be detected by:

• Looking for a “hot” disk• Unexpectedly poor response time

• Can be fixed by:• Ensuring log is isolated, write-cache supported• Configure logs for maximum performance

• Use a dedicated controller, use a reasonable number of dedicated log disks• In exceptional circumstances you can use raw logs or solid-state disks

42

42

#4: SORTHEAP too large for OLTP

• Large sortheap encourages sorts• Beware of scaling users up with large sortheaps• Can be detected by:

• Heavy memory consumption, paging• Heavy temp usage• Large number of sorts, long sort time in monitor

• Can be fixed by:• Reducing sortheap size, increase sheapthres• Tuning SQL with large sort requirements

43

43

#5: Tempspace poorly configured

• Important to tune for both DSS and errant OLTP queries• Can be detected by:

• Heavy utilization of disks for temp tablespace• Tablespace full with multiple temp tablespaces

• Can be fixed by:• 1 temp tablespace per page size only• spread over large number of disks

• Often over the same disks that you use for data/indexes• This is the everything over everything philosophy

44

44

#6: Missing/Out-of-Date Statistics• Sophisticated optimizer needs current statistics (even for small tables)

• Keep in mind the “granularity” you can select• Can be detected by:

• Long-running queries• Curious access plans

• Can be fixed by:• Runstats on all tables• Selecting from detailed and distribution stats• Increasing num_quantiles and num_freqvalues

• Automatic Runstats in 8.2

Reorgcheck can issue runstats for all tables

45

45

#7: Poorly Tuned SQL

• The better the SQL the easier it is to evaluate• SQL is very powerful, use the power

• Can be detected by:• Obtaining dynamic SQL snapshots• Finding long-running SQL statements• Compare rows read to rows returned• Further refinement in the application

• Can be fixed by:• Specify exactly what you need/prefer

• Limit columns and rows in the SELECT• Use the Design advisor

E.g., instead of creating temporary tables on the fly (which cost catalog inserts), try using common table expressions or global declared temporary tables at least.

46

46

#8: Concurrency Problems• Many concurrent users require consideration• Can be detected by:

• Long lock waits• Lock timeouts• Poor throughput

• Can be fixed by:• Selecting correct isolation level • Using TYPE2 indexes (default with 8.1)• Increase locklist (sometimes maxlocks)• Set locktimeout to something other than –1• Shorter transactions

DB2_SKIPINSERTED with 8.2.2

47

47

#9: Naive CLI/Java applications• There are a myriad of ways to write CLI/Java applications

• Not all are optimal• Can be detected by:

• Poor hits in the package cache• High client CPU/memory consumption• High network latency• CLI trace• JDBC Tracing implemented by all DB2 data sources

(DB2SimpleDataSource,DB2DataSource,DB2ConnectionPoolDataSource,cc.DB2XADataSource)• Eg: dataSource.setTraceFile(“<path to an existing directory>");

dataSource.setTraceLevel(<trace level>);• Can be fixed by:

• Use of parameter markers in CLI/Java

• SQLJ for static complex SQL statements to avoid prepare time• Increase in package cache size• Maintaining connection pool• Turning off AutoCommit in your Java application and programmatically control the commits

-Auto-commit is a common pitfall for most applications, especially for applications with many consecutive updates and inserts -Using the JDBC batch interface is like growing a bunch updates/ inserts into 1 drdaflow and send it over. On the engine side, each request is still processed separately, unlike array insert or update-If you’re fetching a string, the use getString method to get the data out from the result set and into the application, If you’re fetching a timestamp, use getTimeStamp and not getString. Works functionally, but poor performance!-SQLj (static) to avoid reprepared of complex SQLs-Use parameter markers instead of string literals-For any of the Universal Drivers (type-2 and type-4), we recommend using the DataSource class instead of DriverManager class. Many have asked how trace should be turned on. Here is a snippet of code to do so.

-There is also setTraceFile(“<a file name>”) where all connections will write to the same file.-7 different traceLevels of which you can choose from to control how much information you want to write to the trace.

48

48

#10 Page cleaning/Bufferpools

• Bufferpools have finite amount of memory • sometimes need to tune for the workload

• Can be detected by:• Poor hit ratio• PageWrites-AsyncPageWrites is high (a.k.a page steals/synch writes)

• Can be fixed by:• Increasing the # of cleaners• Adjusting CHNGPGS_THRESH• May need to create separate bufferpools for data and indexes

49

49

Runners-up

• Incorrect use of stored procedures• Too few: too much network traffic / latency• Too much: only 1 statement / procedure• Wrong implementation language

• Overuse of UDF’s• Even trusted UDF’s incur overhead• Consider DB2 built-in functions

• Overuse of lobs, LONG VARCHAR• Consider VARCHAR(32K) with a 32K page

• Mincommit set too high• Should essentially always be left at default of 1!

50

50

Summary

• Early planning will help ensure good performance• Important to build a balanced system

• Observe good design/implementation principles • Look for opportunities to exploit latest DB2 features

• Lots of new goodies in DB2 Version 8.2, including 8.2.2• If you do encounter performance issues

• Don’t Panic!• Look for matches with top 10 list!

• Excellent performance doesn’t require tuning wizards• Watch for more even more wizardry coming in Viper next year!

• Especially self-tuning memory management

51

51

DB2 Performs!

52

52

DB2 Performance Tuning FundamentalsSession: D01

Berni SchieferIBM Toronto Labschiefer@ca.ibm.com