Post on 14-Apr-2018
7/27/2019 Runstats and Reorg
1/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 1
Runstats
and
Reorganization
7/27/2019 Runstats and Reorg
2/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 2
Table of Contents
Why and When to Collect Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Scheduling Runstats, Reorg, & Rebind . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Catalog Tables Updated by RUNSTATS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Collecting Statistics on Non-Indexed Columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
When to Reorganize Using the Catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Online REORG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Incremental Update of Shadow Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
REORG with SORTKEYS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
REORG with Inline COPY and STATISTICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Unloading Rows Using REORG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
REORG INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Appendix A: Reorg Catalog & Directory Tablespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7/27/2019 Runstats and Reorg
3/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 3
Why and When to Collect Statistics
Why collect statistics
Performance improvements for static, dynamic SQL, and reoptimization< Particular important for changing cluster ratios, skewed distribution of values, low
cardinality columns, and range predicates
< Non-indexed column statistics are used for:
S Local predicates in determining the join method and the inner and outer table of thejoin
SNon-leading columns of a composite index when equal predicates are not specified onall columns and more cases expected in future
Provide the optimizer with current statistics for generating an efficient access path Monitor and tune database, plans and packages
< Determine reorganization needs, space management, current documentation
When to collect statistics After a table and indexes have been created and the data loaded After a tablespace or index has been reorganized and statistics have changed significantly After heavy insert, update, and delete activity Before binding and rebinding plans and packages Before monitoring catalog tables
When to collect statistics with utility processing (V6)
During LOAD processing During REORG processing when needed and if additional costs are acceptable During REBUILD INDEX processing if needed Can specify most RUNSTATS parameters to these utilities
7/27/2019 Runstats and Reorg
4/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 4
Timings of a SELECT Before/After RUNSTATS (V1.2)
Timings
3090/200 CPU BeforeRUNSTATS
AfterRUNSTATS
CPU Time > 20 Minutes 1.29 Seconds
Elapsed Time > 75 Minutes 9.14 Seconds
List all customers who do not have accounts
Customer table has 1,100 rows< Unique index on primary key of CUST NO
Account table has 42,000 rows< Index on foreign key of CUST NO
SELECT CT.CUST NO, CUST NAMEFROM CUST TAB CT
WHERE NOT EXISTS(SELECT AT.CUST NOFROM ACCT TAB ATWHERE CT.CUST NO = AT.CUST NO);
7/27/2019 Runstats and Reorg
5/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 5
Default Statistics Used by the Optimizer
If RUNSTATS has not been executed,< Default statistics are used by the optimizer
SYSIBM.SYSTABLES
CARDF 10,000 rows
NPAGESF 501 = CEILING(1 + CARDF / 20)
PCTROWCOMP 0
EDPROC blank = EDITPROC not used
SYSIBM.SYSINDEXES
CLUSTER-RATIOF
0 (95 % IF CLUSTERED = Y)
CLUSTERED Y if CLUSTER used when index was created
FIRSTKEY-CARDF
25
FULLKEY-CARDF
25
NLEAF 33 = CARDF / 300
NLEVELS 2
SYSIBM.SYSCOLUMNS
COLCARDF 25FOREIGNKEY B for SBCS (Single Byte Character Set)
SYSIBM.SYSTABLESPACE
NACTIVEF 501 = CEILING(1 + CARDF / 20)
Uniform distribution of data assumed
Same default statistics are used for the "STATS" tables with partitioned tablespaces to be discussed
7/27/2019 Runstats and Reorg
6/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 6
Catalog Tables in DSNDB06 and SYSHIST Databases
DSNDB06 database contains the catalog tables used by DB2
SYSHIST database can contain a history of most statistics collected by RUNSTATS if requestedwith the HISTORY parameter (V7)
Can accumulate a history of statistics over time in:< SYSIBM.SYSCOLDIST_HIST
< SYSIBM.SYSCOLUMNS_HIST< SYSIBM.SYSINDEXES_HIST
< SYSIBM.SYSINDEXPART_HIST
< SYSIBM.SYSINDEXSTATS_HIST
< SYSIBM.SYSLOBSTATS_HIST
< SYSIBM.SYSTABLEPART_HIST
< SYSIBM.SYSTABLES_HIST< SYSIBM.SYSTABSTATS_HIST
Can insert, update, and delete the history catalog tables
MODIFY STATISTICS utility can be used to clear out the history statistics
7/27/2019 Runstats and Reorg
7/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 7
Scheduling Runstats, Reorg, & Rebind
Avoid using a divining rod
Execute RUNSTATS with the HISTORY parameter periodically, for example,< Daily, weekly, monthly, etc.
Collect statistics inline using LOAD and REORG< Good alternative
Analyze changes in statistics over time and update cycles
VOLATILE can be specified when table is created or added with alter (V8)< Optimizer is very likely to choose to use an index
< Useful for tables that vary in size and other statistics throughout the day, week, etc.
Schedule RUNSTATS, REORG, and rebinds as indicated by analysis
Alternative - Use utility thresholds, develop or purchase software to submit utilities as requiredbased on statistics in the catalog tables
< Beware of automatically submitting utilities taking more time than batch window allows
7/27/2019 Runstats and Reorg
8/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 8
Growth in the Number of Rows and Space
6 Assume history statistics have been collected over a period of time (weeks, months, etc. forexample)< Constant growth rate over time period
< Statistics can be cleared using MODIFY STATISTICS utility and collected over another
period of time
SELECT MAX(CARDF) AS MAXCARDF, MIN(CARDF) AS MINCARDF,
((MAX(CARDF) - MIN(CARDF)) * 100 / MIN(CARDF)AS PGROW,(DAYS(MAX(STATSTIME)) - DAYS(MIN(STATSTIME)) AS NDAYS)
FROM SYSIBM.SYSTABLEPART_HISTWHERE DBNAME = 'DASPJDB'AND TSNAME = 'DASPJTSP';
Alternative - Can do an analysis based on space rather than the number of rows,< Do similar calculations using SPACEF rather than CARDF
7/27/2019 Runstats and Reorg
9/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 9
Estimate Pages Changed
Determine the % of changed pages between 3 full image copies6Assume random changesS Otherwise, analysis suggests executing RUNSTATS more often than neededSELECT C.TIMESTAMP, C.DBNAME, C.TSNAME, C.NPAGESF, C.CPAGESFFROM SYSIBM.SYSCOPY AS C, SYSIBM.SYSTABLESPACE AS TWHERE DBNAME = 'DASPJDB'AND TSNAME = 'DASPJTSP'AND ICTYPE = 'F'
AND C.TIMESTAMP > T.STATSTIME;
Imagecopy
NPAGESF/Integer
CPAGESF/Integer
% Changed % AfterSubtraction
1 1.00E+02100
1.00E+0110
(10/100) * 100 = 10 10
2 1.00E+02100 1.50E+0115 (15/100) * 100 = 15 +(15/100)-((10 %)*(15/100))= 23.5
3 1.00E+02100
5.00E+005
(5/100) * 100 = 5 +( 5/100)-((23.5%)*( 5/100))= 27.3
C.NPAGESF = number of pages in image copy
C.CPAGESF = number of changed pages in image copy
"% Changed" column shows percentage of changed pages since the previous image copy
"% After Subtraction" column shows the formula and results of adjusting for the possibilitythat changes were made at random and the same pages were copied twice
% of changed pages with or without subtractions can be considered in scheduling RUNSTATS,
REORG, and REBIND
7/27/2019 Runstats and Reorg
10/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 10
Scheduling Runstats, Reorg, & Rebind
Executing RUNSTATS has no effect on performance after static plans/packages with
NOREOPT(VARS) have been bound< Static SQL with REOPT(VARS) and dynamic SQL see statistic with each access path
selection
Rebind plans and packages in conjunction with REORG/RUNSTATS if statistics have changed> 20 % and changes are disproportionate, particularly:
< NLEAF, NPAGESF, NACTIVEF< Table, partition, index, and column cardinalities
< CLUSTERRATIOF and FREQUENCYF
< Range of HIGH2KEY and LOW2KEY
< NLEVELS > 2 and increases by >=1
Alternative - analyze changes in paths chosen in test PLAN TABLE to determine if rebinds arenecessary in production
RUNSTATS and rebinds need not be done frequently in most cases
It is worth 1-2-3 days of analysis for a system to
be in production 5-10-15 years< All your ducks will be in a row
STATSTIME column in catalog tables updated by RUNSTATS contains the timestamp of whenstatistics were updated
7/27/2019 Runstats and Reorg
11/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 11
Catalog Tables Updated by RUNSTATS and Used by the Optimizer
SYSIBM.SYSTABLES
CARDF Number of rows in the table from tablespace scan
NPAGESF Number of pages that contain rows of the table fromtablespace scan
PCTROWCOMP Percentage of rows compressed in table from tablespacescan * 100
EDPROC blank = EDITPROC not used
SYSIBM.SYSCOLUMNS
COLCARDF Estimated number of distinct values in column fromtablespace scan or scan of leaf pages if column isindexed
HIGH2KEY First 8 bytes of second highest value in the columncollected like COLCARDF
LOW2KEY First 8 bytes of second lowest value in the columncollected like COLCARDF
SYSIBM.SYSINDEXESCLUSTER-RATIOF
Percentage of rows in clustering sequence from leaf pages* 100. Counted if RID >= RID from previous RID pair andfunction applied
CLUSTERED Y if >=95 % of rows in sequence from CLUSTERRATIOF
FIRSTKEY-CARDF
Number of distinct values of the first key column fromleaf pages
FULLKEY-CARDF Number of distinct values of the full key from leaf pages
NLEAF Number of active leaf pages in the index from leaf pages
NLEVELS Number of levels in the index tree from traversing index
SYSIBM.SYSTABLESPACE
NACTIVEF Number of active pages in the tablespace from HURBA (HighUsed RBA) in VSAM/ICF catalog.
7/27/2019 Runstats and Reorg
12/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 12
Partition Statistics Collected like Nonpartitioned Statistics
SYSIBM.SYSTABSTATS
CARDF~ Number of rows in partition
NPAGESF~ Number of pages that contain rows in the partition
PCTROWCOMP Percentage of rows compressed in the partition * 100
NACTIVE Number of active pages in the partition
SYSIBM.SYSCOLSTATS
COLCARDF~ Estimated number of distinct column values in partition
HIGHKEY*~ First 8 bytes of highest value in the column
HIGH2KEY First 8 bytes of second highest value in the column
LOWKEY*~ First 8 bytes of lowest value in the column
LOW2KEY First 8 bytes of second lowest value in the column
SYSIBM.SYSINDEXSTATS
CLUSTER-RATIOF Percentage of rows that are in clustering sequence
CLUSTERED Y if >=95 % of rows in sequence from CLUSTERRATIOFFIRSTKEY-CARDF
Number of distinct values of the first key column
FULLKEY-CARDF
Number of distinct values of the full key
NLEAF Number of active leaf pages
NLEVELS Number of levels in index
KEYCOUNT Number of index keys/entries in index
IOFACTOR Not used
PREFETCH-FACTOR
Not used
SYSIBM.SYSINDEXPART
LIMITKEY~* The upper range value specified when the partitioningindex is created.
~Used by the optimizer if parallelism is considered*Statistics that cannot be updated with UPDATE statement
7/27/2019 Runstats and Reorg
13/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 13
Frequency and Correlation Statistics
Frequency statistics are used for reoptimization, dynamic SQL, and predicates with literals on
nonpartitioned tablespaces< INDEX parameter is required for statistics
< Can specify number of values (default of 10)
< > 10 COLVALUE has FREQUENCYF < 9 %
< Uniform distribution is assumed for remaining percentage of rows
Correlation statistics provide the number of distinct values for a column group in a compositeindex
SYSIBM.SYSCOLDIST and SYSIBM.SYSCOLDISTSTATS
COLVALUE One of up to "n" values of the first and additional
(optional) key columns that occur most frequently
FREQUENCYF Percentage of rows that contain the value specified in
COLVALUE * 100 from leaf pages
TYPE 'F' for frequency statistics
'C' for correlation statistics
CARDF Number of distinct values for a column group
COLGROUP-
COLNO
Each SMALLINT array element contains the sequence number
of the column in the row for which statistics arecollected
NUMCOLUMNS Number of columns associated with the statistics
Can update, delete, and insert additional rows< RUNSTATS deletes all inserted rows when executed
7/27/2019 Runstats and Reorg
14/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 14
Statistics Updated by RUNSTATS
Used to determine if reorganization is needed and for space management
SYSIBM.SYSTABLEPART
CARDF Number of rows in the table space or partition from
tablespace scan
SPACEF (V7) KB of space as determined by RUNSTATS or STOSPACE
NEARINDREF Number of rows moved to a nearby page (within 2 segments
or 16 pages for a nonsegmented tablespace) due to row
length increase from tablespace scan
FARINDREF Number of rows moved to a faraway page due to row length
increase from tablespace scan
PERCACTIVE Percentage of space occupied by active rows containing
actual data from active tables (not space from deleted
rows). Calculated as active row space bytes /SYSTABLEPART active page bytes.
PERCDROP Percentage of space occupied by rows from dropped tables.
Dropped simple tablespace bytes / SYSTABLEPART active
page bytes.
PAGESAVE Percentage of pages saved in the tablespace or partition
as a result of using ESA compression * 100.
SPACEF (V7) Kilobytes of DASD storage.
DSNUM (V7) Number of data sets for a tablespace or partition.
EXTENTS (V7)Number of data set extents.
7/27/2019 Runstats and Reorg
15/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 15
Statistics Updated by RUNSTATS - continued - 1
SYSIBM.SYSINDEXPART
CARDF Number of rows referenced by the index or index partition
from leaf pages
NEAROFFPOSF Number of times the next row was not in sequence of index
but was in a near page from leaf pages. Counter
incremented by 1 if RID >1 and < cutoff pages away from
last RID sequence or in a previous page from the RID.
FAROFFPOSF Number of times the next row was not in sequence of index
but was in a faraway page from leaf pages. Counter
incremented by 1 if RID > cutoff pages away from last RID
sequence.
LEAFDIST 100 times the average number of pages between successive
leaf pages during scan of leaf pages.
SPACEF (V7) KB of space as determined by RUNSTATS or STOSPACE
DSNUM(V7) Number of data sets for index.EXTENTS (V7)Number of extents in index data set.
PSEUDO_DEL_
ENTRIES (V7)
Number of pseudo deleted index entries.
LEAFNEAR V7 Number of leaf pages out of physical position near by.
LEAFFAR (V7)Number of leaf pages out of physical position far off.
7/27/2019 Runstats and Reorg
16/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 16
Statistics Updated by RUNSTATS - continued - 2
AVGROWLEN in SYSTABLES is the average row length, useful for:< Variable length rows due to VARCHAR columns prior to V8 and compression< Segmented tablespaces with >1 table< UNLOAD space allocation< REORG and LOAD space allocationS Work data setsS Sort space
PCTPAGES in SYSTABLES and SYSTABSTATS is percentage of total pages that containsrows
< NPAGESF/NACTIVE for nonsegmented tablespace< NPAGESF/(Number of segments * SEGSIZE)
7/27/2019 Runstats and Reorg
17/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 17
The RUNSTATS Statement
Update statistics in database DASPJDB tablespace DASPJTSP and all indexes
< Most catalog table columns updated by RUNSTATS contain -1 until it is executed
RUNSTATS TABLESPACE DASPJDB.DASPJTSPINDEX (ALL)SHRLEVEL CHANGEREPORT YES
UPDATE ALL
Statistics collected on all indexes with an index scan after a tablespace scan to collect statisticson data
If only index statistics need to be recalculated,
< A tablespace scan can be avoided by executing RUNSTATS INDEX
RUNSTATS INDEX (AUTHID.SPJX, AUTHID.QTYX)
RUNSTATS can be executed against the catalog tables and indexes< Provides information for the optimizer to choose a good access path for SQL DML
< Statistics are not used for internal manipulation of the catalog tables when objects arecreated, altered, and dropped, plans and packages are bound and freed, privileges are
granted and revoked, and other operations performed on the catalog tables within DB2
7/27/2019 Runstats and Reorg
18/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 18
SHRLEVEL Parameter
SHRLEVEL CHANGE allows others to update data
< Uses UR and a read claim- Ignores locks held by programs while collecting statistics
< Collects statistics on data currently being updated
- Does not adversely effect statistics in most cases
- Serializes on mass delete processing
< Good choice
SHRLEVEL REFERENCE allows others to select data while utility is executing (default)< Share lock taken at table/tablespace level
REPORT YES causes statistics to be written to SYSPRINT in addition to updating catalogtables
< Good choice (REPORT NO is the default)
7/27/2019 Runstats and Reorg
19/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 19
Parameters Indicate Which Statistics to Update
UPDATE ALL updates statistics for space analysis and those used by the optimizer (default)
UPDATE SPACE updates statistics used for space management analysis< Avoids the possibility of losing current access paths with rebinds
UPDATE ACCESSPATH updates statistics used by the optimizer to determine a good accesspath
< Collection of partial statistics reduces the CPU time
UPDATE NONE can be used with REPORT YES< Can review statistics before having RUNSTATS change statistics, for example
HISTORY parameter causes statistics in the SYSHIST database to be updated in addition to
the catalog tables in the DSNDB06 database< Can be used in conjunction with the ALL, SPACE, and ACCESSPATH (not NONE)
parameters
Catalog tables in the DSNDB06 database are updated when tables in the SYSHIST database areupdated
< There may be an option to update only history statistics in the future
7/27/2019 Runstats and Reorg
20/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 20
Update Statistics on a Partition
Can update statistics on a specific partition
RUNSTATS TABLESPACE DASPJDB.DASPJTSPPART 12INDEX (SPJX PART 12)
PART 2 indicates that statistics should be collected for partition 2 of a partitioned tablespace
and the partitioning index
Good choice for specific partitions that are updated frequently< Reduces costs
Advisable to execute RUNSTATS on the entire partitioned tablespace before execution on
individual partitions
Otherwise, tablespace statistics are not updated until statistics have been gathered on allpartitions if part is named
Can name a specific index and its partition
RUNSTATS INDEX (AUTHID.SPJX PART 12)
7/27/2019 Runstats and Reorg
21/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 21
Collecting Statistics on Non-Indexed Columns
COLCARDF, HIGH2KEY, and LOW2KEY can be collected on some or all non-indexed
columns (not the default)< Simulation method used
RUNSTATS TABLESPACE DASPJDB.DASPJTSPTABLE (S) COLUMN (ALL)TABLE (SPJ) COLUMN (PN,JN) -- SN not required
SAMPLE 25INDEX (ALL)SHRLEVEL CHANGEUPDATE ALLREPORT YES
If TABLE parameter is specified,< COLUMN (ALL) is the default
< Can name specific columns (10 is the maximum)
Non-indexed column statistics are used for:< Local predicates in determining the join method and the inner and outer table of the join
< Non-leading columns of a composite index when equal predicates are not specified on allcolumns
< More cases expected in future
7/27/2019 Runstats and Reorg
22/117
7/27/2019 Runstats and Reorg
23/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 23
Sampling Test Results
Total RUNSTATS CPU time for 15 large tables on a 6-way 600J processor tested by IBM
0 % = No collection of non-indexed column statistics
20 complex SELECT rebound with sampling received the same or better access path except:< 1 had a bad access path with 20 % sampling< 2 had a bad access path with 10 % sampling< 5 had a bad access path with 5 % sampling
Do analyze the access paths your SQL before and after sampling
7/27/2019 Runstats and Reorg
24/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 24
Collection of Additional Frequency Statistics
RUNSTATS INDEX (AUTHID.SPJX)
FREQVAL NUMCOLS 1COUNT 15FREQVAL NUMCOLS 2COUNT 25FREQVAL NUMCOLS 3COUNT 35
If FREQVAL is specified,< NUMCOLS must be specified for each group of columns for which frequency statistics are
required and< COUNT indicates the number of frequently occurring values to collect for the group of
columns
Beware of using only< FREQVAL NUMCOLS 3 COUNT 35
< Frequency statistics can be used only if a matching index scan can be used on all 3columns of the composite index
If FREQVAL is not specified,< Frequency statistics are collected on the 1st column of a composite index for the 10 most
frequently occurring values in the column if there is a non-uniform distribution of values
(default)
Also applies to RUNSTATS TABLESPACE
Frequency statistics are particularly important for a skewed distribution of values
7/27/2019 Runstats and Reorg
25/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 25
Collection of Correlation Statistics
RUNSTATS TABLESPACE DASPJDB.DASPJTSP
INDEX (ALL) - Can name an index & partitionKEYCARD
Column correlation statistics are written to CARDF in SYSCOLDIST andSYSCOLDISTSTATSif KEYCARD is specified (not the default)
< 1 row for each column group with TYPE='C'
If a composite index has 4 columns,< CARDF contains the cardinality of columns:
- 1 and 2 concatenated- 1, 2, and 3 concatenated
< FULLKEYCARDF has the cardinality of columns 1, 2, 3, and 4 (all columns in compositeindex)
KEYCARD is not needed if< Composite index has
7/27/2019 Runstats and Reorg
26/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 26
Report on Correlation Statistics
Summarize the correlation and distribution statistics of indexes on the SPJ table
< WHEN condition needed for each column in the composite index with the most columnsminus 1
SELECT NAME, TYPE, INTEGER(CARDF),CASEWHEN NUMCOLUMNS = 2 THEN
HEX(SUBSTR(COLGROUPCOLNO, 1, 4))WHEN NUMCOLUMNS = 3 THEN
HEX(SUBSTR(COLGROUPCOLNO, 1, 6))ELSE NULLEND AS COLGRPCOLNO,CASEWHEN FREQUENCYF -1 THEN
INTEGER(FLOAT(FREQUENCYF) * 100)ELSE -1
END AS FQY, COLVALUEFROM SYSIBM.SYSCOLDIST
WHERE TBNAME = 'SPJ'AND TBOWNER = 'AUTHID'ORDER BY NAME, TYPE, COLGRPCOLNO,
FQY DESC, COLVALUE;
COLGROUPCOLNO contains an array of SMALLINT columns< Each array element contains the sequence number of the column in the row for which
statistics are collected
NUMCOLUMNS is the number of columns for which the statistics are collected
KEYCARD can result in improved access path selection if the columns of a composite index arecorrelated
7/27/2019 Runstats and Reorg
27/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 27
Real-Time Statistics
Statistics are always collected in real time for inserts, updates, and deletes on tablespaces,
partitions, indexes, and logical parts of NPI (V7, APAR PQ48447, PQ48448, and PQ46859)< Written to 2 tables, if they exist, at specified time intervals (30 minutes by default)
Statistics written by LOAD REPLACE and RESUME, REORG, REBUILD INDEX,RUNSTATS UPDATE ALL, COPY, and RECOVER to currency and RECOVER to PIT
< Beware - statistics not written if utility is specified in a utility list
< Specific statistics collected by utilities with various options are detailed in the manuals:< Utility Guide and Reference
< DB2 for z/OS and OS/390 Version 7 Using the Utilities Suite, SG24-6289
Statistics not written if STOP DATABASE(DSNRTSDB) and STOP DB2 MODE(QUIESCE)
Overhead is thought to be less than 1 %< Mostly for recording statistics in 2 tables (statistics are always collected)< Consider placing the objects in a separate buffer pool sized to page fix the objects in
central, data space, or hiperspace storage
Statistics can be analyzed using:
< Control Center with WLM or< DSNACCOR stored procedure using WLM or
< SELECT statements
Useful for analyzing when to execute REORG, RUNSTATS, and COPY utilities< Goal is to have DB2 be self-managed
7/27/2019 Runstats and Reorg
28/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 28
-Continue create tableReorgLastTime TIMESTAMP ,ReorgInserts INTEGER ,ReorgDeletes INTEGER ,ReorgUpdates INTEGER ,
ReorgUnClustIns INTEGER ,ReorgDisOrgLob INTEGER ,ReorgMassDelete INTEGER ,ReorgNearIndRef INTEGER ,ReorgFarIndRef INTEGER ,StatsLastTime TIMESTAMP ,StatsInserts INTEGER ,StatsDeletes INTEGER ,StatsUpdates INTEGER ,StatsMassDelete INTEGER ,CopyLastTime TIMESTAMP ,CopyUpdatedPages INTEGER ,CopyChanges INTEGER ,CopyUpdateLRSN CHAR(6) FOR BIT DATA,CopyUpdateTime TIMESTAMP )
IN DSNRTSDB.DSNRTSTS CCSID EBCDIC;
Creation of Real-Time Statistics Objects
DSNTESS in SDSNSAMP contains create statements (objects must be named as indicated)
CREATE DATABASE DSNRTSDB CCSID EBCDIC;
CREATE TABLESPACE DSNRTSTS IN DSNRTSDBCCSID EBCDIC -- requiredLOCKSIZE ROW -- requiredLOCKMAX 0SEGSIZE 32
CLOSE NOUSING STOGROUP SYSDEFLTPRIQTY 1600SECQTY 160;
CREATE TABLE SYSIBM.TABLESPACESTATS(DBNAME CHAR(8) NOT NULL,NAME CHAR(8) NOT NULL,PARTITION INTEGER NOT NULL,
DBID SMALLINT NOT NULL,PSID SMALLINT NOT NULL,UpdateStatsTime TIMESTAMP NOT NULL WITH DEFAULT,TotalRows FLOAT ,Nactive INTEGER ,Space INTEGER ,Extents SMALLINT ,LoadRLastTime TIMESTAMP ,
CREATE UNIQUE INDEX SYSIBM.TABLESPACESTATS_IXON SYSIBM.TABLESPACESTATS(DBID, PSID, PARTITION)CLUSTER CLOSE NO;
Definition of columns can be found in the manualsUtility Guide and Referenceand DB2 for z/OS and
OS/390 Version 7 Using the Utilities Suite, SG24-6289
7/27/2019 Runstats and Reorg
29/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 29
CREATE UNIQUE INDEXSYSIBM.INDEXSPACESTATS_IXON SYSIBM.INDEXSPACESTATS(DBID, ISOBID, PSID, PARTITION)CLUSTER CLOSE NO;
Must START DATABASE(DSNRTSDB) aftercreating objects
< DB2 places the database in a stopped state Each sysplex data sharing member collects and
writes statistics
Creation of Real-Time Statistics Objects - continued
CREATE TABLE SYSIBM.INDEXSPACESTATS(DBNAME CHAR(8) NOT NULL,INDEXSPACE CHAR(8) NOT NULL,PARTITION INTEGER NOT NULL,DBID SMALLINT NOT NULL,ISOBID SMALLINT NOT NULL,PSID SMALLINT NOT NULL,UpdateStatsTime TIMESTAMP NOT NULL WITH DEFAULT,TotalEntries FLOAT ,Nlevels SMALLINT ,
Nactive INTEGER ,Space INTEGER ,Extents SMALLINT ,LoadRLastTime TIMESTAMP ,RebuildLastTime TIMESTAMP ,ReorgLastTime TIMESTAMP ,ReorgInserts INTEGER ,ReorgDeletes INTEGER ,
ReorgAppendInsert INTEGER ,ReorgPseudoDeletes INTEGER ,ReorgMassDelete INTEGER ,ReorgLeafNear INTEGER ,ReorgLeafFar INTEGER ,ReorgNumLevels INTEGER ,StatsLastTime TIMESTAMP ,StatsInserts INTEGER ,StatsDeletes INTEGER ,StatsMassDelete INTEGER ,CopyLastTime TIMESTAMP ,CopyUpdatedPages INTEGER ,CopyChanges INTEGER ,CopyUpdateLRSN CHAR(6) FOR BIT DATA,CopyUpdateTime TIMESTAMP )IN DSNRTSDB.DSNRTSTS CCSID EBCDIC;
7/27/2019 Runstats and Reorg
30/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 30
Control Center
Control Center can be used to set thresholds
Calls DSNACCOR stored procedure to report on recommendations for execution of utilities
Calls DSNUTILS stored procedure to execute utilities
7/27/2019 Runstats and Reorg
31/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 31
Degree of Checking and Scope
DSNACCOR degree of check can be:
0 = No Additional checking, base function only1 = Do not include any deleted objects in the DSNACCOR result set2 = Record table space names related to index spaces
4 = Use the exception table ( if one exists )
8 = Return status on restricted objects16 = Include any deleted objects in the result set but mark "status" as orphaned
32 = Scale up to table space recommendation on reorg if appropriate
Degree of check option is set to the sum of the categories needed< If you would like category 1, 2, and 4,S Use 7 which is the sum of 1, 2, and 4(default)
Scope of objects can be controlled with a WHERE clause (
7/27/2019 Runstats and Reorg
32/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 32
--Continue call parameters:RRIAPPENDINSERTPCT :RRIAPPENDINSERTPCT-IND,:RRIPSEUDODELETEPCT :RRIPSEUDODELETEPCT-IND,:RRIMASSDELLIMIT :RRIMASSDELLIMIT-IND,:RRILEAFLIMIT :RRILEAFLIMIT-IND,:RRINUMLEVELSLIMIT :RRINUMLEVELSLIMIT-IND,:SRTINSDELUPDPCT :SRTINSDELUPDPCT-IND,
:SRTINSDELUPDABS :SRTINSDELUPDABS-IND,:SRTMASSDELLIMIT :SRTMASSDELLIMIT-IND,:SRIINSDELUPDPCT :SRIINSDELUPDPCT-IND,:SRIINSDELUPDABS :SRIINSDELUPDABS-IND,:SRIMASSDELLIMIT :SRIMASSDELLIMIT-IND,:EXTENTLIMIT :EXTENTLIMIT-IND,:LASTSTATEMENT :LASTSTATEMENT-IND,:RETURNCODE :RETURNCODE-IND,
:ERRORMSG :ERRORMSG-IND,:IFCARETCODE :IFCARETCODE-IND,:IFCARESCODE :IFCARESCODE-IND,:XSBYTES :XSBYTES-IND)END-EXEC.
Calling DSNACCOR Stored Procedure
Recommendations can be generated by calling stored procedure DSNACCOR in SDSNSAMP
EXEC SQLCALL SYSPROC.DSNACCOR(:QUERYTYPE :QUERYTYPE-IND,:OBJECTTYPE :OBJECTTYPE-IND,:ICTYPE :ICTYPE-IND,:STATSSCHEMA :STATSSCHEMA-IND,:CATLGSCHEMA :CATLGSCHEMA-IND,:LOCALSCHEMA :LOCALSCHEMA-IND,
:CHKLVL :CHKLVL-IND,:CRITERIA :CRITERIA-IND,:CRUPDATEDPAGESPCT :CRUPDATEDPAGESPCT-IND,:CRCHANGESPCT :CRCHANGESPCT-IND,:CRDAYSNCLASTCOPY :CRDAYSNCLASTCOPY-IND,:ICRUPDATEDPAGESPCT :ICRUPDATEDPAGESPCT-IND,:CRINDEXSIZE :CRINDEXSIZE-IND,:RRTINSDELUPDPCT :RRTINSDELUPDPCT-IND,
:RRTUNCLUSTINSPCT :RRTUNCLUSTINSPCT-IND,:RRTDISORGLOBPCT :RRTDISORGLOBPCT-IND,:RRTMASSDELLIMIT :RRTMASSDELLIMIT-IND,:RRTINDREFLIMIT :RRTINDREFLIMIT-IND,:RRIINSERTDELETEPCT :RRIINSERTDELETEPCT-IND,
DB2 for z/OS and OS/390Administration Guide, SC26-9931, manual contains: OFFPOSLIMIT (10 % default) assuming that limit parameter is specified,
< A reorganization of the tablespace or named partitions are performed along with the
indexes on the objects
NEAROFFPOSF is the number of times it is necessary to access a different, "near-off" pagewhen processing rows in sequence according to the index
< Segmented tablespace row is near-off position if the difference between the 2 page IDs is>= 2 and < SEGSIZE * 2 (16 rather than SEGSIZE * 2 is used for nonsegmented
tablespaces)
FAROFFPOSF is the number of times it is necessary to access a different, "far-off" page whenprocessing rows in sequence according to the index
< Segmented tablespace row is far-off position if the 2 page IDs differ by SEGSIZE * 2 or
more (16 rather than SEGSIZE * 2 is used for nonsegmented tablespaces)
0AND ((NEARINDREF+FARINDREF) * 100 / CARDF) > 0;
If REORG finds that (NEARINDREF + FARINDREF) * 100 / CARDF > INDREFLIMIT (10% default) (V6) assuming that limit parameter is specified,
10 % of an index is occupied by pseudo deleted RIDs,< Consider reorganizing or rebuilding the index
SELECT IXCREATOR, IXNAME, PSEUDO_DEL_ENTRIES, PGREATER10FROM SYSIBM.SYSINDEXPART
WHERE (PSEUDO_DEL_ENTRIES * 100 / CARDF) > 10 AS PGREATER10
Many more parameters discussed in the remainder of section that you will want to include inthe REORG TABLESPACE and REORG INDEX statements
Determining Free Space Available/Unavailable
7/27/2019 Runstats and Reorg
40/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 40
Determining Free Space Available/Unavailable
Determine the amount of free space currently on pages for tablespaces in the DASPJDB database
SELECT TSNAME, DBNAME, PERCACTIVEFROM SYSIBM.SYSTABLEPARTWHERE DBNAME = 'DASPJDB';
If PERCACTIVE > 95, some of 5 % free space (default) has been used 0,< Space from a dropped table is not reusable
SELECT TSNAME, DBNAME, PERCDROPFROM SYSIBM.SYSTABLEPARTWHERE PERCDROP > 0;
Alternative is to use a segmented tablespace=
(SELECT COALESCE(MAX(STATSTIME),
'0001-01-01.00.00.00.000000')FROM SYSIBM.SYSTABLEPART)ORDER BY EXTENTS;
Can determine number of pseudo deleted RIDs for indexesSELECT IXNAME, EXTENTS, DSNUM, SPACEF, PSEUDO_DEL_ENTRIES,STATSTIMEFROM SYSIBM.SYSINDEXPARTWHERE IXCREATOR = 'MKTGWX'AND STATSTIME >=(SELECT COALESCE(MAX(STATSTIME),
'0001-01-01.00.00.00.000000')FROM SYSIBM.SYSINDEXPART)ORDER BY EXTENTS;
Alternatives to Determine Secondary Extents
7/27/2019 Runstats and Reorg
45/117
Gabrielle & Associates, http://www.GabrielleDB2.com, (972)412-8866, Copyright 1984, 2004 - All Rights Reserved RR 45
LISTCAT (AMS (Access Method Services)) command can be used to determine secondaryextents from the ICF catalog, for example
LISTCAT LEVEL(DSNP.DSNDBD.DSNDB07.*) ALL
ISPF 3.4 can be used, for example< Dsname Level . . . DSNP.DSNDBD.DSNDB07.* with option 2
S Gives the space allocated for work tablespaces
< SORT XT lists the data sets in multiple extents first
If real-time statistics are recorded,< SYSIBM.TABLESPACESTATS contains space information
for DSNDB07 and TEMP tablespaces in columns:
S
NACTIVE, SPACE, EXTENTS
Cannot use catalog tables to analyze space for DSNDB07< RUNSTATS does not collect statistics on DSNDB07
Frequently used for partitioned tablespaces