©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA 1 An Expert System designed to...

44
1 ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cp An Expert System designed to evaluate IBM z/OS systems

Transcript of ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA 1 An Expert System designed to...

Page 1: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

1©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

An Expert System

designed to evaluate

IBM z/OS systems

Page 2: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

2©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

Product Overview

• Helps analyze performance of z/OS systems.

• Written in SAS (only SAS/BASE is required).

• Runs as a batch job on mainframe (or on PC).

• Processes data in a standard performance data base (either MXG, SAS/ITRM, or NeuMICS).

• Produces narrative reports showing results from analysis!

• Product is updated every six months

• 45-day trial is available (see license agreement for details).

Page 3: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

3©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

Components Delivered• SRM Component * March 1991

• TSO Component * April 1991

• MVS Component * June 1991

* These legacy components apply only to Compatibility Mode

• DASD Component October 1991

• CICS Component May 1992

• WLM Component April 1995

• DB2 Component October 1999

• WMQ Component June 2004

Page 4: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

4©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

Product Documentation

Each component has an extensive User Manualavailable in hard-copy or CD, and web-enabled

• Describes the likely impact of each finding

• Discusses the performance issues associated with each finding

• Suggests ways to improve performance and describes alternative solutions

• Provides specific references to IBM or other documents relating to the findings

• More than 4,000 pages for all components

Page 5: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

5©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WLM Component•Checks for problems in service definition

•Identifies reasons performance goals were missed

•Analyzes general system problems:

• Coupling facility/XCF

• Paging subsystem

• System logger

• WLM-managed initiators

• Excessive CPU use by SYSTEM or SYSSTC

• IFA/zAAP, zIIP, and IOP/SAP processors

• PR/SM and LPAR problems

• Intelligent Resource Director (IRD) problems

Page 6: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

6©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WLM Component - sample reportRULE WLM103: SERVICE CLASS DID NOT ACHIEVE VELOCITY GOAL

DB2HIGH (Period 1): Service class did not achieve its velocity goal during the measurement intervals shown below. The velocity goal was 50% execution velocity, with an importance level of 2. The '% USING' and '%TOTAL DELAY' percentages are computed as a function of the average address space ACTIVE time. The 'PRIMARY,SECONDARY CAUSES OF DELAY' are computed as a function of the execution delay samples on the local system.

‑‑‑‑‑‑LOCAL SYSTEM‑‑‑‑‑‑‑‑ % % TOTAL EXEC PERF PLEX PRIMARY,SECONDARY MEASUREMENT INTERVAL USING DELAY VELOC INDX PI CAUSES OF DELAY 21:15‑21:30,08SEP1998 16.6 83.4 17% 3.02 2.36 DASD DELAY(99%)

RULE WLM361: NON‑PAGING DASD I/O ACTIVITY CAUSED SIGNIFICANT DELAYS

DB2HIGH (Period 1): A significant part of the delay to the service class can be attributed to non‑paging DASD I/O delay. The below data shows intervals when non‑paging DASD delay caused DB2HIGH to miss its performance goal:

AVG DASD AVG DASD ‑‑AVERAGE DASD I/O TIMES‑ MEASUREMENT INTERVAL I/O RATE USING/SEC RESP WAIT DISC CONN 21:15‑21:30,08SEP1998 31 1.405 0.010 0.003 0.004 0.002

Page 7: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

7©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WLM Component - sample reportRULE WLM601: TRANSPORT CLASS MAY NEED TO BE SPLIT

You should consider whether the DEFAULT transport class should be split. A large percentage of the messages were too small, while a significant percentage of messages were too large. Storage is wasted when buffers are used by messages that are too small, while unnecessary overhead is incurred when XCF must expand the buffers to fit a message. The CLASSLEN parameter establishes the size of each message buffer, and the CLASSLEN parameter was specified as 16,316 for this transport class. This finding applies to the following RMF measurement intervals:

SENT SMALL MESSAGES MESSAGES TOTAL MEASUREMENT INTERVAL TO MESSAGES THAT FIT TOO BIG MESSAGES 10:00‑10:30,26MAR1996 JA0 4,296 0 57 4,353 12:00‑12:30,26MAR1996 Z0 2,653 6 762 3,421 12:30‑13:00,26MAR1996 Z0 2,017 0 109 2,126

RULE WLM622: THE NUMBER OF OUTBOUND PATHS MAY NEED TO BE INCREASED

The PATH BUSY (when selected for transfer) was high relative to the PATH AVAILABLE for the C605 path on System JB0, sending messages to the C611 path on System JA0 in transport class DEFSMALL. This usually means that you need to add more OUTBOUND paths to the transport class. This finding applies to the following RMF measurement intervals:

TOTAL PCT OUTBOUND PCT PATH MEASUREMENT INTERVAL MESSAGES PATH BUSY AVAILABLE 12:00‑12:30,26MAR1996 2562 21.1 78.9

Page 8: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

8©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WLM Component - sample reportRULE WLM705: STAGING DATA SETS NOT EFFICIENTLY USED FOR LOG STREAM

GRP1.CICS1T9A.DFHLOG: The bytes deleted after data was offloaded from staging data sets to DASD log data sets was high relative to the bytes deleted before data was offloaded. More than 10% of the bytes deleted were deleted after being offloaded to DASD, with associated unnecessary I/O operations. This indicates that the space allocated for the DASD staging data sets was not efficiently used for the log stream data. If this finding is consistently produced, you should consider increasing the size of the DASD staging data set for log stream GRP1.CICS1T9A.DFHLOG. This finding applies to the following SMF measurement intervals:

BYTES BYTES DELETED BYTES DELETED PERCENT MEASUREMENT INTERVAL DELETED BEFORE OFFLOAD AFTER OFFLOAD AFTER 14:45,03OCT2002 844K 0 844K 100.0

RULE WLM537: ZAAP-ELIGIBLE WORK HAD HIGH GOAL IMPORTANCE

Rule WLM530 or Rule WLM535 was produced for this system, indicating that a relatively large amount of zAAP-eligible work was processed on a central processor. One possible cause of this situation is that the zAAP-eligible work was assigned a relatively high Goal Importance (the Goal Importance was either Importance 1 or Importance 2). Please see the discussion in the WLM Component User Manual for an explanation of this issue.

Page 9: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

9©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 Component

•Analyzes standard DB2 interval statistics

•Applies analysis from DB2 Administration Guide

and DB2 Performance Guide (with DB2 9.1)

•Analyzes DB2 Versions 3, 4, 5, 6, 7, 8, and 9

•Evaluates overall DB2 constraints, buffer pools,

EDM pool, RID list processing, Lock Manager,

Log Manager, DDF, and data sharing

•All analysis can be tailored to your site!

Page 10: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

10©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 Component

• There might be insufficient buffers for work files

• There were insufficient buffers for work files in merge passes

• Buffer pool was full

• Hiperpool read requests failed (pages stolen by system)

• Hiperpool write requests failed (expanded storage not available

• Buffer pool page fault rate was high

• Data Management Threshold (DMTH) was reached

• DWQT and VDWQT might be too large

• DWQT, VDWQT, or VPSEQT might be too small

Typical DB2 local buffer constraints

Page 11: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

11©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 ComponentTypical DB2 I/O prefetch constraints

• Sequential prefetch was disabled, buffer shortage

• Sequential prefetch was disabled, unavailable read engine

• Sequential prefetch not scheduled, prefetch quantity = 0

• Synchronous read I/O and sequential prefetch was high

• Dynamic sequential prefetch was high (before DB2 8.1)

• Synchronous read I/O was high

Page 12: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

12©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 Component

• Parallel groups fell back to sequential mode

• Parallel groups reduced due to buffer shortage

• Prefetch quantity reduced to one‑half of normal

• Prefetch quantity reduced to one‑quarter of normal

• Prefetch I/O streams were denied, shortage of buffers

• Page requested for a parallel query was unavailable

Typical DB2 parallel processing constraints

Page 13: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

13©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 ComponentTypical DB2 EDM pool constraints

• Failures were caused by full EDM pool

• Low percent of DBDs found in EDM pool

• Low percent of CT Sections found in EDM pool

• Low percent of PT Sections found in EDM pool

• Size of EDM pool could be reduced

• Excessive Class 24 (EDM LRU) latch contention

Page 14: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

14©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 ComponentTypical DB2 Lock Manager constraints

• Work was suspended because of lock conflict

• Locks were escalated to shared mode

• Locks were escalated to exclusive mode

• Lock escalation was not effective

• Work was suspended for longer than time-out value

• Deadlocks were detected

Page 15: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

15©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 ComponentTypical DB2 Log Manager constraints

• Archive log read allocations exceeded guidance

• Archive log write allocations exceeded guidance

• Waits were caused by unavailable output log buffer

• Log reads satisfied from active log data set

• Log reads were satisfied from archive log data set

• Failed look-ahead tape mounts

Page 16: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

16©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 ComponentTypical DB2 Data Sharing constraints

• Group buffer pool is too small

• Incorrect directory entry/data entry ratio

• Directory reclaims resulting in cross-invalidations

• Castout processing occurring in “spurts”

• Excessive lock contention or false lock contention

• GBPCACHE ALL inappropriately specified

• GBPCACHE CHANGED inappropriately specified

• Conflicts between applications

Page 17: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

17©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 Component - sample report

RULE DB2-208: VIRTUAL BUFFER POOL WAS FULL Buffer Pool 2: A usable buffer could not be located in virtual Buffer Pool 2, because the virtual buffer pool was full. This condition should not normally occur, as there should be ample buffers. You should consider using the -ALTER BUFERPOOL command to increase the virtual buffer pool size (VPSIZE) for the virtual buffer pool. This situation occurred during the intervals shown below:

BUFFERS NUMBER OF TIMES MEASUREMENT INTERVAL ALLOCATED POOL WAS FULL 10:54-11:24, 15SEP1999 100 12 11:24-11:54, 15SEP1999 100 13

RULE DB2-216: BUFFER POOLS MIGHT BE TOO LARGE Buffer Pool 1: The page fault rates for read and write I/O indicated that the buffer pools might be too large for the available processor storage. This situation occurred for Buffer Pool 1 during the intervals shown below:

BUFFERS PAGE-IN FOR PAGE-IN FOR PAGE MEASUREMENT INTERVAL ALLOCATED READ I/O WRITE I/O RATE 11:15-11:45, 16SEP1999 25,000 36,904 195 41.2 11:45-12:15, 16SEP1999 25,000 30,892 563 35.0 12:45-13:15, 16SEP1999 25,000 23,890 170 26.7

Page 18: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

18©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DB2 Component - sample report

RULE DB2-601: COUPLING FACILITY READ REQUESTS COULD NOT COMPLETE Group Buffer Pool 6: Coupling facility read requests could not be completed because of a lack of coupling facility storage resources. This situation occurred for Group Buffer Pool 6 during the intervals shown below:

GROUP BUFFER POOL TIMES CF READ MEASUREMENT INTERVAL ALLOCATED SIZE REQUESTS NOT COMPLETE 11:01-11:31, 14OCT1999 38M 130

RULE DB2-610: GBPCACHE(N0) OR GBPCACHE NONE MIGHT BE APPROPRIATE Group Buffer Pool 4: This buffer pool had a very small amount of read activity relative to write activity. Pages read were less than 1% of the pages written. Since so few pages were read from this group buffer pool, you should consider specifying GPBCACHE(NO) for the group buffer pool or specifying GBPCACHE NONE for the page sets using the group buffer pool. This situation occurred for Group Buffer Pool 4 during the intervals shown below:

GROUP BUFFER POOL PAGES PAGES READ MEASUREMENT INTERVAL ALLOCATED SIZE READ WRITTEN PERCENT 10:34-11:04, 14OCT1999 38M 14 18,268 0.07%

Page 19: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

19©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

CICS Component

•Processes CICS Interval Statistics contained inMXG Performance Data Base (standard SMF 110)

•Analyzes all releases of CICS (CICS/ESA,CICS/TS for OS390, and CICS/TS for z/OS)

•Applies most analysis techniques contained inIBM’s CICS Performance Guides

•Produces specific suggestions for improving CICS performance

Page 20: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

20©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

CICS Component(Major areas analyzed)

•Virtual and real storage (MXG/AMXT/TCLASS)•VSAM and File Control (NSR and LSR pools)•Database management (DL/I, IMS, DB2)•Journaling (System and User journals)•Network and VTAM (RAPOOL, RAMAX)•CICS Facilities (temp storage, transient data)•ISC/IRC (MRO, LU61., LU6.2 modegroups)•System logger•Temporary Storage •Coupling Facility Data Tables (CFDT)•CICS-DB2 Interface•Open TCB pools•TCP/IP and SSL

Page 21: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

21©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

CICS Component - sample report

RULE CIC269: EXCESSIVE GETMAIN/FREEMAIN ACTIVITY FOR MRO SESSIONS

The IOAREALEN parameter specifications for the below connections might cause excessive GETMAIN/FREEMAIN activity. The GETMAIN/FREEMAIN rate was greater than 25 per second to the Storage Domain SMTP Subpool. This high GETMAIN/FREEMAIN activity normally means that the IOAREALEN parameter was set to the default (zero), or was set to a smaller value than the TIOA (Terminal I/O Area) required to support the MRO traffic. The information shown below describes the total GETMAIN and FREEMAIN activity, the Connection ID, the percent of message activity to each Connection ID, and TIOA size (and the percent of message activity at that TIOA size for the Connection ID) for connections with the highest total function shipping activity.

GETMAIN+ CONN PERCENT APPLID FREEMAIN ID ACTIVITY TIOA--PCT TIOA--PCT TIOA--PCT CICS1TAA 191,502 1AAA 34.2 624 96.2% 120 3.7% CICS1TAA 191,502 1AAB 33.0 600 97.0% 264 2.9% CICS1TAA 191,502 1AAC 32.7 600 97.1% 344 2.8%

Page 22: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

22©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

CICS Component - sample report

RULE CIC267: INSUFFICIENT SESSIONS MAY HAVE BEEN DEFINED

CPExpert believes that an insufficient number of sessions may have been defined for the CICS DAL1 connection, or the application system could have been issuing ALLOCATE requests too often. The number of ALLOCATE requests returned was greater than the value specified for the ALLOCQ guidance variable in USOURCE(CICGUIDE). CPExpert suggests you consider increasing the number of sessions defined for the connection, or you should increase the ALLOCQ guidance variable to cause CPExpert to signal a potential problem only when you view the problem as serious. For APPC modegroups, this finding applies only to generic ALLOCATE requests. This finding applies to the following CICS statistics intervals:

STATISTICS ALLOCATE REQUESTS COLLECTION TIME APPLID RETURNED TO USERS 10:00,26MAR1996 CICSDTL1 335 11:00,26MAR1996 CICSDTL1 12 12:00,26MAR1996 CICSDTL1 27

Page 23: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

23©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DASD Component•Processes SMF Type 70(series) to automaticallybuild model of your I/O configuration.

•Identifies performance problems with deviceswhich have most potential for improvement.

• PEND delays• Disconnect delays• Connect delays• IOSQ delays• Shared DASD conflicts

•Analyzes SMF Type 42(DS) and Type 64 toidentify VSAM performance problems.

Page 24: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

24©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DASD Component - sample report

RULE DAS100: VOLUME WITH WORST OVERALL PERFORMANCE

VOLSER DB2327 (device 2A1F) had the worst overall performance during the entire measurement period (10:00, 16FEB2001 to 11:00, 16FEB2001). This volume had an overall average of 56.8 I/O operations per second, was busy processing I/O for an average of 361% of the time, and had I/O operations queued for an average of 1% of the time. Please note that percentages greater than 100% and Average Per Second Delays greater than 1 indicate that multiple I/O operations were concurrently delayed. This can happen, for example, if multiple I/O operations were queued or if multiple I/O operations were PENDing. The following summarizes significant performance characteristics of VOLSER DB2327:

I/O --- AVERAGE PER SECOND DELAYS--- MAJOR MEASUREMENT INTERVAL RATE RESP CONN DISC PEND IOSQ PROBLEM 10:00-10:30,16FEB2001 59.1 1.308 0.316 0.004 0.988 0.000 PEND TIME 10:30-11:00,16FEB2001 57.2 3.792 0.300 0.004 3.483 0.006 PEND TIME 11:00-11:30,16FEB2001 54.2 5.769 0.279 0.004 5.464 0.023 PEND TIME

Page 25: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

25©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DASD Component - sample report

RULE DAS130: PEND TIME WAS MAJOR CAUSE OF I/O DELAY.

A major cause of the I/O delay with VOLSER DB2327 was PEND time. The average per-second PEND delay for I/O is shown below:

PEND PEND PEND PEND PEND TOTAL MEASUREMENT INTERVAL CHAN DIR PORT CONTROL DEVICE OTHER PEND 10:00-10:30,16FEB2001 0.492 0.000 0.000 0.000 0.495 0.988 10:30-11:00,16FEB2001 1.927 0.000 0.000 0.000 1.556 3.483 11:00-11:30,16FEB2001 2.840 0.000 0.000 0.000 2.624 5.464

RULE DAS160: DISCONNECT TIME WAS MAJOR CAUSE OF I/O DELAY.

A major cause of the I/O delay with VOLSER DB26380 was DISCONNECT time. DISC time for modern systems is a result of cache read miss operations, potentially back-end staging delay for cache write operations, peer-to-peer remote copy (PPRC) operations, and other miscellaneous reasons.

--PERCENT-- DASD CACHE ----CACHE---- READ WRITE TO TO MEASUREMENT INTERVAL READS WRITES HITS HITS CACHE DASD PPRC BPCR ICLR 8:30- 8:45,22OCT2001 14615 932 19.2 100.0 11825 903 0 0 0 8:45- 9:00,22OCT2001 14570 921 20.7 100.0 11567 907 0 0 0

Page 26: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

26©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DASD Component - sample report

RULE DAS300: PERHAPS SHARED DASD CONFLICTS CAUSED PERFORMANCE PROBLEMS

Accessing conflicts caused by sharing VOLSER DB2700 between systems might have caused performance problems for the device during the measurement intervals shown below. Conflicting systems had the indicated I/O rate, average CONN time per second, average DISC time per second, average PEND time per second, and average RESERVE time to the device. Even moderate CONN, DISC, or RESERVE can cause delays to shared devices. .. I/O MAJOR OTHER -------OTHER SYSTEM DATA-------- MEASUREMENT INTERVAL RATE PROBLEM SYSTEM I/O RATE CONN DISC PEND RESV 8:30- 8:45,22OCT2001 31.3 QUEUING SY1 35.0 0.041 0.001 0.455 0.000 SY2 88.2 0.100 0.003 0.714 0.000 SY3 109.0 0.123 0.003 0.723 0.000 TOTAL 232.2 0.264 0.006 1.892 0.000 8:45- 9:00,22OCT2001 25.7 QUEUING SY1 46.4 0.054 0.001 0.565 0.000 SY2 98.2 0.112 0.003 0.836 0.000 SY3 119.0 0.136 0.003 0.846 0.000 TOTAL 263.5 0.303 0.007 2.247 0.000

Page 27: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

27©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DASD Component - sample reportRULE DAS607: VSAM DATA SET IS CLOSE TO MAXIMUM NUMBER OF EXTENTS

VOLSER: RLS003. More than 225 extents were allocated for the VSAM data sets listed below. The VSAM data sets are approaching the maximum number of extents allowed. The below shows the number of extents and the primary and secondary space allocation: .. TOTAL EXTENTS ---ALLOCATIONS--- SMF TIME STAMP JOB NAME VSAM DATA SET .. EXTENTS THIS OPEN PRIMARY SECONDARY 10:30,11MAR2002 CICS2ABA RLSADSW.VF01D.DATAENDB.DATA................. 229 4 30 CYL 1 CYL

RULE DAS625: NSR WAS USED, BUT LARGE PERCENT OF ACCESS WAS DIRECT

VOLSER: MVS902. Non-Shared resources (NSR) was specified as the buffering technique for the below VSAM data sets, but more than 75% of the I/O activity was direct access. NSR is not designed for direct access, and many of the advantages of NSR are not available for direct access. You should consider Local Shared Resources (LSR) for the below VSAM data sets (perhaps using System Managed Buffers to facilitate the use of LSR). The I/O RATE is for the time the data set was open. The SMF TIME STAMP and JOB NAME are from the last record for the data set. .. I/O OPEN -ACCESS TYPE (PCT)- SMF TIME STAMP JOB NAME VSAM DATA SET .. RATE DURATION SEQUENTIAL DIRECT 13:19,19SEP2002 NRXX807. SDPDPA.PK.MVSP.RT.NDMGIX.DATA............... 8.4 0:07:08 0.0 100.0 13:19,19SEP2002 NRXX807. SDPDPA.PR.MVSP.RT.NDMGIXD.DATA.............. 11.2 0:06:42 0.0 100.0 13:33,19SEP2002 TSJHM... SDPDPA.PR.MVSP.RT.NDMRQFDA.DATA............. 0.3 2:21:58 0.0 100.0 13:33,19SEP2002 TSJHM... SDPDPA.PR.MVSP.RT.NDMRQF.DATA............... 2.8 3:37:53 0.0 100.0 13:33,19SEP2002 TSJHM... SDPDPA.PK.MVSP.RT.NDMTCF.DATA............... 11.1 6:24:10 0.1 99.9

Page 28: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

28©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

DASD Component(Application Analysis)

•Requires simple modification to MXG or MICS

• Modification collects job step data while processing SMF Type 30 (Interval) records

• Typically requires less than 10 cylinders

• Data is correlated with Type 74 information

•CPExpert associated performance problems tospecific applications (jobs and job steps)

•CPExpert can perform “Loved one” analysis ofDASD performance problems

Page 29: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

29©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WMQ ComponentAnalyzes SMF Type 115 statistics, as processedby MXG or NeuMICS and placed into performancedata base.

• MQMLOG - Log manager statistics

• MQMMSGDM - Message/data manager statistics

• MQMBUFER - Buffer Manager statistics

• MQMCFMGR - Coupling Facility Manager stats

Type 115 records should be synchronized withSMF interval recording interval.

IBM says overhead to collect accounting data is negligible.

Page 30: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

30©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WMQ ComponentOptionally analyzes SMF Type 116 accounting

data, as processed by MXG or NeuMICS and

placed into performance data base.

• MQMACCTQ - Thread-level accounting data

• MQMQUEUE- Queue-level accounting data

Type 116 records should be synchronized withSMF interval recording interval.

IBM says overhead to collect accounting data is 5-10%

Page 31: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

31©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQTypical queue manager problems

Assignment of queues to page sets

Assignment of page sets to buffer pools

Queue manager parameters

Index characteristics of queues

Characteristics of messages in queues

Characteristics of MQ calls

CPExpert analysis uses SMF Type 116 records

Page 32: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

32©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQTypical buffer manager problems

Buffer thresholds exceeded for pool

Buffers assigned per pool (too few/too many)

Message traffic

Message characteristics

Application design

CPExpert analysis uses SMF Type 115 records

Page 33: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

33©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQTypical log manager problems

Log buffers assigned

Active log use characteristics

Archive log use characteristics

Tasks backing out

System paging of log buffers

Excessive checkpoints taken

CPExpert analysis uses SMF Type 115 records

Page 34: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

34©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQTypical DB2-interface problems

Thread delays

DB2 server processing delays

Server requests queued

Server tasks experienced ABENDs

Deadlocks in DB2

Maximum request queue depth was too large

CPExpert analysis uses SMF Type 115 records

Page 35: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

35©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQTypical Shared queue problems

Structure was full

Large number of application structures defined

MINSIZE is less than SIZE for CSQ.ADMIN

SIZE is more than double MINSIZE

ALLOWAUTOALT(YES) not specified

FULLTHRESHOLD value might be incorrect

CPExpert analysis uses SMF Type 115 records and Type 74 (Coupling Facility) records

Page 36: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

36©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQ – sample reportRULE WMQ100: MESSAGES WERE WRITTEN TO PAGE SET ZERO

More than 0 messages were written to Page Set Zero during the intervals shown below. Messages should not be written to Page Set Zero, since serious WebSphere MQ system problems could occur if Page Set Zero should become full. This finding relates to queue SYSTEM.COMMAND.INPUT

MESSAGES WRITTEN STATISTICS INTERVAL TO PAGE SET ZERO 13:16-14:45, 28AUG2003 624

RULE WMQ122: DEAD.LETTER QUEUE IS INAPPROPRIATE FOR PAGE SET ZERO

Buffer Pool 0. The DEAD.LETTER queue was assigned to Page Set Zero. A dead-letter queue stores messages that cannot be routed to their correct destinations. If the DEAD-LETTER queue grows large unexpectedly, Page Set Zero can become full, and WebSphere MQ can enter a serious stress condition. You should redefine the DEAD.LETTER queue to a page set other than Page Set Zero. This finding relates to queue SYSTEM.DEAD.LETTER.QUEUE

Page 37: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

37©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQ – sample reportRULE WMQ110: EXPYRINT VALUE IS OFF OR TOO SMALL

Buffer Pool 3. There were more than 25 expired messages skipped when scanning a queue for a specific message. Processing expired messages adds both CPU time and elapsed time to the message processing. With WebSphere 5.3, the EXPYRINT keyword was introduced to allow the queue manager to automatically determine whether queues contained expired messages and to eliminate expired messages at the interval specified by the EXPYRINT value. This finding applies to queue: DPS.REPLYTO.RCB.IVR04

GET BROWSE EXPIRED MESSAGES STATISTICS INTERVAL SPECIFIC SPECIFIC PROCESSED 13:41-13:41, 03JUL2003 0 0 313

RULE WMQ320: APPLICATIONS WERE SUSPENDED FOR LOG WRITE BUFFERS

Applications were suspended while in-storage log buffers are being written to the active log. This finding normally means that too few log buffers were assigned. However, the finding could mean that there is an I/O configuration problem and the log buffer writes to the active log are delayed for I/O reasons. This finding applies to the following statistics intervals.

NUMBER OF SUSPENSIONS STATISTICS INTERVAL WAITING ON OUTPUT BUFFERS 14:19-14:44, 12SEP2003 139

Page 38: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

38©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQ – sample reportRULE WMQ201: BUFFER POOL ENCOUNTERED SYNCHRONOUS (5%) THRESHOLD

Buffer Pool 0. This buffer pool encountered the Synchronous Write threshold (less than 5% of the pages in the buffer pool were "stealable" or more than 95% of the pages were on the Deferred Write queue). While the Synchronous Page Writer is executing, updates to any page cause the page to be written immediately to the page set (the page is not placed on the Deferred Write Queue, but is written immediately to the page set as a synchronous write operation). This situation harms performance of applications, and is an indicator that the buffer pool is in danger of encountering a Short on Storage condition.

BUFFERS TIMES AT IMMEDIATE STATISTICS INTERVAL ASSIGNED 5% THRESHOLD WRITES 17:08-17:09, 07OCT2003 1,050 19 19

RULE WMQ205: HIGH I/O RATE TO PAGE SETS WITH SHORT-LIVED MESSAGES

Buffer Pool 0. This buffer pool had short-lived messages assigned. The total I/O rate (read and write activity) to page sets for the short-lived messages was more than 0.5 pages per second. Writing pages to the page set and subsequently reading the pages from the page set cause I/O overhead and delay to the application. This finding applies to the following intervals:

BUFFERS PAGES PAGES I/O RATE STATISTICS INTERVAL ASSIGNED WRITTEN READ WITH DASD 11:32-11:32, 24JUL2006 50,000 101 0 50.5

Page 39: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

39©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

WebSphere MQ – sample reportRULE WMQ300: ARCHIVE LOGS WERE USED FOR BACKOUT

WebSphere MQ applications issued log reads to the archive log file for backout more than 0 times during the WebSphere MQ statistics intervals shown below. Most log read requests should come from the output buffer or the active log. Using archive logs for backout purposes often indicates that either the active log files were too small or long-running applications were backing out work. NUMBER OF LOG READS STATISTICS INTERVAL FROM ARCHIVE LOG 4:30- 5:00, 12SEP2003 192

RULE WMQ611: LARGE NUMBER OF APPLICATION STRUCTURES WERE DEFINED

SMF TYPE74 (Structure) statistics showed that more than 5 application structures were defined to a coupling facility. IBM suggests that you should have as few application structures as possible. Having multiple application structures in a coupling facility can degrade performance.

WEBSPHERE MQ COUPLING FACILITY STRUCTURES DEFINED CF1 8 CF2 9 CF3 8

Page 40: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

40©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

License fees(Site license)

Components First Year Additional year

WLM Component 7,500 5,000

DB2 Component 7,500 5,000

CICS Component(see note) 5,000 3,000

WMQ Component 5,000 3,000

DASD Component 3,000 1,500

Note: Fees shown for the CICS Component are for analyzing no more than 50 CICS regions.

Page 41: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

41©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

CPExpert Release 17.1(Issued April 2007)

Major enhancements with this update:• Provided email alert feature for serious problems

• Provided additional analysis of paging problems

• Provided additional analysis of XCF and coupling facility problems

• Redesigned CICS Component to use wild-card feature (with significant reduction in processing time for large CICS users of CICS Component)

• Provided additional analysis of WebSphere MQ performance problems

Page 42: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

42©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

Major enhancements with this update:• Provided support for z/OS Version 1, Release 9

• Provided additional analysis of z/OS performance problems (in WLM Component), including “blocked workload” analysis

• Provided support for CICS/TS Release 3.2 and added significant additional analysis of CICS performance problems

• Provided support for DB2 Version 9.1 and additional analysis of DB2 performance problems

• Provided support for NeuMICS in WebSphere MQ

CPExpert Release 17.2(Issued October 2007)

Page 43: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

43©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

Summary

• The major objective is to share solutions and provide insight into new z/OS features.

• CPExpert is updated every six months; support for new versions of z/OS has been available within 30 days after General Availability of the new z/OS release.

• CPExpert is offered at a low cost (affordable by all z/OS shops).

• 45-day trial is available (see license agreement for details).

Page 44: ©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA  1 An Expert System designed to evaluate IBM z/OS systems.

44©Copyright 1998, Computer Management Sciences, Inc., Alexandria, VA www.cpexpert.com

For more information, please contact

Don DeeseComputer Management Sciences, Inc.634 Lakeview DriveHartfield, VA 23071-3113

Phone: (804) 776-7109Fax: (804) 776-7139email: [email protected]

Visit www.cpexpert.com for more information, to review sample output, to review documentation in SAS ODS “point-and-click” format, to download license agreements in .pdf “form” mode, etc.