Lunch n Learn - Solving CICS performance issues
Transcript of Lunch n Learn - Solving CICS performance issues
© 2015 IBM Corporation
APA/CICS PA Webinar
August 04 2015
Solving CICS performance issues….if you’re looking for trouble, you’ve come to the right place.
Lamar Woods
Sr. Client Technical Professional, Z series Problem Determination Tools
Eric Higgins
Client Technical Specialist, CICS tools
© 2015 IBM Corporation
The following terms are trademarks or registered trademarks of the
International Business Machines Corporation in the United States and/or
other countries:
– CICS, CICS Explorer, CICSPlex SM, DB2, QMF, IMS, MQSeries, WebSphere,
IBM, SupportPac, OS/390, z/OS, zSeries, RMF, Tivoli, Tivoli Enterprise,
OMEGAMON XE
Java and all Java-based trademarks and logos are trademarks or
registered trademarks of Sun Microsystems, Inc. in the United States
and/or other countries.
Microsoft, Windows, and the Windows logo are trademarks of Microsoft
Corporation in the United States, other countries, or both.
Other company, product, service names, and logos may be trademarks or
service marks of others.
Preface
2
© 2015 IBM Corporation3
Abstract
Identifying and solving CICS performance issues before they get out of
hand can be viewed as looking for trouble before it finds you.
IBM’s CICS Performance Analyzer is a historical performance analysis tool that provides
the detailed performance information required to understand the impact of application
performance. When paired with the performance measurement features of the IBM
Application Performance Analyzer, there’s no better combination of capabilities for
uncovering and solving CICS performance issues – right down to the source code level.
What’s the value of identifying performance issues before they have a chance to
escalate?
• How often do you go looking for tuning opportunities?
• What can a performance measurement tool do to help resolve bad application
behavior?
In this webcast we combine capabilities of CICS Performance Analyzer and Application
Performance Analyzer to:
• Identify performance tuning candidate applications at a CICS system level
• Perform a subsequent deep-dive performance analysis right down to the application
source code level
© 2015 IBM Corporation4
For our scenario, we are using a version of the supplied Catalog manager example (ECAT) that is supplied with CICS.
The example is constructed around a simple sales
catalog and order processing application, in which a user
can perform various tasks such as:
List items in a catalog.
Inquire on individual items in the catalog.
Order items from the catalog.
Administrative functions
We will use CICS Performance Analyzer and Application Performance Analyzer to look for performance issues with ECAT.
For details on the ECAT sample, see https://www-01.ibm.com/support/knowledgecenter/SSGMCP_5.2.0/com.ibm.cics.ts.exampleapplication.doc/topics/dfhxa_t100.html
Let’s go looking for some trouble!
Our Application
© 2015 IBM Corporation5
What is CICS PA? A Comprehensive Performance Reporting and
Analysis tool for CICS
Provides DB2 and MQ subsystem performance reporting related to CICS workload
Provides ongoing system management and measurement reports on all aspects of CICS application performance
Historical performance complimenting real-time Omegamon monitoring
How does CICS PA work? Uses SMF data as input
Easy to use interface for report generation (over 250 supplied report forms)
Performance and Statistical analysis
Graphical performance analysis via the explorer
CICS
PERFORMANCE ANALYSIS
CICS Performance Analyzer for z/OS
Why this tool is important• Reduce both time/resource required to analyze off-
line performance data (usually massive) for tuning
and capacity planning purposes.
• Enables deep-dive CICS performance analysis and
understanding of usage trends
• Aids capacity planning and tuning
• Helps quickly identify and eliminate trends leading to
online performance problems
© 2015 IBM Corporation
CICS PA – Review Performance Alerts – Demo Backup Slide
6
Performance Alerts:
• Define alerts with specific threshold values
• Critical, warning, and informational levels
• Only report on activity exceeding the threshold
In this case we see several alerts on 2015-07-08 for Tran ECAT where the CPU and
Response time thresholds were exceeded
Next we’ll look at some of the performance intervals for ECAT
© 2015 IBM Corporation
CICS PA – Review Summary data – Demo Backup Slide
7
Transaction Summary:
• Interactive view of performance data
• Filter & sort by column
• Customizable layouts
• The data is summarized into 5 minute intervals.
• 2 of the intervals shown have high average response times and high average CPU
utilization
Next we’ll look the response time data graphically for ECAT
© 2015 IBM Corporation
CICS PA – Create charts to analyze data – Demo Backup Slide
8
Transaction Response Time Bar Chart:
• Visual presentation makes it easy to see anomalies
• Most intervals have a good response times, but a couple are way out of range
Right click on one of the poor response time bars to look into more performance
summary details
© 2015 IBM Corporation9
Transaction Response Summary Pie Chart:
• Shows dispatch time is 99.92% of Response time
• Shows CPU time is 90.75% of dispatch time
• Something is using a lot of CPU
CICS PA – Create charts to analyze data – Demo Backup Slide
© 2015 IBM Corporation10
Transaction CPU Summary Pie Chart:
• 77% of CPU is on L8
• Let’s look at our RMI usage to see what’s driving L8 CPU
CICS PA – Create charts to analyze data – Demo Backup Slide
© 2015 IBM Corporation11
RMI Usage Pie Chart:
• 100% of the RMI activity is related to DB2
Let’s look at one more thing with CICS PA
CICS PA – Create charts to analyze data – Demo Backup Slide
© 2015 IBM Corporation
CICS PA – Review Statistic Alerts – Demo Backup Slide
12
• This alert shows that we exceeded out DB2 entry thread wait threshold.
• At this point we need to look deeper into ECAT to see what the programs are doing
to sometimes cause a lot more DB2 activity
• We will use Application Performance Analyzer to drill into ECAT’s performance at the
program level.
Statistic Alerts:
• Define alerts with specific threshold values for SMF 110 sub-type 2 (stats) records
• CICS PA includes hundreds of sample alerts
• Critical, warning, and informational levels
• Only report on activity exceeding the threshold
© 2015 IBM Corporation13
For our scenario, we are using a version of the supplied Catalog manager example (ECAT) that is supplied with CICS.
The example is constructed around a simple sales
catalog and order processing application, in which a user
can perform various tasks such as:
List items in a catalog.
Inquire on individual items in the catalog.
Order items from the catalog.
Administrative functions
Problem: ECAT transaction is taking too long and we will use Application Performance Analyzer to look for performance issues with ECAT
For details on the ECAT sample, see https://www-01.ibm.com/support/knowledgecenter/SSGMCP_5.2.0/com.ibm.cics.ts.exampleapplication.doc/topics/dfhxa_t100.html
Some trouble has been found!
Our Application (Problem Detected)
© 2015 IBM Corporation14
What is APA?
A tool that helps you perform application
performance tuning for applications running on
z/OS systems
Measures and report on how system resources
are used by applications running in virtually any
z/OS address space such as TSO and batch
How does APA work? Automatically starts to monitor job performance
when the job or program becomes active
Analyzes transactions running under IMS, CICS, WAS, as well as DB2 Stored procedures
Displays overall system activity enabling you to check job execution online and select which active job to monitor
Provides multiple summary reports to assist in identifying key areas of performance
Online analysis and reports that can be created as PDF or XML files that can be viewed on workstations or transferred easily to other applications
Proven 3270-based interface and free graphical interface
APPLICATION PERFORMANCE
ANALYSIS
Application Performance Analyzer for z/OS
Why this tool is important• Helps maximize the performance of your
applications and improve the response time of your online applications and batch turnaround
• Identifies constraints and improves the entire
application’s performance no matter where the
problem resides (CICS, IMS, DB2, MQ, COBOL,
PLI, ASM, JAVA)
• Isolate performance problems in new and existing
applications
• Aids application design, development and
maintenance lifecycle
© 2015 IBM Corporation
APA – Open the APA Perspective - Demo Backup Slide
15
Start an APA monitoring session
We will monitor activity in a specific CICS region where the ECAT transaction is invoked.
New Observation
© 2015 IBM Corporation
APA – Specify Measurement Criteria - Demo Backup Slide
16
Specify the Number of Samples you want taken and the Duration.
In this example, 25,000 samples will be taken over a duration of 10 minutes.
Specify CICS Region pattern and select Get Active Jobs button
© 2015 IBM Corporation
APA – Select CICS Region to Monitor - Demo Backup Slide
17
Select one or more active CICS Regions to monitor activity.
In this example, CICSAOR5 region will be monitored.
Select CICSAOR5
© 2015 IBM Corporation
APA –Region Selected - Demo Backup Slide
18
Selected regions are added to the Selected Jobs List.
CICSAOR5 added to the selected jobs list and will be monitored during this observation session.
© 2015 IBM Corporation
APA – Select Data Extractors and Submit - Demo Backup Slide
19
Data Extractors are selected in order for APA to collect additional information specific to a
subsystem environment.
CICS and DB2 data extractors are needed in order for APA to collect specific information.
Click Submit to start APA monitoring
© 2015 IBM Corporation
APA – Observation Session Submitted - Demo Backup Slide
20
New request is added to the Observation List panel.
APA is now monitoring CICS Region CICSAOR5. Now let’s go to that region
to invoke the ECAT transaction that has the performance problem.
© 2015 IBM Corporation
APA – Invoke the CICS Transaction - Demo Backup Slide
21
EnterEnter
Logon to the CICS Region being monitored.
Note: You must select the CICS data extractor in the Schedule New Measurement
panel, and enter the transaction name(s) or patterns you want to observe.
CICSAOR5 region activity is being monitored where the ECAT transaction is
being invoked.
© 2015 IBM Corporation
APA – Test the Application- Demo Backup Slide
22
EnterEnter
Test the online application for performance issues while APA monitors the
region activity.
Select Option 1 to display the catalog items.
© 2015 IBM Corporation
APA – Select an Item from the Catalog - Demo Backup Slide
23
EnterEnter
Select an item from the catalog to order.
There are two distinct types of data that Application Performance Analyzer
gathers when the CICS data extractor is active: Session statistics, and
Transaction measurement data.
© 2015 IBM Corporation
APA – Problem Detected - Demo Backup Slide
24
EnterPF5
After pressing PF5 so that the application suggests other products, I noticed
the clock at the bottom of the screen and that it was taking longer than
expected to display the results.
A transaction is only sampled if it was specified when the Observation
Request was created.
© 2015 IBM Corporation
APA – Back to APA for Analysis - Demo Backup Slide
25
Return to APA to see what could possibly be going on in the application.
APA’s observation session on CICSAOR5 has ended. The Analysis Reports can now be
accessed to determine what happened.
© 2015 IBM Corporation
APA – Performance Analysis Reports - Demo Backup Slide
26
Performance Analysis Reports enables in-depth analysis of the performance data collected.
Start your Analysis with the S01 Measurement Profile Report.
© 2015 IBM Corporation
APA – S01 Breakdown of CPU Activity - Demo Backup Slide
27
The S01 Measurement Profile Report provides an at-a-glance summary of various
aspects of the measurement data and helps you choose which other reports to
concentrate on. This Report shows an overall percentage of time in three states: CPU
time, Wait time, and Queued time.
Here, it collected 25,000 samples. With each sample, it determined which one of the
states it was in. Then drilled in to get the details about exactly what it was doing at that
moment. Over the duration, it collected a statistical sample that we can use to determine
where that time was actually spent.
© 2015 IBM Corporation
APA – S09 Measurement Analysis Report - Demo Backup Slide
28
The S09 Measurement Analysis Report provides recommendations of where some
performance improvements might exist. It is also suggests other reports to obtain more
detail about the area in question.
In this case, the report shows that there is a High CPU percentage of activity
observed in a single load module…and that there is high average response time for
a terminal attached CICS transaction.
© 2015 IBM Corporation
APA – CICS Measurement Reports - Demo Backup Slide
29
The CICS Reports show CICS performance related activity occurring in the selected
region during the Observation Session.
© 2015 IBM Corporation
APA-E04 CICS Mean Service Time by TXN - Demo Backup Slide
30
The E04 Report shows a list of all the transactions that were observed during the
monitoring session.
Click module to expand it
You can see here that, out of the average 28 second response time, over 20
seconds of that was caused by SEQUEL processing…
© 2015 IBM Corporation
APA – Expand DBRM Module ECP120- Demo Backup Slide
31
Use this report to see how time was spent by the CICS transactions that were
executing during the observation session.
Expanding that, you can see that an OPEN statement was the cause.
© 2015 IBM Corporation
APA – Click on OPEN to get details- Demo Backup Slide
32
Right click an item for more options.
Select Details to see more detail information regarding the OPEN
SQL statement.
© 2015 IBM Corporation
APA – SQL Statement Information Details- Demo Backup Slide
33
This panel shows the additional detail pertaining to the SQL statement to help you in
your analysis. This information can be provided to a developer or DBA to help you
tune this statement.
We can see here that the cause of the bulk of this response time was
this one SQL statement in program ECP120. Scroll down for further
information.
© 2015 IBM Corporation
APA – Formatted SQL Statement - Demo Backup Slide
34
A formatted version of the SQL statement can be displayed.
We can also see the fully formatted SQL statement in program ECP120.
© 2015 IBM Corporation
APA – DB2 Reports - Demo Backup Slide
35
There are other reports you can use for further analysis. The DB2 Performance
Reports displays measurement data directly related to SQL activity.
© 2015 IBM Corporation
APA –F03 DB2 SQL Activity by DBRM- Demo Backup Slide
36
The DB2 SQL activity by DBRM report shows DBRM’s that were running
during the measurement.
That DBRM is expanded and we can see that same SQL OPEN statement
that was displayed in the CICS report we saw earlier. So you can reach the
same conclusions from different reports.
© 2015 IBM Corporation
APA – Back to E04 CICS Report - Demo Backup Slide
37
Some reports, such as the E04 CICS Mean Service Time Report, also show
other information that you can drill down into for further analysis.
Besides the SQL statement, you can also see that program ECP110 is
responsible for just over 7 Seconds of the average response time.
Expanding that program, you see other programs and system modules that
program ECP110 called.
© 2015 IBM Corporation
APA – Module ECP110 Analyzed - Demo Backup Slide
38
You can drill down into source statements of application programs.
Right click on module ECP110 and select “Source Program Mapping” to get
program source statement information for application programs.
© 2015 IBM Corporation
APA – Drill Down into the Source - Demo Backup Slide
39
Source Program Mapping shows CPU time right down to the individual
statement level.
Statement 625 was the cause of most of the CPU time…The pink line
shows the CPU time of a system module that was executed because of
that statement. In other words, the CPU time of the system module was
attributed to that statement.
© 2015 IBM Corporation40
For our scenario, we are using a version of the supplied Catalog manager example (ECAT) that is supplied with CICS.
Problem: CICS PA Transaction Performance Alerts indicated that a threshold was exceeded by the ECAT Transaction
Action: A developer was notified to take a closer
look at the application.
Cause: Using APA, developer determined that an OPEN Sequel Statement in module ECP120 needs to be tuned and is the main cause of the slow response time for the ECAT transaction
For details on the ECAT sample, see https://www-01.ibm.com/support/knowledgecenter/SSGMCP_5.2.0/com.ibm.cics.ts.exampleapplication.doc/topics/dfhxa_t100.html
Trouble is contained!
Our Application Conclusion
© 2015 IBM Corporation
Summary
41
Use the combined capabilities of CICS Performance Analyzer and
Application Performance Analyzer for z/OS to:
• Identify performance tuning candidate applications at a CICS system
level
• Perform a subsequent deep-dive performance analysis right down to
the application source code level
Uncover and solve your CICS performance issues—right down to the source
code level TODAY!
© 2015 IBM Corporation42
Exploitation of GUIs made Simple!
Simplicity
All PD Tools GUIs talks to PD Tools Common Server (a quick to start and extensible!).
Additional capabilities added to the server by a simple configuration script (APA V13.1 exploits this feature)
PD Tools Studio: All PD Tools GUIs package together including z/OS Explorer
You can integrate into CICS Explorer or IMS Explorer for Development.
Flexibility
Select capabilities needed to create your own workbench.
Select capabilities you need from a single update site.
© 2015 IBM Corporation4343
Download CICS Explorer from the IBM web site
www.ibm.com/software/htp/cics/explorer/
© 2015 IBM Corporation4444
Download the PD Tools plug-ins or the PD Tools Studio at:
www.ibm.com/software/awdtools/deployment/pdtplugins
© 2015 IBM Corporation
More information
Web pages
– http://ibm.com/cics
– http://ibm.com/cics/showcase
– http://ibm.com/cics/tools
– http://ibm.com/software/awdtools/deployment
Analyst reports
– IBM CICS Tools: Discovery and Optimization for the Next Generation link
– IBM PD Tools – leads the pack again link
Threadsafety & Consolidation - extracting optimum performance from CICS Prezi YouTube
Social media
– Like CICS on Facebook at CICS Hursley or WebSphere and CICS Support
– Watch videos on YouTube - CICS and CICS Tools and IBM System z PD Tools
– View presentations on Slideshare
– Follow IBM_CICS on Twitter
http://www.ibm.com/support/docview.wss?uid=swg21384915
– See multiple channels by using the CICS Social Media Aggregator.
Subscribe to
– CICSbuzz for the latest newshttp://www-01.ibm.com/software/htp/cics/enews/
– Debug Tool newsletter http://www-01.ibm.com/support/docview.wss?uid=swg21422089
© 2015 IBM Corporation46
© 2015 IBM Corporation47