2011 sept 13 performance webinar with notes (5b)

®

Find the Needle in the Haystack…

Pinpoint performance problems before anyone

knows they are there!

Please Dial In For Audio:

1-617-614-3800

Passcode:

45748157#

Copyright © 2011 Macro 4. All rights reserved.

Pinpoint Performance Problems

Hello!!Presenter:Morrie Meyer

Macro 4/UNICOM Panel of Experts:

John Cosmas Phil Mann



‘Handouts’

We are recording this webinar and will send you a link to that

recording



Please use Webex chat to ask questions

Click the Chat Icon on the Webex Toolbar (see bottom right corner)



Using the Chat Window

Please enter questions in bottom section of chat window and send to ‘All Participants’

We will monitor the chat area throughout and ensure all questions

are covered by the end



Agenda

• Macro 4/UNICOM Intro

• Impact of Poor Application Performance

• Application Performance Management– What is APM?– Why APM is not a standard (yet)

• Adopting a Holistic APM Philosophy

• Implementing an APM Strategy

• Macro 4’s APM approach– Toolset offerings– Case studies



Who is Macro 4/UNICOM? • 40 years in the industry• Global Company• Part of the UNICOM Group of Companies• IBM Mainframe Solutions

– Application Availability & Performance Management– Systems Management– Document Archiving & Output Management– Security & Tape Management– Internet-enablement of mainframe solutions

• Open Systems Solutions– Document Archiving & Output Management– Application Decommissioning– Asset Management & Application Performance

• i Series Products– Change Management/Automation/Performance Tuning

• ITPowerPac– Asset, Change and Incident Management



Poor Application Performance Will Cost YOU

• The impact of poorly performing applications can not be

overstated:– Missed SLAs (often resulting in penalties)– Possible regulatory fines– Increased IT costs– Decreased staff productivity– Loss of customers = lost revenue– Damage to reputation and brand



What are built in causes of performance problems?

In addition to business demands and even without taking business growth into account, performance can be impacted by:

– CPU creep from existing applications

– Changes to databases and DB access methods

– New middleware competing for CPU resources

– On average, a 7-10 % annual MIPS growth occurs just to support new releases of Operating Systems

Often the MIPS increase rate is higher than the business growth rate!



Poor Performance = Lost Revenue and Lost Customers ……….But Don’t Just Take Our Word For It

• Retail Bank: “Every hour of downtime [on our website] impacts $19

million in sales”

• Online Insurance Company: “Our competition is just a mouse click

away – we can’t make our customers wait”

• Call Center Rep: “If applications performed more quickly, I could

handle more calls, and our customers wouldn’t have to wait so long

on hold”



What Causes Downtime?

• What do managers and senior leaders view as the primary cause of downtime? – 61% - Applications

• Increased to 80% in organizations with higher use of internally developed applications

– 21% - Hardware – 18% - Don’t know

Survey of more than 200 U.S. IT managers and senior leaders conducted by Managed Objects

• 73% of problems reported by end users through the service desk are not

detected by infrastructure management tools Forrester

• One quarter of one percent (0.25%) of revenue, can be considered a quick, but

credible benchmark cost to businesses of poor software performance and

availability

Survey of more than 200 U.S. IT managers and senior leaders conducted by Managed Objects



Poor Performance = Increased Resource Usage

• Inefficient, resource consumptive applications and systems

create excessive demands: – Extra MIPS– Extra upgrades– Extra servers– Extra memory– Extra disk space– Extra man hours



What Industry Watchers Are Saying

• “Application performance still takes a back seat to speedy development these

days, even though slow application performance is commonplace. [We] see the

minority of organizations creating and maintaining cradle-to-grave application

lifecycle performance strategies, even though most businesses are so

dependent on software today”– Scott Barber/Theresa Lanowitz (Performance Testing For Web Applications)

• “Organizations are moving applications to the Cloud at an impressive rate as the

Cloud is perceived as giving organizations a competitive edge by offering lower

IT costs. However, organizations fail to analyze the ownership of Application

Performance Management in the Cloud and what this change….will do to the

customer’s overall experience”– Kay Diller (The Cloud: APM Ownership Challenges)



What Industry Watchers Are Saying

• “Today’s enterprise is dependent on the superlative operation of any number of

applications across it’s integrated IT structure. Non-availability or poor

performance has significant consequences in today’s competitive 24x7 global

market place, impacting efficient transaction management and mission-critical

operations”– Butler Group

• “Through 2012, 80% of application performance and availability problems will be

blamed on the network, but the network will only represent 20% of the root

cause”– Gartner



Web Application Servers Windows

Workstations,iPads, Smartphones, etc.

Other Platforms

Applications

Tools

Interfaces

Mainframes

DB2

Performance Management

In

fra

structure

iSeries

So, what’s the problem ? Until a few years ago:



Everything was internal and easily accessible

• The Data Center – Mainframes– Mid-range (iSeries)– Distributed systems/networks

• IT performance was measured and tuned from a mainly system-centric

view

• Application and system problems could usually be separated fairly quickly

• As technologies evolved, application performance became the focus as

customers became more savvy and expected:– On-Demand services– Instant information availability– Web access– Low cost



So, what’s the problem now? The Cloud!



The Cloud Changes Everything

• In the Cloud, actions can be automated more easily than in traditional

environments

• Cloud resources are paid for based on usage

• Cloud resources are created and allocated dynamically based on

application requirements

• Web sites have evolved into web applications, with multiple users on

shared back-end services

• The line between Client-side and Server-side continues to blur

• Application performance is more critical than ever, not only in terms of

customer satisfaction, but also in terms of cost



The Problem

Development

Users

Systems

Executives

Works independently, according to spec, but does not consult with

Whose main concern is keeping everything running, but have no

application knowledge

Are focusing on the business, and don’t

understand why customer satisfaction

is down

Which leaves users frustrated and

ready to take their business elsewhere

(if they haven’t already!!)



Application Performance Management (APM) What exactly is it?

• Application Performance Management, Measurement, Monitoring or

Tuning

– depends on who you are and your involvement in it

• It is looking at the performance of your computer systems from the

point of view of the applications that run on them and provide the

service that you deliver to your users,

– and then trying to improve (or tune) them

• It is different from the normal process of system performance and

tuning

– which looks at things from a system point of view

– tunes from a system point of view

– and only has limited success in controlling costs and delivering service improvements



So, why APM?

• Because there are huge benefits to be had if you practice Application

Performance Management properly

– Both in terms of cost control; avoiding additional expenditure and making real significant savings on your existing costs

– And service improvements; improved response times, service levels, more operational flexibility

• Yes, you can have both. Make savings (cut-backs) and improve service

at the same time

• A rare ‘win, win’ situation



APM – So why isn’t everyone doing it?

• APM is the ‘fair-haired stepchild’

– always comes second to system performance measurement and tuning

• Why?

• Is it?

– Not as important

– Not as beneficial

• No, but

– Its not as easy to do (where do you start?)

– difficult to fit into existing organizational structures

– Not my job, someone else’s job, so nobody’s job



APM – why it doesn’t get done - SUCCESSFULLY

• Difficulty in deciding whose job it is

– If it’s application related then only developers can do it …

but

– … developers don’t like dealing with production systems

• Confusion between the main things needed for application tuning

– Finding and analyzing the performance problems in applications …

– … and fixing them

• Should be a joint effort

– With ideally operational staff taking the lead role



Application vs. System TuningHow’s It Different?

• You’ll need to take an application centric view– What service does the application get from the system …– ... rather than what an application does to the system as it

executes

• System monitors – Can help you identify application processes in need of

tuning …... but can’t show you what’s happening (in any detail) as they execute

• You need to look ‘under the hood’ of the application to focus your performance tuning efforts



Development versus Systems Viewpoint

• Development

– On other projects, no resources available

– People who wrote it are no longer here

– It works according to the specs, performance wasn’t part of the specs

– What do you mean by ‘performance’ problem?

• Systems

– Can’t resolve application issues

– No application knowledge

– No development support

– Some applications are 3rd party packages

– ‘It’s not our job’



Adopt An Application Performance Management Philosophy

• Organizations need to recognize that APM :

– Is more than ‘just another software purchase’

– Has a Lifecycle directly parallel to the Application lifecycle

– Can avoid production deployment issues by addressing performance proactively (instead of reactively)

– Has a positive impact on organizational efficiency

– Is probably the most direct route to customer satisfaction



Develop Application Performance Management Strategy

• Organizations need to develop and adopt a coherent performance management strategy:

– Invest in appropriate technology and services to enable greater visibility and control over performance

– Gain insight into performance and valuable know-how that can be transferred back into development

– Assess all applications and subsystems, prioritize according to how much the business is dependent on them, and how likely they are to fail

– You can’t improve what you can’t measure!



Implement the APM Strategy

• Performance Management Strategy across the entire enterprise:– integrate the performance strategy with other enterprise

initiatives (CMM, ITIL)– obtain Executive level commitment– joint responsibilities of all stakeholders (e.g. systems,

development, testing, production) play a critical role

• Implement a standard & measureable process throughout the

entire lifecycle– Start in development – proactive is less expensive than

reactive– It is ten times more expensive to correct a performance issue

in production then it is to correct it in testing



APM Philosophy + APM Strategy = APM Best Practices

• Understand

– Know your applications, users, and requirements

• Measure

– Properly measure key aspects of application performance

• Communicate

– Provide relevant performance reports to management

• Link

– Show specific business-performance links

Source: NetForecast, Inc.



APM = Continual Service Improvement

Understand Measure

CommunicateLink

APM Best Practices



Benefits Of Performance Management Strategy

• Among those enterprises that have adopted a performance

management strategy:

– 74% are perceived as an industry leader compared to only 44% that have not adopted a strategy

– 83% are ranked in the top third of their industry compared to 52% that have not adopted a strategy

– 93% boast clear agreement on strategy among senior management compared to 37% that have not adopted a strategy

John H. Lingle and William A. Schieman – “Is Measurement Worth It?”

• Companies …. maintain exemplary records of continuous IT efficiency

and effectiveness improvements – with some organizations reducing

costs by up to 30% while increasing value by 2x – 3xMETA Group



The Solution - Application Performance Management

Development Users

Systems Executives

TOOLS



Application Performance Management Solutions New Approach for a Multi-Platform World

• Understand mainframe, iSeries

and distributed infrastructures

• Help a wide range of users – not

just experts

• Address cross-platform issues



Our Driving Philosophy:

To be a trusted, long-term partner to our customers in

the process of Application Performance Management by:

Listening to their needs

Understanding their operational and strategic goals

Helping them achieve theirplans by applying creative, innovative and supportiveintellect

Showing care and consideration for their problems

By being fair and reasonable

Supplying quality specialist advice and solutions

Delivering product support that is second to none



UNICOM Perspective

• To make a real difference we believe that a 360o view must be taken

• Rather than focus only on the supporting infrastructure and then try to guess

what the application is really doing – focus on the application and work down to

the infrastructure

Application Centric – Cross Platform

Workstation

Internet

OtherPlatforms

Applications

Tools

Data and Applications

Infrastructure Centric

ApplicationServers Web

Servers



Business Drivers defining our Approach

• MIPS reclamation

• Reduce Carbon Footprint– Power Consumption– Paper consumption

• Improve Response times

• Improve Application Uptime

• Improve Service levels to users

• Improve User Satisfaction

• Reduce Operational Costs of Service delivery



UNICOM APM Tools

• z/OS Performance - ExpeTune Suite

– Pinpoint application performance issues with precision - FreezeFrame

– Identify candidates for performance analysis - ExpeTune

– Target, analyze & improve problem SQL - ExpeTune DB

• Multi-Platform Performance Monitoring – SUPERMON® for Java

– JEE application monitoring with pre-emptive alerts, correlation across multiple tiers (DB2, Oracle & CICS)

– Application centric drill down to infrastructure

• iSeries Performance Monitoring – SUPERMON® for iSeries

– Auto discovery, continuous performance metrics collection

– Manage job priority based on CPU consumption

APDEX Certified



The ExpeTune SuiteFreezeFrame ExpeTune ExpeTune DB

• Determines which applications will benefit from tuning

• Automatically sets up observations for applications of interest

• Precisely determines, down to the line of code, large resource users

• Sets off observations based on thresholds, both in batch and CICS

• Knowledge Base - Provides recommendations for tuning actions

• Determines DB2 inefficiencies

• Analyzes Dynamic Statement Cache activity



FreezeFrame



FreezeFrame – Source Listing Support



ExpeTune Analysis Based on SMF Records

Macro 4 Combined MIPCPU - MIPS by Application Grouping

0

10

20

30

40

50

60

70

9:00 AM 10:00 AM 11:00 AM 12:00 PM 1:00 PM 2:00 PM 3:00 PM 4:00 PM 5:00 PM 6:00 PM 7:00 PM 8:00 PM

Monday

21/04/2008

MIP

S

18 - Unknown

15 - IXGLOGR

14 - BATCH

13 - *AGYJOB

12 - DB2

11 - DBM1

10 - COMMS

9 - O/EDITION

8 - TSO

7 - HSM

6 - MONITORS

5 - IMS-JOB

4 - IMS-STC

3 - CICS

2 - STC

1 - Uncaptured

Plex (All) System (All)

Sum of MIPS

Date Day Hour

Order

Category

Total MIPS



SUPERMON® for Java

• Automatic discovery (Java EE), continuous collection and storage of volume and high level performance metrics, across complex multi-server environments

• Correlate that with detailed resource usage analysis in other Enterprise Information Systems, such as …– DB2, CICS, Oracle

• A modern, Web-launched rich client, providing roles and goals based functionality covering the areas of:– Ongoing monitoring with management dashboard– Post event analysis and comparison– Summary reporting– Scheduling and analysis of probe sessions

• Apdex rating is built into UNICOM reporting



SUPERMON® for Java



SUPERMON® for iSeries

• Automatic discovery, continuous collection and storage of volume and high level performance metrics form a single IBM I machine or multiple networked IBM I machines

• A substantial reduction in disk space occupied by performance data and significant paper savings over standard IBM performance tools

• Takes advantage of the SUPERMON® rich client interface for graphic display of iSeries performance data

• Facilities to produce a comprehensive range of performance related reports based on historical data

• Ability to automatically lower the priority of jobs consuming more than a pre-set amount of CPU

• The means to produce reliable long term projections to tackle potential performance issues before they arise



SUPERMON® for iSeries



UNICOM’s Guiding Principles for Performance

• Simple, intuitive, and user friendly interfaces

• Multiple application technologies and hardware platforms

• Usable by a wide range of users – not just experts– Much greater return on investment

• Application-centric monitoring and viewpoint– Automatic discovery of applications– Ability to drill down into infrastructure issues

• Capable of detecting and pinpointing problems quickly – Proactive monitoring and alerting– Analysis of historical performance– Collaboration facilities

• Production ready– In both senses – highly robust and negligible impact on monitored

systems– Effective in preproduction and development



What our Customers Think:

“When the new application was deployed it started to burn CPU after having been in and running for a few hours early morning, if it continued like that then we would have had no choice but to back out the code and deploy again at a later date. However with SUPERMON® for Java we were able to spot the CPU intensive SQL statement and modify this then redeploy within the same day thus not having a high impact on production.”Catalog & Internet Retailer

“During the initial FreezeFrame training, a programmer identified a performance issue in a COBOL program. The programmer corrected the problem and found that CPU processing time was reduced from over 50 minutes to 1 to 2 minutes, a savings of over 96 %.”Telecommunications Enterprise - US



Real Life Performance Challenges

• Supermarket Chain

• Manufacturer of Industrial Cabling

• Trade, Travel and Tourism Company



Challenge #1

• Biggest Supermarket chain in UK

– huge growth in the last 10-20 years

• Problem – Constant Replenishment System

– Forcing CPU upgrades every year in line with Christmas peak

– Taking so long to run that it was causing real operational problems

• Goals

– To break this cycle of forced CPU upgrades

– regain control of the processing cycle for the system

– save money - preferably lots of it



What had they already done?

• Tried to address application performance issues internally

– tried to get internal groups ‘interested’

– and place overall responsibility for application performance tuning

• Some tuning efforts had been made

– Using internal staff

– And external DB2 tuning experts

• Some, but only limited success



What did they do?

• Introduced a ‘step change’ program

– the ‘catalyst’ that made everything else work

• Got the right tools

– an application performance monitoring tool [FreezeFrame]

– DB2 access path analysis tool [ExpeTune DB]

– etc.

• Got assistance in setting up the application tuning process

– training from UNICOM



Results?

• Huge CPU savings in the cost of running the ‘constant replenishing’

system– several thousand MIPS in the first pass– no need for normal Christmas time machine upgrade

• Most savings came from identifying SQL that was using lots of CPU– … and then tuning it

• Other savings from improving logic and execution of date routine– tiny savings on every execution …– …. but executed many millions of times a day

• Run times dramatically reduced– giving back operational control of the system



Challenge #2

• Major U.S. supplier of industrial cabling– Annual revenue $5 billion + per year

• Problem – Poor J2EE application performance– Both internal and customer facing applications (home-grown and

vendor supplied)– Issues ranged from poor performance to complete failure– Lost revenue + fines for late deliveries

• Goals – Find an Application Performance Management tool that:– Has an application centric approach– Allow collaborative problem diagnosis– Verify the impact of application changes– Utilize a small footprint in their production environment



What had they already done?

• Tried to address application performance issues internally

– Attempts to develop internal APM tools did not prove successful

– Trials of other APM tools did not meet their requirements

• Some were not cost effective

• Some were not user-friendly or easily deployed

• Some tuning efforts were provided by the third party application

providers but:

– Internal staff were kept ‘out of the loop’

– Changes had minimal performance impact

• Still needed an APM tool that met their requirements



What did they do?

• Engaged UNICOM in a Proof of Concept

– Defined explicit acceptance criteria

– Evaluation used both home grown and third party applications

• The right tool for the task

– SUPERMON® for Java APM tool which includes:

• DB2 – JDBC correlation analysis

• Flexible reporting facilities

• Method level profiling and analysis

• CPU, JVM utilization

• zIIP and zAAP engine performance statistics



Results?

• Support team– Increased productivity– Able to pinpoint and trouble shoot performance issues faster

• Development team– More granular detail available during pre-deployment phase– Time period comparison revealed the impact of code changes in

production

• Application business owners– Improved product quality– Higher availability and fewer errors

• Customers– Improved performance of customer facing applications enabled

contractual obligations to be met, avoiding costly penalties



Challenge #3

• One of the leading trading and travel companies in Europe– Annual revenue $50 billion + per year

• Problem – IT infrastructure growth on HTTP and Web Application Servers– Needed comprehensive monitoring of thousands of Java methods– Long SQL statements caused unacceptable application response

times– Complex distributed Web applications required event notification

for high availability

• Goals – Find an Application Performance Management tool that:– Provides auto discovery for ‘Robotic monitoring’– Can correlate SQL statements back to the calling Java class and

method– Supports DB2, Oracle and CICS



What did they do?

• By implementing SUPERMON® for Java they were able to:– Monitor the performance of Java programs, database SQL calls and

back end CICS transaction processing through a single, straightforward, user friendly product.

– Perform monitoring functions and historical comparisons more efficiently. These take less time and use fewer resources.

– Make time savings on analysis and reporting as compared with previous products:• SUPERMON® for Java is intuitive and easy to work with and

requires minimal training • Data is clearly arranged and displayed in tables and can easily

be exported in CSV format when further processing is required • All online reports and graphics can be produced in PDF and

other formats to meet the company's ongoing reporting requirements



Results?

• Can now obtain a thorough overview of resource consumption for all

applications across multiple environments via a single product

• Now able to:

– save resources

– lessen usage of applications

– Reduce CPU costs

– Provide high performance applications to all users

• Minimize business risks

• Increase customer satisfaction



Recap

• Application performance impact

– Why it matters

– Why organizations should rethink and re-prioritize

• Application Performance Management (APM)

– Philosophy + strategy = APM Best Practices

• M4’s APM approach

– Tool sets

– Real world examples



In Conclusion:

This is your system without APM



This is your system WITH APM



Questions?

Please enter questions in bottom section of chat window and send to ‘All Participants’

®Thank You!!!For further information please go to:

www.unicomglobal.com or www.macro4.com

2011 sept 13 performance webinar with notes (5b)

Technology

Transcript of 2011 sept 13 performance webinar with notes (5b)