Real World Problem Solving Using Application Performance Management 10

Pre-Con Education: Real World Problem Solving Using Application Performance Management 10

Haroon Ahmed, Andreas Reiss

DevOps: Agile Ops

CA Technologies

CA APM SWAT

DO5X114E

@CAWorld

#CAWorld

The content provided in this CA World 2015 presentation is intended for informational purposes only and does not form any type of

warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.

For Informational Purposes Only

Terms of this Presentation

Abstract

CA Application Performance Management 10 dramatically reduces the time needed to find and solve app problems. In this session you will learn about common problem-solving techniques used by experts to solve real-world app problems. You will get a chance to put these techniques to the test in a hands-on lab that mimics an interesting application performance problem.

HaroonAhmed

CA Technologies

Senior Director Engineering Services

Agenda

TRIAGE AND DIAGNOSIS STRATEGY

USE CASE 1 – PIG IN A PYTHON

USE CASE 2 – DEATH BY 1000 PAPER CUTS

USE CASE 3 – OUT OF GAS

Typical Application Flow

OS/HWOS/HW

RUNTIMEJVM, CLR, Java Script, PHP

APPLICATION SERVERJ2EE, .Net, Node.js, PHP

SynchronousAsynchronousSynchronous

Asynchronous

MainframeMainframe

MessagingMessaging

ERP, CRMERP, CRM

Web ServicesWeb Services

DatabasesDatabases

Micro ServicesMicro Services

CICS / IMS / TuxedoCICS / IMS / Tuxedo

Response

Request

SOAPRESTJMS

JDBC, ADO.NET

SOAPRESTJMS

JDBC, ADO.NET

SynchronousAsynchronous

Mainframe

Messaging

ERP, CRM

Web Services

Databases

Micro Services

CICS / IMS / Tuxedo

Response

Request

SOAPRESTJMS

JDBC, ADO.NET

What to Monitor in Production

OS/HWOS/HW

SynchronousAsynchronousSynchronous

Asynchronous

MainframeMainframe

MessagingMessaging

ERP, CRMERP, CRM

Web ServicesWeb Services

DatabasesDatabases

Micro ServicesMicro Services

CICS / IMS / TuxedoCICS / IMS / Tuxedo

Response

Request

SOAPRESTJMS

JDBC, ADO.NET

SOAPRESTJMS

JDBC, ADO.NET

SynchronousAsynchronous

Mainframe

Messaging

ERP, CRM

Web Services

Databases

Micro Services

CICS / IMS / Tuxedo

Response

Request

SOAPRESTJMS

JDBC, ADO.NET

FRONTEND BACKEND

RESOURCES

Key Focus Areas

• Business Transaction, URL, Web Service

• Servlet, ASP.Net, JMSFrontend

• Web Service, EJB, Enterprise Service

• POJOMuck

• Database, Mainframe, LDAP, Web Services

• JDBC, ADO.Net, CTG etc…Backend

• Thread Pools, JDBC Pools, Memory, CPU

• GC MonitorResources

Metrics

Events

Relationships

Stall & Concurrency Count Brewing problems, Hot spots

Response Time and Throughput Lagging indicator

Resources GC, GC per Interval/time

Thread Utilization

Error Count

Instance Counts

Metrics

10 Minute

Interpreting Performance Metrics (Production)D

Ability to RespondReactive Proactive

Heap DumpsThread Dumps

Log Files

PingersProcess

BrewingLaggingHosed

Mean Time To ResolutionHigh Low

Response TimeThroughput

GC HeapCPU Utilization

Stall CountConcurrency Error Count

Instance Counts

GC Monitor

Thread PoolsJDBC Pools

Traces

Events

Status

Topology

Attribute

Relationships

Dynamic

Correlated View

Decorated

Timeline

Historical

Trending

Use Cases

1. PIG IN A PYTHON

2. DEATH BY A THOUSAND PAPER CUTS

3. OUT OF GAS

USE CASE 1

PIG IN A PYTHON

Symptoms: – Slower and slower over time or under load

– Consistent slowness

Root Cause:– External bottleneck

– Over-usage of external system

– Bad Coding: CPU bound component

– Network Saturation

– Threading: Chokepoint

Common Symptoms & Root Cause

In Team Center - relationship view, timeline and spark graphs

Look at frontend metrics, enable min/max to see outliers

Search tab and sort on average response time, enable min/max to see outliers

Run Transaction Tracer and look for long running components (to get single user transaction)

Where to Start…

Backend Bottleneck

Muck – Business Component Bottleneck

77% of the transaction time is spent waiting on the billing system

Framework Bottleneck – Bad XML

Use Case 2

DEATH BY A THOUSAND PAPER CUTS

Symptoms:– Consistent slowness

– Slower and slower under load

Root Cause:– Over-usage of external system

– Layer-itis

– The unending retry

Common Symptoms and Root Cause

In Team Center - relationship view, timeline and spark graphs

Look at frontend metrics, enable min/max to see outliers

Run Transaction Tracer and look for a bar code

Look at the Summary View and sort on # of calls. Look for high

call counts for one or more components in a single user

transaction

Where to Start…

Over Usage of External SystemNote: 4666 SQL calls in a single user txn - 2 SQL statements account for ~95% of SQL calls and ~50% of the total txn time!

Abuse of J2EE Framework

Misconfigured Resource – JDBC Adapter

Note the barcode of SQL calls. Also note commits after every JDBC Select. COMMIT call taking ~5 seconds!!!

Use Case 3

OUT OF GAS

Symptoms:– Sudden chaos

– Foreseeable lockup

– Slower and slower under load or over time

– Consistent slowness

– Sporadic hangs or aberrant errors

Root Cause:– External bottleneck

– Bad Coding: infinite loop

– The unending retry

– Internal resource bottleneck: over usage or under allocation

– Network Saturation

– Threading: Chokepoint, Deadlock/Livelock

Common Symptoms and Root Cause

APM– Sudden/gradual increase in stall counts and stalled events

– Increasing concurrency counts and decreasing responses per interval

– Investigator/search tab used to find components with stalls or high concurrency

Application Server Resources– Look for sudden or slow thread pool / JDBC pool depletion

OS Resources– Look for changes in CPU and Heap utilization

– Increases in CPU and Full GC time/count could point to bad code

– CPU Utilization decreases and JVM halts – out of application server resources

Where to Start…

External Bottleneck – DB Stored Procedure is the Root Cause

Thread Depletion

Increase Count against a DB stored procedure

Stored Procedure execution time from 25 min and UP

Stalls That Degrade Performance Over Time

Surge in Application Activity

Stalled threads degrading application reliability over time

Stall Events With Smart Instrumentation

Hands-on Lab

Challenge: Triage and Diagnose Using APM 10

2. Webview1. EM

3. Start

MathApp

4. Run

JMeter

4. Run Lab5

problem

scenario

Hands-on Exercise

1. Log into the lab VM (Administrator/CAdemo123)

2. Start the Enterprise Manager

3. Start Webview

4. Start MathApp

5. Start Baseline Load on MathApp

6. Wait 5 minutes

7. Start Lab5 problem scenario

8. Triage and Diagnose the problem

– Using ATC, Webview, Investigator tree, events etc.

Recommended Sessions

SESSION # TITLE DATE/TIME

DO5X189SHow to Achieve a Customer-Centric View in an Omni-

Channel World11/18/2015 at 1:00 pm

DO5X194SMonitor Microservices, Containers, Cloud Foundry and

Node with CA Application Performance Management11/18/2015 at 4:30 pm

DO5X193S

Customize CA Application Performance Management

with Tips for Using the CA Application Performance

Management Open APIs

11/19/2015 at 4:30 pm

Must See Demos

Application Performance Management and DevOps, featuring APM use in preproduction scenarios

Application Performance Management

Theater 5

Application Performance Management, Modern Monitoring, featuring the new APM Team Center

Theater 5

Ensuring a “5 star” mobile app experience with CA Mobile App Analytics

Mobile App Analytics

Theater 5

Unified Monitoring: APM Integrations including UIM

Theater 5

Follow On Conversations At…

Smart Bar

Theater 5

Tech Talks

Theater 5

For More Information

To learn more, please visit:

http://cainc.to/Nv2VOe

CA World ’15

Real World Problem Solving Using Application Performance Management 10

Technology

Transcript of Real World Problem Solving Using Application Performance Management 10

Solving the word problem in real time - Warwick Insitehomepages.warwick.ac.uk/.../realtimewp2/rtwp2.pdf · Solving the word problem in real time Derek F. Holt and Sarah Rees Mathematics

Problem Solving Answers 4.1-4.8.gwb - 1/8 - Wed Jan 14 ... Solving... · Problem Solving REAL WORLD Model Reason ... $0.12 tor 1 ounce $1.92 for 16 ounces ... Problem Solving 1 problem

Problem Solving with Quadratics. Problem Solving Guide:

Problem Solving REAL WORLD . roblem Solving REAL WORLD . Problem Solving

Solving a Real-world Supply-Chain Management Problem Using ...kdeb/papers/c2017009.pdf · Solving a Real-world Supply-Chain Management Problem Using a Bilevel Evolutionary Approach

explorefourth.weebly.comexplorefourth.weebly.com/uploads/8/3/4/2/83423860/chapter_11_answer_key.pdf · REAL WORLD roblem Solving . REAL WORLD roblem Solving . Problem Solving

Examples of real-life problem solving with Mascot Integra

· 2019-09-02 · Problem Solving REAL WORLD . Problem Solving . REAL WORLD roblem Solving

· Problem Solving REAL WORLD . REAL WORLD roblem Solving . REAL WORLD roblem Solving

oliver4.weebly.comoliver4.weebly.com/uploads/1/3/3/6/13361905/... · REAL WORLD roblem Solving . roblem Solving REAL WORLD . Problem Solving REAL WORLD

Solving a Large Real-world Bus Driver Scheduling Problem ... · Solving a Large Real-world Bus Driver Scheduling Problem with a Multi-assignment based Heuristic Algorithm ... (itinerary)

One at a Time Real Time Non-Clinical Problem-solving.

Technological Problem Solving Unit Five Technological Problem Solving Problem Solving vs. Design Problem solving Model Design Briefs Creative Brainstorming.

Real World Problem-Solving - Tufts UniversitySarathy Real World Problem-Solving and the process of forming and executing a solution plan (Robertson, 2016). 2.2.1. Problem Deﬁnition

The C Programming Lecture 24. Summary of Previous Lecture Programming in real life. Introduction What is Problem Solving? Problem Solving process.

Climate-Change Problem Solving: Structured Approaches Based on Real-World Experiences

explorefourth.weebly.comexplorefourth.weebly.com/uploads/8/3/4/2/83423860/... · 2018. 12. 30. · Problem Solving REAL WORLD . Problem Solving REAL WORLD . Problem Solving REAL WORLD

explorefourth.weebly.comexplorefourth.weebly.com/uploads/8/3/4/2/83423860/chapter_1_answer_key.pdf · Problem Solving REAL WORLD . Problem Solving . REAL WORLD roblem Solving

Problem solving models Rational problem solving Creative problem solving.

· Problem Solving REAL WORLD . REAL WORLD roblem Solving . Problem Solving REAL WORLD