Application Performance Management - Uni...

Institute of Software Technology

Reliable Software Systems

ApplicationPerformanceManagement

State of the Art andChallenges for the Future

André van HoornMario MannDušan OkanovićChristoph Heger

Tutorial @ 8th ACM/SPEC International Conference on Performance Engineering,

April 22, 2017, L‘Aquila, Italy

André van Hoorn

University of Stuttgart

Inst. of Softw. Techn.

Reliable SW Systems

Mario Mann

NovaTec Consulting

Application

Performance

Management

Dušan Okanović

University of Stuttgart

Inst. of Softw. Techn.

Reliable SW Systems

Christoph Heger

NovaTec Consulting

Application

Performance

Management

Who Are These Guys?

Performance Problems are Omnipresent

An unexpected

error occured

Temporarily

not available

… more capacity

is on the way

Try again later

Please visit us

again later

We are

experiencing

heavy demand

4/22/2017Application Performance Management: State of the Art and Challenges for the Future

Influence of Poor Performance on the Success of Businesses

“Application performance management (APM), as a core IT operations discipline, aims to achieve an adequate level of performance during operations. To achieve this,

APM comprises methods, techniques, and tools for

• continuously monitoring the state of an application system and its usage, as well as for

• detecting, diagnosing, and resolving performance-related problems using the monitored data.”

C. Heger, A. van Hoorn, D. Okanović, M. Mann.:

Application performance management: State of the art and challenges for the future.

In: Proc. 8th ACM/SPEC ICPE, ACM (2017)

Agenda

Introductionto APM

APM Tools

APM Advanced

Discussion

45 min

45+15 min

60 min

15 min

André van Hoorn und Stefan Siegl.

Application Performance Management (APM). Continuous Monitoring of Application Performance.

(Poster in ObjektSpektrum magazine; in German)

Order for free: http://www.sigs-datacom.de/wissen/fachposter.html

• Agents collect data from all system levels

• On application level the agents are often technology-dependent

Collecting Data from All System Levels1.

Where? What? How?

Trace-based Metrics (Selection)

Metric

Response Time

CPU Time

Method Name

Return Type

Logging Level

SQL Statement

Error Message

Okanović, D., van Hoorn, A., Heger, C., Wert, A., Siegl, S.:

Towards performance tooling interoperability: An open format for representing execution traces.

In: Proc. EPEW ’16. LNCS, Springer (2016)

Monitoring (Measurement-based Performance Evaluation)

Database

Server

Client

Browser

Request

Response

Result

Application

Server

Recorder

0.2s 0.3s 0.1s 3.0s

0.1s0.4s

VisualizationData Analysis

(fictio

naltim

• Event-driven measurement

• Event: a change in system state

• e.g., incoming requests, access to HDD, operation execution, throwing exceptions

• Calculate performance metrics whenever an event occurs

• Simplest metric: counter

• Tracing (also event-driven)

• Recording a certain part of the system state when an event occurs

• Example: Code adjustments

• printf – easy, but with questionable maintainability

• Bytecode engineering

• Aspect-oriented Programming (AOP)

• Example: Sampling

• Measurement (state or counter) is performed in specified time intervals

• Through overhead and increased resource consumption, measurements

influence the system!

Measurement Approaches and Techniques (Examples)

• Data is collected from the system…

• represented as time series…

Reconstructing Information from Data2.

• Data is collected from the system…

• represented as time series…

• … and as detailed execution traces, and

used to support problem analysis

Reconstructing Information from Data2.

Traces in an APM Tool

• High quantity of information has to be pre-processed

• It has proven useful to use different views to show the data

• Views are navigable and can be categorized by both scope and detail level

Visualization Through Navigable Views3.

Example: Application Topology Discovery and Visualization

Manual or automated conclusions and actions can be derived from the

information, e.g.,

• Problem detection and alerting

• E.g., increased response times and resource utilization

• Detection, for instance, based on thresholds and baselines

• Problem diagnosis and root cause isolation

• E.g., N+1 problem, too many remote calls, poor DB queries

• Detection based on monitoring information

• System refactoring and adaptation

• E.g., auto-scaling in cloud-based architectures

Interpreting and Using the Information4.

• http://kieker-monitoring.net

1/20/2016 22

Dynamic Software Analysis and ApplicationPerformance Management

Application Performance Management: State of the Art and Challenges for the Future

Part 2 – APM Tools

• Commercial APM tools

• Magic quadrant

• Timeline of APM tools

• Goals of APM tools

• Architecture

• EUM, Server, Database Monitoring

• How APM tools can help you to identify problems

• Open Source APM tools

Contents

Commercial APM Tools

Magic Quadrant

4/22/2017

Timeline of Tools

Wily – CA APM

DynaTrace

AppDynamics

inspectIT

2005 2008

Kieker

Commercial tools

Open source tools

Goals of APM Tools

End-User

PerspectiveEnd to End

Monitoring

Analytics

Lifecycle by

Design

@Dynatrace

Architecture – Dynatrace

Java .NET

Mainframe,

Native, … Database

Dynatrace Sessions

Performance WarehouseDynatrace Frontend ServerDynatrace Backend Server

Agent/PurePath

Collector GroupMonitoring Collector

Web Server /Node.js

/ NGINX / PHP

Dynatrace

Client

Web UI Dashboards

@Dynatrace

Sensors – How Do They Work?

• Every sensor (custom and OOTB) instruments Java/.NET methods

• Code gets added to each method that matches the sensor rule to

• measure execution time

• capture method arguments and return values

• count method invocations

• capture exceptions

Uninstrumented Application Instrumented Application

Execution Time

of Diagnosis Code

@Dynatrace

4/22/2017

Architecture – AppDynamics

Source:

https://docs.appdynamics.com/download/attachments/34271888/dbmondplussingleapp_appd_architecture.png?version=4

&modificationDate=1487053783658&api=v2

10 ms 10 ms 10 ms 10 ms

Sensors – How Do They Work?

@Dynatrace

Applications and Infrastructure Overview

Analysis dashboards

UEM Visits grouped byChannel and Application

Show the total Tier

time with Process,

Host, and

Transaction

Deeper analysis

Application Overview – Dynatrace

Application Overview – AppDynamics

Tiers with technology

Databases

Remote Services

@AppDynamics

Real User Monitoring – AppDynamics

Infrastructure

Visibility

End User Monitoring Application Performance Management

@AppDynamics

Real User Monitoring – Dynatrace

Source: https://qph.ec.quoracdn.net/main-qimg-cd5a4b6e8cc29465d2cea98e055f606e.webp

End User Monitoring – AppDynamics

Geolocation

CrashesRequests

Server Monitoring – AppDynamics

Availability CPU Usage

Memory Usage

Network

VolumesProcesses

Database Monitoring – AppDynamics

Alerting – AppDynamics

Dashboarding

@Dynatrace

How APM tools can help to identify problems?

BusinessTransaction

Potential Issues

User Experience

@AppDynamics

Top findings to optimize

performance

Filter

How APM tools can help to identify problems?

@Dynatrace

• Share configurations between stages

• Identify business transactions and map this to functional level

Challenges

Open Source APM tools

Licensing costs

Vendor Lock-In

• Typical model: x $ per agent scaling?

Weaknesses of Commercial APM Tools

Microservices Internet of Things

[Abbildung: http://blog.wso2.com] [Abbildung: Christian Hinkelmann, http://nahverkehrhamburg.de]

Flexibility Interoperability Sustainability

Mobile Revolution

[Abbildung: https://uxmag.com]

Adaption to own

Bug fixing

Sources

Tools for analyse

OtherAPM tools

Stopp development

Change of strategy of

APM vendor

4/22/2017Application Performance Management: State of the Art and Challenges for the Future 4/22/2017Application Performance Management: State of the Art and Challenges for the Future

Open Source APM tools

Load Testing

Performance

Modeling

Web Performance

Analysis

System &

Resources Monitoring

Real User Monitoring

JRatJMemProf

Low-Level

Performance Profiling

Monitoring &

Application Deep Dive

What is inspectIT?

pen Source

Platf rm

Pareto-principle

• Focus on main functionality

• Experience of APM projects

Platform-principle

• Integration with tools

• Extensibility

Open-Source APM solution

• Development since 2005

• Open Source since 2015

Gathering and Visualizing Timeseries Data

Time Series Databases

Graphing ToolsData Collectors

Custom Code…

Persist data Query & Visualize

Dashboards

The Flaw of Averages

The Flaw of Averages: Why We Underestimate Risk in the Face of

Uncertainty

Sam L. Savage, with illustrations by Jeff Danziger –

http://flawofaverages.com

Gil Tene – https://www.youtube.com/watch?v=lJ8ydIuPFeU

Dashboards

Alerting

Anomaly Detection & Alerting

End User Experience?

[Image | http://cdn2.hubspot.net/hub/2224117/file-4141070752-jpg/blog-files/three-

e1409846588737.jpg]

Satisfied Tolerating Frustrated

Performance User Actions Users

Performance issues

Functional Errors

Network (Local ISPs, Mobile network carriers)

Third party content providers

First user action

Last user action

Did the visit convert?

Did the visit bounce?

Device

Resolution

Browser Versions

Geolocation

What is Web Performance Analysis about?

Measuring End-User Experience

Real User

Testing

Synthetic

Monitoring

Continuous external testing

Known testing nodes

Different Locations (ISP, Networks)

No baseline traffic required

Competitor Benchmark

Availability testing (incl. 3rd Party)

e.g., every 30 mins from different nodes

Real User

Monitoring

Real End Users

Real Interaction

Real traffic

External testing with all major

browsers,

operating systems,

mobile devices

and real world data

QoS Validation

QoS ExpectationQoS Optimization

Anomaly

Testing and Monitoring Web Performance

[http://www.webpagetest.org/performance_optimization.php?test=160623_DF_MV7&run=1

&cached=0]

webpagetest.org

Server Monitoring

Source: https://www.icinga.com/wp-content/uploads/2016/08/Screen-Shot-2016-08-29-at-16.11.23.png

• No licensing costs

• Growing community

• Pick the tools which work for your needs and combine them

Benefits of Open Source Tools

• From perspectives of industry and research…

Challenges

Part 3 – APM Advanced

Problem Diagnosis

Component

Deep-Dive

Interpretation of

measurements

Detect problem

Application Performance Management: State of the Art and Challenges for the Future 4/22/2017

Problem: Too Many Traces to Analyze

Lots of data to

analyze

http://diagnoseit.github.io/

diagnoseIT Overview

Node 1

Node n

Traces

Instrumentation

Refinement

Request

Result

Monitoring Tool

Dynamic

Instrumentation

API Labeling

Location

Identification

Instrumentation

Quality Manager

Result Query

4/22/2017

C. Heger, A. van Hoorn, D. Okanović, S. Siegl, and A. Wert.

Expert-guided automatic diagnosis of performance problems in enterprise applications.

In Proc. EDCC ’16. IEEE, 2016.

Agenda

• APM interoperability

• Anti-patterns and trace based detection

• Mobile

• Overhead control

• Results aggregation

• Business integration

convert

APM Interoperability

.dat X .itds

4/22/2017

Solution

APM Interoperability

OPEN.xtrace

.dat .itds .txt

convert

convert convert

4/22/2017

Data Export in APM Tools

Source

Export

possible

✔ ✔

✘ ✔

Source

Export

possible

✘ ✔

✘ ✘

4/22/2017

Metric OPEN.xtrace Kieker inspectIT AppDynamics Dynatrace New Relic

Response

Time ✔ ✔ ✔ ✔ ✔ ✔

CPU Time ✔ ✔ ✔ ✔

Method

Name ✔ ✔ ✔ ✔ ✔ ✔

Return Type ✔ ✔ ✔

Logging

Level ✔ ✔ ✔

Statement ✔ ✔ ✔ ✔ ✔

Message ✔ ✔ ✔ ✔ ✔

Comparison table (excerpt)

4/22/2017

D. Okanovic, A. van Hoorn, C. Heger, A. Wert, and S. Siegl.

Towards performance tooling interoperability: An open format for representing execution traces.

In Proc. EPEW ’16, pages 94–108, 2016.

• Execution trace is a data structure that captures control flow of method

execution for a request served by the system (Ammons et al., 1997)

OPEN.xtrace Traces

SubTrace

Callable

Location

Callables

• Method executions

• DB calls

• Remote calls

• Logging

• Errors

• What is available:

• Trace model based on the available data in these tools

• Adapters to convert the data between OPEN.xtrace and tools

• Serialization

• Planned: OPEN.timeseries

• More on OPEN.xtrace

• D. Okanovic, A. van Hoorn, C. Heger, A. Wert, and S. Siegl. Towards performance

tooling interoperability: An open format for representing execution traces. In Proc.

EPEW ’16, pages 94–108, 2016.

• https://research.spec.org/apm-interoperability/

• https://github.com/spec-rgdevops

OPEN.xtrace

• Track what is going on in a complex, heterogeneous systems

• Capture important events during processing of client request

• Avoid vendor lock-in

• Wide industry support (http://opentracing.io)

OpenTracing

diagnoseIT Overview

Node 1

Node n

Traces

Instrumentation

Refinement

Request

Result

Monitoring Tool

Dynamic

Instrumentation

API Labeling

Location

Identification

Instrumentation

Quality Manager

Result Query

C. Heger, A. van Hoorn, D. Okanović, S. Siegl, and A. Wert.

Expert-guided automatic diagnosis of performance problems in enterprise applications.

In Proc. EDCC ’16. IEEE, 2016.

Anti-patterns are conceptually

similar to patterns, however

describe recurrent solutions to

design problems which, however,

may have a negative effect on

different software quality

attributes.

Koenig, A. (1998). “Patterns and Antipatterns”. In: The Patterns

Handbooks. Ed. by L. Rising. New York, NY, USA: Cambridge

University Press

• Performance Antipatterns

(Performance) Antipatterns

• Problem

„[…] a point in the execution where one, or only a few, processes

may continue to execute concurrently. All other processes must wait. […]”

• Cause

• „It frequently occurs in applications that access a database. Here, a lock ensures

that only one process may update the associated portion of the database at a

• It may also occur when a set of processes make a synchronous call to another

process that is not multi-threaded; all of the processes making synchronous

calls must take turns “crossing the bridge.”“

• Solution

“Shared Resources principle[:] responsiveness improves when we minimize the

scheduling time plus the holding time. Holding time is reduced by reducing the

service time for the One-Lane Bridge, and by rerouting the work.”

[Smith and Williams, 2001]

Example 1: One-Lane Bridge

• Synchronization in source-code:

• Thread- and connection- pools in application servers

Example 1: One-Lane Bridge – Causes

public synchronized

void buyItems (Collection<Item> items) {

<Connector ... maxThreads="300" acceptCount="150"

... />

Example 1: One-Lane Bridge – Symptoms

1 10 20 30 40 50

Number of users

Example 2: N+1-Problem

Wert, A. (2015). Performance Problem

Diagnostics by Systematic

Experimentation, PhD thesis.

Parsons, T. (2007): “Automatic Detection of

Performance Design and Deployment

Antipatterns in Component Based

Enterprise Systems”. PhD thesis.

Performance-Antipatterns– Classifications

Analysis automation

4/22/2017

List of Currently Detectable Antipatterns

• N+1 Query Problem

• Circuitous Treasure Hunt

• The Stifle

• The Ramp

• Traffic Jam

• More is Less

• Application Hiccups

• Garbage Collection Hiccups

Detecting Antipatterns in Mobile Environment

Buffer

OpenTracing

OPEN.xtrace

4/22/2017

Antipatterns in Mobile Environment

• Anti-patterns

• Too many remote calls

• Too many remote calls to the same URL

• Too many remote calls to the same server

• Hard-disk utilization too high

• Hard-disk usage increases too fast (ramp)

• RAM utilization too high

• RAM usage increases too fast (ramp)

• Response is available after timeout (timeout occurred too early)

• Glitches

• No connection available during a request

• Latency between back end and mobile device is too high

diagnoseIT Overview

Node 1

Node n

Traces

Instrumentation

Refinement

Request

Result

Monitoring Tool

Dynamic

Instrumentation

API Labeling

Location

Identification

Instrumentation

Quality Manager

Result Query

Overhead Control

• diagnoseIT can request more data from the monitoring tool

• Tool has to support changing of monitoring at runtime

• Instrumentation Quality Monitor can deny this request

• How to assess?

• Modeling?

• Machine learning?

diagnoseIT Overview

Node 1

Node n

Traces

Instrumentation

Refinement

Request

Result

Monitoring Tool

Dynamic

Instrumentation

API Labeling

Location

Identification

Instrumentation

Quality Manager

Result Query

diagnoseIT – Diagnosis Results

Problems

Overview

Natural language

description

Problem instance

details

Aggregating the Results

PIPI PI

PI PIPI

T. Angerstein, D. Okanović, C. Heger, A. van Hoorn, A. Kovačević, T. Kluge:

Many flies in one swat: Automated categorization of performance problem diagnosis results.

In: Proc. ACM/SPEC ICPE ’17

Categorization Process

Diagnosis results Categorization

Optimization

Categorization result

T. Angerstein, D. Okanović, C. Heger, A. van Hoorn, A. Kovačević, T. Kluge:

Many flies in one swat: Automated categorization of performance problem diagnosis results.

In: Proc. ACM/SPEC ICPE ’17

Aggregation in diagnoseIT

4/22/2017

Missing Business Context

CPU utilization over time

How critical is this?

Asking the right questions:

How are the users affected?

Which business transactions are affected?

Are any time-critical batch jobs affected?

11pm 0am 1am 2am 3am 4am 5am

Defining Business Context

Conclusion

Node 1

Node n

Traces

Instrumentation

Refinement

Request

Result

Monitoring Tool

Dynamic

Instrumentation

API Labeling

Location

Identification

Instrumentation

Quality Manager

Result Query

Application Performance Management - Uni...

Documents

Transcript of Application Performance Management - Uni...

EasyGen 2000 Configuration Manual 37427.pdf

Treaty Series - United Nationstreaties.un.org/doc/Publication/UNTS/Volume 2144/v2144.pdf · 2014. 11. 22. · No. 37426. Germany and Romania: ... . 6 7 No. 37427. Germany and Romania:

FISH AND MORE - Uni Kieleprints.uni-kiel.de/275/1/324_Froese_2005_UseOfFishbase...Aristotle University of Thessaloniki, School of Biology, Department of Zoology, Laboratory of Animal

Generic Research Data Infrastructure - Uni Kieleprints.uni-kiel.de/38774/1/2017-07-07-FDM-Kolloquium.pdf · JIRA API. JIRA Storage. GeRDI’s Architectural Style. Harvest. Harvest

AMP 200N1 - Accelonix · 2018. 5. 23. · data sheet > amp 200n1.1 > 20170422 technical details ford fmc1278, ri 140, ri 150, ci 210 magnetic field testing (ri 140), continuous distubance

easYgen-2000 Series Genset Control · 37427 . Configuration Software Version 1.xxxx Manual 37427 easYgen-2000 Series Genset Control

London CAM 20170422 FINAL - Bengmarkbengmark.com/.../uploads/2017/04/LondonCAM20170422.pdf · 2017-04-12 · Jon BrowerMinnoch, USA 1941 ‐1983, 635 kg ... 2020 to be 4.8 trillion

easYgen-2000 Series Genset Control - DSF TECHNOLOGIES€¦ · 37427 . Configuration Software Version 1.xxxx Manual 37427 easYgen-2000 Series Genset Control

CHAPTER 3 - Shodhgangashodhganga.inflibnet.ac.in/bitstream/10603/37427/8/08_chapter 3.pdf · fractional telegraph equations using Adomian decomposition method (ADM), discussed in

Kieker 1.3 User Guide - Uni Kieleprints.uni-kiel.de/16537/10/kieker-1.3_userguide.pdf · and use custom monitoring records, monitoring probes, and monitoring writers. A more detailed

CHAPTER 0shodhganga.inflibnet.ac.in/bitstream/10603/37427/5/05_chapter 0.pdf · nt1, 1, 0. 0 aa z (0.3.2) If either a or b is a non-positive integer, the function reduces to a polynomial.

PMAY NON VALID APPLICANT'S LIST FROM SUDA...alambag, lucknow gen n 9 18968 sanjay agrahari late ram chandra agrahari 78/82 new ganesh ganj lko gen no 10 37427 shrimati manju shiv kumar

Demag Standard Cranes · 2017-12-28 · 4 39559 40242-6 37427-4 38972-2-1 Travelling hoists DR rope hoist of C-shaped design optimised for crane applications with increased hoist

Research Challenges and Directions - Uni Kieleprints.uni-kiel.de/37428/7/20170422-WOSPC-Microservices-handout.pdf · 4/22/2017 · 3. appropriate performance modeling concepts for

Pliocene oceanic seaways and global climate - Uni Kieleprints.uni-kiel.de/35614/1/Karas.pdfCyrus Karas1,2,3,4, Dirk Nürnberg3, André Bahr5, Jeroen Groeneveld6, Jens O. Herrle1,2,

Weston - Amazon S3s3.amazonaws.com/webgen_einsteinwebsites/public/assets/37427/JCD_Cover... · strate recent advances in wear proper-ties and polishability that make them suitable

63525 63526 63528 63555 · 63525 – 6.68 grams 63526 – 2.9 grams 63528 – 4.1 grams 63555 – 3.4 grams Impact Melt Breccia Figure 2: Photo of 63526. Scale in mm. S80-37427 Figure

2017 HKDSE Math M2 CTL(20170422)(20170427-V1) Mathematics Module … · 2018-10-26 · 2017 HKDSE Math M2 CTL(20170422)(20170427-V1) 2017-DSE-MATH-EP(M2)-1 Mathematics Module 2 (Algebra

Diaspora as aesthetic formation: community sports events ...vuir.vu.edu.au/37427/1/Diaspora as aesthetic formation community s… · nation-derived ones (Vertovec 2001), and potentially

BJT Bias Voltage Divider Bias (H.5) - Wikimedia Commons · Voltage Divider Bias (H.5) 20170422. Based [1] Floyd, Electronic Devices 7th ed [2] Cook, [2] en.wikipedia.org References.