Automatic Analysis for Load Test Results

by Andrey PokhilkoChief Scientist, BlazeMeter Inc.

February 3rd, 2016Mountain View, CA

About Me

My key interest: tools that help people enjoy their job

Current: Chief Scientist for BlazeMeter

Past: Performance Engineering in Yandex

JMeter Plugins project founder http://jmeter-plugins.org/

Loadosophia author http://loadosophia.org/

Taurus Tool lead dev http://gettaurus.org/

Look Out of the Window

1. It’s the era of total speed-up

2. Which is solved by total automation

3. Which leads to CI/CD as standard practices

You can’t have a human in the loop anymore

Understand how humans do it, then automate the process

Analysis

Raw DataInformation DecisionsFacts

True Analysis is Here

What is our “Load Test Results Analysis” ?

1. Decide if results are reliable

2. Break complex test timeline into components - surprisingly important

3. Decide if it had hit bottleneck, had memory leak etc

4. Understand APM correlations

5. Decide if something has changed since the last test / over recent N tests

When do we do it?

1. Online while test is still running

2. Post-test

3. Comparing tests to a specific baseline

4. Observing KPI history from a series of tests

Unconsciously most of the time

Speed Constraint

mainchallenge

ResourcesUsed

Challenge Accepted

Isn’t it trivial?

Wasn’t it done before?

Why can’t we reuse what APM vendors do these days?

How can we do it?

“Automated” vs “Automatic”

Disambiguation:

Apply the right test signal

First and foremost:

Right test signal

Load Profile

Real test intention

MeasuredResponseSystem

Common Test Signals aka Load Profiles

Stress, Max Throughput => Capacity, “how much”

Constant, Soak, Sustain => Quality, “how good”

Spike, recovery

Combinations => reality requires to adapt

Stress

Constant

Mixed

Recovery

Each testing process stage has own techniques

In-Test Automation

Instant threshold

Windowed threshold => average / instant

Steadiness threshold => stop constant test earlier

Decisions to be made: stop test, mark it failed, mark it passed

In-Test Automation: Implementations

JMeter-Plugins Autostop - show it

Taurus Pass/Fail criteria - show it

BlazeMeter Thresholds

Post-Test Analysis

Stage breakdown

1. Ramp-Up Stage

2. Constant Stage

3. Leftovers

POI detection

APM correlation- “measure everything” costs you in the analysis stage

Post-Test Analysis: Implementations

BlazeMeter App => Thresholds

Loadosophia.org:

stages breakdown

test classification

refined KPIs - it’s all about speed

monitoring correlation

85% to 95% correct

Test Comparison & Series Trend

Have reference test (aka “baseline”)

Raise flag on too much difference

Notify on trend outlier

Notify on trend reaching limit

It’s all easy to automate with APIs + simple scripts

We’ve Learned Today

1. Why to automate results analysis

2. What our “result analysis” is

3. Prerequisites to make it automatable

4. Particular techniques & implementations

Now Q&A (my favorite!)