Automated Quality Gates in Performance Testing

Roman Alexander Ferstltriscon

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Agenda

• Live demo of automated performance tests using NeoLoad, Dynatrace and Keptn

• Part A – Happy case

• Part B – Error case

• Evaluation of performance tests based on SLIs & SLOs

• How Dynatrace helps you to define your SLIs & SLOs

• Data exchange between NeoLoad and Dynatrace during a load test

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Brief introduction

… entirely dedicated to performance engineering

… the link between tool providers and companies

… focused on performance tests

… providing workshops & trainings (e.g. NeoLoad instructors)

… setting up & utilizing APM solutions (Dynatrace professionals)

… are based in Vienna

… we provide services across sectors in DACH.

Vienna, Millennium Tower

WE ARE

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Our clients & partners

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

What’s covered in the live demo

Today you are going to see…

…a load test being performed by NeoLoad...

…triggered from a pipeline in Azure DevOps...

…firing at a web application (UI) called „TTT -Triscon Testing Turf“...

…that is monitored by Dynatrace...

…and automatically evaluated by Keptn.

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Benefits of automating performance tests

• Improve performance test quality - repeatability

• Seamless CI/CD flow

• Less time in test evaluation

• Keptn-based Quality Gate

• Prerequisite for performance testing as a service

• Foundation for chaos engineering

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

A quick reminder to SLIs, SLOs & SLAs

SLI SLO SLA

Metrics over time

Examples:

• Error Rate

• Response time

• Throughput

• Request latency

Objectives for your SLIs

Examples:

• Response time must be less

than 400ms over a 30-day

period

• Throughput of at least 400

Requests/h

Business level agreement that

defines the service availability

including penalties.

Examples:

• Registers must be reliable &

fast 99% within a 30-day

window, otherwise you…

SLIs drive SLOs which inform SLAs

Service Level Indicator Service Level Objective Service Level Agreement

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Demo ingredients

Live Demo – Part A

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Keptn

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Keptn overview

Continuous Delivery with Deployment Validation Automated Operations

Performance as a Self-Service Performance Testing as a Self-ServiceDeployment Validation

(aka Quality Gates)

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Keptn quality gate

Keptn triggers Lighthouse for evaluation

Queries SLIs from providers Adds evaluation result to Bridge

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Convert performance engineering dashboards to SLIs & SLOs

Take these metrics that work for you

And list them in your SLI.yaml

And start with a simple SLO.yaml, e.g: within 10% of last build

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

An idea for improvement was born

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Define a load testing dashboard containing your SLIs/SLOs

SLI-Provider takes automatically all SLI/SLO information from a Dynatrace dashboard

Live Demo – Part B

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

NeoLoad - Dynatrace integration

Just add your tenant in the project settings:

Request Header is automatically added:

The requests from NeoLoad can be easily identified in Dynatrace:

See when load tests were performed

(on service level):

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

NeoLoad - Dynatrace integration

Define how the load test target is tagged in Dynatrace By setting the tag, the load test information will also be visible for all underlying services:

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Future perspectives and further possibilities

• Dynatrace• Metric ingest API - push NeoLoad transaction duration

• Keptn• Multiple SLI-Providers

• Deploy your application

• Neoload-service

• Set up dynamic load generators

• Trigger load test

• NeoLoad-Web

• Resource reservation – planning load tests

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

Upcoming talks

Automated Quality Gates in Performance TestingRoman Alexander Ferstl

References

1. NeoLoad Web - https://www.neotys.com/neoload/features/neoload-web

2. Dynatrace - https://www.dynatrace.com/platform/

3. Azure DevOps Keptn Integration: https://marketplace.visualstudio.com/items?itemName=RealdolmenDevOps.keptn-integration• Special Thanks to Bert van der Heyden

4. K3s: https://k3s.io/

5. Keptn: https://keptn.sh/

6. Keptn/dynatrace-sli-service: https://github.com/keptn-contrib/dynatrace-sli-service

7. Keptn/neoload-service: https://github.com/keptn-contrib/neoload-service

8. Twitter – SLI/SLO Dashboard Origin: https://twitter.com/grabnerandi/status/1279071519000408064

9. Youtube -SLIs, SLOs, SLAs, oh my! (class SRE implements DevOps): https://www.youtube.com/watch?v=tEylFyxbDLE

Thanks for listening!