© Utopia Solutions

Performance Testing in the Agile Lifecycle

Lee Barnes, CTO Utopia Solutions @USI_LeeBarnes

© Utopia Solutions 2

Key Questions

What needs to change?

How do we make those changes?

Why don’t traditional performance testing models fit nicely with agile?

© Utopia Solutions 3

Case Study

Back story…

• Large global manufacturer • Performance Testing CoE in operation for ~10 yrs • Seen as a huge value to the enterprise • Began moving toward agile a few years ago…

and the luster of the CoE began to dull…

© Utopia Solutions 4

Traditional Performance Testing Model

Center of Excellence

Project 1

Project 2 Project N

Provides Consistency in… • Resources / Skills • Processes • Deliverable Quality • Technology

Request Test

Project Team

Center of Excellence

Perform Discovery

Develop & Execute

Deliver Results Evaluate Results

Info / Environment

© Utopia Solutions 5

CoE in the Waterfall Years

Plan Design Build Test Deploy

• One size fits all • Strict entry criteria • Performed late in SDLC • One-way communication of

results

Service Characteristics

Performance Testing

Too late!

© Utopia Solutions 6

Agile Development

Analyze

Develop

Integrate

Acceptance Test

Deploy

Where does performance testing fit?

© Utopia Solutions 7

Early Attempts on Agile Projects

Sprint 1 Sprint 2 Sprint 3 … Deploy

Agile

Performance Testing

Same Process…

Same Risk!

© Utopia Solutions 8

Early Attempts on Agile Projects

Sprint 1 Sprint 2 Sprint 3 … Deploy

Agile

Performance Testing

Performance Testing

Performance Testing

Performance Testing

CoE Approach • Rigid • 4 week test cycle

Agile Approach • Flexible • 2 week sprints

+ =

© Utopia Solutions

Back to the drawing board…

© Utopia Solutions 10

Early Cycle / Ongoing

Performance Analysis

Establish Objectives

Ultimate Goal…

Late Cycle Performance

Testing Shift Left

© Utopia Solutions 11

Identify Gaps

What’s stopping us from analyzing performance continuously?

• Performance objectives weren’t part of user stories

• Agile teams lacked performance engineering expertise

• APM tools / skills were non-existent

© Utopia Solutions 12

Breakdown the Barrier Between CoE and Agile Teams

© Utopia Solutions 13

Disclaimers

It was not easy.

The changes discussed on the next few slides were implemented over a two year duration.

It was not painless.

© Utopia Solutions 14

Performance Testing in an Agile Environment

Capture non-functional requirements

Make performance everyone’s job

Create a role to coordinate performance activities

CoE performance test architects become mentors

Begin developing a performance focused culture

© Utopia Solutions 15

Incorporating Performance Requirements…

… as acceptance criteria for existing user stories

As a dealer I want to create a product configuration for an order so that my customer can purchase the equipment they need

• All steps in the product config. process will have a response time of less than 2 seconds

• 1,500 dealers are concurrently using the system to configure products for customer orders

… as technical stories As a dealer, I want the response time of any step in the product configuration process to be less than 2 seconds

… as constraints

The system will respond in less than 2 seconds for all steps in the product configuration process for a maximum of 1,500 concurrent users configuring orders

© Utopia Solutions 16

Performance Testing in an Agile Environment

Unit performance testing

Component / system level performance testing

Include performance testing in the CI chain

Incorporate performance testing / analysis into each sprint

© Utopia Solutions 17

Performance Unit Testing

Start with existing unit tests “Decorate” them with

timers and load

Monitor results across builds

Build 1 Build 2 Build 3 Build 4

© Utopia Solutions 18

Beyond Unit Testing

Map user stories onto system components

Benchmark in development / QA environment

Look for trends in measurements vs. absolutes

Use service virtualization for 3rd party or yet to be developed components

Incorporate as part of the CI chain

Harden scripts against application changes

Component and system level testing

© Utopia Solutions 19

Adding Performance Tests to CI

Lessons Learned

Custom plug-ins exist to run performance tests as part of the CI chain – very useful

People get mad when you unnecessarily break the build

They get really mad when you can’t figure out why you broke the build

Add error handling and logging to scripts to enable quick failure investigation

Allow for non-production-like environment Set your response time tolerances and client timeouts accordingly

© Utopia Solutions 20

Waterfall Agile

Responsibility for Performance

Performance CoE Everyone

Role of CoE End of cycle PT Provide mentorship to agile team Pre-deployment system PT

Role of development

Throw system over wall Wait for results

Continuous focus on performance

Tools Traditional load test tools APM tools Service virtualization Continuous integration Traditional load test tools

Environment Dedicated / production-like Development / desktop Deliverable Boiler plate results delivered

just prior to release Continuous feed of actionable data

Waterfall Agile

Responsibility for Performance

Performance CoE

Role of CoE End of cycle PT

Role of development

Throw system over wall Wait for results

Tools Traditional load test tools

Environment Dedicated / production-like Deliverable Boiler plate results delivered

just prior to release

Performance Testing in an Agile Environment

Transformation Summary

© Utopia Solutions 21

Summary

Key takeaways… • Agile performance testing is not more

/ quicker performance tests it’s a continuous focus on performance

• Performance is part of everyone’s job • Success requires a shift in people,

process and technology

© Utopia Solutions 22

Questions…

…. and Answers!

Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: lee.barnes@utopiasolutions.com Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com

?

© Utopia Solutions

Links to Unit Performance Testing Frameworks

• JUnitPerf – http://www.clarkware.com/software/JUnitPerf.html • P-unit – http://p-unit.sourceforge.net/