Performance Testing in the Agile Lifecycle

4/23/15

1

© Utopia Solutions

Performance Tes3ng in the Agile Lifecycle

Lee Barnes, CTO Utopia Solu3ons @USI_LeeBarnes

© Utopia Solutions 2

Key Questions

What needs to change?

How do we make those changes?

Why don’t tradi3onal performance tes3ng models fit nicely with agile?

4/23/15

2


Case Study

Back story…

•  Large global manufacturer •  Performance Tes3ng CoE in opera3on for ~10 yrs •  Seen as a huge value to the enterprise •  Began moving toward agile a few years ago…

and the luster of the CoE began to dull…


Traditional Performance Testing Model

Center of Excellence

Project 1

Project 2 Project N

Provides Consistency in… •  Resources / Skills •  Processes •  Deliverable Quality •  Technology

Request Test

Project Team

Center of Excellence

Perform Discovery

Develop & Execute

Deliver Results Evaluate Results

Info / Environment

4/23/15

3


CoE in the Waterfall Years

Plan Design Build Test Deploy

•  One size fits all •  Strict entry criteria •  Performed late in SDLC •  One-‐way communica3on of

results

Service Characteris?cs

Performance Tes3ng

Too late!


Agile Development

Analyze

Develop

Integrate

Acceptance Test

Deploy

Where does performance tes3ng fit?

4/23/15

4


Early Attempts on Agile Projects

Sprint 1 Sprint 2 Sprint 3 … Deploy

Agile

Performance Tes3ng

Same Process…

Same Risk!


Early Attempts on Agile Projects

Sprint 1 Sprint 2 Sprint 3 … Deploy

Agile

Performance Tes3ng

Performance Tes3ng

Performance Tes3ng

Performance Tes3ng

CoE Approach •  Rigid •  4+ week test cycle

Agile Approach •  Flexible •  2 week sprints

+ =

4/23/15

5

© Utopia Solutions

Back to the drawing board…


Early Cycle / Ongoing

Performance Analysis

Establish Objectives

Ul?mate Goal…

Late Cycle Performance

Tes?ng ShiG LeG

4/23/15

6


Identify Gaps

What’s stopping us from analyzing performance con?nuously?

•  Performance objec3ves weren’t part of user stories

•  Agile teams lacked performance engineering exper3se

•  Performance tes3ng knowledge and tools were locked in the CoE

•  APM tools / skills were non-‐existent


Breakdown the Barrier Between CoE and Agile Teams

4/23/15

7


Disclaimers

It was not easy.

The changes discussed on the next few slides were implemented over a two year dura3on.

It was not painless.


Performance Testing in an Agile Environment

Build performance objec3ves in to user stories

Make performance everyone’s job

Create a role to coordinate performance ac3vi3es

CoE performance test architects become mentors

Begin developing a performance focused culture

4/23/15

8


Incorporating Performance Requirements…

… as acceptance criteria for exis3ng user stories

As a dealer I want to create a product configura3on for an order so that my customer can purchase the equipment they need

•  All steps in the product config. process will have a response 3me of less than 2 seconds

•  1,500 dealers are concurrently using the system to configure products for customer orders

… as technical stories As a dealer, I want the response 3me of any step in the product configura3on process to be less than 2 seconds

… as constraints The system will respond in less than 2 seconds for all steps in the product configura3on process for a maximum of 1,500 concurrent users configuring orders



Unit performance tes3ng

Component / system level performance tes3ng

Include performance tes3ng in the CI chain

Incorporate performance tes3ng / analysis into each sprint

4/23/15

9


Performance Unit Testing

Start with exis3ng unit tests “Decorate” them with

3mers and load

Monitor results across builds

Build 1 Build 2 Build 3 Build 4


Beyond Unit Testing

Map user stories onto system components

Benchmark in development / QA environment

Look for trends in measurements vs. absolutes

Use service virtualiza3on for 3rd party or yet to be developed components

Incorporate as part of the CI chain

Harden scripts against applica3on changes

Component and system level tes3ng

4/23/15

10


Adding Performance Tests to CI

Custom Maven plug-‐ins exist to run JMeter (and other tools) performance tests – very useful

People get mad when you unnecessarily break the build

Set your response 3me thresholds and client 3meouts accordingly

They get really mad when you can’t figure out why you broke the build

Add error handling and logging to scripts to enable quick failure inves3ga3on

Lessons Learned


Waterfall Agile

Responsibility for Performance

Performance CoE

Role of CoE End of cycle PT

Role of development

Throw system over wall Wait for results

Tools Tradi3onal load test tools

Environment Dedicated / produc3on-‐like Deliverable Boiler plate results delivered

just prior to release

Waterfall Agile

Responsibility for Performance

Performance CoE Everyone

Role of CoE End of cycle PT Provide mentorship to agile team Pre-‐deployment system PT

Role of development

Throw system over wall Wait for results

Con3nuous focus on performance

Tools Tradi3onal load test tools APM tools Service virtualiza3on Con3nuous integra3on Tradi3onal load test tools

Environment Dedicated / produc3on-‐like Development / desktop Deliverable Boiler plate results delivered

just prior to release Con3nuous feed of ac3onable data


Transforma?on Summary

4/23/15

11


Summary

Key takeaways… •  Agile performance tes3ng is not more / quicker performance tests it’s a con3nuous focus on performance

•  Performance is part of everyone’s job •  Success requires a shio in people, process and technology


Questions…

…. and Answers!

Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: [email protected] Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com

?

4/23/15

12

© Utopia Solutions

Links to Unit Performance Testing Frameworks

•  JUnitPerf – hpp://www.clarkware.com/sooware/JUnitPerf.html •  P-‐unit – hpp://p-‐unit.sourceforge.net/

Performance Testing in the Agile Lifecycle

Software

Transcript of Performance Testing in the Agile Lifecycle