Incorporating Performance Testing in Agile Process

Michael VaxPresident, onmaas Software

www.onmaas.com

Need Performance Numbers for the Web Site

Types of Performance Testing

Performance Testing

Load Testing Stress Testing

Capacity Testing Focused Testing

Performance Project Phases

Proactive Reactive

Sizing and Capacity Planning

Performance Engineering

Performance Testing

Monitoring & Troubleshooting

What do we need to test performance?

Performance testing

Test Environment

Skills

Working System

Tools

Problems- No functional system until the very end- Bugs prevent test execution- Last minute changes across the board- Test script development is delayed - Testing is delayed to the end of the project- Too late to fix performance unfriendly design decisions- Developers may not be available by the time the performance

testing starts

Performance Testing & WaterfallSizing and Capacity Planning

Performance Engineering

Performance Testing

Monitoring & Troubleshooting

Performance Testing in Agile Project

Iter.0 Iter.1 Iter.2 Iter.3 .... Iter.x-1 Iter.x

Benchmark Test Benchmark Test Benchmark Test

• Test environment setup• Tools selection• Performance

requirements• Test data preparation Focus Test Focus Test Focus TestFocus Test

• Load tests• Stress tests• Capacity

testing• Resilience

testing

Performance Testing is Iterative Process

Analyze

Fix

Test

Doing performance test right before the launch day is not helpful.You need Calendar time!

Multiple Test

Cycles are Needed

Done in collaboration with Senior Developers

Performance Testing is Team Activity

• Performance analyst• Performance tester • Senior developer / architect• DBA• Network engineer

Performance Testing and Iterations

Iter. n-1Feature A

Iter. nFeature B

Iter n+1

Feature C

Feature A• Test requirements• Test development

PerformanceSpike

Feature A - Focus test

Feature B• Test requirements• Test development

Feature B - Focus test

Feature C• Test requirements• Test development

Performance Testing - Agile Best Practices

• Put performance tasks on a SCRUM board• Make fixing functional bugs that block

performance tests a high priority• Do not delay fixing performance issues identified

by tests• Retest after fixes are applied to know if they

worked• You cannot guess where the next bottleneck is• Don’t pre-optimize before testing

Performance Testing-Agile Best Practices (2)

• Understand the production environment• Analyze logs• Know when performance is good enough• You can always add more h/w • Apply low level load to the system under test

during manual testing• Use load test scripts:

• to find functional and stability bugs• to look for memory leaks• to reproduce intermittent problems• as part of smoke test

Skills

• Networking

• Databases

• Application servers

• Development environments

• Application domain

Performance Scripting is software development

– It should follow the same rules and be treated the same

way

– Design, modularization, documentation, guidelines, source

versioning, code reviews should all be carried out

– Allows for long term use and quick adaptability to changes

in the system under test

– Allows for easy creation of new tests

– Invest in automation of test setup, data population,

monitoring, and analysis of results

• Break down your system-interaction model

into a modular framework that can be re-used

Design Smart

Session X

Use Case A

Use Case B

User Transaction AUser Transaction BUser Transaction C

User Transaction C

This modularization allows quick test changes and creation

• This model allows Focused Tests or any sort of testing to be

created quickly

Design Smart

“Long Running” Stress Test

“Check-out” Focused Test

Session Type Y Session Type Z Session Type ASession Type X

Types of Performance Testing

Performance Testing

Load Testing Stress Testing

Capacity Testing Focused Testing

Load Testing

• Load Testing can encompass a large number of tests specific

to some area of performance, such as:

• Throughput testing

• Saturation testing

• End-user response

• Server response

Capacity Testing

• Capacity Testing is typically more specific than general load

testing

• Involves measuring “sizing” statistics, such as:

• Peak Active Sessions

• Sessions / Hour

• Transaction / Hour

Stress Testing

• Stress Testing is typically based off load testing but pushes

things even further until the break point

• Resilience and Availability Testing

• Disaster recovery

Focused Testing

• Not a ‘typical’ performance category

• Allows us to put strain on a specific aspect or functionality of

the system such as product views, check-out, or user

registration.

• Short duration

• Heavy load

• Usually run repeatedly with small tweaks

In Conclusion

Agile Development Process enables performance testing throughout the

entire development cycle and ensures the adequate performance

by the end of the project

Q & A

?