Establishing an effective performance testing environment.

Gordon McKeown www.facilita.com TMF 2010

TMF "Creating an effective performance testing environment" 2

Agenda Introductory tour of the issues. Share our experiences & identify best

practice. In the spirit of the TMF...

You will drive the the session!

27/1/2010

TMF "Creating an effective performance testing environment"

What do we mean by "Environment“ ?

A wide scope. But not too wide! In

"Systems" Network Tool Load Injectors System Under Test (SUT)

Software installation & configuration Test data

Out People (except for responsibilities / roles ) Tool selection

327/1/2010

TMF "Creating an effective performance testing environment"

Performance test elements

27/1/2010 4

load injectors System Under Test "SUT"

load

controller

server metrics

timings

control

scripts test def. test data

TMF "Creating an effective performance testing environment"

Desirable characteristics

Few impediments to tester activities. Low management & bureaucratic

overheads. “Realistic tests” are possible. SUT as close as possible to production. Experimentation is possible. Isolation

Reproducible tests Reduced risk of impact on corporate ICT

27/1/2010 5

TMF "Creating an effective performance testing environment"

Some of the issues

Who controls the test infrastructure? Security standards. Test data. Relationship to "production“. Resource sharing. Technical innovations. Any other major areas?

27/1/2010 6

TMF "Creating an effective performance testing environment"

Should the test team declare independence?

27/1/2010 7

TMF "Creating an effective performance testing environment"

Who should control the environment?

"Lack of environmental control by testers is a common, serious impediment to testing efficiency." That is my experience. What is yours?

Testing is an exploratory activity, we often don't know what the next change should be.

Is it easier if load testing is integrated with development?

Are things worse in organisations conducting infrequent tests? E.g. medium sized organisation with limited IT capability who have commissioned a new system.

27/1/2010 8

TMF "Creating an effective performance testing environment"

Corporate security standards

We have the characteristics of hackers! Denial of Service Attack. Recording interactions. "Man in the middle." Looking inside systems. Reverse engineering in extremis.

27/1/2010 9

TMF "Creating an effective performance testing environment"

Test data Quantity & realism

Depending on DB design too little data can give unrealistically fast or slow timings.

An "unrealistic" database can distort results Managing "real" data extracted from a live

system. Legal & ethical issues Anonymizing

27/1/2010 10

TMF "Creating an effective performance testing environment"

Test's relationship to “Production"

Naive scaling But who can afford full duplication? Are sizing models fit for purpose? Systems are not linear.

The "unknown unknowns" are a risk. An example: licensing

One of our client's load tested against a near replica system. We recommended a subset of tests should be run against actual system (with test db) before going live. Discovered default server license for 50 users had not been upgraded to 300.

Significant but unforeseen configuration issues. Changing configuration settings from scaled down

replica to production: should this be tested?

27/1/2010 11

TMF "Creating an effective performance testing environment"

Resource sharing Functional testing & UAT sharing

resources with performance testing. Mutual interference & frustration Reproducible tests may not be possible.

Switching between configurations & databases.

Out of hours testing: a solution or another set of problems?

27/1/2010 12

TMF "Creating an effective performance testing environment"

Technical innovation Cloud computing. Software as service. Virtualization. What else might change the

game?

27/1/2010 13