Load and Performance Testing in the Cloud: Myth vs. Reality

T5 Cloud Testing

5/2/2013 9:45:00 AM

Load and Performance Testing in the

Cloud: Myth vs. Reality

Presented by:

Steve Weisfeldt

Neotys

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Steve Weisfeldt

Steve Weisfeldt is a senior performance engineer at Neotys, a provider of load testing software for web applications. In the load and performance testing space since 1999, Steve’s expertise lies in enabling organizations to optimize their ability to develop, test, and launch high-quality applications efficiently, on-time, and on-budget. Prior to Neotys, Steve was president of Engine 1 Consulting, a services firm specializing in all facets of test automation. At an earlier position he spent seven years at automated testing vendor Segue Software (acquired by Borland), where he delivered professional services and training.

5/13/2013

1

© 2013 Neotys. All Rights Reserved. CONFIDENTIAL. www.neotys.com

Load and Performance Testing in the Cloud

Myth versus Reality

StarEAST 2013 Steve Weisfeldt, Neotys USA

May 2, 2013

www.neotys.com© 2013 Neotys. All Rights Reserved. CONFIDENTIAL.

Biography

Neotys USA

• Consulting Manager

• Senior Performance Engineer

Former President/Founder, Engine 1 Consulting

Borland / Segue Software

• Consulting lead / Engagement Manager

• Product Marketing Manager, SilkPerformer

Blogger on neotys.com

5/13/2013

2


Agenda

Load and Performance Testing

• What is it?

• Architecture

The “Cloud”

Load Testing from the Cloud

• Load Generation

• Test execution

Applications In The Cloud

Key Considerations


Load and Performance Testing - Overview

What?

• The process of putting demand on a system or device and measuring its response

• Non-functional

• Bridges, roller coasters, elevators44

• Software applications: Generating network traffic

Why?

• To avoid:

o Loss of revenue

o Customer “migration”

o Decreased productivity

• To understand an application’s

o Scalability –

o Stability –

o Ability to adhere to Service Level Agreements, etc.

5/13/2013

3


Load Testing Architecture


The Cloud

Wikipedia:

• Use of computer technology stored on servers rather than the client computers

• 4computing resources (hardware and software) that are delivered as a service over

a network (typically the Internet).

In Load Testing1.

• Provides several key advantages but not the only solution

• Can “host” the individual parts of the infrastructure

- Load Generators

- “Controller”

- Application under test

5/13/2013

4




Cloud Load Generation Benefits

(Very) Large scale load tests

• Additional infrastructure

• No need for internal data center

• CAN be cost effective (cloud provider vs. your own cloud)

• Improved maintenance (cloud provider)

Greater level of realistic testing

• Test full delivery chain

• Complete Geographical coverage

- Generate load closer to real-world end-user locations

- Compare performance data across multiple geographies

- Important for mobile

5/13/2013

5


AUT’s Visibility From The Cloud

Application “publicly” facing

• Essentially no challenges

• Generate load directly against the Application Under Test (AUT)

Application behind a firewall

• Load needs to go through the firewall!

• How do we “hit” the application?


Dealing With The Firewall

VPN?

• Not effective for load testing

- Client needed on each Load Generator

- End up load testing the VPN connection

• Nor for Controller <-> Load Generator communication

Open firewall

• Completely (i.e. for all external traffic)

• For certain IP addresses

• Requires involvement of admin teams / substantial lead time

• Still the best solution

Private cloud

• May not give the “outside the firewall” load generation

• Hard to set-up/maintain

5/13/2013

6


Challenges of “Cloud-only” Load Generation

Difficult to isolate root causes

• Only testing the FULL delivery chain

• Hard to reproduce results

Impossible to test IP-based load balancers

• 1 IP Address per Load Generator

• Public cloud solutions can’t “spoof” IP addresses

- Unless using IPv6

Data security

• Don’t want to expose “real world” data


Recommended Load Generation: Hybrid Approach

Employ a “two-stage” process

• Test first from “inside the firewall”

- Isolate the application

- Identify root causes

• Test next from The Cloud

- Test the full delivery chain

- Understand the implication of the “last mile”

Employ a “combination” approach

• Generate load from both inside AND outside the firewall

• Keep security-sensitive data tied to Load Generators inside the firewall

• Exercise IP-based load balancers via internal Load Generators

5/13/2013

7




Controller In The Cloud – Why?

Firewall considerations

• Firewall in between Controller and Load Generators

External Resources

“Large” load tests

• Controller needs to remain connected to Load Generators

• Not enough data center throughput to support lots of Load Generators

“Big” Test Data

• Large quantity, real-world data

• Needs to be pushed to load generators

5/13/2013

8




What If Your Application Is In The Cloud?

Load Generators in the cloud

• Same Cloud

- The same as “inside the data center” load generation

• Different cloud

- Just as if your application was inside your data center

Load Generators inside data center

• Oy vey!!

5/13/2013

9


Key Considerations / Recommendations

Use a Cloud Solution:

• Support for multiple geographies

• Ability to support network parameters

- Packet loss, Network latency

- Important for mobile!

• In “synch” with internally driven load testing

• Based on multiple cloud providers

- Avoids delays if a provider has downtime

• Easy accessibility / maintainability

• Services?


Summary

The cloud can be very useful in load testing

Can host different “pieces” of the load testing infrastructure

Should not be the sole source for load generation

5/13/2013

10


Questions?

Contact me at: [email protected]

Visit me here at StarEAST: Booth 40

Load and Performance Testing in the Cloud: Myth vs. Reality

Technology

Transcript of Load and Performance Testing in the Cloud: Myth vs. Reality