Testing Applications—For the Cloud and in the Cloud

8/19/2015

1

© 2013 IBM Corporation

Continuous delivery of software innovation on hybrid cloud

Testing ApplicationsFor the Cloud and in the Cloud

@Al_Wagner

IBM Technical Evangelist

[email protected]

1

Why do we care?

In the Cloud; For the Cloud

8/19/2015

2

2

Delivering

in today’s DevOps world


3

DevOps – the three principles

Left to Right – Continuous Delivery

Emphasize the performance of the entire system

Right to Left – Continuous Feedback

Create the right to left feedback loops

Culture of Experimentation and Learning

Continually experiment, take risks and learn from failure;

Understand that repetition and practice is the pre-req to mastery.

Continuous Delivery

Whole team approach;

No blame culture;

Encourage experimentation & learning

Continuous Feedback


8/19/2015

3

4

The 1st step is solving the “legacy” challenges

Before DevOps, test

teams were fairly

successful at keeping

up with development

velocity

Test teams need to

change their behavior

in order to keep up

with increased

development velocity

Traditional test teams

can no longer keep up

with continuous delivery


5

Today’s reality for Testers

Testers want to spend LESS time

ibm.biz/TestingSurveyResults

In the Cloud

8/19/2015

4

6

Virtualizing today’s test labs

Modern test labs consist of…

Test Data

Management

Service

Virtualization

Deployment

Automation

Dynamic Infrastructure

In the Cloud

7

Service virtualizationmaking the unavailable available for testing

Virtual components simulate the behavior of a service or application during testing

– System dependencies

– Unavailable/inaccessible services

– Costly 3rd party access fees

Virtual components run on commodity hardware, private cloud, public cloud

Each developer and tester can easily have their own test environment

Developers and testers can continue to use current testing procedures and tools

Streamline creation of production-like environments and negative testing

Heterogeneous Environments

Public CloudPrivate Cloud

Data Warehouse MainframeEnterprise

Service Bus

Directory

Identity

File

systems

Collaboration

App Under TestRouting

Service

Third-party

Services Portals

Content

ProvidersEJB

Shared

ServicesArchives

Business

Partners

Messaging

Services

Databases Mainframe

applications

App Under Test

Third-party

Services

Packaged apps, messaging services, etc.

Virtual Components

In the Cloud

8/19/2015

5

8

Continuous deploymentautomating testing includes automating deployment

Consistent deployments and rollbacks of applications

Orchestration of changes across servers, tiers and components

Configuration and security differences across environments

Clear visibility: what is deployed where and who changed what

• Multi-Tier Application Models

• Scalable distributed automation

• Quality gates and approvals

Deployment Automation

In the Cloud

9

In the Cloud

“yes...BUT!”

I still need networking

and computing

resources to run the

deployed software!

Dynamic infrastructure

8/19/2015

6

10

Clouds are rolling in

Clouds are not a commodity.

Cloud computing doesn’t come out of the sky.

Clouds comes from physical hardware inside brick and

mortar facilities connected by hundreds of miles of

networking cable.

And no two clouds are built the same way.


11

Clouds deliver networking and computing power with speed!


privatepublic

On-Premises IT

hybrid

8/19/2015

7

12

Cloud services save you time!

Infrastructure (IaaS) Platform (PaaS) Software (SaaS)

more consumer control more vendor control


13

Automated test execution in the cloud

Many organizations are looking to the clouds to execute:

Functional Tests Drive execution of functional tests in datacenters around the world without having

to deploy physical servers

Reduce latency issues and long running tests

Performance Tests Drive load around from datacenters around the world to measure global user

experience

Scale user load on demand

In the Cloud

• Setup; tear down test execution

environments on demand at lower cost

• Distribute test execution around the

world with ease

Launch test execution

Log execution results

8/19/2015

8

14

Deployment automation and cloud accelerates testing

Virtualizing Hybrid Test Labs - deploy applications in hybrid clouds

Simultaneous execution of test stages – integration, functional,

performance, security testing

Eliminates scheduling delays

Eliminates hardware delays

Setup; Tear Down on demand

In the Cloud

15

Deployment automation and clouds eliminate test lab bottlenecks

15

Rational Test

Virtualization

Server

- virtualized services

CICS

Tx

SAP External

Services

DB

Service

Virtualization

Deployment Automation

Env1

Env2

Customers

DB

App

Server

SAP

CICS

PaaS(private cloud)

External

Services

SaaS(public cloud)

Off premise

Off premise

DB

App

Server

App

Server

SAP

CICS

PaaS

3rd Party

Service

3rd Party

ServiceDatabase

CICSSAP

Physical devices,

emulators, devices

clouds

Development Test Production

On premise

In the Cloud

8/19/2015

9

16

Deployment automation and cloud accelerates testing

Virtualizing Test Workstations - deploy pre-configured virtual

desktops in the cloud

Eliminate need to install, configure, license and maintain

workstations

Standardize on a test automation platform

Increases productivity; reduces down time

Setup, Tear Down on demand

In the Cloud

17

Virtualizing testing workstations

Image

Repository

Artifact

Repository

Deployment

Automation

Private

Request Cloud

instanceRequests

Testing

Workstation

Returns

Workstation

I/P address

Dynamic Infrastructure

Request

workstation

image

Request specific

workstation configuration

Remote access to

virtual workstation

Operating system

& software

Configuration

files

Test

Assets

In the Cloud

8/19/2015

10

18

For the Cloud

What is quality software

in the cloud?

Clouds can be quickly stood up in Data Centers

anywhere in the world. This capability can expose risk to

the business if not considered in the test effort.

You need to make sure the software being released is:

• Resilient

• Reliable

• Secure

19

Testing for high availability

• Does failover routing work as expected?

• Does the system automatically scale?

• Are databases redundant and backed up?

• Are faults logged & people notified of outages?

Data Center

Pod

Cluster

Server/Data

For the Cloud

8/19/2015

11

20

Testing for resiliency

Quality is not only measured by how well code survives during

a long test run, but how quickly normal operations can be

returned after an error.

Testers need to assume remote services:

will respond slowly,

will respond with an error,

or won’t respond at all

The most important production issue metric is

not “root cause” but “time to recovery”.

For the Cloud

21

Patterns for the cloud - Retry

For the Cloud

8/19/2015

12

22

Testing Retry pattern

Service A Service B500ms (timeout)

500ms (timeout)

RetryCount=2RetryCount=1

Server LogOperation Exception – (date/time)

….


Operation Exception – (date/time)

….

For the Cloud

Timeout = 500ms

Max # of Retries = 3

extends Retry

23

extends Retry

Testing Retry pattern

Service A Service B


Operation Exception – (date/time)

….

500ms (timeout)

500ms (timeout)

200ms (passed)

• Use a virtual service to simulate

“Service B” responses along with

timeouts and retry attempts

• Confirm log entries are captured

RetryCount=0

For the Cloud

Timeout = 500ms

Max # of Retries = 3

8/19/2015

13

24

Patterns for the cloud – Circuit Breaker states

For the Cloud

25

Patterns for the cloud – Circuit Breaker

For the Cloud

8/19/2015

14

26

Testing Circuit Breaker pattern

Service A Service B

For the Cloud

State: closed

Circuit

Breaker

success

Parameters

invocationTimer = 0.1

resetTimer = 60_000

retryThreshold = 1

Testing the “closed” state:• Straight forward and easy to test

• Validate “closed” circuit breaker state using real

service implementation or virtual service

requestIDState = closed

getState(requestID) returnState()

27


Service A Service B

For the Cloud

State: closed

Circuit

Breaker

failure

Parameters


resetTimer = 60_000

Service A Service B

State: half open

Circuit

Breaker

timeout

testCall()

success

success

updateState(halfopen)

updateState(closed)

resetTimer()

success

executeCall()

Testing for “half open” state:• Hard to test negative in test labs and production

• Virtual services can help simulate the necessary

“negative” responses required

8/19/2015

15

28


For the Cloud

Service A Service B

State: open

Circuit

Breaker

fast fail

Parameters


resetTimer = 60_000

retryThreshold = 1

requestIDState = open

getState(requestID) returnState()

Testing the “open” state:• Little harder if Service B is deployed

• Validate “open” circuit breaker state

• Setting requestIDState to “open”

• Routing request to stopped virtual service

29

Testing feature toggles in a continuous integration world(aka feature flags; dark launches)

Release Toggles

Hides incomplete features (dark launch)

Short lived (removed when feature deployed)

Pre-decided values across environments

Business Toggles

Enables A/B testing

Release to small control groups of people to get

feedback

Enable and disable on the fly

For the Cloud

8/19/2015

16

30

Release toggles

For the Cloud

//pseudo-code

featureToBeReleased = true;

If “featureToBeReleased” {

do this():

} else {

do that();

}

Hide features from being exposed until development and testing

is “Done”.

Remove feature toggle from code base across all modified

components.

Check to ensure “feature” is hidden or available as intended.

Be sure to toggle the UI layer!

31

Business toggles

For the Cloud

//pseudo-code

checkUser() {

boolean testgroup = checkDatabaseForUserGroup(userID);

if (testgroup) {

accessNewFeature();

} else {

useLegacyFunctionality();

}

}

Enables organizations to “test” features before releasing to the

entire user community. (A/B Testing)

Test for all user groups.

Need to determine what combinations of features need to be

tested. It could be difficult and time consuming to test them all.

Be sure to check the UI presentation as well!

8/19/2015

17

32

Continuous feedback delivers insight on software quality

Build a system that allows you to make decisions

based on real data instead of just market direction.

User behavior

Customer sentiment analysis

Performance metrics

For the Cloud

33

Testing for security

Data Breaches

Encrypt the data exchanged between services

Data Loss

Deploy redundant systems and test backups

Insecure APIs

Develop and Test 3rd party access using authorization services

Denial of Service attacks

Your organization could be billed for all the cloud services

Malicious Insiders

Protect your encryption keys

Shared Technology

You are not alone in “public” clouds

to name a few…

For the Cloud

Don’t forget to scan your code for insecure coding practices.

Penetration test your applications.

If you are hacked; ALL your customers are impacted!

8/19/2015

18

34

Remember

It is important to test:

what a product/service does

how a product/service performs

as well as how that product/service is managed

For the Cloud

35

ibm.com/devops

8/19/2015

19

36

ibm.com/devops

Testing Applications—For the Cloud and in the Cloud

Software

Transcript of Testing Applications—For the Cloud and in the Cloud