Marc van Lint

Consultant

Email: marc_van_lint@nl.ibm.com

Find me on: http://business.vanlint5.nl/

Continuous Testing:

Removing the Bottlenecks

Business stakeholder expectationsBetter – Cheaper - Faster

In the past

• Organizations looked to find a

balance between speed, cost &

risk.

• Development teams were not as

agile as they are today; testing

never seems to have enough

time.

• Software quality suffered.

Today

• Organizations are asking teams to

release higher quality software

sooner.

• Development teams are becoming

more and more agile; testing still

never seems to have enough

time.

• Software quality continues to

suffer.

Bottom line – current approaches are not helping!Today’s testing challenges still remain

Development VelocityTesting Labor Test Lab Complexity

Outsourcing labor is no longer

a sustainable model as global

wages are increasing

Product and application

complexity are driving

test lab cost increases

Test teams can no longer

keep up with agile

development

Heterogeneous Environments

Public CloudPrivate Cloud

Data Warehouse MainframeEnterprise

Service Bus

Directory Identity

File systems

Collaboration

Web/Internet

Routing Service

Third-partyServices Portals

ContentProviders

EJB

SharedServicesArchives

Business Partners

Messaging Services

Traditional

Testing

Other costs

Testing costs

Accelerate

software delivery –

for faster time to value

Balance speed, cost,

quality and risk –

for increased capacity

to innovate

Reduce time to

customer feedback –

for improved customer

experience

IBM DevOps point of viewEnterprise capability for continuous software delivery that enables organizations to seize

market opportunities and reduce time to customer feedback

Our DevOps POV is resonating with clients and they are delivering

measurable business outcomes with DevOps

Continuous

Customer Feedback

& Optimization

Collaborative

Development

Continuous Release

and Deployment

Continuous

Monitoring

Continuous

Business Planning

Continuous

Testing

Operate Develop/ Test

Deploy

Steer

DevOpsContinuous Feedback

All the world’s a “test” stage,and all Testers merely players; They have their exits and their entrances;

The Continuous Delivery PipelineDevelopment Testing Staging Production

Business

Owners Customers

From ideation to implementation…

everyone contributes to quality!

Continuous Release & Deployment

All the world’s a “test” stage,and all Testers merely players; They have their exits and their entrances;

The Continuous Delivery PipelineDevelopment Testing Staging Production

Business

Owners Customers

Collaborative

Development

The goal:

create the

build package

Continuous

Monitoring

The goal:

production

deployment

Continuous Testingsimultaneously across multiple test environments

Everything between dev &

prod is a test stage

Quality

Gate

Security

FVT

Performance

SIT

UAT

Quality

GateBVT

Dev UT

At least 1/3 of is testing!

66

The Five Imperatives of Application Lifecycle

Management Improve organizational productivity

1. Accelerate time to delivery with Real-time Planning

2. Improve quality with Lifecycle Traceability

3. Maximize product value with In-Context Collaboration

4. Achieve predictability with Development Intelligence

5. Reduce costs with Continuous Improvement

What Testers really need!Validating today’s complex applications

Continuous Testing

DevOps Foundation

Test Automation

• Run unattended

• Validate business flows

across channels

• Durable, maintainable,

resilient to change

Orchestration from build to production deployment

• Reliable • Repeatable • Traceable

Test Environments

• Realistic & complete

- (production like)

• Available on demand

• Inexpensive to setup

What Testers really need!Validating today’s complex applications

Continuous Testing

DevOps Foundation

Test Automation

• Run unattended

• Validate business flows

across channels

• Durable, maintainable,

resilient to change

Orchestration from build to production deployment

• Reliable • Repeatable • Traceable

Test Environments

• Realistic & complete

- (production like)

• Available on demand

• Inexpensive to setup

Solving the Test Environment dilemmaMaking production-like test environments a reality

Traditional test environments are both underutilized and insufficient– Environments are expensive: hardware, software, and labor

– Tester and developer time wasted waiting for appropriate environment

Agile test environments require four enabling capabilities

Dynamic Infrastructure

Test Data

Management

Deployment

Automation

Test Environments

DevOps Foundation

Service

Virtualization

Organization Process Technology

an agile test organization and testers need continuous access to test data

with agile development you have continuous integration and delivery and have to test often

test data management software needs to support agile method by streamlining access to test dataand having insight into test data

Test Data ManagementAgile testing needs continuous access to test data

Solving the Test Environment dilemmaMaking production-like test environments a reality

Traditional test environments are both underutilized and insufficient– Environments are expensive: hardware, software, and labor

– Tester and developer time wasted waiting for appropriate environment

Agile test environments require four enabling capabilities

Dynamic Infrastructure

Test Data

Management

Deployment

Automation

Test Environments

DevOps Foundation

Service

Virtualization

Reality checkWhich answer would you pick?

How does your organization deal with testing bottlenecks due to unavailable or

missing software and services?

1. Manually write simulation stubs to emulate missing functionality.

2. Use commercial service virtualization solution.

3. Defer testing until later when the code is available to test.

4. De-scope testing of that functionality.

5. None of the above.

9%

9%

56%

6%

19%

Responses from GRUC Webcast – October 22nd, 2013

The Modern Application

•Light

•Fresh

•New

•Simple

…but still

unfortunately

connected to 20+

years of IT

legacy!

14

What is Test Virtualization?

Test Virtualization enables to create “virtual

services”:

–Virtual Services simulate the behavior of an

entire application or system during testing

–Virtual Services can run on commodity

hardware, private cloud, public cloud

–Each developer, tester can easily have their

own test environment

–Developer and testers continue to use their

testing tools (Manual, Web performance, UI test

automation)

Cap

ture & M

od

el

System dependencies are a key challenge in

setting up test environments:

Unavailable/inaccessible: Testing is constrained

due to production schedules, security restrictions,

contention between teams, or because they are still

under development

Costly 3rd party access fees: Developing or testing

against Cloud-based or other shared services can

result in costly usage fees

Impractical hardware-based virtualization:

Systems are either too difficult (mainframes) or remote

(third-party services) to replicate via traditional

hardware-based virtualization approaches

Heterogeneous Environments

Public CloudPrivate Cloud

Data Warehouse MainframeEnterprise

Service Bus

Directory Identity

File systems

Collaboration

App Under TestRouting Service

Third-partyServices Portals

ContentProviders

EJB

SharedServicesArchives

Business Partners

Messaging Services

Databases Mainframeapplications

App Under Test

Third-partyServices

Packaged apps, messaging services, etc.

Virtual Services

TestAutomation

TestManagement

GUI Appl Service Serv/DB

TestAutomation

TestManagement

GUI Appl – V2BPM

ServiceServ/DB

TestAutomation

TestManagement

BPMService

Serv/DB

TestAutomation

TestManagement

BPMService

ServiceVirtualization

TestAutomation

TestManagement

GUI Appl – V2Service

Virtualization

IBM Selenium Supporting Services incl datbases

Virtual Components replacing entire applicationsEnables testers to continuously test what they want, when they want, how they want

Continuous Testing improves software quality

Actual Service/App

Virtual Service/App

Service Virtualization is an enabler for continuous testing

Services, applications, systems are introduced into the continuous integration cycle in a prioritized, controlled fashion.

Controlled integration helps isolate defects for faster resolution.

Service virtualization enables

continuous integration testing

Virtual Components mature over timeEnables testers to continuously test what they want, when they want, how they want

• ActiveMQ

• Email (SMTP, IMAP)

• Files

• FTP/S

• HTTP/S

• WebSphere MQ

• Java RMI

• JMS

• SAP IDoc, BAPI, RFC

• Software AG’s IB & IS

• Sonic MQ

• TCP/IP

• TIBCO Rendezvous, Smart

Sockets & EMS

• CICS TG

• IMS Connect

• CentraSite

• Oracle Fusion

• SCA Domain

• Software AG IS, Broker, BPMS

• WebSphere Application Server

• WebSphere Message Broker

• WebSphere MQ

• Websphere SI Bus

• Sonic ESB

• System z (CICS and IMS)

• TIBCO ActiveMatrix

• UDDI

• Web Services

• WebSphere RR

• WSDL

• BPM

• Databases

• Log Files

• .Net Objects

• Bytes

• COBOL Copybook

• ebXML

• ASC X12

• UN/EDIFACT

• DFDL

• Fixed Width

• Java Objects

• MIME

• SOAP

• Software AG Broker Docs

• SWIFT

• CHIPS

• FedWire

• TIBCO ActiveEnterprise

• XML (DTD, XSD, WSDL)

• ISO8583

• Custom

Messaging Protocols SOA, ESB, Others Message Formats

Supported Technologies In v8.6

V8.6 – July 2014

Sample Deployment:

MQ Exit (and z MQ Exit) Recording

Component

to Simulate/

SUT

RIT(s)

MQ Queue

Manager

Exit

Client that initiates

transaction

MQ Exit is installed on the MQ

Queue Manager. Captures

inbound and outbound traffic for

recording purposes. For tests and

stubs.

Q1 Q2

Okay great… but we use “mocks”?

• “Mock” or ad-hoc stub

– A piece of code used to stand in for some other

programming functionality.

– A stub may simulate the behavior of existing code or be a

temporary substitute for yet-to-be-developed code.

This approach is:

• Labor intensive to write and maintain mocks.

• Requires deep knowledge of both technology and the business

logic to be emulated.

• Requires changes to application configuration when switching

to live services for testing.

• Hard to share across the entire team.

• Test data is not maintained by Testers.

OAT

Performance

UAT

Sys Tests

Unit Tests

The Old World

Requirements

Quality Gate

Big Bang

• Multiple defects introduced at once

• More unknowables – greater risk

• More expensive

Performance

Service Virtualization

Requirements

Unit Tests

UAT

OAT

Integration Tests Sys TestsSys Tests

• Accelerates testing

• Reduces costs

• Lower risk

Big Bang

Virtualization - Reduces Sys Test dependencies

Solving the Test Environment dilemmaMaking production-like test environments a reality

Traditional test environments are both underutilized and insufficient– Environments are expensive: hardware, software, and labor

– Tester and developer time wasted waiting for appropriate environment

Agile test environments require four enabling capabilities

Dynamic Infrastructure

Test Data

Management

Test Environments

DevOps Foundation

Service

Virtualization

Deployment

Automation

Deployment AutomationAccelerating application delivery across the lifecycle

UrbanCode Deploy

• Automated, consistent deployments and rollbacks of applications

• Clear visibility: what is deployed where and who changed what

• Integrated with middleware, provisioning and service virtualization

What Testers really need!Validating today’s complex applications

Continuous Testing

DevOps Foundation

Orchestration from build to production deployment

• Reliable • Repeatable • Traceable

Test Environments

• Realistic & complete

- (production like)

• Available on demand

• Fast & inexpensive to

standup & setup

Test Automation

• Unattended execution

• Validation of business

flows across channels

• Durable, maintainable,

resilient to change

Unattended test executionTeams need to automate tests at every layer – integration, functional, & performance

Continuous Testing

Orchestration

Continuous Testing

InfoSphere Optim

Test Data ManagementRational Test

Virtualization

Server

Rational Test

Workbench

• Capture service

behavior and model

virtual services

• Test services and

applications long

before their user

interfaces becomes

available and do

integration testing

• Performance test,

regression test,

functional test &

integration test

• Virtualize

heterogeneous

hardware, software,

and services to

provide 24x7 testing

capabilities

• Reduce the cost of

traditional testing

environments

• Build Virtual Services

to simulate over 70+

protocols

Rational

Performance

Test Server

• Reuse Rational Test

Workbench scripts to

drive performance

testing

• Combine virtual

services and the real

implementations to

performance test

earlier

• Probe for

identification of

system bottlenecks

• Create referentially intact, “right-sized” test databases

• Automate test result comparisons to identify hidden errors

• Protect confidential data used in test, training & development

• Refresh test environments faster shortening iterative testing cycles and accelerating time to market

IBM’s continuous testing solution

Rational Quality

Manager

• Track quality using a

comprehensive and

customizable test

plan

• Link test cases with

requirements,

defects, development

work items and builds

• Author and execute

automated or manual

test scripts

• Provide up-to-date

project quality status

using customizable

dashboards

Test Data

ManagementService

Virtualization

Test

Automation

with integration to

Collaborative

Development

& Test

The IBM Continuous Testing solutionHelping organizations get higher quality software to market

Orchestration

Continuous Testing

InfoSphere Optim

Test Data ManagementRational Test

Virtualization

Server

Rational Test

Workbench

• Capture service

behavior and model

virtual services

• Test services and

applications long

before their user

interfaces becomes

available and do

integration testing

• Performance test,

regression test,

functional test &

integration test

• Compound Tests

embeds

Selenium-Web

• Virtualize

heterogeneous

hardware, software,

and services to

provide 24x7 testing

capabilities

• Reduce the cost of

traditional testing

environments

• Build Virtual Services

to simulate over 70+

protocols

Rational

Performance

Test Server

• Reuse Rational Test

Workbench scripts to

drive performance

testing

• Combine virtual

services and the real

implementations to

performance test

earlier

• Probe for

identification of

system bottlenecks

• Create referentially intact, “right-sized” test databases

• Automate test result comparisons to identify hidden errors

• Protect confidential data used in test, training & development

• Refresh test environments faster shortening iterative testing cycles and accelerating time to market

IBM’s continuous testing solution

Rational Quality

Manager

• Track quality using a

comprehensive and

customizable test

plan

• Link test cases with

requirements,

defects, development

work items and builds

• Author and execute

automated or manual

test scripts

• Provide up-to-date

project quality status

using customizable

dashboards

The IBM Continuous Testing solutionHelping organizations get higher quality software to market

What Testers really need!Validating today’s complex applications

Continuous Testing

DevOps Foundation

Orchestration from build to production deployment

• Reliable • Repeatable • Traceable

Test Environments

• Realistic & complete

- (production like)

• Available on demand

• Fast & inexpensive to

standup & setup

Test Automation

• Unattended execution

• Validation of business

flows across channels

• Durable, maintainable,

resilient to change

IBM UrbanCode Deploy orchestrates and automates the deployment

of applications, middleware configuration, and database changes into

development, test and production environments.

Databases InternalMessages

Third-partyServices

virtual components

Simultaneously

test across

multiple test

stages

Dev QA

IBM Rational Test

Virtualization Server

Combining continuous testing withOrchestration from build to productionReliable, repeatable, & traceable deployment automation

IBM UrbanCode

Deploy

IBM Rational

Test Workbench

integrated with

Test Environments

Dynamic Infrastructure

Deploy what is ready,

virtualize the rest

Continuously test in

production-like env.

Deploy private, secure

right sized test data

IBM InfoSphere

Optim Test Data Mgmt

UC Deploy Plugins

IBM Rational

Quality Manager

Manage quality across

the delivery lifecycle

Conclusion on Tools

30

BUILD

DEPLOY

Operate Develop/ Test

Deploy

Steer

DevOpsContinuous Feedback

What we covered …

33

• Tools

– ALM

– Environments

– Automation

– Deployments

• Goals

– Integration

– Standardization

– Automation

IBM Software Group | Rational software

34

IBM Positioned in Leaders Quadrant Gartner, Inc.’s 2014 Magic Quadrant for Integrated Software Quality Suites

Download the full report

* Gartner does not endorse any vendor, product or service depicted in our research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

"This Magic Quadrant graphic was published by Gartner, Inc. as part of a larger research note and should be evaluated in the context of the entire report. The Gartner report is available upon request from IBM."

Figure 1. Magic Quadrant for Integrated Software Quality SuitesIBM has built a broad portfolio of technology products and services over many years. The products have made strong strides during the past few years in becoming more consistent and improving ease of use.

Gartner, Inc. positioned IBM as a leader in the published

2014 Gartner Magic Quadrant for Integrated Software

Quality Suites (ISQ) report. For the report, Gartner

evaluated 14 worldwide vendors, recognizing IBM moving

to the number one spot in the leaders quadrant based on

its completeness of vision and ability to execute.

IBM are helping companies across the globe unlock the full potential of quality management & testing with our solutions:

DevOps - Continuous Testing

Continuous testing provides project teams with continuous feedback and enables them to test earlier, with greater coverage removing traditional testing bottlenecks such as unavailable

Continuous Engineering – Continuous Verification

Continuous verification is a key practice in continuous engineering. Physical and systems behavior are modeled early in the product development lifecycle, and then simulation and test technologies are applied as the design is matured—to cut rework and achieve faster time to quality.

Where to find more information?

35

http://business.vanlint5.nl

http://jazz.net

Proof of Technology

Service Virtualization

NL: 30 September

BE: 25 September

Other POTs:

- DevOps / Urbancode

- Agile Development

- SAP Testing with Rational