Automate virtualize and smart test the new testing realities

©2011, Cognizant

by

Manoj NarayananDirector –Testing Services, Cognizant

Automate, Virtualize & Smart Test

Understanding the new realities of Testing

@manoj7698

http:\\everydaytesting.blogspot.com

Requirements Gathering

Architecture & Design

Development

Testing Release

1

What does traditional testing offer?

Discrete, independent

Waterfall Methodology

Focused on defect identification

©2011, Cognizant

Reducing risk by preventing poor quality

from impacting satisfaction

Quickly responding to

customer requests and requirements

22%Reducing costs by improving

our development and testing productivity

17%Delivering

applications on time and on

budget

14%Improving

collaboration between

development and QA

13%Delivering

applications before

competition

7%Compliance

and regulatory

issues

5%

2

• 32% of organizations say they release software with too many defects

• 38% of organizations believe they lack an adequate quality assurance program

• 27% of organizations do not conduct any formal quality reviews

What does business want today?

Source- QAI –Edista Testing More than 60%

24%

©2011, Cognizant

Reducing risk by preventing poor quality

from impacting satisfaction

Quickly responding to

customer requests and requirements

22%Reducing costs by improving

our development and testing productivity

17%Delivering

applications on time and on

budget

14%Improving

collaboration between

development and QA

13%Delivering

applications before

competition

7%Compliance

and regulatory

issues

5%

3

• 32% of organizations say they release software with too many defects

• 38% of organizations believe they lack an adequate quality assurance program

• 27% of organizations do not conduct any formal quality reviews

What does business want today?

Source- QAI –Edista Testing

Reducing Risk

Responding Fast

Beat Competitors

More than 60%

24%

©2011, Cognizant



Development

Testing Release

4

Traditional testing approach

Discrete, independent


Focused on defect identification

©2011, Cognizant

We need to move testing to the beginning of SDLC – focus on prevention than identification

5



Development Testing Release

From Discrete and Independent Testing to…

…Continuous and Integrated Testing


Scrum

©2011, Cognizant

The “Shift-Left” Philosophy

6

Identify QA skill sets needed

and initial effort estimate

Early execution of QA, Non-GUI testing

Automated end to end regression

testing

Non-functional testing

Focus on Test Driven Development or a

variant

Performance & Security PoCs

©2011, Cognizant

Shift-Left Philosophy is accompanied by the need for a new testing approach

7 Automation becomes critical

Need to move away from traditional

approach focusing on GUI based testing..…to an approach focusing on increased

defect capture through Unit and Service

Layer Testing

Unit

Testing

Service Layer Testing

GUI Testing

Unit Testing


GUI

Testing Perc

en

tag

e

eff

ort

sp

en

t

©2011, Cognizant

This results in the need for seeding Quality Engineering into testing

8

QE Approach QA Approach

• A QE approach focused on testing components

• A QA approach focused on regression and UAT

©2011, Cognizant

And “Smart Testing” dissolving boundaries of traditional system & integration testing

9

QE Approach

Unit Testing


Functional Testing

(automation)

Regression Testing

(automation)

Beta Testing

User Acceptance

Testing

QA Approach

Fast merging into a social media and

“crowd” aided Community Testing

©2011, Cognizant

And “Smart Testing” dissolving boundaries of traditional system & integration testing

10

QE Approach

Unit Testing


Functional Testing

(automation)

Regression Testing

(automation)

QA Approach

Community Testing

©2011, Cognizant

Leverage optimal mix of automation across the lifecycle

11

Unit Testing


Regression Testing

• Re-usable, automated test cases used by developers• Leveraging xUnit Frameworks

• Leverage re-usable frameworks like RIDE, Fitnesse, Cucumber

• Web Services testing • Leverage Service Virutalization

• >90% automated regression test cases• 100% automated smoke testing• Continuous validation of test flow relevancy

Manual Testing

©2011, Cognizant

Prioritizing Automation: ROI driven approach

12

Repetitive Testing

Verify application paths that have high degree of frequency and other high volumeactivities where software failures would occur

Validate applications with a high degree of risk which define or control the core ofa company’s business; Failure can cause extreme disruptions in critical operations

Automating applications that are or will be in production for a long time

Greater benefits can be reaped from Test Automation if the test executionfrequency is high

Test Automation will be an ideal testing solution if the testing has to be performed onmultiple environments

Any flow that needs to be run repeatedly in a build

©2011, Cognizant

Prioritizing Automation: ROI driven approach

13

Repetitive Testing

Verify application paths that have high degree of frequency and other high volumeactivities where software failures would occur

Validate applications with a high degree of risk which define or control the core ofa company’s business; Failure can cause extreme disruptions in critical operations

Automating applications that are or will be in production for a long time

Greater benefits can be reaped from Test Automation if the test executionfrequency is high

Test Automation will be an ideal testing solution if the testing has to be performed onmultiple environments

Any flow that needs to be run repeatedly in a build

©2011, Cognizant

The need for early testing and automation also triggers environment related challenges

14

OrderingSwing App

Processing Web App

COM App

ESB, JMS, MQ

Mainframe

Business Process App

Additional Apps Now & Future

Database

Database

App ServerEJB, WS,

Environment availability challenges –productivity losses, inability to duplicate

defects, performance testing

Need for testing individual components as part of “shift-left”

philosophy

©2011, Cognizant

Work Around: Virtualization & Service Virtualization

15

OrderingSwing App

Processing Web App

COM App

ESB, JMS, MQ

Mainframe

Business Process App

Additional Apps Now & Future

Database

Database

App ServerEJB, WS,

5

LISA Virtualize

UI Testing no longer the dominant testing method

Defects identified at each component level

Early testing is now feasible. Enables organization to adopt

“Shift Left” testing

Environment availability challenges mitigated – simulate performance

environment, component transactions

Re-usability increases, limited dependency on development teams

Picture: Courtesy ITKO

©2011, Cognizant

Explaining Service Virtualization

• Mainframe applications are consumed

through queues – in this case, MQ

• The MQ sends messages to the

application which gets transformed

through the adapter

“Listener”

©2011, Cognizant

This has to be complemented by a centralized Test Environment Management approach

17

Cloud too is an option – even after the Amazon EC2 fiasco

18

And an organizational transition plan based on current maturity level

Focus on• Process & Methodology• Tools• Infrastructure• Operating Model• People: Skill set/Roles

Evaluate Current QA

State

Define Future State

Stakeholder Discussions

Artifact Analysis

Inputs

Industry best practices

Methodology Adopted

Inputs

Identify implementation priorities

Focus on• Risk appetite• Cost constraints• Available time• Priority areas• Current & Future

state gap

Draw Implementation

Roadmap

Stakeholder Validation

External dependencies

Inputs

©2011, Cognizant

But do not let the total cost creep up…

19

• The QA organization needs to keep a constant watch on the

Total Cost of Quality as it makes the transition

• During the transition phase, the Cost of Testing goes up due to

the increased focus in areas like skill ramp-up, automation and

process re-alignment

• The resultant increase in Total Cost of Quality needs to be

continuously adjusted to ensure that it falls within best practice

levels of 25–30% within 3 to 6 months of transition

Cost of TestingCost of Production

Defects

Cost of Compliance Audits

Total Cost of Quality

And most importantly, do not forget to communicate

20

• Ensure Executive & Stakeholder buy-in on the

change

• Establish a communication channel for regular

updates and feedback

Summary

21

• Traditional testing approaches are being replaced by a “shift-

left” approach to accommodate current business realities

• Shift away from GUI based testing to component based testing

• Focus on both QE and QA based testing – prevention rather

than identification: Need for “Smart Testing”

• Automation becomes critical to maintain the ever reducing

testing window

• New facilitators have risen: Community Testing, Service

Virtualization, Test Environment Management

• However, it is very critical that the organization has a well

thought out transition plan to accommodate these changes

And, DO NOT forget to communicate!

Automate virtualize and smart test the new testing realities

Technology

Transcript of Automate virtualize and smart test the new testing realities