The Survey Says: Testers Spend Their Time Doing...

9/11/2014

1

© 2013 IBM Corporation

@Al_Wagner

IBM Rational

Testers spendtheir time doing…

The

1 © 2013 IBM Corporation

Session Abstract

How can testers contribute more to the success of their project and their company? How can they focus on asking the right questions, improving test planning and design, and finding defects so the business releases a quality product―even though there’s always one more fire to extinguish or one more request to fulfill? There aren’t enough hours in the day to do it all.

Join Al Wagner as he reveals recent survey results showing where testers actually spend their time and where testers think their time would be better spent. Compare your own experience with what 250 test professionals from around the world reported. You may be surprised how prevalent testing challenges really are. Learn what techniques and technologies are available to help today’s test professionals execute what they were actually hired to do—test software.

Return to your organization with an increased understanding of how other testers are dealing with their testing bottlenecks and what activities your peers view as the best use of their valuable time.

9/11/2014

2


Why run a survey?

Aside from talking to directly to customers, we were…

watching forums and blogs

reading papers

following social media channels

listening to the Analysts

What we found was a small sampling of data harvested from anecdotes and discussions between small groups.

We wanted to hear directly from the Testers!

We wanted to hear from a lot of Testers!


Agenda – Survey Says

Setting the stage

Challenges testers face

Help is available

Putting it in context

9/11/2014

3


Who responded?250 Testing Professionals

from around the world!

Testing Experience

Testing Roles

Quality Managers, Test

Managers, and Test Leads

with more than 6 years

experience!


What and how are they testing?

250 Testing Professionals

dedicated to improving

software quality

Types of applications being tested

Development methodology followed

43% cite Scrum as their main

development methodology.

56% of organizations are

using some variation of

iterative development.

9/11/2014

4


What tools and techniques are being used?

61% of Testers surveyed use

documents & spreadsheets to manage

software quality and test execution.

Is this effective? You decide!


Agenda

Setting the stage


Help is available


9/11/2014

5


What’s The Rush?Better – Cheaper - Faster

In The Past

• Organizations looked to find a

balance between speed, cost &

risk.

• Development teams were not as

agile as they are today; testing

never seemed to have enough

time.

Today

• Organizations are asking teams to

release higher quality software

sooner.

• Development teams are becoming

more and more agile; testing still

never seems to have enough

time.


Change in Release Frequency

How have Release Cycles changed?

…over last 3 years

32% of Test Professionals surveyed have

seen an increase in release frequency

over the past 3 years.

34% of those who

expressed no change were

already releasing quarterly

or better.

9/11/2014

6


What is the ratio of Testers to Programmers on a delivery team in your organization?


Testing takes as long as it takes! Really?

Testing will take

longer when you are

not “actually” testing.

How Much of Your 40-Hour Workweek Is

Spent on Unplanned, Non-Testing Activities?

Most Disruptive

non-Testing Activities!

35% of Testers are

spending at least ½ of their

week not testing!

9/11/2014

7


How Confident Are You in Your Product's Quality?

44% of Test Professionals

surveyed have little to no confidence

in the quality of the software

being released.


Where and How Often Do You Find Defects?

40% of Testers shared that defects are

frequently found in production!

Shouldn’t it be a priority to find defects sooner?

9/11/2014

8


What change do you feel will make the biggest difference in your organization?


Agenda

Setting the stage


Help is available


9/11/2014

9


Testers want to spend

more time on…

1

2

3

4

5

6

7

8

We asked 250 Testers where they would like to spend less time and where they felt their time would be better spent?


1

2

3

4

5

6

7

8

Survey Says!

I want to Spend MORE time

9/11/2014

10


Things to remember when automating tests…

Not everything should be automated!

Not everything can be automated!

Test automation is not a silver bullet!


Are you testing what’s under the surface?

Build and Deploy

Software components

built by developers.

Software being

checked by Testers

9/11/2014

11


Test Automation

Are we automating the right tests?

Are we automating the tests rights?

Traditional Test Automation*

* http://www.agilecoachjournal.com/index.php/2014-01-28/testing-2/the-agile-testing-pyramid/

User Interface

tests

Service

tests

Unit

tests


Prioritizing your test automation effort

Focus on…

Manual /

Exploratory

Testing

pushing

tests

lower

Unit Tests

Tests written before code (TDD);

instant feedback to the developer on

code being written; defects isolated to

specific line(s) of code; run as part of

build automation process

Integration Tests

Tests message exchange, behavior, and

performance between integrated and dependent

components/systems; defects isolated to

specific application component, continuously

executed as part of deployment automation;

lower effort to maintain as application changes

User Interface tests

Checks functionality via the user interface without

understanding the application architecture (black box);

most fragile of automated test types; lacks ability to

isolate defect; most time consuming to create and

maintain; dependent on user interface being available

and stable

9/11/2014

12


Unattended test executionTeams need to automate tests at every layer


Separate what the test accomplishes from how it is executed

–Keywords abstract the details of how to execute a logical unit of function

–Business scenarios compose keywords into a flow

Better leverage the skills in your organization

–Subject matter experts can focus on the business scenarios independent of implementation

–Test automation engineers can focus on automating

Reduce test (scenario and automation) maintenance

–Keyword behavior isn’t reproduced in multiple tests; update fewer places when application changes

–Reduce test cases because overlap becomes apparent when abstraction is raised to keyword level

Durable, maintainable, resilient to changeTesters want maintainable test automation

9/11/2014

13


A business process may be available through multiple channels– The goal is the same across channels

– The flow is similar across channels

– The UI gestures differ depending on channel

A business process may span multiple channels– Different steps in the flow occur on different

channels

– Consider applications taking advantage of mobile capabilities such as location and near field communication

Multichannel testing is about effectively creating, managing, and executing tests of these kinds of processes

Validating business flows across multiple channelsEffectively creating, managing and executing tests

Keywords make supporting channels easy!


Testers want to spend

LESS time on…

1

2

3

4

5

6

7

We asked 250 Testers where they would like to spend less time and where they felt their time would be better spent?

9/11/2014

14


1

2

3

4

5

6

7

8

Survey Says!

I want to Spend LESS time


Standing up Test Environments is a key inhibitorApplications are getting more complex

• Increasing expense of standing up

complex environments

• Delays caused by lack of system

availability

• Third-party services can be

especially costly

• Testers spend more time setting up

tests than testing

30-50%The average amount of time testing teams

spend on setting up test environments,

instead of testing

$5-30 million

The typical investment to build a single

test lab for a Fortune 500 company. Most

have dozens …

9/11/2014

15


How do you mitigate delays?

Beware… these approaches are risky and

potentially damaging to the business.


Solving the Test Environment dilemmaMaking production-like test environments a reality

Today’s test environments must be: Realistic and complete (production-like)

Inexpensive to stand up

Available on demand

Agile test environments require four enabling capabilities Test Data Management

Service Virtualization

Deployment Automation

Orchestration

9/11/2014

16


Time Spent on Test Data Management(more than one response allowed)

64% creating test data manually!

34% cloning production and manually cleansing!


Refresh test data

speeding testing and

application delivery

Reduce Risk

Mask sensitive

information for

compliance and

protection

Reduce Cost

Automate creation of

realistic “right sized” test

data to reduce the size

of test environments

• Understand what test data is needed for test cases

• Create “right-sized” test data by subsetting

• Ensure masked data is contextually appropriate to the data it replaced so as not to impede testing

• Easily refresh & maintain test data by developers and testers

• Automate test result comparisons to identify hidden errors

• Support custom & packaged ERP applications in heterogeneous environments

Improve application quality and delivery Agile testing needs continuous access to test data

Speed Delivery

9/11/2014

17


Dealing With Unavailable Software & Services(more than one response allowed)

Survey says… “Top 2 tactics”:

• Manual creation and maintenance of mocks

• Expensive; lack of scalability; takes time away from testing

• Defer testing until dependent software is available

• Adds risk; decreases test coverage; potential project slip


What is Service Virtualization?Agile teams need to make the unavailable available for testing

Virtual components simulate the behavior of a service or application during testing

– System dependencies are a key challenge in setting up test environments

– Unavailable/inaccessible services slow down the testing process

– Costly 3rd party access fees can significantly impact testing availability

Virtual components run on commodity hardware, private cloud, public cloud

Each developer and tester can easily have their own test environment

Developers and testers can continue to use current testing procedures and tools

Streamline creation of production-like environments and negative testing

Heterogeneous Environments

Public CloudPrivate Cloud

Data Warehouse MainframeEnterprise

Service Bus

Directory

Identity

File

systems

Collaboration

App Under TestRouting

Service

Third-party

Services Portals

Content

ProvidersEJB

Shared

ServicesArchives

Business

Partners

Messaging

Services

Databases Mainframe

applications

App Under Test

Third-party

Services

Packaged apps, messaging services, etc.

Virtual Components

9/11/2014

18


Deployment AutomationAgile teams need to accelerate application delivery across the lifecycle

Automated, consistent deployments and rollbacks of applications

Orchestration of changes across servers, tiers and components

Configuration and security differences across environments

Clear visibility: what is deployed where and who changed what

Integrated with middleware, provisioning and service virtualization

Key Features:

• Multi-Tier Application Models

• Easy Process Designer

• Track what is where:

Inventory

• Scalable Distributed

Automation

• Quality Gates and Approvals

• Integrations Replace Custom

Scripting

Deployment Automation


Agenda

Setting the stage


Help is available


9/11/2014

19


Dev-Test-Ops (aka. DevOps)Enterprise capability for continuous software delivery that enables organizations to seize market opportunities and reduce time-to-customer feedback

Accelerate

software delivery –

for faster time to value

Balance speed, cost,

quality and risk –

for increased capacity

to innovate

Reduce time to

customer feedback –

for improved customer

experience

Continuous

Customer

Feedback &

Optimization

Collaborative

Development

Continuous Release

and Deployment

Continuous

Monitoring

Continuous

Business Planning

Continuous

Testing

Operate Develop/ Test

Deploy

Steer

DevOpsContinuous Feedback


All the world’s a “test” stageand all Testers merely players; They have their exits and their entrances;

The Continuous Delivery Pipeline

Development Testing Staging ProductionBusiness

Owners Customers

From ideation to implementation…• everyone contributes to quality!

• everyone contributes to a positive end user experience!

• everyone contributes to the success of the business!

9/11/2014

20


Continuous Testing:

• helps extend agile and lean practices across

the delivery lifecycle

• removes traditional testing bottlenecks such as

unavailable test environments to increase

efficiency

• enables teams to test earlier and with greater

coverage at lower cost

• provides project teams with continuous

feedback on software quality to reduce

business risk

A comprehensive Continuous Testing solution

covers quality management, functional UI and

integration test automation, performance testing,

and service virtualization.

Organizations need Continuous TestingAccelerating the delivery of higher quality software

Resolve today’s testing

challenges with confidence


Orchestration and automation for the deployment of applications,

middleware configuration, and database changes into development,

test and production environments.

Databases InternalMessages

Third-partyServices

virtual components

Simultaneously

test across

multiple test

stages

Dev QA

Service

Virtualization

Combining continuous testing withContinuous Testing enables Continuous DeliveryRepeatable, & traceable test and deployment automation

Test

Automation

Test Environments

Dynamic Infrastructure

Virtualize unavailable

yet dependent software

& systems

Continuously test in

production-like env.

Quality

Management

Manage quality across

the lifecycle

Deployment

Automation

Deploy private, secure

right sized test data

Test Data

Management

Deploy just built

applications

Continuous

Delivery

Orchestrate the

delivery pipeline

integrated with

9/11/2014

21


Top 6 take aways

1

2

3

4

5

Not every test can be automated.

Not every test should be automated.

At least 1/3 of the delivery lifecycle

is testing. Testers do matter!

Test what matters earlier.

Test it continuously!

Deploy what you have.

Virtualize the rest!

6

Everyone contributes to quality!

You can’t test quality in.

Continuous Testing enables

Continuous Delivery


The

Download your copy today!

ibm.biz/TestingSurveyResults

How do you stack up to what

the others are saying?

9/11/2014

22

42

ibm.biz/continuoustesting

43

The Survey Says: Testers Spend Their Time Doing...

Software

Transcript of The Survey Says: Testers Spend Their Time Doing...