September 25, 2015

Introduction to QASymphony for

[INSERT COMPANY NAME]

Relieving the Testing Bottle Neck in Your

Projects

AGENDA

• How Testing Becomes a Bottleneck• Removing the Bottleneck with a New Process• Increasing Manual Testing Efficiency• Improving Automated Testing Coverage &

Reliability• Build Testability into the Application• Managing an Testing Strategy• Measuring a Development Team on Quality• Q&A (based on interest)

September 25, 2015

Introduction to QASymphony for

[INSERT COMPANY NAME]

Why does Testing become a Bottleneck?

Past development cycles often modeled the Rational Unified Process:

Source: http://www.psa-software.com/_img/_knowledge_center/rup.jpg

WHERE WE CAME FROM

If testers cannot test efficiently with predictable estimates of effort, testing can become the bottleneck for development (less features deployed):

TESTING CAN BECOME A BOTTLENECK

Design Code Test Deploy

Features that couldn’t be tested, so not deployed

Given a fixed level of depth in the testing , we can test and deploy more features but it will be less coverage

WHAT IF WE TEST LESS?

Design Code Test Deploy

Reduced depth of testing on features

If given the choice, we would have moved testing earlier in the cycle, but we were constrained by many environmental factors:

WHY DOES TESTING NEED TO BE LAST?

Environment Creation Code Merges On-Premise

Prevalence

Desktop Focus

Lack of Collaboration

Off-Shore Development

Many of our prior limitations have been replaced, based on macro trends around technology and industry:

WE CAN MOVE THE TESTING UP FRONT

• Containers have simplified the process dramaticallyEnvironment Creation

• Git has replaced Subversion as the industry standardCode Merges

• Cloud adoption is at an all time high, increased securityOn-Premise Prevalence

• Prevalence of Web, Mobile, Internet of thingsDesktop Focus

• Increase in teamwork, chat and collaboration technologyLack of Collaboration

• Shifts towards rural sourcing, onshoring of laborOff-Shore Development

Now that we have freed ourselves of past limitations, the process can be shifted to one that aligns more with our needs:

Traditional Approach

Test-First Approach

HOW THE PROCESS CAN LOOK NOW

Design Requirements Code Test Deploy

Design (Automated) Test Code Refactor Deploy

Creating tests up front means code can be deployed almost right upon completion, removing the testing bottleneck:

HOW THE FUNNEL LOOKS NOW?

CreateTests

Code Deploy

Moving testing to the front of the process is desirable, but many realities of software teams restrict them from doing so: • Lack of automation coverage for legacy features• Lack of testability (API’s, UI frameworks) in legacy

applications• Lack of automation talent within the organization• Finite project length (i.e. package software

implementation)• Regulatory compliance (need to document requirements)

For these organizations – we will also cover ways to make a traditional requirements driven approach more efficient

FLIPPING THE FUNNEL MAY NOT BE POSSIBLE

September 25, 2015

Introduction to QASymphony for

[INSERT COMPANY NAME]

For Manual Testers

Manual test execution can be optimized by implementing exploratory testing. A 2007 controlled study found that:

– Testing with test cases vs. exploratory testing take almost 7 times longer, due to the amount of time needed to write the tests and report on them – TEST FASTER

– Testing with test cases vs. exploratory testing doesn’t find any more defects, and does not miss many (if any) critical or severe defects in comparison to test case testing – TEST BETTER

– Testing with test cases causes more false defect reports vs. exploratory testing – TEST SMARTER

Study link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.3696&rep=rep1&type=pdf

HOW TO OPTIMIZE MANUAL TESTING

1. Parallel test planning, test design, and test execution

2. Specific yet flexible

3. Aligned towards investigation of potential opportunities

4. Values depth and attention to detail during testing

5. Fosters knowledge sharing and awareness

WHAT IS EXPLORATORY TESTING?

Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approach

PARALLEL PLANNING, DESIGN & EXECUTION

Plan Design Execute Report

Plan

DesignExecute

Report

Exploratory testing provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc.

However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it

SPECIFIC, YET FLEXIBLE

Manual Scripted Testing

I tested the application as the script prescribed

Exploratory Testing

I tested the application as the end user would

Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters

INVESTIGATING OPPORTUNITIES

Manual Scripted Testing Exploratory Testing

Exploratory testing relies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity

KNOWLEDGE SHARING

Plan

DesignExecute

Report Transfer Learning

Example Questions to Ask

• Have you seen this before?

• What am I not considering?

• Why would someone do this?

• How would you have tested this?

September 25, 2015

Introduction to QASymphony for

[INSERT COMPANY NAME]

For Automated Testers

Moving towards TDD will also demand a more complete automated testing strategy focused on more than just UI testing:

BUILDING A COMPLETE TESTING STRATEGY

UI Tests

Integration TestsU

nit Tests

UI TestsIntegration

Tests

Unit Tests

UI automation should be only be used for key features and functionality, given the following limitations:

• SPEED – tests take much longer to run when waiting for pages to load, objects to interact, etc.

• RELIABILITY – tests break frequently for reasons besides failing functionality (like renamed or moved objects)

• MAINTENANCE – tests require frequent updating as UI elements are changed, requiring more maintenance

• COST – many UI automation tools are expensive and based on perpetual licensing

WHY UI AUTOMATION IS LIMITING

The ideal testing pyramid would look like the following:

THE IDEAL RATIOS

UI TestsIntegration

Tests

Unit Tests 70% of coverage, effort, maintenance

20% of coverage, effort, maintenance

10% of coverage, effort, maintenance

Focus on the missing box (es) in the Agile Testing Matrix - typically the performance and load tests:

Source: http://image.slidesharecdn.com/agiletestingandtheroleoftheagiletester2008-09-26rev4-090615163750-phpapp01/95/agile-testing-the-role-of-the-agile-tester-40-728.jpg?cb=1245084455

ADDING OTHER MISSING TESTS

September 25, 2015

Introduction to QASymphony for

[INSERT COMPANY NAME]

For Developers

Moving to TDD process moves testing activity from solely UI based to focusing more testing at the services/API layer:

BUILDING TESTABLE SOFTWARE

Need for Testability

Testing in the Unit Layer will require more mock objects to “unhinge” features that are being tested, removing them from dependencies:

Source: http://zeroturnaround.com/wp-content/uploads/2015/12/PUZZLE-1-min.png

BUILDING TESTABLE SOFTWARE

Many teams struggle to pick an automation framework, but the reality is most frameworks are similar and differentiate in a few key areas:

PICKING AN AUTOMATION FRAMEWORK

•Does it support dev centric languages (i.e. Java) or easier to use languages (i.e. Ruby)

•Does it support the concept of features, or stories?

•Does it have features to support large enterprises, as well as small teams?

• Is it more focused on BDD, TDD, or ATDD?

Dev Style Scalability

LanguageWorkflow

September 25, 2015

Introduction to QASymphony for

[INSERT COMPANY NAME]

For Test Managers

There is a myth that EVERY test should be automated immediately for a team to be as agile as possible

It is impossible that creating an automated test would the most efficient approach in all scenarios

Some examples:• Tests being used one time• Tests on difficult parts of an application• Tests for parts of the system that change frequently

THE AUTOMATE EVERYTHING MYTH

A more balanced approach is to estimate the total cost of creating AND maintaining a test over a course of its life vs the cost of manual testing:

AN ROI BASED APPROACH

Automation Manual

Script Creation Cost $500 $200

Script Maintenance Cost $200/year $50/year

Script Execution Cost $120/year $480/year

Cost after Year 1 $820 $730

Cost after Year 2 $1140 $1260

Cost after Year 3 $1460 $1790

Manual

Even

Automate

For teams that are looking to build automation coverage on legacy applications, another approach might be to automate based on time saved

A SAVINGS BASED APPROACH

Test Case A Test Case B Test Case C

Manual Execution $500/year $300/year $400/year

Manual Maint. $200/year $100/year $200/year

Auto. Creation $300 $150 $480

Automation Maint. $100/year $50/year $100/year

Year 1 Savings $400 $250 $120

September 25, 2015

Introduction to QASymphony for

[INSERT COMPANY NAME]

For Dev Managers

Agile promotes delivering software quickly, so development managers often incentivize developers based on features, story points, or lines of code delivered

However, this approach can often incentivize bad behavior – pushing incomplete or unstable code over to testers, who are then responsible to identify the issues developers missed

Development managers should incentivize their teams for building good code quickly, which is understandable and easy to test and debug

MANAGING ENGINEERS TO QUALITY METRICS

Coupling Between Objects (CBO) – A measure of how closely or loosely coupled methods are. 1-4 is good, higher than that may mean risk in trying to refactor or resolve issues

Comment Ratio – Ratio of comments to total lines of code, indicating how much the developer has documented the code for future debugging or refactoring

Associated Critical Defects Ratio – Ratio of critical defects opened on specific features divided by the effort (story points) for that feature. Often normalized for feature complexity

SOME GOOD QUALITY METRICS TO TRACK

Kevin Dunnekevindunne@qasymphony.com

Twitter: @KevinDunneQA

QUESTIONS?