Build the "right" regression suite using Behavior Driven Testing (BDT)

BUILD THE “RIGHT” REGRESSION SUITE USING BEHAVIOR DRIVEN TESTING (BDT) vodQA

ABOUT US

about.me/anand.bagmar [email protected]

@BagmarAnand

[email protected]

WHAT DO YOU EXPECT FROM THIS SESSION?

CASE STUDY

•  Multiple, long running projects

•  Legacy applications

•  Integrated

•  Limited / flaky / long-running Automation

•  Long regression cycle

•  Huge cost of fixing defects

FUNCTIONAL TESTING CHALLENGES

•  Brittle

•  Little / less value

•  No visibility into what is tested

•  Maintenance nightmare

•  Outdated

•  Expensive

AS A RESULT

•  Trust deficit on the team •  In existing Automation

•  In manual testing

•  Finger pointing / blame game

BIGGEST PROBLEM

My Case Study

Concept #1

Given -> When -> Then

Expected Functionality:

The customer should be able to withdraw money from his account via an ATM machine

BDD Example:

Given the account has balance of 5000

When the customer requests 1000

Then the account is debited by 1000

Workshop Begins – Iteration 1

Case Study

PHASE 1 REQUIREMENTS

•  Guest User is able to search for a flight for a single traveller

•  Select option from Search Results page

•  Specify contact information for person booking the flight

A Guest User can …

WORKFLOW

Search Search Results

Contact Information

SEARCH MODULE

SEARCH RESULTS

CONTACT DETAILS


•  Guest User is able to search for a flight for a single traveller

•  Select option from Search Results page

•  Specify contact information for person booking the flight


WORKFLOW


Contact Information

LETS GET STARTED … WRITE 2 TEST SCENARIOS

Iteration #1

Showcase

Concept #2

Test Specification Styles

•  Imperative

•  Declarative

TEST SPECIFICATION STYLES


AN EXAMPLE


Contact Information

IMPERATIVE STYLE

Given I am a guest user on the home page

And I choose “round” trip option

And I select “Chicago” from the origin dropdown

And I select “San Francisco” from the destination dropdown

And I select departure date as “5 December 2013”

And I select returning date as “25 December 2013”

When I click on Search

Then I should see the search results page

And I should see at least 1 option for my criteria

...

IMPERATIVE STYLE …

…

When I select the “first” option

Then I am on the Contact Information page

When I enter first name as “Anand”

And I enter last name as “Bagmar”

…

…

And I click the “Select and Continue” button

Then I should be on the next page

DECLARATIVE STYLE

Given I am a guest user

When I search for flight options for a “one-way” trip for “1” “Adult” from “Chicago” to “San Francisco”

And I select the “first” flight

And I enter “valid” contact details for “traveller1”

Then I am able to Save and Continue

•  Ben Mabey - http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html

•  Alister Scott - http://watirmelon.com/2010/12/10/cucumber-imperative-or-declarative-that-is-the-question/

REFERENCES

Workshop – Iteration 2

Case Study Continues

•  User registration from Contact information page

•  Registered user can login from home page (below search criteria)

•  For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information


Login Search Search Results

Contact Information

•  Pre-filled

WORKFLOW … ENHANCED

REGISTERED USER

•  User registration from Contact information page

•  Registered user can login from home page (below search criteria)

•  For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information


Login Search Search Results

Contact Information

•  Pre-filled

WORKFLOW … ENHANCED

LETS GET STARTED … WRITE 2 TEST SCENARIOS

Iteration #2

Showcase

WHY IS THIS IMPORTANT?

Concept #3

The Test Pyramid

AUTOMATION – A SAFETY NET

•  “Succeeding with Agile” – Mike Cohn

•  Martin Fowler – Test Pyramid

martinfowler.com/bliki/TestPyramid.html

THE TEST PYRAMID

Time Value Cost

IDEAL TEST AUTOMATION PYRAMID

Manual / Exploratory

UI

Web Service

View

JavaScript

Integration

Unit

Technology-facing Tests

Business-facing Tests

REALITY

UI

Web Service

View

JavaScript

Integration

Unit


ICE CREAM CONE ANTI PATTERN

Developers Test Pyramid QA Team Test Pyramid

Web Service

View

JavaScript

Integration

Unit


UI

Web Service

DUAL TEST PYRAMID ANTI-PATTERN

Concept #4

Behavior Driven Testing (BDT)

BEHAVIOR DRIVEN TESTING (BDT)

•  How is this functionality going to be used by the end-user?

•  What is their thought process going to be when using this product?

•  What “core-business-value” should be delivered to the end-user?

THE BIG PICTURE


UI

Web Service

View

JavaScript

Integration

Unit

Technology-facing Tests

Business-facing Tests

Identify the “right” type of tests that sit on top of the Test Pyramid

My Case Study

HOW TO SOLVE THE PROBLEM?

•  Remove the ambiguity

•  Same visibility for all

SOLUTION

•  Whiteboards

•  Mind maps

•  Flow charts

•  Collaboration!!

TOOLS TO ENABLE BDT

HOW DID WE ACCOMPLISH THIS?

IDENTIFY BUSINESS FLOWS

•  Identify business flows based on personas, and how the personas use the business functionality, or,

•  Identify business flows, and how different user personas can use the business flows

NOTE: Business flows are different from UI flows

HOW TO IDENTIFY BUSINESS FLOWS ?

PRIORITIZE BUSINESS FLOWS

1st candidate •  Automate! •  High Value, Low Cost

2nd candidate •  Automate! •  High Value, High Cost

Can Automate? •  Manual Regression? •  Low Value, Low Cost

Do Not Automate •  Manual Regression! •  Low Value, High Cost

High Value

High Cost

Low Value

Low Cost

AUTOMATE THE TOP LAYER USING

•  BDD Tools, or,

•  Directly in any appropriate programming language / tools

•  Identification of Tests

•  Automate it!

WHAT’S NEXT?

AUTOMATE TESTS THAT ARE VALUABLE, NOT BECAUSE YOU CAN OR LIKE TO!

CAUTION!

Who is doing the action?

What is the business functionality?

Depicts a user flow / user journey

Actions & Verifications happen implicitly

Given auctioneer creates an auction And 5 vehicles are added to the auction When auctioneer starts the auction And auctioneer sets a starting bid of 5000 And buyer_1 bids And buyer_2 bids Then auctioneer Sells the vehicle And buyer_2 wins the vehicle

EXAMPLE

Delete profile

UPDATE BUSINESS FLOWS

1st candidate •  Automate! •  High Value, Low Cost

2nd candidate •  Automate! •  High Value, High Cost

Can Automate? •  Manual Regression? •  Low Value, Low Cost

Do Not Automate •  Manual Regression! •  Low Value, High Cost

High Value

High Cost

Low Value

Low Cost

Project Management tool Mingle

Test case repository (manual + automated)

Cucumber .feature files with appropriate tagging

Feature files / Scenarios 33 / 65

# of Manual Tests 0

Tests running on CI (Jenkins) Since the 1st test was automated

Smoke tests execution time 15 minutes

Regression tests execution time

45 minutes

FINAL STATE … IN MY CASE STUDY

•  Builds and deployments were automated

•  Tests ran on demand, automatically! •  Smoke / Regression / Javascript / Unit

•  Functional Performance

•  Concurrent Testing

ADDITIONALLY …

How did this Technique (BDT) become successful?

CONTEXT IS KING!

Domain

Distributed Teams

Skills

Capabilities Product-under test

Tech Stack

Process

Tools

NOT work in isolation

BDT WILL …

BDT REQUIRES …

Lower layers of Test Pyramid to be well automated!

BDT REQUIRES …

Dedicated and Focused testing (manual / exploratory)

BDT helps identify the “right” type of regression tests!

Executable Specs

BDT Scenarios

Business Idea

Features

Epics

Stories

Iterations

Releases

BDT VALUE PROPOSITION

BDT VALUE PROPOSITION …

Incrementally build

Business Flows

Regress Business Value

Test in-sync with Product Functionality

Effective Tests

BDT VALUE PROPOSITION …

Remove ambiguity

Understand new functionality

Living Documentation

Onboarding

ABOUT US

about.me/anand.bagmar [email protected]

@BagmarAnand

[email protected]

Build the "right" regression suite using Behavior Driven Testing (BDT)

Software

Transcript of Build the "right" regression suite using Behavior Driven Testing (BDT)