Building the "right" regression suite using Behavior Driven Testing (BDT)
61
Building the “right” regression suite using Behavior Driven Tes5ng (BDT) Anand Bagmar So#ware Quality Evangelist Principal Consultant,
-
Upload
anand-bagmar -
Category
Technology
-
view
113 -
download
1
description
Presented in World Conference - Next Generation Testing, Bangalore on 11th July 2013
Transcript of Building the "right" regression suite using Behavior Driven Testing (BDT)
Building the “right” regression suite using Behavior Driven Tes5ng
(BDT) Anand Bagmar
So#ware Quality Evangelist Principal Consultant,
What do you expect from this session?
Case Study
• MulHple, long running projects
• Legacy applicaHons
• Integrated
• Limited / flaky / long-‐running AutomaHon
• Long regression cycle • Huge cost of fixing defects
FuncHonal TesHng Challenges
• BriRle • LiRle / less value • No visibility into what is tested • Maintenance nightmare • Outdated • Expensive
As a result
Biggest problem
• Trust deficit on the team – In exisHng AutomaHon – In manual tesHng
• Finger poinHng / blame game
My Case Study
Concept # 1
The Test Pyramid
AutomaHon – A Safety Net
• “Succeeding with Agile” – Mike Cohn
• MarHn Fowler – Test Pyramid – hRp://marHnfowler.com/bliki/TestPyramid.html
The Test Pyramid
Func%onal Tests (GUI)
Web Service Tests
View Tests
JavaScript Tests
Integra%on tests
Component tests
Unit Tests
Test AutomaHon Pyramid
Manual / Exploratory TesHng
Product under test
Ideal Test Pyramid
Business-‐facing Tests
Technology-‐facing Tests
• Cost • Effort • Time
Test Pyramid reality
Development team Test Pyramid QA team
Test Pyramid
Func%onal Tests (GUI)
Web Service Tests
…
…
Unit Tests
Manual / Exploratory TesHng
Test Ice-‐cream cone
Business-‐facing Tests
Technology-‐facing Tests
Test Pyramid – AnH PaRern
Product under test
Func%onal Tests (GUI)
Web Service Tests
…
…
Unit Tests
Manual / Exploratory TesHng
Product under test
Ideal Test Pyramid
Business-‐facing Tests
Technology-‐facing Tests
IdenHfy the right type of tests that sit on top of the Test Pyramid
Concept # 2
BDD & BDT
Behavior Driven Development (BDD)
RED
GREEN REFACTOR
Behavior Driven TesHng (BDT)
The Big Picture
• How is this funcHonality going to be used by the end-‐user?
• What is their thought process going to be when using this product?
• What is the “core-‐business-‐value” I am delivering to the end-‐user?
Concept # 3
Test Specifica5on Styles
Test specificaHon styles
• ImperaHve • DeclaraHve
Search Search Results
Contact Informa%on
An example For SomeAwesomeAirlines website – A Guest User can …
ImperaHve style Given I am a guest user on the home page And I choose “round” trip opHon And I select “Chicago” from the origin dropdown And I select “San Francisco” from the desHnaHon drop down And I select departure date as “5 December 2013” And I select returning date as “25 December 2013” When I click on Search Then I should see the search results page And I should see at least 1 opHon for my criteria ...
… When I select the “first” opHon Then I am on the Contact InformaHon page When I enter first name as “foo” And I enter last name as “bar” … … And I click the “Select and Con%nue” buRon Then I should be on the next page
Given I am a guest user When I search for flight opHons for a “one-‐way” trip for “1” “Adult” from “Chicago” to “San Francisco” And I select the “first” flight And I enter “valid” contact details for “traveller1” Then I am able to Save and ConHnue
DeclaraHve style
References
• Ben Mabey -‐ hRp://benmabey.com/2008/05/19/imperaHve-‐vs-‐declaraHve-‐scenarios-‐in-‐user-‐stories.html
• Alister ScoR -‐ hRp://waHrmelon.com/2010/12/10/cucumber-‐imperaHve-‐or-‐declaraHve-‐that-‐is-‐the-‐quesHon/
My Case Study
• Remove the ambiguity • Same visibility for all
How did we accomplish this?
• IdenHfy business flows based on personas, and how the personas use the business funcHonality, or,
• IdenHfy business flows, and how different user personas can use the business flows
1st candidate • Automate! • High Value, Low Cost
2nd candidate • Automate! • High Value, High Cost
Can Automate? • Manual Regression?
• Low Value, Low Cost
Do Not Automate • Manual Regression!
• Low Value, High Cost
High Value
High Cost
Low Value
Low Cost
Who is doing the acHon?
What is the business funcHonality?
Depicts a user flow / user journey
Given auc%oneer creates an aucHon And 5 vehicles are added to the aucHon When auc%oneer starts the aucHon And auc%oneer sets a starHng bid of 5000 And buyer_1 bids And buyer_2 bids Then auc%oneer Sells the vehicle And buyer_2 wins the vehicle
What next?
• IdenHficaHon of Tests
• Automate it!
Automate tests that are valuable, not because you can or like to automate!
Who is doing the acHon?
What is the business funcHonality?
Depicts a user flow / user journey
Ac5ons and verifica5ons happen implicitly
Given auc%oneer creates an aucHon And 5 vehicles are added to the aucHon When auc%oneer starts the aucHon And auc%oneer sets a starHng bid of 5000 And buyer_1 bids And buyer_2 bids Then auc%oneer Sells the vehicle And buyer_2 wins the vehicle
Delete profile
1st candidate • Automate! • High Value, Low Cost
2nd candidate • Automate! • High Value, High Cost
Can Automate? • Manual Regression?
• Low Value, Low Cost
Do Not Automate • Manual Regression!
• Low Value, High Cost
High Value
High Cost
Low Value
Low Cost
Final state Project Management tool Mingle
Test case repository (manual + automated)
Cucumber .feature files with appropriate tagging
Feature files / Scenarios 33 / 65
# of Manual Tests 0
Tests running on CI (Jenkins) Since the 1st test was automated
Smoke tests execuHon Hme 15 minutes
Regression tests execuHon Hme 45 minutes
How did this technique ever become successful?
Context is KING!
Domain
Product-‐under-‐test
Distributed Teams Skills
Capability
Tech Stack
Process Tools
BDT will ….
• NOT work in isola5on
Requires more dedicated and focused tesHng (manual / exploratory)
Requires lower layers of Test Pyramid to be automaHon and implemented well!
Func%onal Tests (GUI)
Web Service Tests
…
…
Unit Tests
Manual / Exploratory TesHng
Product under test
Ideal Test Pyramid
Business-‐facing Tests
Technology-‐facing Tests
IdenHfy the right type of tests that sit on top of the Test Pyramid
AddiHonally …
• Builds and deployments are automated
• FuncHonal Performance and Concurrent TesHng automated and run on demand
BDT Value ProposiHon
?
Incrementally build User Flows
Regress required Product Business
Value
Tests in-‐sync
Effec%ve Tests
Remove ambiguity
Understand new
func%onality
Living Documenta%on
Onboarding
Executable Specs
BDT Scenarios
Func%onal Tests (GUI)
Web Service Tests
…
…
Unit Tests
BDT helps iden5fy the “right” type of regression tests!
Manual / Exploratory TesHng
Product under test
Ideal Test Pyramid
Tools to enable BDT
• Whiteboards • Mind maps • Flow charts • Collabora5on!!
Automate the top layer
Automate using -‐ • BDD Tools, or, • Directly in any appropriate programming language / tools
![Data-driven Fluid Simulations using Regression …regression forests [Breiman 2001], a powerful approach success-fully applied to various real-time tasks such as Kinect body-part regression](https://static.fdocuments.in/doc/165x107/5f0f06747e708231d4421db0/data-driven-fluid-simulations-using-regression-regression-forests-breiman-2001.jpg)
