End-to-End Test Automation for Both Horizontal and Vertical Scale
-
Upload
erdem-yildirim -
Category
Technology
-
view
96 -
download
3
Transcript of End-to-End Test Automation for Both Horizontal and Vertical Scale
End to End Test Automation for Both Horizontal and Vertical Scale
Erdem YILDIRIM
2000 – Software Developer
2006 – Discovered Test Automation &
Decided to be Test Engineer
2010 – Test Lead at Defence Technologies and Engineering (STM)
2013 – Test Lead at İnnova(Telecommunication)
About Me
Linkedin: https://tr.linkedin.com/in/erdem-yildirim-0b5ab466
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales)
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Agenda
Agenda
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales)
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Machines are great for runningrepetetive / straightforward scenariosthat don’t require any intelligence orexperience.
Test Automation
End to end testing is verifying that all units of anapplication and all subsystems of a system interact asexpected with each other that the system as a wholeworks as intended.
E2E Test Automation
Ideal Software Test Automation Pyramid
UI
Tests
Integration
Tests
Unit Tests
Manual
(Scripted)
Test
It is OK that automating whole Unit, Inregration, UI levels is great.
Can we improve it?
More Ideal One
CI –
Co
ntin
uo
us In
teg
ratio
n
Exploratory
Test
An Extra Level: Test Data Preparation
How to prepare test dataset and bringdatabase to known state for test cases before
every test run?
Is Teardown Method OK?
Test Data Preparation Automation
Teardown Method with Automation
Setup
Exercise
Verify
Teardown
SUT
Install
Create
Return
ValuesIndirect
Inputs
Record
Mock
Object
Advantages of AutomatingTest Data Preparation Phase
Faster coding,
Faster test executions,
Clean test environment,
Testing with real and more complex test data,
Loose coupling between data and code layers
Facilitates testing for multiple customers (each with a DB)
So, we were able to automate theentire vertical E2E testing:
Unit, Inregration, UI +Test Data Preparation levels
Can we improve it further?
E2E Test AutomationHorizontal & Vertical Scale
UI
Tests
Integration
Tests
Unit Tests
Test Data Preparation
CI –
Co
ntin
uo
us In
teg
ratio
nHorizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay
Vertical E2E
Implementing
tests on all level
of the pyramid
E2E Test AutomationHorizontal & Vertical Scale
UI
Tests
Integration
Tests
Unit Tests
Test Data Preparation
CI –
Co
ntin
uo
us In
teg
ratio
nHorizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay
Vertical E2E
Implementing
tests on all level
of the pyramid
Prerequisites of Horizontal E2E Tests
All system and subsystems’ test environments shouldbe ready.
If they are not ready, mocking work should be performed.
Prerequisites of Vertical E2E Tests
Having a supporting test/development strategy likeTDD, BDD, Continuous Testing
It should be driven by all stakeholders (dev, tester, pm, tl) in Project.
Advantages of Vertical andHorizontal E2E Tests
Advantages of Horizontal E2E Tests:
High business logic coverage,
Testing from the user perspective,
Prevents production incidents,
Gives the confidence that everything is OK
Advantages of Vertical E2E Tests:
High code coverage,
Fast execution,
More determined and focussed tests,
Useful for especially safety critical softwares
UI
Tests
Integration
Tests
Unit Tests
Test
Type
Low
Medium
High
Business Logic
Coverage
Code
Coverage
Low
Medium
High
Execution
Time/Costs
Short/
Low
Medium
Long / High
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Agenda
E2E Test Strategy
Which projects’ tests strategy should be designed as E2E?
Can all tests be written as E2E ?
Test Strategy
I Put That Sh t on Everthing*
Alternate Test Automation Approach
Continuous Testing
To develop software within shorter delivery cycles; Agile, DevOpsand Continuous Delivery approaches are on the forefront.
Continuous Delivery supports Continuous Testing which is a testing strategy that consists of a large number of automated unit and acceptance tests but a small number of automated end-to-
end tests.
Continuous Testing claims that E2E tests are inefficient and theyshould be limited.
It is a means of using Vertical E2E Test Automation effectively but avoiding Horizontal E2E Test Automation
Which Test Automation Strategyis the Best?
There isn’t one absolute strategy that is valid forevery SUT
Criteria / PrinciplesTo Form Test Strategy
Test Types / Methodologies that your staff is strong,
Coverage Strategy: Business Logic Coverage or CodeCoverage
Execution Time Priority
Cost
These Criteria should be considered with the dynamics and needsof Project, Staff, Company, Customer.
It is not an obligation to select & implement a strategystrictly.
You can mix the approaches and form your ownstrategy based on your needs.
It’s about finding the right balance.
Criteria / PrinciplesTo Form Test Strategy
Key Factors To Form & Implement a Successful Test Automation Strategy
Manage test releases effectively
Establish test measurement strategy & metrics
Establish test management strategy
Choose your test delivery model
Determine & implement test tools strategy
Assess your test & process maturity
Feasibility Analysis
Evaluation of Tools / Methodologies
Forming Automation Strategy (Plan, Design)
Implementation
Test & Maintain
Key Factors To Form & Implement a Successful Test Automation Strategy
Based on Delivery Strategy
Automated test executions should be planned andclear CI roadmap should be outlined based on
needs.
An ExampleAutomated Test Execution/CI Strategy
Based on Below Criteria
Delivery Velocity; should be fast so
smoke Unit test first, smoke UI test second,
full test sets last-after deploy in order.
Test Risk Level: Medium; smoke tests for all test types runsbefore deployment. But it is risky to run full sets after
deployment, since bug could stay undetected.
Dev TeamVersionControl
Build & UnitTests
UI Test Automation
(Smoke)Deploy (Prod)
UI Test Automation
(Smoke)
UI Test Automation
(Full Set)Monitoring
Dev CI Pre-Production Production
Check inTrigger
Trigger
Approval
Approval
Approval
Approval
Feedback
Feedback
Feedback
Feedback
FeedbackAutomated Test RunsAfter Nightly Build [1]• Unit Test
Automation• Web Service Test
Automation• UI Test Automation
(Full Set)• Cross Browser &
Platform Tests
Automated Test RunsAfter Weekly Build [2]• Automated Non
Functional Tests(Performance & Load Tests)
• Static Code QualityAnalysis
Feedback
Feedback
Feedback
Communication
Collaboration
Automation
Living Documentation
Drives whole project team
Healthy test basis / requirements
Replaces scripted test with automation scripts
To Strengthen the Automation Process:BDD – Behaviour Driven Development
Benefitsof E2E Test Automation
Velocity
Stability
Efficiency
Money
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Agenda
E2E Test AutomationExperiences
7 active projects’ tests that are being automated
as E2E
ROI
Project Name # of E2E Test
Scenarios
Manual Test
Execution Time
Automation
Execution Time
Ratio of Saving
Legalite 235 6 Days (48 hrs) 1,5 Hour 32 times
HOPE 85 2 Days (16 hrs) 1 Hour 16 times
PROTTON 1200 5 Days (40 hrs) 2 Hours 20 times
TTSis 2000 8 Days (64 hrs) 3 Hours 21 times
Planor 110 3 Days (24 hrs) 2 Hours 12 times
TTS 120 2 Days (16 hrs) 1 Hour 16 times
ATS 30 1 Day (8 hrs) 1 Hour 8 times
Challenges / Problems FacedIn the Process
TechnicalStaff Training and MotivationChoice of Tool and Methodology
PlanningDoing Feasibility Analysis Unproperly.Wrongly designed test automation strategy.
ManagerialDoing incrementally and showing the work & results.
Test Team (Agile) Organizational Structure
PM
PM
PM
PM
Team
Team
Team
Team
T L
Test Engineers TL: Test Lead
PM: Project Manager
Me!
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Agenda
Testing cost: testing accounts for roughly 26% of IT budgets
*According to a 2014 industrial survey of 1543 executives from 25
countries
Lack of testing is even more costly; the global cost of locating
and removing bugs from software has risen to $312 billion
annually and it makes up half of the development time of the
average project.
* A 2013 study by the Cambridge University
Testing Cost vs Lack of Testing Cost
Trying to find defects on System UnderTest using human intelligence andexperience.
Manual testing
Is Manual Test Still Needed?
Mostly: Yes!
Avoid Scripted Testing;
Automate it instead
Concantrate on Exploratory Testing;
Machines are great for running straigtforward
tests
Humans are great finding new defects
Support
your team
by moving testers
across teams
and a little
bit of fun
At the End of Bug Fest
Team Facility;Playing Tabu, Surprise Gifts,
Coffee, Free Time
• V. Garousi and M. V. Mäntylä, "When and what to automate in software testing? A multi-vocal literature review," Information and Software Technology, vol. 76, pp. 92-117, 2016.
• Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl
• V. Garousi and D. Pfahl, "When to Automate Software Testing? A Decision-Support Approach Based on Process Simulation," Wiley Journal of Software: Evolution and Process, vol. 28, pp. 272-285, 2016.
Effective Software Testing: 50 Ways to Improve Your Software Testing, Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA ©2002
End-to-End Integration Testing Design, Computer Software and Applications Conference, 2001. COMPSAC 2001. 25th Annual International
Distributed End-to-End Testing Management, Enterprise Distributed Object Computing Conference, 2001
HP Capgemini. Sogetti, World quality report 2014-2015. www.sogeti.com/solutions/testing/wqr/, Last accessed: Sept. 2015.[2] T. Britton, L. Jeng, G. Carver, P. Cheak, and T. Katzenellenbogen, “Reversibledebugging software,” University of Cambridge, Judge Business School, TehnicalReport, 2013
How Google Tests Software, James A. Whittaker
Behavior Driven Development Webapps mit Groovy Spock und Geb
http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/
http://techarcis.com/services/end-to-end-test-strategy/
References