Post on 09-Aug-2020
© 2018 by .
Thomas Hadorn
The Future of Testing: Continuous Enterprise Testing ANZTB Test Conference, Canberra, 1 June 2018
© 2018 by
Dev Ops
© 2018 by
Critical Physical Limit Technical Complexity
& Rate of Change
Years Months
Months Weeks
Weeks Days
Days Hours
Hours Minutes
Delivery Cycle Time
Perceived Disruption Software Quality
Challenge
Today
© 2018 by
Requirements Design Implementation Testing Acceptance Deployment
Inception Elaboration Construction Transition
Development Operations
DevOps
Agile Event Horizon
Digital Singularity
Incremental Rational Unified
Process
4
Agile Scrum, Kanban
2
Continuous DevOps
1
6+
Big Bang Waterfall
= wall of confusion
Flexible Framework
Rigid Rules
© 2018 by 5
Present
80%
Man
ual t
estin
g
20%
Aut
omat
ed te
stin
g
UI
Load
API
The reliance on manual testing is the
top technical challenge in app development
World Quality Report, 2015/16
Test automation requires developers
© 2018 by
Present Future
80%
20
%
Man
ual t
estin
g A
utom
ated
test
ing
Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT)
Optimize
!
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
80%
20
%
Man
ual t
estin
g A
utom
ated
test
ing
Progression testing
t
Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Optimize
!
Test the unknown. Session-Based Exploratory Testing
Explore
!
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
80%
20
%
Man
ual t
estin
g A
utom
ated
test
ing A
utom
ated
test
ing
+85%
Progression testing
t
Regression testing t
Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT)
Optimize
!
Test the unknown. Session-Based Exploratory Testing
Explore
!
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
automated UI tests
automated load tests UI and API
automated API tests
80%
20
%
+85%
Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Optimize
!
Test the unknown. Session-Based Exploratory Testing
Explore
!
Automating the automation framework for UI and API. Model-based Test Automation (MBTA) Automate
!
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
automated UI tests
automated load tests UI and API
automated API tests
Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Optimize
!
Test the unknown. Session-Based Exploratory Testing
Explore
!
Automating the automation framework for UI and API. Model-based Test Automation (MBTA) Automate
!80%
20
%
+85%
Increase the testing window. Test-Driven Service Virtualization
Provide matching test data in time. Stateful Test Data Management
Manage
!
Integrate
!Automate build to release process. Continuous Integration/Testing/Delivery (CI, CT, CD) Distributed Execution
+
Virtualize data and systems key enabler for high automation
© 2018 by
risk
cove
rage
60%
80%
100%
40%
20%
0%
𝑴/𝑵 =
0
defe
ct ra
te
max
100% test cases
20% 0% 40% 60% 80%
critical defects
M N 𝟏 weight 𝟏 𝟏 𝟏 𝟏 𝟏 𝟏 … …
[%]
© 2018 by
60%
80%
100%
40%
20%
0%
risk
cove
rage
100% 20% 0% 40% 60% 80%
test cases
critical defects 0
defe
ct ra
te
max
M N 𝒘↓𝟏 𝒘↓𝟐 𝒘↓𝟑 𝒘↓𝟒 𝒘↓𝒊 𝒘↓𝑵 … … … … 𝒘↓𝒊 ≥ 𝒘↓𝒊+𝟏 weight
© 2018 by
60%
80%
100%
40%
20%
0%
Intuitive Test Design
Methodical Test Design
100% 20% 0% 40% 60% 80% Test Cases Risk
Coverage
Achieve maximum risk coverage with a minimal number of powerful test cases.
© 2018 by
choose type of vehicle
display premium
choose insurance
cover
provide insurant details
provide vehicle details
Truck
Car
Bike
Private
Commercial
Purchase no
Purchase - yes
Get a Quote
Purchase yes
Auto generate… Orthogonal Arrays Linear Expansion
Pairwise Tests All Combinations Negative Tests Boundary Tests
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
automated UI tests
automated load tests UI and API
automated API tests
80%
20
%
+85%
Test the unknown. Session-Based Exploratory Testing
Explore
!
© 2018 by
Formal Testing Informal Testing
Exploratory Testing Manual Testing
Automated Testing
Ad-hoc Testing
Testing Continuum
© 2018 by
Risks Performance Issue
Usability Issue
Functional Issue
Stability Issue
Reliability Issue Scalability Issue
Coherence Issue
Understandability Issue
Testability Issue
Convenience Issue
Security Issue
Accessibility Issue
© 2018 by
Risks
I’m an automated test case.
Usability Issue
Stability Issue
Reliability Issue Scalability Issue
Coherence Issue
Understandability Issue
Testability Issue
Convenience Issue
Security Issue
Accessibility Issue
Performance Issue
Functional Issue
Automation is doing, what automation does.
© 2018 by
1 2 3 4 5 … … … … … … …
Risks
© 2018 by
I’m an automated test case. I’m the same test case executed manually.
© 2018 by
Plan as you test!
Story-Based
Motivating
Credible
Exploratory Branching The art of deciding on what to test next.
New Testing
Idea
Test Cases vs. Scenarios Pre-specified inputs vs. hypothetical situation
© 2018 by
Specification Based Testing
Exploratory Testing
Analyze Potential Risks Problem vs. No Problem
Monitor Known Risks Pass Result vs. Fail Result
*That’s an illustration. Don’t confuse it with reality.
© 2018 by
Exploratory Testing « Testing as performance »
Formal Testing « Testing as artifact creation »
Monitor Known Risks Confirm what you already know (measurable things)
Intelligent Testing Create new test ideas based on what you have learned
Mechanical Testing Process pre-defined data in pre-designed steps
Analyze Potential Risks Focus on the things you don’t know
Low Information Value Repeat what you have already learned
High Information Value Learn something new
High Risk Coverage Easy to scale because it’s parallelizable
Low Risk Coverage Hard to scale continuously because it relies on humans
Change Detector Problem Detector
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
automated UI tests
automated load tests UI and API
automated API tests
80%
20
%
+85%
Automating the automation framework for UI and API. Model-based Test Automation (MBTA) Automate
!
© 2018 by
100%
0%
Sprint Start
Sprint End
API
UI
Start testing early, shift left test automation Degree of
Completion
Time Efficiency
Gain
+4x Creation
+6x Maintenance
+20x Execution
© 2018 by
1
n
Test Automation Framework
(TAF)
Record Replay
Test Data Keywords …
Manually, coded framework.
Scr
ipt-B
ased
(Cod
ed T
AF)
B T
Automating the automation framework.
Mod
el-B
ased
(aut
omat
ed T
AF)
B T Automation
Model
Model-Based Test Automation
(MBTA)
Scr
ipt-B
ased
B Business Information T
Technical Steering
Information
B and/or T change requires re-recording.
Test Coverage Design (MBT)
© 2018 by
Customer Portals
Websites & CMS
Systems of Engagement
Web Server, API Gateway
Social
Mainframe
COTS
Legacy
Systems of Record
AppSrv/ESB
BI / Analytics
Open-Source
Model-Based
© 2018 by
Typical Enterprise QA Staff
Programmer Scripter Non Technical SME
Effective Velocity
Script-Based Open Source
Script-Based
Commercial
Model-Based
Typical Startup
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
automated UI tests
automated load tests UI and API
automated API tests
80%
20
%
+85%
Automating the automation framework for UI and API. Model-based Test Automation (MBTA) Automate
!
© 2018 by
Browser-Level Users
Cloud Age Before Cloud
2011
60% web pages < 50 requests/page
80% web pages < 200KB JS/page
Release cycle > weeks
Memory Footprint
Browser-Level
Testing Complexity
2018
60% web pages > 100 requests/page
80% web pages > 200KB JS/page
Release cycle < days, minutes, sec
© 2018 by
15 GEOGRAPHIC
REGIONS
+36 +14& hybrid
Cloud economy of scale
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
automated UI tests
automated load tests UI and API
automated API tests
80%
20
%
+85%
Increase the testing window. Test-Driven Service Virtualization
Provide matching test data in time. Stateful Test Data Management
Manage
!
© 2018 by
*voke, market snapshot report on service virtualization - 2012
On average, organizations require access to 33 systems for development or testing.
18 Average # of systems with
unrestricted access
96 % of testers have
restricted test lab access
Testing Present
Enterprise system landscapes are alike disease gene networks.
Welcome to the tester’s hell.
© 2018 by
Insurance App
CRM CAR CRM RATING ORDER
Business Logic API (e.g. Web-Services)
API (e.g. Web-Services)
Instable or not available applications
Test Data search & preparation
(expensive) 3rd party or Cloud based services
Decoupled, outsourced Development & coordinated
Deployment, multiple Vendors
© 2018 by
Insurance App
CRM CAR CRM RATING ORDER
Business Logic API (e.g. Web-Services)
API (e.g. Web-Services)
Virtualising systems replicates
system complexity
Virtualisation
Virtualising components creates
inconsistencies
Virtualisation is not
a fallback plan
© 2018 by
Insurance App
CRM CAR CRM RATING ORDER
Business Logic API (e.g. Web-Services)
API (e.g. Web-Services)
Orchestrated Service Virtualisation
© 2018 by
Insurance App
CRM CAR CRM RATING ORDER
Business Logic API (e.g. Web-Services)
API (e.g. Web-Services)
Test-Driven (Orchestrated) Service Virtualisation
Virtualising for automated tests eliminates
system complexity
Virtualising for automated tests solves
inconsistencies
Model-based Test Automation
Virtualisation is not a fallback plan,
it’s the first line of defence
© 2018 by
80’ and 90’ Millennia 10’…
Record/Replay Record/Replay Record/Replay
Keyword + Data Driven Frameworks
Model-Based
Automation (MBTA)
Test-Driven Service Virtualisation
Model-Based Test Design (MBT)
Exploratory Testing
Script-Based Model-Based 20’…
Machine Learning
10%
30%
80%
R I
S K
C
O V
E R
A G
E
Framework + Traditional
Commercial Automation Tool
Cucumber
+ Selenium
Systems of Record
Systems of Engagement
© 2018 by
Present Future
remaining manual tests manual spec-driven and exploratory
automated UI tests
automated load tests UI and API
automated API tests
Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Optimize
!
Test the unknown. Session-Based Exploratory Testing
Explore
!
Automating the automation framework for UI and API. Model-based Test Automation (MBTA) Automate
!80%
20
%
+85%
Increase the testing window. Test-Driven Service Virtualization
Provide matching test data in time. Stateful Test Data Management
Manage
!
Integrate
!Automate build to release process. Continuous Integration/Testing/Delivery (CI, CT, CD) Distributed Execution
+
Virtualize data and systems key enabler for high automation
© 2018 by
Jan 2011
Aug 2014
Dec 2015 Nov 2017
“It should be considered by enterprises that have struggled to make test automation work, and by those seeking to support agile continuous automation practices.”
© 2018 by 41 41
Model-Based Test
Automation
Risk-Based Test
Management
Browser-Based Load
Testing
Test-Driven Service
Virtualization
Model-Based Test Coverage
Design
Session-Based Exploratory
Testing
End-to-End Functional & Load Testing
Optimize Automate Explore
Optimize
10X Compared to other testing tools.
30-40% Cost Reduction
2X Risk Coverage
2-3X Release Speed
The Tricentis Tosca X-Factor Continuous Testing Platform
© 2018 by 42
APAC fast-growing customer footprint
42
© 2018 by 43
Focus Area M1 M2 M3 M4 M5
Most enterprise customers are on level M1 when they decide for Tosca.
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization Manage
!
Integrate
!
Optimize
!
Automate
CTMM Continuous Testing Maturity Model
© 2018 by
?Thomas Hadorn
Managing Director APAC at Tricentis +61 424 282 757
t.hadorn@tricentis.com