Don't put your eggs in one basket - Lessons learnt from automation testing
-
Upload
brynley-scully -
Category
Technology
-
view
192 -
download
1
Transcript of Don't put your eggs in one basket - Lessons learnt from automation testing
![Page 1: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/1.jpg)
Agile Testing, Test Automation & BDD, Singapore www.testingmind.com October 25, 2016
Don’t put your
Lessons learnt from automation testing
EGGS in onebasket
Brynley ScullyHead of Testing, South East AsiaHewlett Packard Enterprise
![Page 2: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/2.jpg)
About the Speaker
2
Brynley Scully
Head of Testing, South East Asia
Hewlett Packard Enterprise
https://sg.linkedin.com/in/brynleyscully
@BrynleyScully
Project Testing Delivery
Testing Pre Sales
Testing Design & Review
Testing Solutioning
Project Remediation
Testing Governance
Test Management
Automation Test Management
Performance Test Management
Financial Information
Banking & Financial Services
Government
![Page 3: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/3.jpg)
Agenda
Automation Testing Today
Five Lessons Learnt
Summary
3
![Page 4: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/4.jpg)
Automation Testing Today
4
![Page 5: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/5.jpg)
Automation Testing Today
Valuation of Global Automation Testing Market
5Source : Transparency Market Research
$ 12.9 B
Year 2015
$ 85.8 B
Year 2024
Asia Pacific will have largest growth rate
![Page 6: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/6.jpg)
Automation Testing Today
What skills do you need to be a good tester ?
6Source : 2016 State of Testing Report. Testing survey of more than 1000 participants in more than 60 countries.
of survey respondents feel that it is very important to have functional automation & scripting skills.61%
![Page 7: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/7.jpg)
Automation Testing Today
Does your organization have automation ?
7Source : 2016 State of Testing Report. Testing survey of more than 1000 participants in more than 60 countries.
![Page 8: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/8.jpg)
Automation Testing Today
Automation is key when transitioning to DevOps
8
The biggest change in testing as a result of the transition to DevOps, is Automation, as it is the core of the successful DevOps cycle.
VP of Applications
Logistics Provider, Italy.
“
”
![Page 9: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/9.jpg)
Automation Testing Today
Yet we have many challenges in automation testing:
Pitching the case for automation testing
What to automate
How much to automate
Re-use of automated test scripts
Maintaining automation test cases
9
![Page 10: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/10.jpg)
Lesson #1:Don’t over-promise
10
![Page 11: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/11.jpg)
Lesson #1Don’t over promise
11
Increased market share
Faster
Zero defects
Cheaper
100% automation
Fewer testers
Higher revenue
Automation testing promise basket
Magic
Eradicate world poverty
![Page 12: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/12.jpg)
Lesson #1Don’t over promise
Why have Automation Testing in your projects ?
Understand the business objectives
Understand the technical landscape
12
We have business pressure to reduce our implementation cycles, for our retail banking app, from once every 2 months to once every 3 weeks.
Test Manager,
Banking, Thailand
“
”
![Page 13: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/13.jpg)
Lesson #1Don’t over promise
Challenges faced in applying testing in agile development
13
29%
43%
0% 10% 20% 30% 40% 50% 60%
Lack of a good testing approach that fits withagile development method
Source : World Quality Report 2016-2017.
2016
2015
![Page 14: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/14.jpg)
Lesson #1Don’t over promise
Build a realistic Automation Testing Approach/Strategy:
Scope - new automation test cases to build
Scope - existing automation test cases to maintain
Composition - manual vs automated test cases
Groupings of automated test cases
Frequency of automation test execution
Testing metrics to collect and publish
14
![Page 15: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/15.jpg)
Lesson #1Don’t over promise
Be realistic in Automation Testing ROI
15
Effort
/ C
ost
Current – Manual Testing
Desired – Automation Testing
time
![Page 16: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/16.jpg)
Lesson #1Don’t over promise
Be realistic in Automation Testing ROI
16
Effort
/ C
ost
Current – Manual Testing
Desired – Automation Testing
time
![Page 17: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/17.jpg)
Lesson #1Don’t over promise
Be realistic in Automation Testing ROI
17
Effort
/ C
ost
Current – Manual Testing
Desired – Automation Testing
time
When will this happen ? How
much savings ?
How long will this take ?
![Page 18: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/18.jpg)
Lesson #1Don’t over promise
Don’t put your eggs in one basket.
Don’t over-promise in automation testing, else you will have egg-static customers with unrealistic expectations.
18
![Page 19: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/19.jpg)
Lesson #2:Don’t concentrate on automating UI test cases only
19
![Page 20: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/20.jpg)
Lesson #2Don’t concentrate on automating UI test cases only
20
UI test case
UI test case
UI test case
UI test case
UI test case
UI test case
Automation test case basket
![Page 21: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/21.jpg)
Lesson #2Don’t concentrate on automating UI test cases only
Challenges faced in applying testing in agile development
21
23%
41%
0% 10% 20% 30% 40% 50% 60%
Inability to apply test automation at differentlevels
2016
2015
Source : World Quality Report 2016-2017.
![Page 22: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/22.jpg)
Lesson #2Don’t concentrate on automating UI test cases only
Mike Cohen
Automation Test Pyramid
22
Some
Lots
Fewer
API test cases
Unit tests
UI test cases
![Page 23: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/23.jpg)
Lesson #2Don’t concentrate on automating UI test cases only
Mike Cohen
Automation Test Pyramid
Avoid the:
Inverted pyramid
Or
Ice Cream cone
23
Some
Lots
Fewer
API test cases
Unit tests
UI test cases
Some
Fewer
Lots
API test cases
Unit tests
UI test cases
![Page 24: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/24.jpg)
Lesson #2Don’t concentrate on automating UI test cases only
Don’t put your eggs in one basket.
Egg-span automation testing beyond just UI test cases.
24
![Page 25: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/25.jpg)
Lesson #3:Don’t forget about automation frameworks
25
![Page 26: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/26.jpg)
Lesson #3Don’t forget about automation frameworks
26
Automation test case
Automation test case
Automation test case
Automation test case
Automation test case
Automation test case
Automation test framework
![Page 27: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/27.jpg)
Lesson #3Don’t forget about automation frameworks
27
Automation test case
Automation test case
Automation test case
Automation test case
Automation test case
Automation test case
Missing Automation test framework
![Page 28: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/28.jpg)
Lesson #3Don’t forget about automation frameworks
28
Why:
Re-useable scripts
Easier script maintenance
What:
Data parameterization
Data separate from code
Log test results
Error Handling
Auto recovery/retry
![Page 29: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/29.jpg)
Lesson #3Don’t forget about automation frameworks
29
Types of automation test frameworks:
Modular
Data Driven
Keyword Driven
Hybrid
![Page 30: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/30.jpg)
Lesson #3Don’t forget about automation frameworks
weak
Don’t put your eggs in one ^ basket.
Egg-norring a good automation test framework will weaken your automation test re-use.
30
![Page 31: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/31.jpg)
Lesson #4:Don’t just automate test cases only
31
![Page 32: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/32.jpg)
Lesson #4Don’t just automate test cases only
32
Automation test case
Automation test case
Automation test case
Automation test case
Automation test case
Automation test case
Automation test basket
![Page 33: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/33.jpg)
Lesson #4Don’t just automate test cases only
Automation testing is not just
the automation of test cases only
33
![Page 34: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/34.jpg)
Lesson #4Don’t just automate test cases only
What else can we automate ?
34
Automate test reporting
Automate test data creation
Automate test environment provisioning
![Page 35: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/35.jpg)
Lesson #4Don’t just automate test cases only
Emerging automation activities
35
Robotics Automation
Test Design Automation
Cognitive Automation
Machine learning
Predictive analysis
![Page 36: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/36.jpg)
Lesson #4Don’t just automate test cases only
Don’t put your eggs in one basket.
Uncover egg-stra opportunities for productivity through automation.
36
![Page 37: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/37.jpg)
Lesson #5:Don’t lose faith in automation testing
37
![Page 38: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/38.jpg)
Lesson #5Don’t lose faith in automation testing
38
Not many defects found
False sense of security
Many test execution failures
Not reliable
Takes time and effort to maintain
Automation Testing
![Page 39: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/39.jpg)
Lesson #5Don’t lose faith in automation testing
39
Apply testing techniques:
Equivalence partitioning
Boundary Value Analysis
State Transition
Regenerate test data
Build randomness in scenarios
Complement with manual testing
Complement with exploratory testing
Use drivers and stubs
![Page 40: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/40.jpg)
Lesson #5Don’t lose faith in automation testing
Don’t put your eggs in one basket.
Egg-sercise complementary test techniques, data regeneration, randomness in your test scenarios, to avoid losing faith in automation testing.
40
![Page 41: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/41.jpg)
Summary
41
![Page 42: Don't put your eggs in one basket - Lessons learnt from automation testing](https://reader031.fdocuments.in/reader031/viewer/2022022413/58ef43ee1a28ab755d8b4601/html5/thumbnails/42.jpg)
Summary
42
1 Don’t over-promise in automation testing,
else you will have egg-static customers
with unrealistic expectations.
2 Egg-span automation testing beyond just
UI test cases.
3 Egg-noring a good automation test
framework will weaken your automation
test re-use.
4 Uncover egg-stra opportunities for
productivity through automation.
5 Egg-sercise complementary test
techniques, data regeneration,
randomness in your test scenarios, to
avoid losing faith in automation testing.