Post on 31-Oct-2014
description
7 Deadly Sins of Agile
Software Test Automation
Craig Smith
Welcome… Image: © Gracie Films / 20th Century Fox Television http://www.paleodietrecipes.org.uk/paleo-images/content/homer-confession.jpg
Adrian Smith
@adrianlsmith
Geeks Hate Repetition
It Started
With…
Image: © New Line Cinema http://www.imfdb.org/images/5/5e/SevenCover.jpg
© United Plankton Pictures https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-ash4/p480x480/429897_565831523456105_395040135_n.jpg
Discussion:
What is
stopping you
from being
great at
automation?
Image: http://toworkandback.com/wp-content/uploads/2012/08/stop-sign.jpg
Image: http://scitascienda.files.wordpress.com/2013/04/barney-stinson-quote-when-i-get-sad-i-stop-being-sad-and-be-awesome-instead-true-story.jpg?w=500&h=353
Image: http://www.cupojoy.com/images/image/Quality%20Inn%20Exterior.jpg
Building Quality In
AUTOMATED TESTING Automated
Testing
Image: http://www.lastwordonnothing.com/wp-content/uploads/2011/12/iStock_000007001634Small.jpg
Envy
Flawed comparison of manual testing & automation Image: http://workingtropes.lmc.gatech.edu/wiki/images/Man-vs-machine.jpg
How Management
See Testing
Image: http://www.craigsfire.com/wp-content/uploads/2010/02/pool.jpg
How Management
Would Like To See
Testing Image: http://us.123rf.com/400wm/400/400/dotshock/dotshock1012/dotshock101200006/8437158-internet-network-server-room-with-computers-racks-and-digital-receiver-for-digital-tv.jpg
Repeat After Me…
Agile
≠
Automated Testing
Software is a Series of Loops Image: http://2.bp.blogspot.com/_4G2ptSlZZn0/R9nRjimkLII/AAAAAAAAAqA/tzYJjRyhLkk/s400/brekky05011049.jpg
End to end automated tests and cover 10 conditions per
test, manual covers 1 condition per test
Automated tests cover all high priority business flows,
manual tests mostly cover lower priority flows
Metrics need context!
% Regression test cases automated
Automated Manual
Discussion:
How Much
Automated
Testing Is
Enough?
Manual vs Automated…
A Flawed Comparison Image: http://fvhs.com/wp-content/uploads/2012/11/Screen-shot-2012-11-15-at-7.49.37-AM.png
Testing is more than a series
of merely repeatable actions Image: http://cdn.themis-media.com/media/global/images/library/deriv/71/71335.jpg
Look
Left…
Look
Right…
Image: http://3.bp.blogspot.com/_RdF7S0TNAHY/TOqn8MLYajI/AAAAAAAAAJI/QDBFdixiq0E/s1600/2006-06-17%252520Falling-piano.gif
Look
Up!
21
Image: © Jerry Bruckheimer Television http://images2.fanpop.com/images/photos/4700000/CSI-Las-Vegas-9x18-Mascara-csi-4772098-2000-1333.jpg
CSI:
Software
Regression testing –
assessing current state
Automation of test support
activities
Ideal Automation Targets
Data generation / sub-setting
Load generation
Non-functional testing -
performance, security, ...)
Deterministic problems
Big data problems
Image: http://media.merchantcircle.com/30134145/mobile%20target_full.png
Common “Envy” Symptoms
Relying on automation as
the basis for all testing
activities
All tests are built by
developers
Absence of code reviews
Absence of exploratory
testing
Absence of user testing
Image: http://snappedshot.com/turbo/sites/default/files/styles/large/public/robot-doctor.JPG
Approach: Desired Role
Automation:
I work with developers to
automate tests that
provide business value
and identify system risks
Collaboration:
I work with analysts and SMEs to
ensure that testable acceptance
criteria are created for all stories
Strategy & Planning:
I am involved in the project at all
stages to ensure that testing
provides the greatest value and
quality objectives are achieved
Tools & Techniques:
I use both manual and
automated techniques using
the preferred testing tools
based on the situation
Architecture:
I have an understanding of the
system architecture and can create
tests that verify individual
components and the system as a
whole
Development:
I take an interest in
development practices
and monitor code quality
metrics
Estimating:
I am involved in developing
estimates for projects at a story
level, for tools and
infrastructure and also for
deployment/release activities
Reporting:
I provide metrics that give
insight into project health
and system quality
Agile:
I understand Agile project
delivery and the differences
between testing in the different
phases of an phase
Qualifications & Training:
I have recognised qualifications
in testing and continually
update and maintain my skills
Recruitment & Development:
I am attracted to the organisation
because testing is a cool career path
that offers heaps of opportunities
and a way of continually developing
my skills
Community & Teams:
I am part of a community
of testers that are
embedded within teams
but share common values
Approach: Report
on Confidence
“Envy” Lessons Learned
Avoid comparison between
manual and automated
testing - both are needed
Distinguish between the
automation and the process
that is being automated
Use automation to provide a
baseline
Use automation in
conjunction with manual
techniques
Image: http://idahoptv.org/dialogue4kids/images/season10/robots/bookbot.gif
•Over indulging on commercial test tools
Gluttony
Over indulging on commercial test tools… Image: © Universal Pictures http://static.ifood.tv/files/external-images/photo-26589.jpg
Discussion: What Automated
Tools Do You Use & Why?
Image: http://www.newnorfolkmitre10.com.au/pics/66/66718b.jpg
Some vendors
believe they
have the
“secret
sauce”
for
automation
Image: http://topnews.ae/images/HP-Sauce.jpg
Promise of Automation
Image: http://thebsreport.files.wordpress.com/2009/10/traveling-salesmen-medicine-show.jpg?w=500&h=364
Image: http://www.melhotornot.com/wp-content/uploads/Rowboat.jpg
Commercial Restricts Usage
Justifying The Expense
I have spent a lot of
money on this tool, we
better be using it for everything!
Image: © Warner Bros Television Distribution / Screen Gems http://3.bp.blogspot.com/-1L3LVOu0cZg/UPs4gju7DJI/AAAAAAAANx0/mrdx5RYSSW0/s1600/jetsons_L74.jpg
Underlying commercial tools
technology often not compatible
with development tool chain
Incompatible Technology
Special file formats or
databases
Lack of version control for tests
and/or cannot be versioned
within the software
Not easily combined with
Continuous Integration
Image: http://virtualmarketingofficer.com/wp-content/uploads/2009/06/square-peg.jpg
Not easily adapted or extended
by the developers
Common “Gluttony” Symptoms
A commercial tool forms
the basis of a testing
strategy
Only certain teams or
individuals can
access a tool or run tests
Developers have not been
consulted in the selection
of a testing tools
“We always use <insert tool-name> for testing!”
Image: http://resources2.news.com.au/images/2008/09/02/va1237328073953/burger-6230645.gif
Approach: Tools Guide
Approach: Craftsmanship
Image: © Wild Dancer Productions / Touchstone Television http://www.tvacres.com/images/props_tool_time_binford.jpg
“Gluttony” Lessons Learned
Favour open software tools
where ever possible
Use tools that can easily be
supported by the
development team and play
nicely with existing
development tool chain
Ensure any commercial
tools can be executed in a
command-line mode so it
can be automated
Image: https://fbcdn-sphotos-b-a.akamaihd.net/hphotos-ak-prn1/p480x480/9346_506859889386718_1005343687_n.jpg
Educate!
•User interface forms the basis for all testing
Lust
User interface forms the basis for all testing… Image: http://stuficionado.net/wp-content/uploads/2012/11/computer_hug.jpeg
Traditionally Test via the UI Image: http://stovallorganizing.com/wp-content/uploads/2013/04/Turtleing.jpg
Manual Exploratory
Collaboratively built around
system behaviour
Developer built optimised for fast
feedback
Co
nfi
de
nc
e
Sp
ee
d /
Fe
ed
ba
ck
Exercises components and
systems
Investment Profile
Understand The Architecture Image: http://harrison.kyschools.us/hcms/TechProjects2011.htm
Test Design
F Fast
I Isolated
R Repeatable
S Self Verifying
T Timely
FIRST Test Properties
Image: http://www.moto-choice.com/PRImages/EditorImages/148330-bathurst-finish-2.jpg
Common “Lust” Symptoms
Testers cannot draw the
application or system
architecture
Large proportion of tests
are run through the UI
Testers have limited
technical skills
No collaboration with
developers
Intent of tests is unclear
Image: http://snappedshot.com/turbo/sites/default/files/styles/large/public/robot-doctor.JPG
Discussion: What
technologies do you need to
understand to automate?
Image: © Paramount Pictures http://americanobjective.com/wp-content/uploads/2012/04/original.jpg
Agile Testing Command Line
Interface Continuous Integration
Version Control
Build Tools Capture Replay
(Selenium) HTML
Test Maintenance &
Data
Specification By Example / ATDD
/ BDD
Concordion / Cucumber
SQL Web Services
Approach: Test
Automation Course
Approach: Use
UI Tools for
Right Purpose
Use tools like Selenium sparingly, good for exploratory testing
Only use tools like HTMLUnit or scrape
screens when no other options available
“Lust” Lessons Learned
Limit the investment in
automated tests that are
executed through the user
interface
Collaborate with developers
Focus investment in
automation at lowest
possible level with clear test
intent
Ensure automation gives
fast feedback
Image: http://foodcandy.com/wp-content/uploads/2013/03/flowers-and-chocolate1.jpg
Pride Too proud to
collaborate
when creating
tests
Pride
Too proud to collaborate when creating tests… Image: http://latimesblogs.latimes.com/.a/6a00d8341c630a53ef0120a955baf0970b-600wi
Poor
Collaboration
= Poor Tests
Image: http://2.bp.blogspot.com/-RYkFdryP-Co/Tj7-0q3DyPI/AAAAAAAABJU/6tETrv8zyb4/s1600/OpsSilo.jpg
Poor
Collaboration
or
Dedicated
Roles
=
Duplication
Image: © Universal Pictures http://cdn3.whatculture.com/wp-content/uploads/2013/03/twins-poster.jpg
No
Definition
of
Quality
Image: https://lh3.googleusercontent.com/-46OEX5ffUX4/TtVNoEnFPmI/AAAAAAAADBo/g55cQRy5im0/s800/zimogs_kvalitates_zime.jpg
Discussion:
What is
something
you regard
as quality?
Image: http://farm7.staticflickr.com/6115/6287632596_5cc5a681c8_z.jpg
Analyst /
Customer
Developer Tester Automation
Elaboration /
Specification Acceptance
Criteria
High
Performing
AGILE
Project
Manager
Good
Collaboration
Developer
Tester
Analyst
More Technical
Less
Technical
Co
de
Des
ign
Un
it T
est
Au
tom
ated
Fu
nct
ion
al /
S
pec
ialis
t
Test
s
Man
ual
Te
st
Exp
lora
tory
Te
st
Req
uir
emen
ts
Cu
sto
mer
C
olla
b.
Developers need more testing involvement
Testers need more technical involvement
Analysts need more testing involvement
Skills
Red
Green Refactor
Red
Green Refactor
ATDD
ATDD
TDD
Image: http://concordion.org/image/example/online-shop/AcceptanceTest.png
Specification
by Example
Common “Pride” Symptoms
Automated tests are being
built in isolation from team
Intent of tests is unclear
or not matched to quality
Poor automation design
(abstraction,
encapsulation, ...)
Maintainability or
compatibility issues
Image: http://farm8.staticflickr.com/7120/7683912508_b23c4c9ff0_b.jpg
Approach:
Defining Quality Quality
Advocates
What does quality mean to the different roles in
the team?
Quality Taxonomy
What are quality attributes?
Quality Prioritisation
How do we know what quality attributes to
include?
Quality Tradeoff Risks
What are risks with the quality attributes we
are trading off?
Quality Measurement
How do we test and measure quality?
Success Sliders
How does quality relate to the sliders?
Quality Definition
What does quality mean?
Next Steps How do we apply quality to our work
1
2
3
4
5
6
7
8
Approach:
Disband Testing
Teams Image: http://prince2pm.files.wordpress.com/2012/01/industry-groups-product-groups.jpg
Approach:
Simple
Strategy
“Pride” Lessons Learned
Collaborate to create good
tests and avoid duplication
Limit the investment in UI
based automated tests
Collaborate with developers
to ensure good technical
practices (encapsulation,
abstraction, reuse, ... )
Test code = Production
code
Image: https://c479107.ssl.cf2.rackcdn.com/files/7178/article/width668/rfwyhksh-1327469166-1327469251.jpg
Sloth
Too lazy to properly maintain automated tests… Image: http://photos.pcpro.co.uk/blogs/wp-content/uploads/2011/06/Asleep-at-computer.jpg
New Feature
System Interface
Change
OS Patch
Reference Data
Changes
Time
Many Causes of
Automated Test Failure
Time
Co
st
/ E
ffo
rt Potential
Value of Maintained
Automated Test Suite
Value of Unmaintained
Automated Test Suite
Manual test execution
Maintained automation
Unmaintained automation
Importance of
Maintenance
Discussion:
How often
are you
red?
How do you
know?
Who cares?
Image: http://www.eriding.net/media/photos/environment/traffic/070328_jbean_mp_environment_traffic_0135.jpg
Continuous Integration
Watch Code
Build / Compile
Run Tests /
Analysis
Publish Results
Stop The
Line!
Failure?
Continuous Delivery
Image: http://www.thoughtworks.com/imgs/drawing-cd-header.png
#notesting Image: http://3.bp.blogspot.com/-jy4Sc0m-oTA/UU9qrGJw56I/AAAAAAAAAC8/zSlPJS9Ed7c/s200/No+testing.JPG
Common “Sloth” Symptoms
Test suite has not been recently
run - state is unknown
Continuous Integration history
shows consistent failures
following
development changes / release
Test suite requires manual
intervention
Duplication within automation
code
Image: http://commons.wikimedia.org/wiki/File:Cairns_Australia_Beer_Run.jpg
Small changes trigger a
cascade of failures
Approach:
Visible
Status /
Stop The
Line
“Sloth” Lessons Learned
Ensure automated tests are
executed using a Continuous
Integration environment
Ensure tests are always
runnable - even if the system in
not being actively developed
Make test results visible -
create transparency of system
health
Ensure collaboration between
developers and testers
Image: http://1.bp.blogspot.com/_yEAvQXpDW6I/S-8HgogJWpI/AAAAAAAAAY0/phpZuCfP610/s400/Linea%2520Berocca.jpg
Rage
Frustration with slow, brittle or unreliable tests… Image: © 20th Century Fox http://kei.dizkartes.nl/static/img/_common/header_404.jpg
Fast
Feedback Image: http://www.digitallanding.com/wp-content/uploads/2012/07/Faster_Internet-e1346361972417.jpg
Discussion:
What is a
good…
Time
benchmark
for a build?
Amount of
coverage? Image: http://www.mdr.de/sachsen/bild235170_v-standardBig_zc-3ad1f7a1.jpg?version=49443
Image: http://1.bp.blogspot.com/-pKB3xu6zOc8/TWU1W_4OEqI/AAAAAAAAAzk/ygEOBNumv-Q/s400/nespresso-clooney.jpg
No Waiting…
100%*
Image: http://shirtoid.com/wp-content/uploads/2013/02/100-percent-sci-fi1.jpg
Common “Rage” Symptoms
Slow Automated Tests
Brittle
Unreliable
Large datasets
Unnecessary integrations
Inadequate environments
Too many / UI / manual tests
Time bound data
External / Prod integrations
Reliance on UI / sequence
False positives / confidence
Failures being ignored
Workarounds investigated
HEALTH
PROJECT
DEVELOPMENT
TESTING
USER
0
1
2
3
4
5
0
1
2
3
4
5
6
7
1 2 3 4 5 6 7 8
New
Ris
ks &
Issues R
ais
ed
Num
ber
of
Ris
ks &
Issues
Iteration
TEST COVERAGE
0
5
10
15
20
25
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8
Num
ber
of
Tests
Num
ber
of
Defe
cts
Iteration
MAINTAINABILITY PERFORMANCE 0
200
400
600
800
1000
1200
1400
1600
1800
4700
4800
4900
5000
5100
5200
5300
5400
5500
5600
5700
1 2 3 4 5 6 7 8
Lin
es o
f T
est C
ode
Lin
es o
f C
ode
Iteration
OVERALL
$-
$5
$10
$15
$20
$25
$30
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8
Busin
ess V
alu
e
Num
ber
of
Featu
res
Iteration
Approach: BVC
Approach:
Cloud /
Central /
Open
Source ALM
“Rage” Lessons Learned
Treat automated tests with
the same importance as
production code
Review, refactor, improve ...
Apply a “Stop the line”
approach to test failure
Eliminate (quarantine)
unreliable tests
Image: http://us.123rf.com/400wm/400/400/edma/edma0907/edma090700217/5226661-businessman-doing-yoga-exercises-in-a-city-park.jpg
Ensure collaboration with
developers
Greed
Trying to cut costs through automation… Image: © 20th Century Fox http://blogs.telegraph.co.uk/news/files/2010/09/gordon-gekko.jpg
Lure of saving labour Image: http://www.caradvice.com.au/thumb/640/556/wp-content/uploads/2010/11/Car-salesman.jpg
Automation is not cheap!
Image: http://designbuildsource.com.au/wp-content/uploads/2012/03/australian-money.jpg
Common “Greed” Symptoms
Investment in commercial
tools using a business-case
based on reducing headcount
Using a predicted ROI as a
way of reducing budget for
testing
Consolidating automated
testing within a special group
Image: © Gracie Films / 20th Century Fox Television http://thegreeneconomy.com/wp-content/uploads/2012/05/greedy-246x220.gif
New Software
Test Engineer
Role
New Graduates,
Recruits, Consultants
& Partners
BA
SME
Other
Up
Skill
As
se
ss
me
nt
Current Test Analysts
Capable + Current Skills +
Desire
Capable + Desire
Not Capable + No Desire
Not Capable + Desire
**Assessment = Technical Test + Interview**
Approach:
Assist & Assess
Approach:
Outsource
Teams Not
Testing
Image: https://sphotos-b-ord.xx.fbcdn.net/hphotos-ash3/p480x480/942876_505183632888249_1411782978_n.jpg
Approach: Avoid Test
Automation Projects
Image: http://www.hifi-writer.com/he/progscan/Files/abcestablish.jpg
“Greed” Lessons Learned
Ensure the reasons for
automation are clear
and are NOT based
purely on saving
money/headcount
Ensure business case
for automation
includes costs for
ongoing maintenance
Image: http://resources3.news.com.au/images/2010/12/02/1225964/296651-piggy-bank-and-coins-chart.jpg
Quality
is Key
Image: http://gs-press.com.au/images/news_articles/cache/key_hand_resized-600x0.jpg
Approach:
Quality
Assessment
Tests are NOT executed on a regular basis
State of current defects is unknown
Functional testing occurs regularly
All testers can execute the functional tests
Production verification testing is used to ensure success
A majority of functional tests have been automated
Exploratory testing forms part of test execution
Customers verify implemented features prior to deployment
A majority of non-functional tests (performance, reliability, ...) are
completed prior to deployment, many are automated
Testers pair with developers to automate tests
Customers verify implemented features as they are completed
Functional and non-functional testing occurs continuously within
development iterations
Developers and testers are performing test first practices
-1
1
2
3
4
Discussion: Test Execution
Wrap
Up
Image: http://www.pria.com.au/sb_cache/priablog/id/924/f/7%20deadly%20sins.jpg
Envy Flawed comparison of manual testing & automation
Gluttony Over indulging on commercial test tools
Lust User interface forms the basis for all testing
Pride Too proud to collaborate when creating tests
Sloth Too lazy to maintain automated tests
Rage Frustration with slow, brittle or unreliable tests
Greed Trying to cut costs through automation
7 Deadly Sins
Why Use Automation?
Image: © Fuzzy Door Productions / 20th Century Fox Television
Don’t Lose
Sight of the
Goal
How geeks really work How Geeks Can Work Together
Questions?
Craig Smith
http://www.craigsmith.id.au
craig@craigsmith.id.au
@smithcdau