Just-In-Time Testing Workshop - Let's...
Transcript of Just-In-Time Testing Workshop - Let's...
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 1
Just-In-Time TestingWorkshop
Robert Sabourin
AmiBug.Com, Inc.
Montreal, [email protected]
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 2
Overview
WelcomeSome PhilosophyBe Pre[paredTesting IdeasTest Triage
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 3
Just In Time Testing
• Robert Sabourin , Software Evangelist
• President
• AmiBug.Com Inc.
• Montreal, Quebec, Canada
• www.amibugshare.com
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 4
Just-In-Time Testing
Welcome
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 5
Just-In-Time Testing
• Pain points?– What hurts?
– How Much?
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 6
Just-In-Time Testing
Just In Time For ________?
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 7
Just-In-Time Testing
Some Philosophy
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 8
Fundamental Question
• How do you know when you are finished?
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 9
Crosby on Quality
• “Quality is defined as conformance to requirements”
• “Quality is not a measure of GOODNESS”– Phil B. Crosby, Quality is Free
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 10
“Quality is fitness for use”
Joseph Juran
Quality Control Handbook
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 11
Gerald M. Weinberg
“Quality is value to some person”
Exploring Requirements Quality Before Design
Dorset House
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 12
Edsger W. Dijkstra
• “Program testing can be used to show the presence of bugs, but never to show their absence”
AmiBug.Com, Inc.
Quiz
• Application screensare selected with three controls:(a) has 5 options
(b) has 6 options
(c) has 2 options
• How many screens can a user choose?
May 7, 2012 © Robert Sabourin, 2012 Slide 13
AmiBug.Com, Inc.
Quiz
• Total Combinations = 6 x 5 x 2 = 60
• To exercise each combination once a total of 60 tests would be required.
May 7, 2012 © Robert Sabourin, 2012 Slide 14
AmiBug.Com, Inc.
• How many tests would be required to exercise all possible screens in every possible order?.
May 7, 2012 © Robert Sabourin, 2012
Slide 15
Quiz
AmiBug.Com, Inc.
• To exercise all screens in every possible order would require
60! Test cases
60! = 60 x 59 x 58 x ... 3 x 2 x 1
60! ≈ 8.32 x 10**81
May 7, 2012 © Robert Sabourin, 2012
Pop Quiz
Slide 16
AmiBug.Com, Inc.
Pop Quiz
From 7.0 × 10**79
To 1.5 × 10**82
May 7, 2012 © Robert Sabourin, 2012
How many atoms are in the observable universe?
Slide 17
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 18
Control Flow Testing
• Model flow– Create control flow
diagram– Find basis paths
• Minimal set of transactions• Exercise at least once
– Each step– Each decision
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 19
Control Flow Testing
– Minimal basis paths
• N – number of nodes
• E – number of edges
• P – number of basis paths
• P = E – N + 2
• McCabe Cyclomatic Complexity
B
A
C D
F G
E
H
I
N=9E=10
P=10-9+2P=3
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 20
Control Flow Testing
– Finding basis paths:
1. Start with a typical baseline2. Flip first decision keep rest as
similar as possible3. Continue flipping decisions on
baseline4. After all decisions on baseline
have been flipped continue on next path
5. Stop when all paths have been exhausted
B
A
C D
F G
E
H
I
N=9E=10
P=10-9+2P=3
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 21
Control Flow Testing
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 22
AmiBug.Com, Inc.
ST
Di
M
R
N
Q
P
L A
B
C
D
E
FG
I
J
K
O
H
S
T
U
V W
X
ZY
ACAD
AEAF AA
AG AH AJ ABAMAL
AIAK
AN
AO APAQ
May 7, 2012 © Robert Sabourin, 2012 Slide 23
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 24
Control Flow Diagram to Identify Basis PathsBasis Paths
p01 Start Dispatch L M N P Q Rp02 Start Dispatch X Y AA AB Rp03 Start Dispatch AC AF AJ AI AK Rp04 Start Dispatch AD AE AF AJ AI AK Rp05 Start Dispatch O P Q Rp06 Start Dispatch K P Q Rp07 Start Dispatch I J C D H P Q Rp08 Start Dispatch A B C D H P Q Rp09 Start Dispatch L M N P S T Q Rp10 Start Dispatch L M N P S T U Rp11 Start Dispatch L M N P S T U V W Rp12 Start Dispatch X Z AA AB Rp13 Start Dispatch X Z AA AM Rp14 Start Dispatch X Z AA AM AN AP AQ Rp15 Start Dispatch X Z AA AM AN AO AQ Rp16 Start Dispatch AC AF AG Rp17 Start Dispatch AC AF AH Rp18 Start Dispatch AC AF AJ AL AM Rp19 Start Dispatch AC AF AJ AL Rp20 Start Dispatch I J C D E F Rp21 Start Dispatch I J C D E G Rp22 Start Dispatch A B R
Nodes 45Edges 65Complexity 22 E-N+2
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 25
Purpose of Testing
• Common definition:– To find bugs before our customers do!
• Broader definition:– The role of testing is to provide objective input to
facilitate business decisions!
– Keeps stakeholders aware of all issues or concerns that relate to shipping a product!
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 26
Bug Defined
• To make our job more fun, whenever we have a concern with software, we call it a “bug”.
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 27
• It’s all about people! (and the occasional bug too)
Just-In-Time Testing
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 28
Just-In-Time Testing
Context Drivers
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 29
Context Drivers - BTO
• Business– Value– To whom?– Why?
• Technology– Solutions
• Organization– Corporate Structure– Team Structure– Roles and Responsibilities
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 30
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 31
Context Listeners
• Find Sources
• Monitor Drivers
• Anticipate Change
• React
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 32
Just In Time Testing
Get Ready, Get Set,Cause here it comes
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 33
Turbulence
(.wav)
Just-In-Time Testing
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 34
Unprepared
Just-In-Time Testing
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 35
Just-In-Time Testing
Sharpen Testing Skills
• Thinker
• Detective
• Reporter
• Diplomat
• Negotiator
• Cheer Leader
• Pragmatist
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 36
Philosophy
We have precious little time to run tests!
We must always be prepared!
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 37
Time
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 38
Just In Time Testing
Test Triage
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 39
"No! Try not, Do. Or do not.
There is no try."
YodaP
lan
to s
uppo
rt c
hang
e
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 40
Testing Ideas
• Collect all testing ideas you can find!– List
– Sort
– Organize
– Shuffle
Pla
n to
sup
port
cha
nge
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 41
Testing Ideas
• How to find them?– Does system do what it is suppose to do?
– Does the system do things it is not supposed to?
– How can the system break?
– How does the system react to it’s environment?
– What characteristics must the system have?
– Why have similar systems failed?
– How have previous projects failed?
Pla
n to
sup
port
cha
nge
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 42
Testing Ideas
• Collect testing ideas
• From testing ideas build a series of testing objectives– Each can be assigned as
work to testers
– Each can include all, part of, or multiple testing ideas
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 43
Testing Ideas
• I often use Index Cards– Unique id
– One testing idea per card
– Colour indicates source
– Shuffled and reviewed
– Organized and reorganized
– Sorted, grouped, prioritized and collected
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 44
Test Idea Sources• Capabilities• Failure Modes• Quality Factors• Usage Scenarios• Creative Ideas• States• Data• Environments• White Box• Taxonomies
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 45
Testing Ideas
• Investigative approaches– We become truffle
snorting pigs and try to find useful information in all evidence we discover
– We can even get good ideas from out of date sources
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 46
Testing Ideas• Capabilities
– Use cases– Functional
requirements– Documented
requirements– Implicit requirements
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 47
Testing Ideas
• Failure Modes– What can break?– Reaction to invalid input?– How does software
behave in constrained environment?
• Memory• Disk Space• Network Bandwidth• CPU capacity• Shared resources
– Stress, Load, Volume
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 48
Ada
ptab
ility
Acc
essi
bilit
y
Aud
itabi
lity
Ava
ilabi
lity
Con
tinui
ty
Dep
enda
bilit
y
Exp
anda
bilit
y
Fun
ctio
nalit
y
Inte
grity
Inte
rope
rabi
lity
Mai
ntai
nabi
lity
Ope
rabi
lity
Por
tabi
lity
Rel
iabi
lity
Re-
usab
ility
Sca
labi
lity
Sec
urity
Ser
vice
abili
ty
Test
abili
ty
Usa
bilit
y
Application service providerAutomatic content generatorCustomized accessDatabase accessDeliveryDocument accessFile sharingInformationalInteractiveTransaction orientedUser-provided contentWorkflow oriented
High FocusMedium Focus
Low Focus
Quality Factors ImportanceDifferent Application Types
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 49
Testing Ideas
• Usage Scenarios– Identify classes of
users
– Identify how users will use system
– Describe scenarios
– Use Story board or similar approaches
– Identify variations
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 50
Testing Ideas
• Creative approaches – Action verbs
– Mind Maps
– Soap Operas
– Lateral Thinking
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 51
State Models
power up
idleinserting
coins
userchoose
makecoffee
serviceneeded
coin insertedreset button
coin return
right amountenteredcoin return
button pushed
no cupsOR no coffeeOR sensor jam
cup removed
Testing IdeasC
aptu
re t
estin
g id
eas
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 52
Testing Ideas• Data
– Flow– Structure– Create– Update– Change
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 53
Testing Ideas
• Environment– Hardware
– Software
– Operating systems
– Locales
– Browsers
– Plug-ins
– Co-dependent software
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 54
Testing Ideas
• White Box– Design
– Internal structure
– Code
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 55
Testing Ideas• Bug taxonomies
– Collections of possible bugs– Appendix A of Testing
Computer Software, Kaner, Falk, Nguyen
– Boris Biezer Taxonomy Otto Vinter manages
– Shopping cart taxonomy Giri Vijayaraghavan
Cap
ture
tes
ting
idea
s
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 56
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 57
Which test?
•Impact estimation–For each test idea guesstimate:
• benefit of implementation
• consequence of implementation
• benefit for not implementing
• consequence of not implementing
–How credible is the information?
Tria
ge te
stin
g id
eas
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 58
Which test?
Test Idea Rejection – What If?–If the cost/benefit does not make business sense then consider implementing:• part of the test, could that lead to part of the
benefit at a more reasonable cost?
• more than the stated test, would that generate more benefit?
• a different test than the stated idea, could that generate more benefit for less cost?
Tria
ge te
stin
g id
eas
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 59
Test Triage
• Test Triage Meeting– Review Context
• Business
• Technical
• Organizational
– New Information• Test results
• Bug results
• New testing ideas
Tria
ge te
stin
g id
eas
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 60
Test Triage
• Allocate Testing Assignments to Testers– Make sure testers know context
– Best thing to test
– Best person to test it
– Best people to explore it
– Best lead
– Are subject matter experts required
Tria
ge te
stin
g id
eas
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 61
Test TriageLife of a test idea
a. Comes into existence
b. Clarified
c. Prioritizeda. Test Now (before further testing)
b. Test before shipping
c. Nice to have
d. May be of interest in some future release
e. Not of interest in current form
f. Will never be of interest
d. Integrate into a testing objective
Tria
ge te
stin
g id
eas
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 62
Which test is next?
• Questions– Given state of project, state of business, state of
technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?
– How much effort are we willing to spend continuing to test this project?
– Can we ship yet?
Tria
ge te
stin
g id
eas
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 63
Deciding what not to test?
• Time pressure
– Should we skip a test?
– If test failed could system still be of value to some stakeholder?
– If test was skipped could important bugs have been otherwise found?
Tria
ge te
stin
g id
eas
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 64
Guidelines and Decisions
• To each stakeholder– risk of failure
– consequence of failure
– value of success
– how much certainty do we have
– is it a wild guess or an absolute truth?
Get
Sta
rted
Rig
ht
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 65
Bottom Line
• My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it!
Get
Sta
rted
Rig
ht • Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?
AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 66
Thank You
• Questions?