Top 10 myths and illusions - SOCO Norge AS...2017/09/28 · company 1: started with 1:8 ratio after...

© Lloyd Roden Illusions120215

illusions-1

1

Dispelling Testing’s Top Ten Myths and Illusions

2

Contents

What are myths and illusions? The top ten myths and illusions

Taking action


illusions-2

What is a myth?

  true or false?   eating lots of carrots helps you see in the dark?   eating the crusts of bread makes your hair curl?   eating fish gives you brains?   you must drink 2 litres of water a day?   chocolate causes acne?   watching TV too much gives you square eyes?   stepping on a rusty nail causes tetanus   feed a cold, stave a fever

3

a popular belief or story (fiction or half-truth) that has become associated with a person, institution, or occurrence,

especially one considered to illustrate a cultural ideal. A made up story that seems believable.

half truth

half truth

false

false false

false

false half truth

What is an illusion?

4

An erroneous perception of reality. A misleading image presented as a visual stimulus. Something we see which is

not there, or want to believe is there.


illusions-3

5 5

Some more optical illusions…

6


illusions-4

7

Contents

What are myths and illusions ? The top ten myths and illusions

Taking action

#1: test estimation involves effort and resource

fact: estimation is based on 2 aspects

effort + people = schedule half truth for testing:

effort + people + quality = schedule we must also estimate the number and severity of the bugs we are going to find to make our estimates meaningful (a direct correlation)

WARNING: be careful of Brook’s law – 9 women cannot have a baby in 1 month! Some activities cannot be accelerated by adding more resource

8


illusions-5

9

Estimate, monitor and report quality

how to estimate quality   guess   use historical data (previous projects, defect density)   use models

what to monitor and report   actual versus expected   trend analysis and forecasting   correlation between graphs

0

10

20

30

40

50

1 2 3 4 5 6 7 8 9 10 0

50

100

150

200

250

1 2 3 4 5 6 7 8 9 10

Cycle #

Actual bugs

Masked rate

Observed bugs

Fix Rate

Fixed bugs

Insert rate

Inserted bugs

Remaining bugs

1 10 30% 7 80% 6 20% 1 5

2 5 30% 4 80% 3 20% 1 3

3 3 0% 3 100% 3 0% 0 0

4 0 full regression test

Count the black dots!

10


illusions-6

#2: large numbers are better than small numbers

  some large numbers   billion 1,000,000,000   trillion 1,000,000,000,000   quadrillion 1,000,000,000,000,000   quintillion 1,000,000,000,000,000,000

  large numbers tend to become meaningless without the context   £1,698,100,000,000

UK Debt - Office of National Statistics Jan 2017   managers want lots of information

  often unsure as to the purpose   often wanting large numbers

  but meaningless without the context 11

when your boss asks for all that

unnecessary information

the only difference between a rut and a grave is the

dimensions

Challenge the information that is requested

  we provide endless information for our managers   number of tests run   number of tests passed/failed   number of scripts automated   number of defects raised/fixed

  we provide this information on a regular basis...   but what is it used for?   who is analyzing the data?   is it really needed?

  is there a better way?   different information   educating managers regards test

12

…the larger the number the better it is


illusions-7

Is the ball closer to the top or bottom?

13

#3: authors know their subject

  signs that they don’t   plagiarism rather than uniqueness   generalization rather than specifics   vagueness rather than detail   avoidance rather than openness   nervousness rather than confidence

14

  authors of what?   requirement specifications   design specifications   code   test documentation   technical books   presentations

they might write about it…but do they know their subject?


illusions-8

If in doubt…challenge!

  study the body language   use walkthroughs rather

than other review types   ask questions

15

remember there is no such thing as a stupid question…only a stupid answer

anyone who produces something for the benefit of others has the right to be challenged

#4: test (development) managers will always be needed

16

1980s 1990s

2000s

prediction for 2020s

the need to change


illusions-9

Test (and development) managers need to change

  traditional Test Manager fulfills 2 main roles:   organizing and leading the test teams   planning, monitoring and controlling what gets tested

  these roles are often covered by:   the Scrum Master   the self-organizing Scrum teams

  how can they change?   become more strategic   adapt to different roles

  scrum masters   enhance their skills

  writing utilities and tools for more efficient testing   data collection, statistical analysis &process improvement   testing of the product!

17

#5: anyone can test

  facts:   anyone can run tests

  this depends on the detail of the script   many organisations encourage different people to test

  business, developers, customer service and customers

  but…   this doesn’t mean they are good testers

  and neither does a certificate make you a good tester!

18

following a recipe doesn’t make me a chef


illusions-10

“An excellent tester this one is”

19

  a “jedi” tester will…   let themselves be distracted by anomalies

and new ideas   encourage variability, avoid repeating the

same test twice   use their confusion as a resource   know their limits and get to know/use

subject matter experts   adapt testing to fit the context   develop resources and tools   train their minds to be curious, cautious

and critical   take notes and report results in a useful

and concise way

#6: outsourcing/offshoring is cost effective

remote teams fall into 4 categories

short-sited view disturbing case studies

company 1: started with 1:8 ratio after 2 years 1:3 ratio company 2: internal offshore, after 3 years spent more in sending UK staff to Mumbai company 3: found external/offshore team wrote 10,000 test cases in excel with just a name change for each test case company 4: in 2016 out-source IT department to India. Major problems resulted in £500 million payout to customers

20

cheaper labour = cost saving

internal external off-shore on-shore

outsourced…external to the company


illusions-11

The need for a longer-term view

  balancing costs and risks

  risks with outsourced/off-shore teams   quality of the work they produce   overhead of hand-holding/knowledge transfer   only do what they are told to do   communication and cultural differences   staff churn/loss of skill and knowledge   management overhead   failure to deliver   security and IPR issues   getting trapped 21

risks costs

test the testers and

put in penalty clauses

Is the brim of the hat smaller than the height?


illusions-12

#7: software quality is achieved through testing

  fact:   testing can find bugs   removing bugs improves software quality

  myth:   conclusion: testing improves software quality

removing bugs is a development activity

23

Testing found 5 bugs

net result: quality is still the same

Development fixed 5 bugs

net result: quality is improved

testing together with

development improves the quality of the

software: good bug

reports are key to success

Testing assesses software quality

24

Low software quality High

we think we are here

but we are really here

Low

te

st q

ualit

y H

igh

To have justified confidence in the

quality of the software we MUST have

justified confidence in the

quality of our tests


illusions-13

#8: detailed scripts better than exploratory testing

A

B 25

the problem with detailed scripts

Dangers of detailed scripts

  reasons to write detailed scripts   describes detailed tests for inexperienced testers   automation   provide an audit trail

  risks   boring to write…boring to run   more time spent writing than running tests   maintenance nightmare   false confidence in the quality of the

software   your best testers will probably leave

26

empirical evidence shows that exploratory testing finds 13 times more defects than scripted testing


illusions-14

#9: we must automate everything for efficiency and effectiveness

fact: automation can provide increased efficiency IF…

automation effort < manual effort myth: automation is effective at finding bugs   how good are your regression tests?

  automated the easy tests   automated the happy path   continued to add to the regression tests

  but who is testing the tests?   a project idea...

  run your regression tests   error seed your system with 100 bugs   run your regression tests again   count how many of the bugs have been found

27

Disturbing case studies...

  company 1:   project spent most of test effort in automation,

testers became automation specialists…and lost their testing skills. Software quality was poor.

  company 2:   automation team so “locked” into providing an

automated solution, spent 1 week of effort. Manual tester took 1 hour to do the same work.

  company 3:   outsourced team had automated those 10,000

test cases all doing the same thing ! all they achieved was “faster rubbish”

  company 4   we want to automate 90% of our tests by the end

of 2017 28


illusions-15

# 10: software testing is a co-dependent behaviour

  fact: testing is dependent on many aspects in a project   good quality requirements   good quality environments   good quality code

  myth: testing is given...   good quality requirements   good quality environments   good quality code

29

The term codependency was coined to describe an unhealthy pattern of coping with life as a result of being involved with someone who is

chemically dependent.

Testing as a co-dependent behaviour

their inappropriate behaviour

testing’s codependent response

Formal requirements for the software were either not written or are inadequate in scope and/or content.

We “do our best” by testing whatever we can.

The time allotted to testing in the project schedule is woefully inadequate.

We “do our best” by testing as much as we can.

Adequate testing process and/or tools are not provided for the testers.

We “do our best” by testing however we can.

The quality of the software was not as good as we anticipated

We “do our best” by testing as much as we can

30


illusions-16

31

Contents

What are myths and illusions ? The top ten myths and illusions

Taking action

Why do we believe in myths or allow ourselves to be illuded?

  don’t know any different – seems plausible   myths are “passed down” through generations   some myths are based on partial truth

  they are often believed by many people   easier to swim with the current than against it   source is often trusted (your grandma is always right!)

  out of sight, out of mind   our attention can be directed to other things

32


illusions-17

What can be done?

  train the eye   watch for the symptoms   recognise early warning signs

  educate the mind   learn how myths and illusions are

formed on projects, gather facts   take action

  it is vital that we provide the right information at the right time to dispel the myths and illusions

  set yourself goals and achieve them   John, Susan, Nora and George

33

34

Summary

  myths and illusions are everywhere on our projects

  my top ten myths and illusions maybe present in your organisations

  educate your mind and train your eye to recognise myths and illusions when they appear and dispel them as soon as you can

Top 10 myths and illusions - SOCO Norge AS...2017/09/28 · company 1: started with 1:8 ratio after...

Documents

Transcript of Top 10 myths and illusions - SOCO Norge AS...2017/09/28 · company 1: started with 1:8 ratio after...