Costs Of Agile Testing

ACCU 2010© Schalk W. Cronjé1 / 45

The Costs of Agile TestingThe Costs of Agile Testing

Schalk W. Cronjé[email protected]

mailto:[email protected]


Agile TodayAgile Today


Agile TodayAgile Today

● Mature● Well-known methodologies● Embraced by many

– Even if only by lip service● Misunderstood by many● Too easy to tick the boxes

than to deliver value● Tyranny of the urgent

– Discipline

agiletoday


InstitutionsInstitutions● Sets of internalised rules

supported by values– Has tacit influence– Rules will be contested

● Shape understanding of social meaning + order– Provides a framework for

performance● Shapes of rights + duties

– Political authority– Economic opportunities

Institution

Organisations

People

An organisation can become "institutionalised"Institutions don't last forever


Scott's Model Scott's Model

Societal Institutions

Organisational Fields

Organisations

Actors(groups/individuals)

diffuse, impose

diffuse, impose

diffuse, impose

invent,negotiate

interpretinnovateerrorinvent,

negotiate

Limitations of cognitive / social rationalitySelective perception

sanctionbehaviour

invent,negotiate



""Excellent system qualities are a Excellent system qualities are a continuous management and engineering continuous management and engineering

challenge, with no perfect solutionschallenge, with no perfect solutions""

Tom Gilb, Overload 85, June 2008


The Economic ViewThe Economic View


""You are not here to produce software, You are not here to produce software, you are here to provide you are here to provide valuevalue

to the businessto the business""

Schalk W. Cronjé


Why take an economic view?Why take an economic view?

● It helps to quantify the effects of multiple interacting variables

● It helps us to understand that the customer is not the only judge of value

● By using an economic framework it can allow us to maximise value, including

– cycle time– product cost– development expense

● It helps to communicate with non-technical decision makers


Cost of QualityCost of Quality

● Appraisal costs– Discovering condition of hardware & 3rd-party

software components● Internal failure costs

– Defects found before shipment● External failure costs

– Defects found after shipment● Prevention costs

– Costs for preventing all of the above


deliveredbuild

Batch DevelopmentBatch Development

Feature 3 DevFeature 2 Dev

Feature 4 Dev

Feature 1 Dev Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA


deliveredbuild

Batch DevelopmentBatch Development


Feature 4 Dev

Feature 1 Dev

BugDB

Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA

defect trickle feed

Total Time = devt1 .. devt

4 + qat

1 …qat

4 + fixdelay + fixtime + retest time

Feature 5+ Dev

Feature 5+ QA

deliveredbuild

Retest fixes QA


deliveredbuild

Time FactorsTime Factors


Feature 4 Dev

Feature 1 Dev

BugDB

Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA

defect trickle feed

Feature 5+ Dev

Feature 5+ QA

deliveredbuild

Retest fixes QA

understandingspecs

time from raising defect until it is available for testing

Building testinfrastructure

time tore-test

basic buildverification


FeedbackFeedback

● Faster feedback makes learning faster and more efficient

● Co-location improves communication● Faster feedback provides a sense of control● Large batches leads to slower feedback


Optimising Batch SizeOptimising Batch Size

● U-curve optimisation● Holding cost includes delay

in time to learn● Transaction cost includes

overhead in releasing a build for testing

Transaction cost

Batch size

Total Cost

Holding cost

Cost

_________________________________________________ 2 x Cost per Batch x Total features in project-------------------------------------------------- Holding cost per feature√

Optimal Batch Size


""If you only quantify one thing, If you only quantify one thing, quantify the cost of delayquantify the cost of delay""

Don Reinertsen


""Time counts more than moneyTime counts more than money""

Don Reinertsen

The Attention PrincipleThe Attention Principle


Building an Agile Testing Building an Agile Testing ModelModel


Simple Flow ModelSimple Flow Model

Specify Develop QA

Ready QA QAComplete

ReleasedSpec Dev DevComplete

SpecComplete

Kanban Board

Isn't this waterfall?


Reducing Learning TimeReducing Learning Time

● Writing test specifications after the development is costly in time

– Knowledge decay– Leads to incomplete designs

● Move the test specification up-front before any development starts

– This is part of requirements discovery– It broadens the perspective of the programmer– Allows us to distinguish between automated checks

and human testing


Reducing Learning TimeReducing Learning Time

Specify Develop QA

+Time to specify

+Time to develop

-Time to re-work

-Time to re-test

Test specification


Quantifying Learning TimeQuantifying Learning Time

● Can you convert time saved to a monetary value? Before After

Avg cycle days to spec 2 4

Avg cycle days to develop 8 10

Avg cycle days to test 13 8

Avg rework days during test

5 1

Total 23 22

Cost $8,118.00 $7,872.00

(*) Data in graph is from an insignificant data set and provides no conclusive proof. It is for illustrative purposes only


Change the Flow ModelChange the Flow Model

Specify Develop & Check Verify

● Ensure the all automated checks are included in the development stage

● Free up a Verification stage for exploratory testing and validating that process has been fulfilled.


Change the Flow ModelChange the Flow Model


+Time to develop

+More testing people

-Time to re-work

-Time to re-test


The Role of the TesterThe Role of the Tester

● Up-front involvement to build in quality● Help to build test infrastructure up-front during

the development phase● Work with developer to critically review what

is being automated.


Apply Lean PrinciplesApply Lean Principles


● Make the system pull-based● Reduce the batch size● Limit the work-in-progress● One feature end-to-end● If the flow breaks, fix it immediately


Apply Lean PrinciplesApply Lean Principles


+Time to develop

+Time to test

-Average Cycle time

-Inherited delay cost

-Waiting time


Team CompositionTeam Composition


The Classic Dev-QA TeamThe Classic Dev-QA Team

● This is the easiest team to adapt● Requires the will to change to a new working

practices● If teams are distributed then groups of

programmers and testers must be co-located.


The "No Testers" TeamThe "No Testers" Team

● All development + testing done by same people.

● Requires T-skilled people● Initial slow down of cycle time due to learning

of new skills OR reduction in quality delivered


Paired TeamsPaired Teams

● Pair up people with different skills to work on same feature

– Feature teams are not a new concept● Could be perceived to have higher person

cost per feature– Instead of distributing the person cost over multiple

queues, the cost is combined into a single queue– Can actually lead to reduced cycle time.


Real-world measurementsReal-world measurements


Real-world measurementsReal-world measurements

M1 M2 NF1 NF2

READY 4 6 5 1h

SPEC 1 0.5 6 0.5

SPEC/Completed 2 0.5 1 0.5

DEV + Check 2 1 13 4

DEV + Check / Completed

3 6 4 18

VERIFY 3 7 8 1

VERIFY/Completed 2 5 6 18

Blockages 0.5 1h 2 1h

Total 17 26 43 42


Average cycle days per feature


What is wrong?What is wrong?

M1 M2 NF1 NF2

READY 4 6 5 1h

SPEC 1 0.5 6 0.5




3 6 4 18

VERIFY 3 7 8 1



Total 17 26 43 42




M1 M2 NF1 NF2

READY 4 6 5 1h

SPEC 1 0.5 6 0.5




3 6 4 18

VERIFY 3 7 8 1



Total 17 26 43 42


Excessive long time between feature availability and testing. Is there enough testing bandwidth? Or is this team just cheating?



M1 M2 NF1 NF2

READY 4 6 5 1h

SPEC 1 0.5 6 0.5




3 6 4 18

VERIFY 3 7 8 1



Total 17 26 43 42




M1 M2 NF1 NF2

READY 4 6 5 1h

SPEC 1 0.5 6 0.5




3 6 4 18

VERIFY 3 7 8 1



Total 17 26 43 42


Is this 2 days due to re-work? Is there a problem in the development phase?


How do these teams compare?How do these teams compare?

M1 M2 NF1 NF2

READY 4 6 5 1h

SPEC 1 0.5 6 0.5




3 6 4 18

VERIFY 3 7 8 1



Total 17 26 43 42



ExerciseExercise

● Analyse the previous listed teams and comment on their process (with liberty)

● Is it possible to identify points where cost of delay can be calculated?

● How would you identify the holding costs and transaction costs?

● Can you improve testing/quality control and what is the expected and hidden costs?


The End GameThe End Game


The institution hits backThe institution hits back


Organisational Fields

Organisations

Actors(groups/individuals)

imposing existing failed processes

diffuse, impose

slow diffusionor no diffusion

attempting to introduce change

speaking at public conferences, publishing articlessuccess

propagates from known champions

Limitations of cognitive / social rationalitySelective perception

sanctionbehaviour

invent,negotiate



Making Testing more AgileMaking Testing more Agile

● Reduce learning time - bring test specification forward

● Pair up developers and testers● Be intelligent about what is being automated.


Investing in Agile TestingInvesting in Agile Testing

● Building infrastructure● Good testers demand equivalent

remuneration to good programmers● Put measurements in place● Cross-train people to create T-skilled sets● Smooth the Flow● Recognise that transition will take months,

even a year


Cost of Agile TestingCost of Agile Testing

● Appraisal costs● Internal failure costs● External failure costs● Prevention costs

– Checks against external deliverables– Automating checks (TDD)– Building the capacity for addressing critical

issues without too much interruption to other development.

– Training & Practising the mind-set


SummarySummary

● Balanced teams reduce costs● Shared problem-solving can reduce feature

cycle time● Reduced cycle time will have a direct

influence on delay cost.● Following a process does not make you

testing agile● Measure to improve● You will operate within an institutional context

Costs Of Agile Testing

Documents

Transcript of Costs Of Agile Testing