Post on 21-Mar-2018
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 1
First practical experience
ISTQB Expert levelImproving the testing process
www.ip-consult.be
Version 1.1
Isabelle Robrechts
●Training content
Agenda
●Practical experience
●Conclusion
2
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 2
ISTQB Structure
IP-Consult bvba 3
Bron : www.istqb.org
Goal ISTQB Expert ITP
• Coaching/Leading test process improvement tracks in own organisationtracks in own organisation
• Goal part 1 : Assessing the test processes• Improvement models• Linking and situate
4
• Goal part 2 : Improving the test processes• Change Management• Organisation and human factors
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 3
IDEAL
Part 1: Assessing
© 2014 Improve Quality Services B.V. 5
Part 2: Implementing
Approaches
Overview Improvement Approaches
Model based
IT/SW Testing
Analytical
GQMMeasuresMetrics
Indicators
Hybrid / Mixed
Analytical & Models
Analytical & Content
Other
People Skills
Standards & Regulations
Infrastructure & Tools
© 2014 Improve Quality Services B.V. 6
CMMIIEEE15504
Processmodels
TMMiTPI Next
Content models
CTPSTEP
(TMap)
CausalAnalysis
Infrastructure & Tools
Evaluations
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 4
●Training content
Agenda
●Training content
●Practical experience
●Conclusion
7
●OverviewEnvironment
Practical experience:
− Environment− Business goals
●Retrospective− Starting point− Transfer to improvement: Causal analysis ?
exercise
● Improvement proposition− Short term− Long term
8
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 5
Environment
•Financial institution•SDLC : Combi V-model / ‘Agile’ (inhouse invention)•SDLC : Combi V-model / Agile (inhouse invention)•IS : tester is not a profession but a role (<-> Bsn)
9
●To go quickly to the market with new products− Risk-based testing !
Business goals
Risk-based testing !− Regression testing !
●Good quality of the requested products− Fit for purpose− Passive testing !g− Less nuisance for the operations department
10
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 6
Retrospective
• Assignment 2009 : IS•Set-up :Set-up :
• test organization• test methodology• test culture• testing
•Representing the department in an organizational
11
g gtesting competence center•Representing the IS testing at business meetings•Report to the department management
• just CTAL
●What is present ?− Test work products ?
Retrospective : where do I start ?
Test work products ?Test PolicyTest Strategy (Master)Test PlanTest Cases
− Test culture− Test process ?− Test Tooling
Test ManagementDefect Logging
12
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 7
●Knowledge through measuring is power...
Retrospective : where do I start ?
●But what are you going to measure ?− Inspired by the circumstances ?− Inspired by others (lecture, management,...)− Inspired by own interestsInspired by own interests
13
● Defects : − # incidents in production
Retrospective : Data Collect
p− # scope changes per phase− # defects per phase (incl.priority)
● Test cases : − # written test cases per phase (incl.priority)− # executed, (in-)correct test cases
● Budget : − Planned & Actual test budget− Planned & Actual total budget− Planned & Actual Realization (Analysis & Development)− Planned & Actual repair budget
14
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 8
●Can we obtain information from this data ?Test Effectiveness
Retrospective : first metrics...
− Test EffectivenessHow many defects are found per test case ?How many defects are found per test phase ?
HOW EFFECTIVE ARE OUR TESTS?
− Test EfficiencyTest EfficiencyWhat’s the price of a defect found during the ST/SIT phase vs a defect found during the UAT phase ?
15
●How effective are our tests ??
Retrospective : Test Effectiveness
16
Beware of the numbers !
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 9
Retrospective : Test Efficiency
17
●Pre-condition : an effective defect and/or incident management process
Transfer to improvement : Causal Analysis ?
management process
●4 steps : 1. Select the items for causal analysis2. Collect and organize the information3. Identify any root causes3. Identify any root causes4. Draw conclusions
18
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 10
●Defects/incidents● Irregularities reported by stakeholders
1. Select the items for Causal Analysis
● Irregularities reported by stakeholders●Problems found during the analysis●Aids:
− The pareto principle : 80/20− Categorization of defects− Analysis of statisticsAnalysis of statistics− Results of retrospectives
19
Analysis of statistics
Reality:• incidents doubled the last two years
• normal requests (no projects) decreased
20
Peaks are indicating the close-out of a project.
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 11
Analysis of statistics
21
●Let someone take the lead●Aid while structuring the data :
2. Collect and organize the information
●Aid while structuring the data :− Ishikawa
22
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 12
2. Collect and organize the informationManagementInfrastructurePlanning
Minimal budgets for resources
Tests are delayed by strict authorizations
Unavailability of the test environment is 1 on 5
UnrealisticProject planning
# Incidents doubles in the last
2 years
No money for trainings
authorizations
Monthly refresh of test dataTesting is a role, no profession
No budget for resources
Lack of van PM capacities
No commitment to any test workMissing milestines is easily Accepted.
Bsn is often seen as the enemy by IS
23
Communication Culture Human
No professional IS testers
Motivation/work spirit often very low
IS : why loose time testing now if it is thoroughly tested later ?
Defects are found much too late
No transparancy in the IS tests
● Identify the ‘most-likely’ causes●Work top down through the causes
3. Identificy the root causes
●Work top-down through the causes●Add (if necessary) extra information to the analysis● Identify any Tipping Points or Broken windows●Maintain brainstorm rules
24
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 13
●Are there any conclusions ?●Be Aware for single events ?
4. Draw conclusions
●Be Aware for single events ?●Do we find any problems in the test process which
can be avoided in the future ?●Are there any other processes involved ?●Mutual causes ?
25
− Short term : some quick-wins are defined: More transparancy between IS and Bsn
Improvement proposal
p y● Visualisation of the businessplan with all milestones● Weekly reporting on planning at the Division Management● Definition of quality gates and its controles● Better communication (1N)● Usage of 1 Test Management Tool (creates responsibility)
26
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 14
− LongTerm : elaborate an improvement track based on TPI Next, with possiblity to expand to CMMi
Improvement Proposal
, p y pWhy TPI Next ? And not TMMi ?● Test process exists (TMAP)● Test process has already a certain maturity● Emotional sensitive to improvement● Clear visibility of the impact on the business goals● Link with CMMi exists
Fit in the model ... ● How does it look dd 2014Q1 ?● TPI model in oktober● Where do we want to go ?
27
Improvement Proposal : Long Term
TPI Next : Test Maturity Matrix (dd 2014Q1)
TPI Next : Test Maturity Matrix (dd 2014Q4)
28
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 15
●TPI Next : Test Maturity Matrix (2016)
Improvement Proposal : Long Term
Maturity Level : - Controlled on all key areas- Evolution from Efficient on the business critical key areas
29
Don’t forget the soft skills !
IP-Consult bvba 30
© 2014 Improve Quality Services BV 01‐01 Course Introduction ‐ 16
●Training content
Agenda
●Practical experience
●Conclusion
31
●Models and causale analyse are complement
Conclusion
− Models for Long Term visions− Causale analyse for selection and short term activities
● In the training links between techniques are shown
●Assignments connect the theory with own practical experiences
●Don’t forget the soft skills32