Test Improvement with TMMiTest Improvement with TMMi

Process Improvement for Present and Future

Erik van Veenendaal

www.erikvanveenendaal.nl

Erik van VeenendaalErik van Veenendaal

� Founder and major shareholder ImproveQS

� In testing since 1989 working for many different clients and in many different roles

� Author “TMap”, “The Little TMMi” and many other books and papers

www. erikvanveenendaal.nl

Improve Quality Services BV 2

other books and papers

� Vice-President International Software Testing Qualifications Board (ISTQB) 2005 - 2008

� Vice-Chair TMMi Foundation

� Keynote speaker, e.g. EuroSTAR, STAR-East

� Winner of the European Testing Excellence Award (2007)

ImproveImprove QualityQuality Services BVServices BV

� Service organisation in the area ofTesting, Requirements Engineering and Quality Management

� Consultancy, Subcontracting and Training

www. improveqs.nl

Improve Quality Services BV 3

� Consultancy, Subcontracting and Training

SW Process ImprovementQuality AssuranceIT-AuditingRequirements Engineering& management (IREB)

Testing (TMap, TMMi)Test Process ImprovementCertification (ISTQB)- incl. Advanced !!Inspections / Reviews

� Increasing size and complexity�Amount of software in consumer

products doubles every 24 months (Philips)

�Number of requirements for mobile phones doubles every 6 months (Nokia)

Computable, Sept.’09

ChallengesChallenges

Improve Quality Services BV 4

every 6 months (Nokia)

� # Defects hardly decreases�Defect density per KLOC is almost constant in the

last 10 years (Les Hatton)

� Deadlines, Competition & Outsourcing

� Testing often takes 30 - 40% of project costs

PeoplePeople

Test Improvement DirectionsTest Improvement Directions

Improve Quality Services BV 5

Test ProcessTest Process

People SkillsPeople Skills

Test knowledgeTest knowledge-- Test principlesTest principles

-- TechniquesTechniques

IT knowledgeIT knowledge-- ScriptingScripting

-- Requirements Requirements (IREB)(IREB)

via (exchanging) via (exchanging) Practical Practical ExperiencesExperiences, , CoachingCoaching

and (formal) and (formal) TrainingTraining

Improve Quality Services BV 6

-- TechniquesTechniques

-- Tools, etc.Tools, etc.

-- Requirements Requirements (IREB)(IREB)

-- Configuration mgt.Configuration mgt.

DomainDomainknowledgeknowledge-- Business processBusiness process

-- User characteristicsUser characteristics

Soft skillsSoft skills-- CommunicationCommunication

-- Critical mindsetCritical mindset

-- Presentation & reportingPresentation & reporting

CCMMMM((II)) MMaattuurriittyy

lleevveell

DDeessiiggnn FFaauullttss // KKLLOOCC

((KKeeeennee))

DDeelliivveerreedd DDeeffeeccttss // FFPP

((JJoonneess))

SShhiippppeedd DDeeffeeccttss // KKLLOOCC

((KKrraassnneerr))

RReellaattiivvee DDeeffeecctt DDeennssiittyy

((WWiilllliiaammss))

SShhiippppeedd DDeeffeeccttss

((RRiiffkkiinn))

55 00,,55 00,,0055 00,,55 00,,0055 11

44 11 00,,1144 22,,55 00,,11 55

Trends in Software Trends in Software QualityQuality

Improve Quality Services BV 7

44 11 00,,1144 22,,55 00,,11 55

33 22 00,,2277 33,,55 00,,22 77

22 33 00,,4444 66 00,,44 1122

11 55--66 00,,7755 3300 11,,00 6611

Readiness for Test ImprovementReadiness for Test Improvement

�� Some fundamental practices availableSome fundamental practices available��Requirements documented and managedRequirements documented and managed

��Project planning existsProject planning exists

��Configuration management Configuration management

Improve Quality Services BV 8

��Configuration management Configuration management

�� ManagementManagement “needs” / supports better testing“needs” / supports better testing��Quality is important enoughQuality is important enough

�� A budget for process improvementA budget for process improvement

��or reverse quality push……..!!or reverse quality push……..!!

TMMi modelTMMi model

�� Test Maturity Model (TMM) FrameworkTest Maturity Model (TMM) Framework��IlinoisIlinois Institute of Technology Institute of Technology -- CMM basedCMM based

�� TMMi SourcesTMMi Sources��CMMI, ISTQB, TMM, IEEE, TPICMMI, ISTQB, TMM, IEEE, TPI

BB

Improve Quality Services BV 9

��CMMI, ISTQB, TMM, IEEE, TPICMMI, ISTQB, TMM, IEEE, TPI

�� Starting pointsStarting points��Staged representation, CMMI structureStaged representation, CMMI structure

��Independent or in combination with CMMIIndependent or in combination with CMMI

��TMMiTMMi V3.1 released November 2010V3.1 released November 2010

AA

TMMi process areas by levelTMMi process areas by level5 : Optimization5 : Optimization-- Test process optimizationTest process optimization-- Quality controlQuality control-- Defect preventionDefect prevention

Helps to set priorities Helps to set priorities and focus the testand focus the testimprovement processimprovement process

Helps to set priorities Helps to set priorities and focus the testand focus the testimprovement processimprovement process

4 : Management and measurement4 : Management and measurement-- Product quality evaluationProduct quality evaluation-- Test measurement programmeTest measurement programme-- Advanced reviewsAdvanced reviews

Improve Quality Services BV 10

3 : Integration3 : Integration-- Test organizationTest organization-- Test training programTest training program-- Test life cycle and integrationTest life cycle and integration-- NonNon--Functional testingFunctional testing-- Peer ReviewsPeer Reviews

2 : Managed2 : Managed-- Test policy and strategyTest policy and strategy-- Test planningTest planning-- Test monitoring and controlTest monitoring and control-- Test design and executionTest design and execution-- Test environmentTest environment

± 2 years

± 1 year

improvement processimprovement processimprovement processimprovement process -- Advanced reviewsAdvanced reviews

CertificationCertification

�� Requirements defined regardingRequirements defined regarding

��Assessment requirements (TAMAR) Assessment requirements (TAMAR) based on ARC (B/C) and ISO 15504based on ARC (B/C) and ISO 15504

��LeadLead--Assessor and Assessor Assessor and Assessor

Improve Quality Services BV 11

��LeadLead--Assessor and Assessor Assessor and Assessor

�� Organizations and persons to be accredited by the Organizations and persons to be accredited by the TMMi FoundationTMMi Foundation

�� Register of (lead) assessorsRegister of (lead) assessors

��Formal TMMi Certification possibleFormal TMMi Certification possible

Lead Assessor / AssessorLead Assessor / AssessorLead assessorLead assessor

(formal assessments)(formal assessments)

AssessorAssessor

(informal assessments)(informal assessments)

TestingTesting > 5 years> 5 years

Multiple levels & domainsMultiple levels & domains

ISTQB AdvancedISTQB Advanced

> 3 years> 3 years

Multiple levels & domainsMultiple levels & domains

ISTQB FoundationISTQB Foundation

Improve Quality Services BV 12

Test Process Test Process ImprovementImprovement

> 2 years> 2 years

(2 years SPI = 1 year TPI)(2 years SPI = 1 year TPI)

> 1 years> 1 years

TMMiTMMi TrainingTraining

Practical experiencePractical experience

TrainingTraining

Practical experiencePractical experience

AssessmentsAssessments Training Training

20 days practical experience20 days practical experience

Training Training

10 days practical experience10 days practical experience

Level 2 : ManagedLevel 2 : Managed

�� Test Policy and StrategyTest Policy and Strategy

�� Test PlanningTest Planning

�� Test Monitoring and ControlTest Monitoring and Control

Improve Quality Services BV 13

�� Test Monitoring and ControlTest Monitoring and Control

�� Test Design and ExecutionTest Design and Execution

�� Test EnvironmentTest Environment

Ensures CMMI complianceEnsures CMMI complianceVerification & ValidationVerification & Validation

TMMi level 2TMMi level 2Lead timeLead time

reduction !!reduction !!

to provide visibility regarding the quality and outstanding risks of software productsto provide visibility regarding the quality and outstanding risks of software products

to contribute to the software product quality by finding defectsto contribute to the software product quality by finding defects

to contribute to customer satifaction regarding the released software productsto contribute to customer satifaction regarding the released software products

….….

Test Policy 20XX

.

Mission / Ambition

Definition Testingbased on

business policy

Improve Quality Services BV 14

Test targets

Employees

Organization

Test process

Customers

Test Process ImprovementTest Process Improvement-- objectives, performance indicators, modelobjectives, performance indicators, model

Example Performance IndicatorExample Performance Indicator

15

20

25

30

35

40Improvement focusImprovement focus

Improve Quality Services BV 15

0

5

10

15

w eek 1-10 w eek 11-23 w eek 24-36

Development.percentage of TOTAL FAT.percentage of TOTAL

UAT.Percentage of TOTAL Production.Percentage of TOTAL

“this is what we are doing !!”“this is what we are doing !!”

Level 3 : IntegrationLevel 3 : Integration

�� Test OrganizationTest Organization

�� Test Training ProgramTest Training Program

�� Test Life cycle and IntegrationTest Life cycle and Integration

InstitutionalizationInstitutionalization

Improve Quality Services BV 16

16

�� Test Life cycle and IntegrationTest Life cycle and Integration

�� NonNon--Functional TestingFunctional Testing

�� Peer ReviewsPeer Reviews

reqreq gdgd dddd implimpl testtest operoper

Life cycleLife cycle

Req.Design

Improve Quality Services BV 17

Code

TestTest planTest design

Test script

Execution

Test ImprovementsTest Improvements

�� Testing as a profession Testing as a profession

�� Clear responsibilitiesClear responsibilities

�� Test plan(ning) Test plan(ning)

�� RiskRisk--based testing based testing

Risk Matrix ISR 2.2

0

8

15

0 25 50

probability

Impa

ct

Q/R

US/MR

ECG

S/R

Vasc

USVW

Lic

Ptrac

Log

Srch

Impd

Merge

DB

DCMlib

Dbkill

Vwregr

Top PriorityTop Priority

engineeringview

PRISMA®

Improve Quality Services BV 18

�� RiskRisk--based testing based testing

�� Testing part of development life cycleTesting part of development life cycle

�� Early involvement Early involvement

�� Test design techniquesTest design techniques

�� Controlled test processControlled test process

view

Reported ResultsReported Results

5

10

15

20alpha / beta test lead time

556065707580859095

100Defect Detection Percentage

Improve Quality Services BV 19

0Y1 Y2 Y3 Y4

0

20

40

60

80

100

120

140

160

% deviation - spent vs. planned

5055

Y1 Y2 Y3

TMMi 1

TMMi 2

TMMi 3

Employee satisfaction

“Getting Started”“Getting Started”

��Test policyTest policy

��Management commitmentManagement commitment

�� Improvement is a projectImprovement is a project

1

Improve Quality Services BV 20

�� Improvement is a projectImprovement is a project

��Resources and effort percentageResources and effort percentage

��Maturity of development organizationMaturity of development organization

2

3

“Getting the job done”“Getting the job done”

�� Set both long term and short term goalsSet both long term and short term goals

�� Use what is availableUse what is available

�� Don’t make an external consultant Don’t make an external consultant responsibleresponsible

Improve Quality Services BV 21

responsibleresponsible

�� Institutionalize the improvementsInstitutionalize the improvements

�� Change management skillsChange management skills

�� Review against the overall objectives Review against the overall objectives (policy) almost continuously(policy) almost continuously

Test Improvement ManifestoTest Improvement Manifesto

•• FlexibilityFlexibility over Detailed Processes

•• Best PracticesBest Practices over Templates

•• Deployment orientationDeployment orientation over Process

Making TMMi work ….Making TMMi work ….

Improve Quality Services BV 22

•• Deployment orientationDeployment orientation over Process orientation

•• ReviewsReviews over Quality Assurance (departments)

•• Business drivenBusiness driven over Model driven

TMMi Supporting OrganizationsTMMi Supporting Organizations

Improve Quality Services BV 23

More information on TMMiavailable at www.TMMiFoundation.org