Ingeniería para la Industria - ÉTS Montréal€¦ · Ingeniería para la Industria Qy of a Large...

Measuring the Cost of Software Quality

Ingeniería para la Industria

g Q yof a Large Software Project

at Bombardier TransportationpPresented by Claude Y. Laporte, Eng., Ph.D.

ProfessorDepartment of Software and IT Engineeringp g g

École de technologie supérieure, Québec, Canada

Project Editor of ISO/IEC 29110 for Very Small Entities

Professor Normand Séguin, Ph.D.

Universitad de Lima, PeruSeptember 13th 2012September, 13th 2012

http://www2.ulima.edu.pe/


Content• IntroductionIntroduction

• Business Rationale for Software Quality Assurance and the Measurement of the Cost of Qualityand the Measurement of the Cost of Quality

• Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality

• Measurement of the Cost of Software Quality – A Case Study

• Conclusion

2

École de technologie supérieure

3

École de technologie supérieureOver 6,3000 studentsEach year 2,200 paid industrial internships in over 900 companies.

Students are paid about 36,500$ for 3 internships of 4 months

Undergraduate Programs• Software Engineering • 600 students (400 in Software Eng.)

20 P f i th d t t hg g

• IT Engineering• Construction Engineering• Production Engineering

• 20 Professors in the department have a mean industrial experience of 10 years.

• Electrical Engineering• Mechanical Engineering • Logistics and Operations Engineering

• Graduate Programs• Software Engineering• Information Technology

175 students• Information Technology• Other programs

www.etsmtl.ca4

Overview of my Background

National DefenseDefence Nationale

OerlikonOerlikonAerospaceAerospace

Defence Nationale

(1973‐1991)

(1991‐1999)

(1999‐2000)

f f d i i

5

Department of Software and IT Engineering

(2000 ‐ )

Project Editor of ISO/IEC 29110 Standards and Guides

f

Joint Committee for IT

Sub committee (SC) 7

Standardization of processes, supporting tools and supporting technologies for the Sub committee (SC) 7 technologies for theengineering of software products and systems.

Mandated to developWorking Group (WG) 24

Mandated to develop Standards and Guides for Very Small Entities *

6Very Small Entity - An entity (enterprise, organization, department or project) having up to 25 people.

ISO/IEC 29110 Standards and Guidesfor Very Small Entities (VSEs)for Very Small Entities (VSEs)

• Entry - Targets VSEs typically d l i 6 thdeveloping 6 person-monthprojects and start-up VSEs;

• Basic - Targets VSEs developing only one project at a time;

• Intermediate – Targets VSEs developing multiple projects

Advanced

Intermediate

p g p p jwithin the organizational context;

• Advanced – Targets VSEs which want to sustain and grow as an

Entry

Basicwant to sustain and grow as an independent competitive software development business.

(ISO/IEC 29110)

Very Small Entities are enterprises, projects or departments having up to 25 people.

7


Content

• IntroductionIntroduction



• Measurement of the Cost of Software Quality – A yCase Study

• Conclusion

8

Software Development – What the customer wants

9

Best Approach = Better Faster Cheaper

Why Software Fails ?Why Software Fails ?

‘Studies have shown that software specialists spend about 40% to 50% of their time on avoidable rework rather than on whatto 50% of their time on avoidable rework rather than on what they call value-added work, which is basically work that’s done right the first time.

Once a piece of software makes it into the field, the cost of fixing an error can be 100 times as high as it would have been g gduring the development stage.’

Software managers and software engineers ‘must’ select and use appropriate practices to reduce rework (i.e. waste or ‘scrap’)

Adapted from (Charette 2005) 10

Components of Project Cost

Project Cost

Cost of Quality Cost of PerformanceCost of Quality Cost of Performance

• Generation of plans• Software Development

Cost of NonConformance

Cost of Conformance

• Fixing defects• Re testing • Re‐reviews

Appraisal Costs

• Reviews• Inspections

Prevention Costs

• Training•Methodologies • Updating source code

• Modifying documents

Inspections• Testing• IV&V• Audits

•Methodologies• Tools• Data gathering

11

• Audits

Adapted from (Haley et al., 1995)

Cost of Quality

• Data from Professional Software Engineers Site A

AmericanSite A

AmericanSite B

EuropeanSite C

EuropeanSite D

EuropeanCourse A

2008Course B

2008Course C

2009Course D

2010Course E

2011Course F

2012AmericanEngineers

(19)

AmericanManagers

(5)

EuropeanEngineers

(13)

EuropeanEngineers

(14)

EuropeanEngineers

(9)

2008(8)

2008(14)

2009(11)

2010(8)

2011(15)

2012(10)

Cost of performance 41% 44% 34% 31% 34% 29% 43% 45% 45% 34% 40%

Cost of rework 30% 26% 23% 41% 34% 28% 29% 30% 25% 32% 31%

Cost of appraisal 18% 14% 32% 21% 26% 24% 18% 14% 20% 27% 20%pp

Cost of prevention 11% 16% 11% 8% 7% 14% 10% 11% 10% 8% 9%

Quality 71 8 23 35 17 403 19 48 35 60 55

Quality = Number of Defects/ 1,000 Lines of Code

12

Defect ‘Injection’ During DevelopmentD f t (%)Defects (%)

W id if d f iblWe must identify defects as soon as possible to reduce cost and schedule slippage.

Development PhasesAdapted from (Selby 2007) 13

Defect Detection During DevelopmentDefects Detected/Defects Detected/ Defects Injected

We must implement software engineering practicesWe must implement software engineering practicesto detect and correct 90% of defects as early aspossible to reduce cost and schedule slippage

Adapted from (Selby,2007)

Development Phases14

possible to reduce cost and schedule slippage.

Start of Initiative% f T l

Cost of Quality

40

45

Cost of Non Conformance (Rework)

Start of Initiative

41%% of TotalProject Cost

30

35

Appraisal & Prevention Costs

15

20

25

18%

5

10

15

11%

0

M t it L l

1 2 341988 1990

6%5%

Maturity Level 19901992

19961994

15Adapted from (Haley et al., 1995)


Content



• Undergraduate/Graduate Software Quality Assurance Course and the Measurement of the Cost of Quality


• Conclusion

16

Software Quality Assurance Course at ETS• Fourteen Topicsp

– Chapter 1: Basic Software Quality Assurance Knowledge

– Chapter 2: Quality Culture and Cost of Quality (COQ)

Ch t 3 Q lit R i t

French Textbooks

– Chapter 3: Quality Requirements

– Chapter 4: Software Engineering Standards and Models

– Chapter 5: Reviews and COQ

Vol. 1

– Chapter 6: Software Audit

– Chapter 7: Verification and Validation and COQ

– Chapter 8: Tests and Software Quality AssuranceEnglish Textbook

p Q y

– Chapter 9: Software Configuration Management and COQ

– Chapter 10: Policies, Processes and Procedures and COQ

Chapter 11: Measurement and COQ

Vol. 2

– Chapter 11: Measurement and COQ

– Chapter 12: Management of Suppliers and Contracts

– Chapter 13: Risk Management and COQ

– Chapter 14: Software Quality Assurance Plan

17Publication in 2013

Software Quality Assurance CourseLaboratory

• Software Development Project with ISO/IEC 29110– Put in practice the software quality practices learned in class,

y

– Each team manages and executes a project using ISO/IEC 29110• A process is a component of the COQ (i.e. Prevention)

Statement S ft

CustomerImplementation

ProcessInitiation

Statement of Work

SoftwareConfiguration

Planning Control

Project Management Process

Analysis

Design

Measure the Cost of

Software

Organizational Management

Planning Control

Execution Closure

Construction

Tests

Delivery

Quality

18

Organizational Management Delivery

ISO/IEC 29110 Standard for Very Small Entities

Graduate Software Engineering Program

• Twelve 3-credit courses and a 9-credit Project in Software engineering

Software Engineering Program

engineering• Students are typically from industry with a minimum of 2 years

of experienceof experience– Students typically have an undergraduate degree in either computer

science or management information systems and work for small and di i d i timedium-sized organizations.

• Software Quality Assurance CourseStudents in teams of 3 have to do a Project in an– Students, in teams of 3, have to do a Project in an Organization

• Measurement of the Cost of QualityQ y

19

C t tIngeniería para la Industria

Content• Introduction

• Business Rationale for Software Quality Assurance and the Measurement of the Cost of Qualityy


• Measurement of the Cost of Software Quality – A Case Study

• Conclusion

20

M i th C t f S ft Q litMeasuring the Cost of Software Quality of a Large Software Project

at Bombardier Transportation in Canada

Laporte C Y Berrhouma NLaporte, C. Y., Berrhouma, N., Doucet, M., Palza‐Vargas, E.

(Laporte et al. 2012) 21

Challenges Facing Railway Manufacturers

• Criticality of softwareBetter, Faster, Cheaper

y– Financially, environmentally or for

human safety.• Multi-disciplinary system development,• Integrator-Suppliers Relationships,• Multi-country development,• Multi-cultural teamsMulti cultural teams,• Downsizing/Merger/Turnover,• Off shoring.

ERTMS / ETCS (European Rail Traffic / ( pManagement System / European Train

Control System)

22

Overview of Bombardier• Workforce of some 80,000 people in over 24 countries.

• Bombardier AerospaceW ld l d i h f f b i j d i l– World leader in the manufacture of business jets and regional transport,

• Bombardier Transportation– Leader in the manufacture of rail transport equipment.,p q p

– Manufactures locomotives, freight cars, and propulsion and control systems,

Provides systems and signalling equipment– Provides systems and signalling equipment.

• Modern trains and subways are increasingly complex, and more and more subsystems are computer-controlled, such as propulsion and braking systems.

• At the time of this case study, there were over 30 software development centers within Bombardier Transportation for adevelopment centers within Bombardier Transportation, for a total of about 950 software engineers.


The Software Development Group

f B b di T t ti i Q éb (C d )of Bombardier Transportation in Québec (Canada)• A group of 30 software engineers whose role is to design, develop,

and maintain embedded software for trains and subwaysand maintain embedded software for trains and subways• Software monitoring system used for collecting software

maintenance information, • Software for controlling car inclination.

SD

Media Converter

LINK PWR LINK

5VDC. 1A_ __ __ +

UP LINK

RX

TX


Software Engineering Processof the Software Engineering Group *g g p

25* Located in Québec, Canada (Laporte et al. 2012)

ISO/IEC 15939 ‐ Systems and Software Engineering Measurement Process StandardEngineering ‐Measurement Process Standard

Technical &Management

Measurement Requirements User Feedbackg

Processes InformationProducts

InformationNeeds

Core Measurement Process

Establish Plan Perform EvaluateCommitment Measurement

E i B

Plan PerformanceInformation

Measurement Measurement

Experience Base

Improvement Actions

EvaluationResults

(ISO/IEC 15939)

Scope of ISO/IEC 15939

26

Objectives of the Measurement Projectf h S f D l Gof the Software Development Group

1. Identify a project to measure the Cost of Software Quality y p j Q y(CoSQ),

2. Collect data on costs, 3 Categorize costs related to software quality3. Categorize costs related to software quality,4. Develop a data model for the measurement of CoSQ,5. Analyze data collected, on the selected project, using the y , p j , g

data model,6. Present the CoSQ report to senior management,7 Expand the measurement of the CoSQ to other software7. Expand the measurement of the CoSQ to other software

projects in the Software Development Group.


Overview of the Project SelectedOverview of the Project Selected

• Development of the software to control the subway of a large American city,

• Team of 15 software engineers,• L b f ft t k• Large number of software tasks

– 1,121 tasks• Total effort of the completed projectTotal effort of the completed project

– 88,598 hours


The Five Stages of the Measurement Project/using the ISO/IEC 15939 Standard

1. Identification of the project tasks related to the CoSQ– Identify measurement requirements and resources

2. Development of a list of typical tasks related to the CoSQDefine data measures– Define data measures

3. Categorization of the tasks related to the CoSQ– Select and plan measures

4. Development and application of weighting rules– Define measurement criteria

5 D t i ti f th fid f th i hti l5. Determination of the confidence of the weighting rules– Define data collection and analysis

29(Laporte et al. 2012)

Stage 1. Identification of Project Tasks R l d h C SQRelated to the CoSQ

• Software engineering process is composed of 4 ‘types’Lif C l– Life Cycle,

– Process,

Type Process Element

- Sub-process, - Activity.

ypLife Cycle Primary Life Cycle ProcessProcess DevelopmentSub-process Code and DebugActivity Unit Tests

Representation of BSEP elements

WBS Element Task Name Effort [hours]

Code document Monitoring-Unit Testing 91.1

(Laporte et al. 2012)

Sample registration of a task in the Project accounting system

30

Stage 2. Development of a list of tasks l d h C f S f Q lirelated to the Cost of Software Quality

• A list of tasks related to the CoSQ is developed for this projectA list of tasks related to the CoSQ is developed for this project.

Primary Life Cycle ProcessesSupply

Organizational Life Cycle ProcessesManagementpp y

DevelopmentSystem Requirements AnalysisSystem Architectural DesignS ft R i t A l i

gEstimate the ProjectPlan the ProjectManage RisksT k th P j tSoftware Requirements Analysis

Software Architectural DesignSoftware Detailed DesignSoftware Coding and Testing

Track the ProjectInfrastructure

Plan Infrastructure ActivitiesEstablish the Project Infrastructure

Subset of CoSQ-related Tasks

Software IntegrationSoftware Validation Testing

Establish the Global InfrastructureMaintain the Infrastructure

(Laporte et al. 2012)

Q

31

Stage 3. Categorization of tasks l d h C SQrelated to the CoSQ

• All 1,121 tasks were sorted as follows: – Implementation (I), – Evaluation (E), – Prevention (P)– Prevention (P), – Rework (R) (internal and external anomalies).

TaskTaskIdentification WBS Element Task Name I E P R

2410 Code Trace Requirements - - x -

Example of classification of the Requirements Traceability Task


Stage 4. Development of weighting rules• Many tasks belong to more than one CoSQ category

– e.g. The ‘Test and coding’ task overlaps the Evaluation and Implementation CoSQ categories. p Q g

• Twenty-seven (27) weighting rules have been defined. Rule

Number Name of Rule Typical Tasks Implementation Evaluation Prevention ReworkNumber

3 Process Audit Software Project & Process Audit 100%

7 Review Design review 100%7 ev ew es g ev ew 00%

12Problem Correction and Coding

Corrections, Debugging and Final Coding

30% 70%

17 Training Training of new resources 100%

26 Follow-up & Follow-up & Validation (all 85% 15%26 Validation Validation (all releases)

85% 15%


Stage 5. Determining the Confidence f W i h i R lof Weighting Rules

• What is the level of confidence of the CoSQ measures ? • Another component was added to the rules

– "H" for high precision,– "M" for medium precision and– M for medium precision, and– "L" for low precision.

C fid ith th W i hti R l P t (%)Confidence with the Weighting Rules Percentage (%)

H High 88%

M Medium 11%

L Low 0,2%

Total number of activities 1,121


Cost of Software Quality Measurement Results

Perform Evaluation Prevention Rework CoSQQ

Hours 59,231 18,551 2,025 8,791 28,824

P t 67 % 17 % 4 % 11 % 32 %Percentage 67 % 17 % 4 % 11 % 32 %

• Total effort (i e cost) of the project: 88 598 ho rs• Total effort (i.e. cost) of the project: 88,598 hours• Number of software tasks: 1,121 tasks


Cost of Software Quality at Raytheon compared to Bombardier Transportationp p

70 CMM level 3Start of intiative CMM level 1

50

60

ct c

ost TCoSQ

Rework

Total Cost of Software Quality (40%) 32 %

40

of to

tal p

roje

c

Cost ofConformance Percentage of Rework ( ≈13%)

Percentage of Evaluation ( ≈20%) 17 %

11 %

20

30

Per

cent

age

o Conformance Percentage of Rework ( ≈13%)

Percentage of P ti ( 7%) 4 %

11 %

0

10

P

PreventionRework

AppraisalPrevention ( ≈7%) 4 %

0

Year87 88 89 90 91 92 93 94 95 96

36Adapted from (Haley 1996)

Cost of Quality and Maturity Levels

• A study of the Software Engineering Institute (SEI) showed that rework varies between 15% and 25% of the cost of

CMM M t it L l P t f R k

that rework varies between 15% and 25% of the cost of developing a CMM Maturity Level of 3.

CMM Maturity Level Percentage of Rework

1 ≥ 50 %2 25 % to 50 %3 15 % to 25 %4 5 % to 15 %5 ≤ 5 %

11 %

Relationship between CMM Maturity Levels and percentage of rework

(Gibson et al. 2006) 37

Recommandations1. Continue data collection and CoSQ Measurement

– Continue the data collection using procedures and methods (for example use of tools database structures) within the SDGexample use of tools, database structures) within the SDG.

2. Control activities related to the correction of problems– Most defects are found mainly during the execution of software

i f d i d di krequirements, software design and coding tasks• Reduce the defects by increasing prevention effort

– e.g. Execute more peer reviews (walk-through, inspection)

3. Present the CoSQ measurement results to Management– To enable them to develop better budgets for business process

improvementimprovement. – Establish a scoreboard to display the CoSQ and the results of

process improvement activities to illustrate the relationships between thembetween them.


Recommandations4. Measure the CoSQ at other Bombardier Transportation sites

– To identify opportunities for reducing costs,T id b i b d d l d li i i i– To provide a basis to budget development and quality activities,

– To use the results of CoSQ measures to improve processes.

OITasks

OutputsInputs

Project WBSCoSQ Report

1. Identify CoSQ tasks, using Project Plan and WBS

2. Record CoSQ effort by category

3. Apply weighting rules CoSQ Measurement Process

Typical CoSQ list of tasks

Weighting rules

Findings

Recommendations

pp y g g

4. Compute the CoSQ

5. Develop findings and recommendations

6. Review findings and recommendations with management

Entry Criteria Exit Criteria

management

7. Produce final CoSQ Project Report

8. Archive Report


Entry CriteriaMeasures

Exit Criteria

Project Plan Approved CoSQ Project Report Approved

Effort (staff‐hours)

C t tIngeniería para la Industria

Content





• Conclusion

40

ConclusionConclusion• The concept of Cost of Software Quality is a powerful

tool to help quantify the rework indentify weaknessestool to help quantify the rework, indentify weaknesses and prevention activities,

• The Software Quality Assurance courses we teach help Q y psoftware engineers in understanding the value of the quality assurance practices learned in class and put into

ti i th l b t ipractice in the laboratories,• The Case study at Bombardier Transportation

demonstrated the business value of the measurement ofdemonstrated the business value of the measurement of the Cost of Software Quality.

Best Approach Better Faster Cheaper

41

Best Approach = Better Faster Cheaper


Th k f iThank you for your attention

42

Contact Information

• Claude Y Laporte– Voice: + 1 514 396 8956Voice: + 1 514 396 8956– E-Mail: [email protected]– Web: http://profs.etsmtl.ca/claporte/English/index.html

• Public site of WG 24l k i i l d– Free access to Deployment Packages, presentation material and

articles:• http://profs.logti.etsmtl.ca/claporte/English/VSE/index.htmlp p g p g

43

References• Fanmuy, G., L’Ingénierie et Management des Systèmes pour les PME/TPE et petits projets, Association Française d'Ingénierie Système

(AFIS)/International Council on Systems Engineering (INCOSE) May 24th 2011 Paris France(AFIS)/International Council on Systems Engineering (INCOSE), May 24th, 2011, Paris, France.• Gibson, D.L., D.R. Goldenson, and K. Kost, Performance Results of CMMI®-Based Process Improvement. Technical Report, CMU/SEI-2006-

TR-004, ESC-TR-2006-004, 2006• Haley, T., Ireland, B., Wojtaszek, E., Nash, D., Dion, R., Raytheon Electronic Systems Experience in Software Process Improvement, Technical

Report CMU/SEI-95-TR-017, Software Engineering Institute, November 1995.• Iberle K., « But Will It Work For Me », Proceedings of the Pacific Northwest Software Quality Conference, p. 377-398, Portland, United States , , g Q y , p , ,

2002,• Iberle K., « They Don’t Care About Quality », Proceedings of STAR East, Orlando, United States, 2003.• ISO/IEC JTC1/SC7 N3288, New Work Item Proposal – Software Life Cycles for Very Small Enterprises, Mai 2005.• ISO/IEC 12207:2008, Information technology – Software life cycle processes, International Organization for Standardization/ International

Electrotechnical Commission: Geneva, Switzerland. • ISO/IEC 29110 - Lifecycle Profiles for Very Small Entities (VSEs) – Part 1: Overview. International Organization for

Standardization/International Electrotechnical Commission: Geneva, Switzerland.• ISO/IEC 15289:2006 - Systems and software engineering - Content of systems and software life cycle process information products

(Documentation)• Kabli, S., Conception, réalisation et mise a l’essai de trousses de déploiement pour faciliter et accélérer l’implémentation de la norme ISO/CEI

20000 l è i ÉTS 200920000 par les très petites structures, ÉTS, 2009.• Konrad, M., Overview of CMMI Model, Presentation to the Montréal SPIN, November 21, 2000, Montréal, Canada.• Kroll, P.; Kruchten, P.; The Rational Unified Process Made Easy – A Practionner’s Guide to the RUP.; Addison-Wesley, 2003 • Laporte, Claude Y., Berrhouma, Nabil, Doucet, Mikel, Palza-Vargas, Edgardo, "Measuring the Cost of Software Quality of a Large Software

Project at Bombardier Transportation", Software Quality Professional Journal, ASQ, Vol. 14, number 3, June 2012, p 14-31L t Cl d Y F G thi Pt k K Th D l t f S t E i i I t ti l St d d d S t T l f V• Laporte, Claude, Y, Fanmuy, Gauthier, Ptack, Ken, The Development of Systems Engineering International Standards and Support Tools for Very Small Enterprises, 22nd Annual International Symposium of the International Council on Systems Engineering, Rome, July 9-12, 2012.

• Laporte, Claude Y., Berrhouma, Nabil, Doucet, Mikel, Palza-Vargas, Edgardo, "Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation", Software Quality Professional Journal, ASQ, Vol. 14, number 3, June 2012, p 14-31.

• Laporte, C.Y., Alexandre, S., O’Connor, R., A Software Engineering Lifecycle Standard for Very Small Enterprises, in R.V. O’Connor et al. (Eds.): EuroSPI 2008, CCIS 16, pp. 129–141.(Eds.): EuroSPI 2008, CCIS 16, pp. 129 141.

• Long, L., The Critical Need for Software Engineering Education, Crosstalk - The Journal of Defense Software Engineering, January 2008, pp 6-10.

• Reifer D., « Let the Numbers Do the Talking », Crosstalk – The Journal of Defense Software Engineering, march 2002.• Rogers, Everett M., Diffusion of Innovations, fifth edition, Free Press, New York, 2003. 44

Ingeniería para la Industria - ÉTS Montréal€¦ · Ingeniería para la Industria Qy of a Large...

Documents

Transcript of Ingeniería para la Industria - ÉTS Montréal€¦ · Ingeniería para la Industria Qy of a Large...