1

COSC 4406COSC 4406

Software Software EngineeringEngineering

Haibin Zhu, Ph.D.Haibin Zhu, Ph.D.Dept. of Computer Science and mathematics, Nipissing University, Dept. of Computer Science and mathematics, Nipissing University,

100 College Dr., North Bay, ON P1B 8L7, Canada, 100 College Dr., North Bay, ON P1B 8L7, Canada, haibinz@nipissingu.ca, http://www.nipissingu.ca/faculty/haibinz, http://www.nipissingu.ca/faculty/haibinz

2

Lecture 15Lecture 15

Risk Management Risk Management and Quality and Quality

ManagementManagementRef. Chap. 25 & 26Ref. Chap. 25 & 26

3

Project RisksProject Risks

What can go wrong?What can go wrong?What is the likelihood?What is the likelihood?What will the damage be?What will the damage be?What can we do about it?What can we do about it?

4

Reactive Risk Reactive Risk ManagementManagement

project team reacts to risks when they occurproject team reacts to risks when they occur mitigation—plan for additional resources in mitigation—plan for additional resources in

anticipation of fire fightinganticipation of fire fighting fix on failure—resource are found and applied fix on failure—resource are found and applied

when the risk strikeswhen the risk strikes crisis management—failure does not respond crisis management—failure does not respond

to applied resources and project is in jeopardyto applied resources and project is in jeopardy

5

Proactive Risk Proactive Risk ManagementManagement

formal risk analysis is performedformal risk analysis is performed organization corrects the root causes of riskorganization corrects the root causes of risk

TQM concepts and statistical SQATQM concepts and statistical SQA examining risk sources that lie beyond the bounds examining risk sources that lie beyond the bounds

of the softwareof the software developing the skill to manage change developing the skill to manage change

6

Seven PrinciplesSeven Principles

Maintain a global perspectiveMaintain a global perspective—view software risks within the context of —view software risks within the context of system and the business problem system and the business problem

Take a forward-looking viewTake a forward-looking view—think about the risks that may arise in the —think about the risks that may arise in the future; establish contingency plans future; establish contingency plans

Encourage open communicationEncourage open communication—if someone states a potential risk, don’t —if someone states a potential risk, don’t discount it. discount it.

IntegrateIntegrate—a consideration of risk must be integrated into the software —a consideration of risk must be integrated into the software processprocess

Emphasize a continuous processEmphasize a continuous process—the team must be vigilant throughout —the team must be vigilant throughout the software process, modifying identified risks as more information is the software process, modifying identified risks as more information is known and adding new ones as better insight is achieved.known and adding new ones as better insight is achieved.

Develop a shared product visionDevelop a shared product vision—if all stakeholders share the same —if all stakeholders share the same vision of the software, it likely that better risk identification and assessment vision of the software, it likely that better risk identification and assessment will occur.will occur.

Encourage teamworkEncourage teamwork—the talents, skills and knowledge of all stakeholder —the talents, skills and knowledge of all stakeholder should be pooledshould be pooled

7

RISK

Risk Management Risk Management ParadigmParadigm

controlcontrol

identifyidentify

analyzeanalyze

planplan

tracktrack

8

Risk IdentificationRisk Identification Product sizeProduct size—risks associated with the overall size of the software to —risks associated with the overall size of the software to

be built or modified.be built or modified. Business impactBusiness impact—risks associated with constraints imposed by —risks associated with constraints imposed by

management or the marketplace.management or the marketplace. Customer characteristicsCustomer characteristics—risks associated with the sophistication of —risks associated with the sophistication of

the customer and the developer's ability to communicate with the the customer and the developer's ability to communicate with the customer in a timely manner.customer in a timely manner.

Process definitionProcess definition—risks associated with the degree to which the —risks associated with the degree to which the software process has been defined and is followed by the software process has been defined and is followed by the development organization.development organization.

Development environmentDevelopment environment—risks associated with the availability and —risks associated with the availability and quality of the tools to be used to build the product.quality of the tools to be used to build the product.

Technology to be builtTechnology to be built—risks associated with the complexity of the —risks associated with the complexity of the system to be built and the "newness" of the technology that is system to be built and the "newness" of the technology that is packaged by the system.packaged by the system.

Staff size and experienceStaff size and experience—risks associated with the overall technical —risks associated with the overall technical and project experience of the software engineers who will do the and project experience of the software engineers who will do the work.work.

9

Assessing Project Risk-IAssessing Project Risk-I

1.1. Have top software and customer managers Have top software and customer managers formally committed to support the project?formally committed to support the project?

2.2. Are end-users enthusiastically committed to the Are end-users enthusiastically committed to the project and the system/product to be built?project and the system/product to be built?

3.3. Are requirements fully understood by the Are requirements fully understood by the software engineering team and their customers?software engineering team and their customers?

4.4. Have customers been involved fully in the Have customers been involved fully in the definition of requirements?definition of requirements?

5.5. Do end-users have realistic expectations?Do end-users have realistic expectations?

10

Assessing Project Risk-IIAssessing Project Risk-II

6.6. Is project scope stable?Is project scope stable?7.7. Does the software engineering team have the Does the software engineering team have the

right mix of skills?right mix of skills?8.8. Are project requirements stable?Are project requirements stable?9.9. Does the project team have experience with the Does the project team have experience with the

technology to be implemented?technology to be implemented?10.10. Is the number of people on the project team Is the number of people on the project team

adequate to do the job?adequate to do the job?11. Do all customer/user constituencies agree on

the importance of the project and on the requirements for the system/product to be built?

11

Risk ComponentsRisk Components

performance riskperformance risk—the degree of uncertainty that —the degree of uncertainty that the product will meet its requirements and be fit the product will meet its requirements and be fit for its intended use.for its intended use.

cost riskcost risk—the degree of uncertainty that the —the degree of uncertainty that the project budget will be maintained.project budget will be maintained.

support risksupport risk—the degree of uncertainty that the —the degree of uncertainty that the resultant software will be easy to correct, adapt, resultant software will be easy to correct, adapt, and enhance.and enhance.

schedule riskschedule risk—the degree of uncertainty that the —the degree of uncertainty that the project schedule will be maintained and that the project schedule will be maintained and that the product will be delivered on time.product will be delivered on time.

12

Risk ProjectionRisk Projection

Risk projectionRisk projection, also called , also called risk estimation,risk estimation, attempts to rate each risk in two waysattempts to rate each risk in two ways the likelihood or probability that the risk is real the likelihood or probability that the risk is real the consequences of the problems associated with the the consequences of the problems associated with the

risk, should it occur. risk, should it occur. There are four risk projection steps:There are four risk projection steps:

establish a scale that reflects the perceived likelihood of a riskestablish a scale that reflects the perceived likelihood of a risk delineate the consequences of the riskdelineate the consequences of the risk estimate the impact of the risk on the project and the product,estimate the impact of the risk on the project and the product, note the overall accuracy of the risk projection so that there note the overall accuracy of the risk projection so that there

will be no misunderstandings.will be no misunderstandings.

13

Building a Risk TableBuilding a Risk Table

RiskRiskMitigationMitigationMonitoringMonitoring

& & ManagementManagement

1:Catastropic2: Critical3: marginal4: negligible

14

Building the Risk TableBuilding the Risk Table

Estimate the Estimate the probabilityprobability of occurrence of occurrence Estimate the Estimate the impactimpact on the project on a on the project on a

scale of 1 to 4, wherescale of 1 to 4, where 4 = low impact on project success4 = low impact on project success 1= catastrophic impact on project success1= catastrophic impact on project success

sort the table by probability and impactsort the table by probability and impact

15

Risk Exposure (Impact)Risk Exposure (Impact)

The overall risk exposure, RE, is determined using the following relationship [HAL98]:

RE = P x C

where P is the probability of occurrence for a risk, and C is the cost to the project should the risk occur.

16

Risk Exposure ExampleRisk Exposure Example

Risk identification.Risk identification. Only 70 percent of the software Only 70 percent of the software components scheduled for reuse will, in fact, be integrated into components scheduled for reuse will, in fact, be integrated into the application. The remaining functionality will have to be the application. The remaining functionality will have to be custom developed.custom developed.

Risk probability.Risk probability. 80% (likely). 80% (likely). Risk impact.Risk impact. 60 reusable software components were planned. If 60 reusable software components were planned. If

only 70 percent can be used, 18 components would have to be only 70 percent can be used, 18 components would have to be developed from scratch (in addition to other custom software that developed from scratch (in addition to other custom software that has been scheduled for development). Since the average has been scheduled for development). Since the average component is 100 LOC and local data indicate that the software component is 100 LOC and local data indicate that the software engineering cost for each LOC is $14.00, the overall cost (impact) engineering cost for each LOC is $14.00, the overall cost (impact) to develop the components would be 18 x 100 x 14 = $25,200.to develop the components would be 18 x 100 x 14 = $25,200.

Risk exposure. Risk exposure. RE = 0.80 x 25,200 ~ $20,200. RE = 0.80 x 25,200 ~ $20,200.

17

mitigationmitigation—how can we avoid the risk?—how can we avoid the risk? monitoringmonitoring—what factors can we track that —what factors can we track that

will enable us to determine if the risk is will enable us to determine if the risk is becoming more or less likely?becoming more or less likely?

managementmanagement—what contingency plans do —what contingency plans do we have if the risk becomes a reality?we have if the risk becomes a reality?

Risk Mitigation, Risk Mitigation, Monitoring,Monitoring,

and Management and Management

18

Risks due toRisks due to

Product SizeProduct Size Business ImpactBusiness Impact the Customerthe Customer Process MaturityProcess Maturity TechnologyTechnology Staff/PeopleStaff/People

19

Risk Due to Product SizeRisk Due to Product Size

• • estimated size of the product in LOC or FP?estimated size of the product in LOC or FP?

• • estimated size of product in number of programs, estimated size of product in number of programs, files, transactions?files, transactions?

• • percentage deviation in size of product from percentage deviation in size of product from average for previous products?average for previous products?

• • size of database created or used by the product?size of database created or used by the product?

• • number of users of the product?number of users of the product?

• • number of projected changes to the requirements number of projected changes to the requirements for the product? before delivery? after delivery?for the product? before delivery? after delivery?

• • amount of reused software?amount of reused software?

Attributes that affect risk:Attributes that affect risk:

20

Risk Due to Business Risk Due to Business ImpactImpact

• • affect of this product on company revenue?affect of this product on company revenue?• • visibility of this product by senior management?visibility of this product by senior management?• • reasonableness of delivery deadline?reasonableness of delivery deadline?

• • number of customers who will use this product number of customers who will use this product

• • interoperability constraintsinteroperability constraints

• • sophistication of end users?sophistication of end users?

• • amount and quality of product documentation that amount and quality of product documentation that must be produced and delivered to the customer?must be produced and delivered to the customer?

• • governmental constraintsgovernmental constraints

• • costs associated with late delivery?costs associated with late delivery?

• • costs associated with a defective product?costs associated with a defective product?

Attributes that affect risk:Attributes that affect risk:

21

Risks Due to the Risks Due to the CustomerCustomer

• • Have you worked with the customer in the past?Have you worked with the customer in the past?

• • Does the customer have a solid idea of requirements?Does the customer have a solid idea of requirements?

• • Has the customer agreed to spend time with you? Has the customer agreed to spend time with you?

• • Is the customer willing to participate in reviews?Is the customer willing to participate in reviews?

• • Is the customer technically sophisticated?Is the customer technically sophisticated?

• • Is the customer willing to let your people do their Is the customer willing to let your people do their job—that is, will the customer resist looking over your job—that is, will the customer resist looking over your shoulder during technically detailed work?shoulder during technically detailed work?

• • Does the customer understand the software Does the customer understand the software engineering process?engineering process?

Questions that must be answered:Questions that must be answered:

22

Risks Due to Process Risks Due to Process MaturityMaturity

• • Have you established a common process framework? Have you established a common process framework? • • Is it followed by project teams?Is it followed by project teams?• • Do you have management support for Do you have management support for software engineering software engineering • • Do you have a proactive approach to SQA? Do you have a proactive approach to SQA? • • Do you conduct formal technical reviews?Do you conduct formal technical reviews?

• • Are CASE tools used for analysis, design and Are CASE tools used for analysis, design and testing?testing?• • Are the tools integrated with one another?Are the tools integrated with one another?

• • Have document formats been established?Have document formats been established?

Questions that must be answered:Questions that must be answered:

23

Technology RisksTechnology Risks

• • Is the technology new to your organization?Is the technology new to your organization?• • Are new algorithms, I/O technology required?Are new algorithms, I/O technology required? • • Is new or unproven hardware involved?Is new or unproven hardware involved?• • Does the application interface with new software?Does the application interface with new software?• • Is a specialized user interface required? Is a specialized user interface required? • • Is the application radically different?Is the application radically different?• • Are you using new software engineering methods?Are you using new software engineering methods?

• • Are you using unconventional software development Are you using unconventional software development methods, such as formal methods, AI-based approaches, methods, such as formal methods, AI-based approaches, artificial neural networks?artificial neural networks?

• • Are there significant performance constraints?Are there significant performance constraints?

• • Is there doubt the functionality requested is "do-able?"Is there doubt the functionality requested is "do-able?"

Questions that must be answered:Questions that must be answered:

24

Staff/People RisksStaff/People Risks

• • Are the best people available?Are the best people available?• • Does staff have the right skills?Does staff have the right skills?• • Are enough people available?Are enough people available?• • Are staff committed for entire duration?Are staff committed for entire duration?• • Will some people work part time? Will some people work part time? • • Do staff have the right expectations?Do staff have the right expectations?• • Have staff received necessary training?Have staff received necessary training?• • Will turnover among staff be low?Will turnover among staff be low?

Questions that must be answered:Questions that must be answered:

26

QualityQuality

The The American Heritage DictionaryAmerican Heritage Dictionary defines defines qualityquality as as ““a characteristic or attribute of something.” a characteristic or attribute of something.”

For software, two kinds of quality may be For software, two kinds of quality may be encountered: encountered: Quality of designQuality of design encompasses requirements, encompasses requirements,

specifications, and the design of the system. specifications, and the design of the system. Quality of conformanceQuality of conformance is an issue focused primarily on is an issue focused primarily on

implementation.implementation. user satisfaction = compliant product + good quality + user satisfaction = compliant product + good quality +

delivery within budget and scheduledelivery within budget and schedule

27

Software QualitySoftware Quality

Conformance to explicitly stated functional Conformance to explicitly stated functional and performance requirements, explicitly and performance requirements, explicitly documented development standards, and documented development standards, and implicit characteristics that are expected of implicit characteristics that are expected of all professionally developed software. all professionally developed software.

28

Cost of QualityCost of Quality Prevention costsPrevention costs include include

quality planningquality planning formal technical reviewsformal technical reviews test equipmenttest equipment TrainingTraining

Internal failure costsInternal failure costs include include reworkrework repairrepair failure mode analysisfailure mode analysis

External failure costsExternal failure costs are are complaint resolutioncomplaint resolution product return and replacementproduct return and replacement help line supporthelp line support warranty workwarranty work

29

Software Quality Software Quality AssuranceAssurance

FormalTechnicalReviews

Test Planning& ReviewMeasurement

Analysis&

Reporting

ProcessDefinition &Standards

30

Role of the SQA Group-IRole of the SQA Group-I

Prepares an SQA plan for a project. Prepares an SQA plan for a project. The plan identifiesThe plan identifies

evaluations to be performedevaluations to be performed audits and reviews to be performedaudits and reviews to be performed standards that are applicable to the projectstandards that are applicable to the project procedures for error reporting and trackingprocedures for error reporting and tracking documents to be produced by the SQA groupdocuments to be produced by the SQA group amount of feedback provided to the software project teamamount of feedback provided to the software project team

Participates in the development of the project’s software Participates in the development of the project’s software process description.process description.

The SQA group reviews the process description for compliance with The SQA group reviews the process description for compliance with organizational policy, internal software standards, externally imposed organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project standards (e.g., ISO-9001), and other parts of the software project plan.plan.

31

Role of the SQA Group-IIRole of the SQA Group-II Reviews software engineering activities to verify compliance Reviews software engineering activities to verify compliance

with the defined software process.with the defined software process. identifies, documents, and tracks deviations from the process and identifies, documents, and tracks deviations from the process and

verifies that corrections have been made.verifies that corrections have been made. Audits designated software work products to verify Audits designated software work products to verify

compliance with those defined as part of the software process.compliance with those defined as part of the software process. reviews selected work products; identifies, documents, and tracks reviews selected work products; identifies, documents, and tracks

deviations; verifies that corrections have been madedeviations; verifies that corrections have been made periodically reports the results of its work to the project manager.periodically reports the results of its work to the project manager.

Ensures that deviations in software work and work products Ensures that deviations in software work and work products are documented and handled according to a documented are documented and handled according to a documented procedure.procedure.

Records any noncompliance and reports to senior Records any noncompliance and reports to senior management.management.

Noncompliance items are tracked until they are resolved.Noncompliance items are tracked until they are resolved.

32

Why SQA Activities Pay Why SQA Activities Pay Off?Off?cost to findcost to find

and fix a defectand fix a defect

100100

1010

loglogscalescale

11

Req.Req.DesignDesign

codecodetesttest

systemsystemtesttest

fieldfielduseuse

113-63-6

101015-4015-40

30-7030-70

40-10040-100

34

What Are Reviews?What Are Reviews? a meeting conducted by technical people a meeting conducted by technical people

for technical peoplefor technical people a technical assessment of a work a technical assessment of a work

product created during the software product created during the software engineering processengineering process

a software quality assurance mechanisma software quality assurance mechanism a training grounda training ground

35

What Reviews Are NotWhat Reviews Are Not

A project summary or progress assessmentA project summary or progress assessment A meeting intended solely to impart informationA meeting intended solely to impart information A mechanism for political or personal reprisal!A mechanism for political or personal reprisal!

36

The PlayersThe Players

reviewreviewleaderleader

producerproducer

recorderrecorder reviewerreviewer

standards bearer (SQA)standards bearer (SQA)

maintenance maintenance oracleoracle

user repuser rep

37

Conducting the Conducting the ReviewReviewbe prepared—evaluate be prepared—evaluate

product before the reviewproduct before the review

review the product, not review the product, not the producerthe producer

keep your tone mild, ask keep your tone mild, ask questions instead of questions instead of making accusationsmaking accusations

stick to the review agendastick to the review agenda

raise issues, don't resolve themraise issues, don't resolve them

avoid discussions of style—stick to technical avoid discussions of style—stick to technical correctnesscorrectness

schedule reviews as project tasksschedule reviews as project tasks

record and report all review resultsrecord and report all review results

1.1.

2.2.

3.3.

4.4.

5.5.

6.6.

7.7.

8.8.

39

Sample-Driven Reviews (SDRs)Sample-Driven Reviews (SDRs)

SDRs attempt to quantify those work products that are primary SDRs attempt to quantify those work products that are primary targets for full FTR (Formal Technical Review)s.targets for full FTR (Formal Technical Review)s.

To accomplish this …To accomplish this … Inspect a fraction aInspect a fraction aii of each software work product, of each software work product, i. i. Record the Record the

number of faults, fnumber of faults, fii found within a found within aii.. Develop a gross estimate of the number of faults within work Develop a gross estimate of the number of faults within work

product product ii by multiplying f by multiplying fii by 1/a by 1/aii.. Sort the work products in descending order according to the Sort the work products in descending order according to the

gross estimate of the number of faults in each.gross estimate of the number of faults in each. Focus available review resources on those work products that Focus available review resources on those work products that

have the highest estimated number of faults.have the highest estimated number of faults.

40

Metrics Derived from ReviewsMetrics Derived from Reviews

inspection time per page of documentationinspection time per page of documentationinspection time per KLOC or FPinspection time per KLOC or FP

errors uncovered per reviewer hourerrors uncovered per reviewer hourerrors uncovered per preparation hourerrors uncovered per preparation hour

errors uncovered per SE task (e.g., design)errors uncovered per SE task (e.g., design)number of minor errors (e.g., typos)number of minor errors (e.g., typos)

number of errors found during preparationnumber of errors found during preparation

number of major errorsnumber of major errors (e.g., nonconformance to req.) (e.g., nonconformance to req.)

inspection effort per KLOC or FPinspection effort per KLOC or FP

41

Statistical Statistical SQASQA

ProductProduct& Process& Process

measurement

... an understanding of how to improve quality ...

Collect information on all defectsFind the causes of the defectsMove to provide fixes for the process

Page 728-729

42

Six-Sigma for Software Six-Sigma for Software EngineeringEngineering

The term “six sigma” is derived from six standard deviations—3.4 The term “six sigma” is derived from six standard deviations—3.4 instances (defects) per million occurrences—implying an instances (defects) per million occurrences—implying an extremely high quality standard. extremely high quality standard.

The Six Sigma methodology defines three core steps:The Six Sigma methodology defines three core steps: DefineDefine customer requirements and deliverables and project goals via customer requirements and deliverables and project goals via

well-defined methods of customer communicationwell-defined methods of customer communication MeasureMeasure the existing process and its output to determine current the existing process and its output to determine current

quality performance (collect defect metrics)quality performance (collect defect metrics) AnalyzeAnalyze defect metrics and determine the vital few causes.defect metrics and determine the vital few causes.

Two additional: Two additional: ImproveImprove the process by eliminating the root causes of defects. the process by eliminating the root causes of defects. Control the process to ensure that future work does not reintroduce

the causes of defects.

http://mathworld.wolfram.com/StandardDeviation.html

43

Software ReliabilitySoftware Reliability

A simple measure of reliability is A simple measure of reliability is mean-time-mean-time-between-failurebetween-failure (MTBF), where (MTBF), where

MTBF = MTTF + MTTRMTBF = MTTF + MTTR

The acronyms MTTF and MTTR are mean-time-to-failure and mean-time-to-repair, respectively.

Software availabilitySoftware availability is the probability that a is the probability that a program is operating according to requirements program is operating according to requirements at a given point in time and is defined asat a given point in time and is defined as

Availability = [MTTF/(MTTF + MTTR)] x 100%

44

Software SafetySoftware Safety

Software safetySoftware safety is a software quality assurance is a software quality assurance activity that focuses on the identification and activity that focuses on the identification and assessment of potential hazards that may affect assessment of potential hazards that may affect software negatively and cause an entire system software negatively and cause an entire system to fail. to fail.

If hazards can be identified early in the software If hazards can be identified early in the software process, software design features can be process, software design features can be specified that will either eliminate or control specified that will either eliminate or control potential hazards.potential hazards.

45

ISO 9001:2000 StandardISO 9001:2000 Standard

ISO 9001:2000 is the quality assurance standard ISO 9001:2000 is the quality assurance standard that applies to software engineering. that applies to software engineering.

The standard contains 20 requirements that must The standard contains 20 requirements that must be present for an effective quality assurance be present for an effective quality assurance system. system.

The requirements delineated by ISO 9001:2000 The requirements delineated by ISO 9001:2000 address topics such as address topics such as management responsibility, quality system, contract management responsibility, quality system, contract

review, design control, document and data control, review, design control, document and data control, product identification and traceability, process control, product identification and traceability, process control, inspection and testing, corrective and preventive inspection and testing, corrective and preventive action, control of quality records, internal quality action, control of quality records, internal quality audits, training, servicing, and statistical techniques.audits, training, servicing, and statistical techniques.

46

SummarySummary

Reactive/proactive Risk StrategiesReactive/proactive Risk Strategies Risk IdentificationRisk Identification Risk ProjectionRisk Projection Risk Mitigation, Monitoring and ManagementRisk Mitigation, Monitoring and Management Quality ConceptsQuality Concepts SQASQA Software reviewsSoftware reviews Statistical SQAStatistical SQA Software reliability and safetySoftware reliability and safety ISO 9000 Quality StandardsISO 9000 Quality Standards