Quality in Software IndustryRicha Goel

1

Contents

•Quality : Meaning•Quality in Software Industry•Implementation of Quality•Measuring Quality•Various Types of Metrics•Case Studies•Software Quality Assurance

2

What is quality?

•A product should meet its specification.•This is problematical for software systems

▫Some quality requirements are difficult to specify in an unambiguous way;

▫Software specifications are usually incomplete and often inconsistent.

3

4

Quality is…..

Invisible when GOOD Impossible to ignore when BAD

•Meeting Our Customer’s Requirements

•Doing Things Right the First Time; Freedom from Failure (Defects)

•Consistency (Reduction in Variation)

•Continuous Improvement

•Quality in Everything We Do

Total Quality Is…

5

Quality of a product is satisfied customer

6

5

Software Quality ???

•Software Quality refers to any measurable characteristics such as correctness, maintainability, portability, testability, usability, reliability, efficiency, integrity, reusability and interoperability.

7

Software quality management

•Concerned with ensuring that the required level of quality is achieved in a software product.

•Involves defining appropriate quality standards and procedures and ensuring that these are followed.

•Should aim to develop a ‘quality culture’ where quality is seen as everyone’s responsibility.

8

9

Absence of Quality…

Definition: Software Quality

•What is software quality?•What are the attributes of quality for

software?

?This is high quality software because...

10

Software Quality AttributesPortability

Reliability

Efficiency

Usability

Testability

Understandability

Modifiability

11

Common problems in software processes•Cost overruns•Schedule delays•Low productivity rate•Poor quality - in software, maintenance or

fixes

12

6

Quality Concepts Quality of Design refers to the characteristics

that designer’s specify for an item.

Quality Control is the series of inspections, reviews and tests used throughout the development cycle to ensure that each work product meets the requirements placed upon it.

Quality of Conformance is the degree to which the design specifications are followed during manufacturing.

13

Quality Concepts

• Quality of Design

Design

14

Software Quality MeasurementsWe best manage what we can measure. 

15

Quality Measurement• Measurement enables the Organization to

improve the software process; assist in planning, tracking and controlling the software project and assess the quality of the software thus produced. 

• Metrics are analyzed and they provide a dashboard to the management on the overall health of the process, project and product.

•  The validation of the metrics is a continuous process spanning multiple projects.

16

What to measure?

Why to measure?

How to measure?

17

What to check?

Should we track:•Number of tests?•Pass rate?•Plan versus actual?•Number of defects?•Code coverage?•Functional coverage?•Performance figures?

18

What to measure?

Resource

Defects

Code Process

19

Metric Classification•Products

▫Explicit results of software development activities

▫Deliverables, documentation, by products

•Processes▫Activities related to production of software

•Resources▫Inputs into the software development activities▫hardware, knowledge, people

20

Types of Software Metrics

•Product metrics – e.g., size, complexity, design features, performance, quality level

•Process metrics – e.g., effectiveness of defect removal, response time of the fix process

•Project metrics – e.g., number of software developers, cost, schedule, productivity

21

Why measure? Are we meeting our business objectives? Are our customers satisfied with our products and

services? Are we earning a fair return on our investments? Can we reduce the cost of producing the product or

service? How can we improve the response to our customers’

needs or increase the functionality of our products? How can we improve our competitive position? Are we achieving the growth required for survival?

Without the right information, you are just another person with an opinion

22

Why Measure Software?

•Determine the quality of the current product or process

•Predict qualities of a product/process

•Improve quality of a product/process

23

Why measure?

•By analyzing the metrics the organization can take corrective action to fix those areas in the process, project or product which are the cause of the software defects.

24

How to measure?

25

Purpose of Measuring Cost saving Employee satisfaction Customer satisfaction Quality

26

Purpose of Measuring• Cost saving

▫Quality of Product: Defects per KLOC or FP▫Project Status: Tracking against estimated

schedule, budget, size• Employee satisfaction

▫Work Effort: Each team member utilized• Customer satisfaction

▫Satisfaction: Survey (Six Months)• Quality

▫Quality of Process: COQ, ROI on QA, amount of rework, quality of team time and teamwork

27

Goal of metrics

•to improve product quality and development-team productivity

•concerned with productivity and quality measures· measures of SW development output as

function of effort and time· measures of usability

28

Help in…

•Allows manager to (1) assess status of ongoing project(2) track project risks(3) uncover problem areas(4) adjust tasks or workflow(5) evaluate team’s ability to control quality

29

Define Metrics toBe collected –

Project & Support Group Metrics

Store data forFuture useIn MetricsDatabase

Establish DataCollection Mechanism

And send the metrics data to QA Group

Arrive at Organizational Process Capability Baseline

Analyze Metrics

30

Metrics Data Capture – Frequency & Sources What should be the frequency of collecting the data?

Sources for the data capture of the Metrics

Timesheets Project plan, Schedule (MPP, XLS) Defect tracking system Ticket tracking sheet

31

Goal 1: Improve software project planning How good is the software

effort planning?

How good is the software scheduling?

32

Effort Variance MetricEffort Variance (in %)=[Actual Effort – Planned(Estimated) Effort]*100 [Planned ( Estimated ) Effort ]

Actual Effort (in hrs) = Planning, Tracking, Configuration, Defect Prevention, Requirements, Design, Coding, Review, Rework & Testing

Planned Effort (in hrs) = Planning, Tracking, Configuration, Defect Prevention, Requirements, Design, Coding, Review, Rework & Testing

This is to measure the variance of effort compared with the estimated effort in terms of man days spend with respect to daily hrs

33

Schedule Variance MetricSchedule Variance =[Actual Duration - Planned Duration] *

100 (in %) [ Planned Duration ]

Actual Duration (in Days) : Actual end date - Planned start date

Planned Duration (in Days) : Planned end date - Planned start date

This is to measure the variance of schedule compared with the estimated

schedule in terms of duration in calendar days

34

PhaseRequirementActual

Schedule Planned Schedule

Design Actual Schedule

Planned Schedule

Code/Unit Test Planned Schedule

Actual Schedule

Requirement

Design

Coding

Testing

Delivery / Release

Schedule Metrics - SampleProject Code and Name:

Prepared By : Reviewed By (MR) : Sign Date : Data Given by : Sign/Date :

35

Goal 2 : Improve ProductivityWhat is the productivity in different types of projects?

36

Productivity MetricsDevelopment = ____Total project effort ____

Project size in LOC or FP

Testing = Total effort spent in execution of test cases Total number of test cases executed

Maintenance = Cumulative Actual Effort for a category of CRs completed

Number of CRs completed in the category

37

Goal 3 : Reduce number of defects How many defects are we getting

with respect to the effort spent?

How efficient and Effective are our reviews?

38

Defect Density MetricA "Defect" is a:

Deviation from a standard Deviation from requirement Anything that causes customer dissatisfaction  

  

Defect Density (DD) = Total no. of defects

Total effort (In hrs)

Total no. of defects = Pre Delivery Defects ( reviews & testing ) + Post Delivery Defects ( reported after delivery to customer )

Total Effort ( in hrs ) = Planning, Tracking, Configuration Management, Defect Prevention, Requirements, Design, Coding, Review, Rework & Testing

39

Review Efficiency MetricReview efficiency = Review defects (including unit testing

defects)*100 Review effort ( including unit testing

effort )

Review Defects = No. of defects in ( Requirements, Design, CUT )

Review Effort (in hrs) = ( Requirements, Design, CUT ) Review Efforts

SRS review defects = 1 Design Review defects = 1 Code review defects = 1 Unit Testing defects = 0 Review Effort = 15

Review Efficiency will be 0.2

40

Review Effectiveness Metric Review effectiveness = Total no. of Review defects *

100 Total no. of defects

( Review + Testing )

DEFECT DATA

SRS review defects = 2 Design Review defects = 2 Code review defects = 4 Testing defects = 2 Review effectiveness will be

80%

41

Goal 4 : Reduce Cost of Quality What is the cost of detection (Appraisal Costs)?

What is the cost of Correction (Failure Costs)?

What is the cost of Prevention (Prevention Costs)?

42

Cost Of Quality (COQ) Metric

COQ ( in % ) = ( AC +PRC +FC ) * 100 ( AC +PRC +FC + PDC )

Appraisal Cost ( AC ) = Review + Audit + Testing

Prevention Cost ( PRC ) = Project Management + Training +

CM + Defect Prevention + RFC / change request

Failure Cost ( FC ) = Rework + Idle Time + Complaints +

post sales defects

Production Cost ( PDC ) = Requirements, Design and Coding +

Tools/Scripts Development + manuals

43

Goal 5 : Improve customer satisfaction Are we meeting our commitments to the customer?

How satisfied are customers with our services and products?

44

Customer Satisfaction Metrics Customer Satisfaction Index / Rating

Rating on a scale of 1 – 5 through Customer Satisfaction Surveys

SLA Compliance [ ( Actual Incidents where SLA was met ) * 100 ] / [ Total

No. of incidents ] resolved

SLA Variance [ ( Actual Resolution Time – SLA ) * 100 ] / [ SLA ]

45

Tell me now… Find Cost Of Quality (COQ) ?

Appraisal Cost = 3 % Prevention Cost = 7 % Production Cost = 78 % Failure Cost = 12 %

Review efficiency in a project is 0.1, what does it mean?

  Defect Density helps in……Reducing rework and controlling effort variance and schedule variance

One Review defect has been detected after spending 10 hours of review efforts

COQ = 22 %

46

Case Studies

47

Management review report

•Progress reports •Periodic performance reports•Customer satisfaction feedback•Follow up reports•Review of significant findings

48

Software Development Projects

49

Development metrics

•In software development projects, we capture the efficiency of correctness and robustness of the code.

•We track them via:▫LOC (Line of Code)▫Function Points▫Cyclomatic Complexity 

50

Line of Code analysis

•Derived by normalizing (dividing) any direct measure (e.g. defects or human effort) associated with the product or project by LOC.

•Size oriented metrics are widely used but their validity and applicability is widely debated.

51

Function oriented Metrics• Function points are computed from direct

measures of the information domain of a business software application and assessment of its complexity.

• Once computed function points are used like LOC to normalize measures for software productivity, quality, and other attributes.

• The relationship of LOC and function points depends on the language used to implement the software.

52

Software Testing Projects

53

Check Points

• Main objectives of a project: High Quality & High

Productivity (Q&P)

• Quality has many dimensions

▫ reliability, maintainability, interoperability etc.

• More defects => more chances of failure => lesser

reliability

• Hence quality goal: Have as few defects as possible

in the delivered software!

54

Metrics for Software Testing

• Defect Removal Effectiveness

DRE= Defects removed during development phase x100% Defects latent in the product

Defects latent in the product = Defects removed during development

phase+ defects found later by user

• Efficiency of Testing Process (define size in KLoC or FP, Req.)

Testing Efficiency= Size of Software TestedResources used

55

Support Groups Metrics

56

SEPG & SQA Metrics Person / Process Trainings PI Index - Process Improvement suggestions per quarter SEPG Efforts

Total effort spent in SEPG Activities Total effort spent in SQA Activities

No. of NCs per project

57

ISS Metrics

% Calls resolved same day =

Number of calls ( 24hrs category ) resolved within one day

Number of user calls ( 24 Hrs category )

% Calls resolved in two days =

Number of calls ( 48hrs category ) resolved within two days

Number of user calls ( 48 Hrs category )

% Network up time = Network up time ( for the month )

Total Available time

58

RMG Metrics

Average hiring cost per hire = Total hiring cost for the month

Total no. of joinees

Selection ratio = Total No. of offers Total no. of candidates interviewed

Offer to joinee ratio = Total no. of joinees Total no. of Offers

Offer to Decline ratio = Total no. of Declines Total no. of Offers

59

Some important metricsIndicator Metric (Unit /

s) Formula Base Measure / s Source / s

Effort Effort Variance(%)

(Actual Effort – Planned Effort) * 100 / (Planned

Effort)

Actual EffortPlanned Effort

TimesheetProject

schedule

ScheduleSchedule Variance

(%)

(Actual End Date – Planned End Date) * 100 /

(Planned Project Duration)

Planned project duration

Planned End DateActual End Date

Project schedule

Size Size Variance (%)

(Actual Software Size – Estimated Software Size) * 100 / (Estimated

Software Size

Actual sizeEstimated size

Estimation sheet

ProductivityProductivity

(KLOC / mandayOr

FP / manday)

(Size) / (Effort) EffortSize

Estimation Sheet

60

Some important metricsIndicat

orMetric

(Unit / s) Formula Base Measure / s Source / s

Quality

Defect Density

(Defects / KLOC

Or

Defects / FP)

(Total No. of Defects) / (Size)

No. of total Defects (Pre

delivery + Post delivery)

Size

Review & testing Defect log

Estimation Sheet

Quality

Review Efficiency

(Defects / Manhour)

(Total no. of review defects) / (review

hours)

Total no. of review defects

Review effort in hours

Review & testing Defect log

QualityCost of Quality

(%)

(Appraisal efforts + Failure efforts +

Prevention efforts) * 100 / (Total Project

Effort)

Review efforts

Testing efforts

Rework efforts

Timesheet

Project schedule

61

Sample Tracking Sheet

62

Software Quality Assurance

63

64

What is Software Quality Assurance?

•Used to Monitor and Improve the Software Development Process

•Making Sure That Standards and Procedures are Followed

•Ensures that Problems are Found and Dealt with

•Orientated to ‘Prevention’

65

Standards and Procedures

•Framework for which Software Evolves•Standards

▫Established Criteria to which Software Products are Compared

•Procedures▫Established Criteria to which Development and

Control Procedures are Followed•SQA is based on the Following of

Standards and Procedures

66

Techniques•Audit

▫The Major Technique used in SQA▫Perform Product Evaluation and Process

Monitoring▫Performed Routinely throughout the Software

Development Process▫Look at a Process and/or Product in depth and

compare to Established Standards and Procedures

▫Provide an indication of the Quality and Status of the Software Product

67

Benefit of Software Quality Assurance in Projects

• Without SQA, many Software Groups would not reach their release goals/deadlines on time

• Lowers time spent on mundane areas and lets more time be focused on important areas

• Decreases the time from Development to Deployment

• Can help catch errors before they are too costly to fix

SQA Activities

•Document Auditing•Document Reviews•Metrics Calculation and Analysis•Meetings and Discussions

▫Final Reports Submission

68

Reviews

69

Types of Review

•Formal Review•Informal Review

70

Types of ReviewFormal Review• Planning• Documented• Thorough• Focused on a certain purpose

71

Types of ReviewInformal Review• Undocumented• Fast• Few defined procedures• Useful to check that the author is on track

72

73

Why do peer reviews?•To improve quality.•Catches 80% of all errors if done properly.•Catches both coding errors and design

errors.•Training and insurance.

Thank You!!

74