UNIT III SOFTWARE QUALITY ASSURANCE MODELS ... … · CS621 – Software Quality Management Unit -...

CS621 – Software Quality Management Unit - III

MTech CSE (PT, 2011-14) SRM, Ramapuram 1 hcr:innovationcse@gg

UNIT III SOFTWARE QUALITY ASSURANCE

MODELS Software Quality Assurance; Statistical Quality Assurance - Software Reliability, Models for Quality Assurance-

ISO-9000 - Series, CMM, SPICE, Malcolm Baldrige Award.

SOFTWARE QUALITY ASSURANCE Preventive Approach

It is the planned and systematic activities implemented within the quality systems

Why Software Quality Assurance is important

If team size grows and manager involves in other duties, then the monitoring of work is not done

To motivate the people to monitor each other

To ensure the inadequacies in the project or process or product

To ensure full compliance with the established standards

Responsibilities of SQA

Review the all developments and quality plans for completeness

Review all test plans for adherence to standards

Review a significant sample of all test results to determine adherence to plans

Periodically audit SCM performance to determine adherence to standards

Participate as inspection moderators in design and code inspections

Benefits of Quality assurance

An appropriate development methodology is in place

The projects use standards and procedures in the work

Independent reviews and audits are conducted

Documentation is produced to support maintenance and enhancement

The documentation is produced during the development

Mechanisms are in place and used to control changes

Testing emphasizes all the high risk product areas

Deviation from standards and procedures are exposed as soon as possible

The project is auditable by external professionals

The SQA plan and development plan are compatible

Eight Steps to launch a SQA program

1. Initiate SQA Program – Management commitment, Goal documentation, Identify the responsibilities and

team leader

2. Identify the SQA issues

3. Write a SQA program – Identify the standards and procedure, Define SQA audit and control activities

4. Establish Standards

5. Establish SQA functions

6. Conduct training and promote the SQA program

7. Implement the SQA plan – SQA activity is assigned to SQA personnel

8. Evaluate the SQA program – periodical audit, corrective action and preventive action

SOFTWARE RELIABILITY

Introduction

System – an entity that provides defined behavior at interfaces

o System is a hierarchy of subsystems, each subsystem being a system

Reliability of a system - its ability to provide failure-free operation

Failure – the system behavior is incorrect or not as expected; is a random phenomenon



The probability that an item will perform a required function without failure under stated conditions for a

stated period of time”

(Qualitative) The ability of a program to perform required function under stated conditions for a stated

period of time.

(Quantitative) The probability is a function of the inputs to and use of the system, as well as a function of

the existence of defects in the software.

– Error > Defect > Fault > Failure

Reliability Quantification

Reliability of a system defined as failure probability in a time period

R(t) = Prob that system has not failed by time t

For rel work, often distribution of R(t) is specified

Reliability can also be quantified by Mean Time to Failure (MTTF)

Also by failure rate (no of failures per unit time.)

From R(t), MTTF or failure rate can be determined

Under some assumptions, failure rate and MTTF are inversely related

Software Reliability

Software (un)reliability not caused due to aging but due to bugs

The more the bugs, the lesser the reliability of the software

Still failures seem random, hence rel theory can be applied

Two main threads

Software reliability modeling – how to model and predict sw rel

Improving sw reliability – by removing defects through program checking, verification, testing,…

Software systems often are one-off

o Measuring reliability in lab not practical as too much failure data is needed; requires time

Failures often result in fault removal, leading to reliability improvement

o Predicting future reliability from measured reliability is harder

Hence different models needed

Software Reliability Model

Prediction Model

o Historical data

o Prior to development or test phases, or as early as concept phase

Growth Model

o Data from software development effort

o Test phase

o Defects detected are removed

Determination Model

o Data from acceptance test effort

o Acceptance test phase

o Defects detected are NOT removed

Software Reliability Growth Models

Assume that reliability is a function of the defect level and as defects are removed, reliability improves

Model the failure-fix process of software evolution

Many models have been proposed in the last 3 decades

Model parameters determined from past data on failures and fixes

Probabilistic Model

o Black-Box Model

Stochastic Process

Non-Stochastic Process

o White-Box Model

Coverage-Based

Fuzzy Model

Nerve Network



Average Failure Rate of a MS Product

Reasons for this Phenomenon

Users learn with time and avoid failure causing situation

Users start with exploring more, then limit to some part of the product

o Most users use a few product features

Configuration related failures are much more in the start

These failures reduce with time

Sw Architecture

Architecture is the components in the system and how they are connected

Is decided very early in sw project

If reliability and performance can be modeled from architecture, can improve the architecture

Some work going on in arch. based perf. and rel modeling

Program Verification

Basic goal – to ensure that program is free of defects (bugs) as much as possible

Good program verification leads to higher reliability

Program Verification Techniques

Testing – program is executed with test data to find bugs

Static analysis – program source code is analyzed

Dynamic analysis – program run on some data and assertions made

Model checking

Formal verification

Reliability modelling

To predict reliability, current failure data is collected and used to infer future behaviour.

Examples of the use of such predictions include:-

o to determine at what point in time a particular level of reliability will be reached.

o to determine what level of reliability will have been reached by a certain point in time.

Failure intensity

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

1 2 3 4 5 6 7 8 9 10 11

Months frm release

Fail

ure

s/m

on

th/u

nit



Reliability curve

Measuring reliability

Can measure:-

o MTTF - Mean Time To Failure

o MTTR - Mean Time To Repair

o MTBF - Mean Time Between Failures = MTTF + MTTR

Various approaches exist:

o Use random inputs and measure defects

o Carry out complete observation

o Look at independent sets of tests

SOFTWARE QUALITY MODELS

Philip B. Crosby

Quality is a subjectively identified differently by each individual and institution

As this is not useful in software engineering quality must be defined as "conformance to requirements"

Nonconformance to requirements is the absense of quality, quality problems become nonconformance

problems, and quality becomes definable

W. Endwards Deming

Translating future needs of the user into measurable characteristics

Constantly changing based on "real world" - competitors, solutions, technology, price

Quality can be defined only in terms of the agent



Joseph M. Juran

Quality can be

o Those product features which meet the need of customers and thereby provide product satisfication

o Freedom from deficiencies

Quality in terms of satisfying customer expectations or specifications is not usable as it is very hard to

achieve

Quality is fitness for use

Boehm Quality Factors

The high-level characteristics address three main questions that a buyer of software has:

As-is utility : How well (easily, reliably, efficiently) can I use it as-is?

Maintainability: How easy is it to understand, modify and retest?

Portability : Can I still use it if I change my environment?

The intermediate level characteristic represents Boehm’s 7 quality factors that together represent the

qualities expected from a software system:

1. Portability (General utility characteristics): Code possesses the characteristic portability to the extent that it

can be operated easily and well on computer configurations other than its current one.



2. Reliability (As-is utility characteristics): Code possesses the characteristic reliability to the extent that it can

be expected to perform its intended functions satisfactorily.

3. Efficiency (As-is utility characteristics): Code possesses the characteristic efficiency to the extent that it

fulfills its purpose without waste of resources.

4. Usability (As-is utility characteristics, Human Engineering): Code possesses the characteristic usability to

the extent that it is reliable, efficient and human-engineered.

5. Testability (Maintainability characteristics): Code possesses the characteristic testability to the extent that it

facilitates the establishment of verification criteria and supports evaluation of its performance.

6. Understandability (Maintainability characteristics): Code possesses the characteristic understandability to

the extent that its purpose is clear to the inspector.

7. Flexibility (Maintainability characteristics, Modifiability): Code possesses the characteristic modifiability to

the extent that it facilitates the incorporation of changes, once the nature of the desired change has been

determined.

McCall’s Quality Factors

Also called as General Electrics Model.

This model was mainly developed for US military to bridge the gap between users and developers.

It mainly has 3 major representations for defining and identifying the quality of a software product

Product Revision

This identifies quality factors that influence the ability to change the software product.

Maintainability : Effort required to locate and fix a fault in the program within its operating environment.

Flexibility : The ease of making changes required as dictated by business by changes in the operating

environment.

Testability : The ease of testing program to ensure that it is error-free and meets its specification, i.e,

validating the software requirements.

Product Transition

This identifies quality factors that influence the ability to adapt the software to new environments.

Portability : The effort required to transfer a program from one environment to another.

Re-usability : The ease of reusing software in a different context.

Interoperability: The effort required to couple the system to another system.

Product Operations

This identifies quality factors that influence the extent to which the software fulfills its specification.

Correctness : The extent to which a functionality matches its specification.

Reliability : The systems ability not to fail/ the extent to which the system fails.

Efficiency : Further categorized into execution efficiency and storage efficiency and generally means the

usage of system resources, example: processor time, memory.

Integrity : The protection of program from unauthorized access.

Usability : The ease of using software.

Deming’s Fourteen Points

Refer to 5th unit



Deming’s Seven Deadly Diseases

Refer to 5th unit

Juran’s Contributions

Juran’s Three Basic Steps to Progress

Achieve structured improvements on a continual basis combined with dedication and a sense of urgency

Establish an extensive training program

Establish committment and leadership on the part of higher management

Juran’s Ten Steps to Quality Improvement

Build awareness of both the need for improvement and opportunities for improvement

Set goals for improvement

Organize to meet the goals that have been set

Provide training

Implement projects aimed at solving problems

Report progress

Give recognition

Communicate results

Keep score

Maintain momentum by building improvement into

the company's regular system

Juran’s Contributions

The Pareto Principle

80/20 Rule: 80% of the trouble comes from 20% of

the problems

The Juran Trilogy

Refer diagram

Quality Planning

Determine who the customers are:

Identify customers’ needs.

Develop products with features that respond to customer needs.

Develop systems and processes that allow the organization to produce these features.

Deploy the plans to operational levels

Quality Control

Assess actual quality performance.

Compare performance with goals.

Act on differences between performance and goals

Quality Improvement

Develop the infrastructure necessary to make annual quality improvements.

Identify specific areas in need of improvement, and implement improvement projects.

Establish a project team with responsibility for completing each improvement project.

Provide teams with what they need to be able to diagnose problems to determine root causes, develop

situations, and establish control that will maintain gains made

Crosby’s 14 Quality Steps

1. Management commitment

2. Quality improvement teams

3. Quality measurement

4. Cost of quality evaluation

5. Quality awareness



6. Corrective action

7. Zero-defects committee

8. Supervisor training

9. Zero-defects day

10. Goal-setting

11. Error cause removal

12. Recognition

13. Quality councils

14. Do it over again

SOFTWARE QUALITY ASSURANCE STANDARDS

Introduction

Software failures: the statistics

How do we achieve QA?

By adopting a disciplined, professional approach - engineering is predictable and repeatable

Quality Management System

o Documented processes & Best Practice

o ISO 9001 & TickIT registration

Continuous Improvement

o New methods (e.g. DSDM, UML) & tools

o Better & more focussed Training

The Quality Management System

Defines the way we work

o focuses on what rather than how

o flexible

o broadly applicable

Procedures and guidelines

o mandatory elements and recommendations

Mix of best practice & common sense

ISO STANDARDS ISO 9000 is a quality system standard that:

o Is a three-part, continuous cycle of planning, controlling, and documenting quality in an organization.

o Provides minimum requirements needed for an organization to meet its quality certification standards.

o Helps organizations around the world reduce costs and improve customer satisfaction.

ISO 15504, sometimes known as SPICE (Software Process Improvement and Capability dEtermination), is

a framework for the assessment of software processes.



What is ISO 9001: 2000

2nd revision of Quality Management System Requirement Standard from International Organization for

Standards

Replacement for previous ISO 9001 / 9002 and 9003 standards of 1994

Introduced considerable conceptual changes

Applicable to all types of Organizations with possible permissible omissions of certain requirements

New ISO 9001

ISO 9001:2000 – Model



Principles of new standard

Customer focus

Organization depends customers

Understand current & future customer needs.

Meet / exceed customer expectations

Leadership

Leaders establish purpose & direction of the organization

Should create & maintain environment to achieve organization’s objectives

Involvement of People

People of all levels are essence of an organization

Their full involvement for organization’s benefit

Process approach

Desired results are achieved more efficiently when activities and resources are managed as process

System approach to Management

Identifying, understanding and managing interrelated process as a system contributes to the organization’s

effectiveness & efficiency

Continual improvements

Continual improvement of the organization’s overall performance should be a permanent objective of the

organization

Factual approach to decision making

Effective decisions are based on the analysis of data and information

Mutually beneficial supplier relationships

An organization & its suppliers are interdependent

Mutually beneficial relationship enhances the ability of both to create value

Expectations of the new Standard

Avoid the application of systems that are separate from the organization’s business process

Enable the development of a Quality system that is fully integrated into the normal operations of

organization’s business

Enable Continual improvements of the system for enhanced customer satisfaction

Enable compliance to statutory & regulatory requirements



Important changes

Criteria Previous version New Version

Main focus Products Customer satisfaction

Approach 20 quality elements Value adding processes

Product requirements Requirements specified by customer /

organization + Statutory & regulatory requirements

Involvement of people What to do, When, Whom & How to do + Why it is to be done

Improvements Maintain the system requirements Continual improvements should be

achieved

Process approach

Continual improvements of Process



System Requirements / Structure of the Standard

4 - Quality management system

General Requirements

o Identification of process required

o Criteria and methods to ensure operation and control

o Availability of information & resources for operation & control

o Monitoring and Measuring of processes

o Continued improvements

Document Requirements

o Quality Policy

o Quality Objectives

o Quality Manual

o Procedures required by the standard

o Procedures required for planning, operation and control of organization activities

o Records

5 - Management Responsibility

Top Management’s commitment

Development, implementation and continually improvement of QMS



Establishment of

o Quality Policy

o Quality Objectives

Identification of Customer requirements

Communication of importance of

o Regulatory & statutory requirements

o Meeting customer requirements

o Quality Policy & Quality objectives

o Responsibilities & authorities

Appointment of Management Representative

Conducting Management Reviews

Providing required resources

Evidence must be provided to show that the Management is committed to the above requirements

Auditors could speak to and audit Top Management (E.g. MD / Directors) to establish their commitment to

the management system

6 - Resource Management

Resources required to

o Implementing, monitoring & continual improvements

o Enhance Customer satisfaction by meeting customer requirements

Human Resources

Infrastructures

o Infrastructures needed to achieve product conformity

Work environment

o Work environment needed to achieve product conformity

6.2 Human Resources

Competent on the basis of appropriate education, skill and experience

Define competencies for people performing work affecting product quality

Provide training or actions

Evaluate effectiveness of the training / actions

Employees should aware importance of the activity being performed

7 - Product Realization

7.1 Planning of Product realization

Quality objectives of Products – Specs

Processes, procedures to realize product

Verification, validation, monitoring, inspection and testing of product

Record to demonstrate conformance

7.2 Customer related processes – (Sales)

Identification of Customer / Market requirements

o Specified by customer

o Requirements taken for granted



o Statutory / Regulatory requirements

Review of requirements related product prior to acceptance / commitment to customers - ability to meet

customer requirements

Effective communication with customer in relation to

o Product information

o Sales order handling

o Customer feedback

o Customer complaints

7.3 Design and Development – (Product)

Planning

o Effective & efficient

o Expectations of interested parties

o EHS

Design inputs and outputs

Review and verification, validation and control of changes

o Accuracy

o Potential hazards & faults

o Corrections

o Evaluations against lessons learned

7.4 Purchasing

Purchasing is done in controlled manner to ensure that purchased products conforms to specific

requirements

Degree of control depends on effects of subsequent processes and effect on final product

Supplier evaluation

Verification of purchased product – Inspection and testing

7.5 Production and service provision

Manufacturing / service provision under controlled condition to ensure conformity of product

Product characteristics (Specs)

Procedures and work instructions

Suitable equipments to manufacture. Monitoring and inspection & testing

Product release, delivery and post delivery

Process validation

Identification and traceability

Customers property

o Material supplied by customers – e.g.. 3rd party blending

7.6 Control of monitoring and measuring devices

Control and Calibration of equipments used for monitoring, inspection and testing

8 - Measurement, analysis and improvement

8.1 - To demonstrate

Conformity of the product

Conformity to QMS requirements

Continually improvements and the effectiveness of the system

8.2 - Monitoring and Measurements

Customer satisfaction / perception

Internal audits - conformity planned arrangements of QMS and ISO9001

Monitoring and measurements of processes – to determine / demonstrate ability of processes to achieve

required results

Monitoring and measurements of product – Conformity to product requirements

8.3 - Control of NCP

To assure that NCP products are identified and controlled to prevent unintended use / delivery



8.4 - Analysis of data

Collection and analysis of data generated through QMS activities to verify suitability, effectiveness and

continual improvement of the system

Analysis shall provide information related to

o Customer satisfaction / perception

o Conformity to specs, requirements

o Trends of processes and products

o Opportunities for preventive actions

o Suppliers

8.5 - Improvements

Continual Improvements

o QMS needed to be continually improved

Corrective action

o Actions to prevent recurrence of NCP, NCR etc

o Includes reviews, determination of causes, need of action, implementation of action, review of action

and maintenance of relevant records

Preventive action

o Actions against potential non conformities to avoid their occurrence

o Includes identification of potential non conformities, cause, need for action, implementation of action,

review of action and maintenance of records

Criteria for measurements

System performance

Satisfaction surveys for customers and other interested parties

o Feedback on products

o Customer & market requirements

Financial measurements

o Prevention cost

o Non conforming / failure cost

o Lifecycle cost

Self assessment

Internal audits

o Effectiveness & efficiency of processes

o Opportunities for improvements

o Use of data / information

o Effective & efficient use of resources

o Adequacy, accuracy and performance of measurements

o Relationships with customers/ suppliers/ other interested parties



Processes

Process capability / process validation

Reaction time

Cycle time / throughput (Capacity)

Utilization of technology

Waste reduction

Cost reduction

Products

Inspection and testing of incoming, in process and final product

Product verification

Product validation

SPICE Software Process Improvement and Capability dEtermination

also known as ISO/IEC 15504 is an international standard for SW process assessments

Mainly used in Europe and Australia by the automotive industry

Goal

o To provide assessment results that are repeatable, objective, comparable

Future

o Automotive SPICE launched in April 2006

o its usage will increase mainly driven by HIS (Audi, BMW, Daimler Chrysler, Porsche, Volkswagen),

Ford & Volvo, Fiat

o Each OEM have different target level; if these are not met, then:

Suppliers are requested to improve the development processes

In case of high risks/low capability levels, the suppliers are excluded from sourcing

The Goal of Process Assessment & Improvement

The goal of an improvement is to change an organization’s processes so that they achieve a higher ability

to meet its businesses goals

Assessments deliver the input for any improvement by detecting strength and weaknesses in the

organizations processes

Assessments are also a tool used by customers to ascertain the ability of their suppliers to meet their needs

SPICE Model’s Structure

The reference model architecture for this assessment model is 2-dimensional

o Process dimension → contains processes in groups

o Process Capability dimension → allows the capability of each process to be measured independently



Process dimension

Characterized by set of purpose statements which describe in measurable terms what has to be achieved

in order to attain the defined purpose of the process

Process Capability dimension

Characterizes the level of capability that an organization unit has attained for a particular process or,

May be used by the organization as a target to be attained

Represent measurable characteristics necessary to manage a process and improve its capability to perform

Capability Dimension overview

Level 1 Performed process

Base practices of the process are performed ad hoc and poorly controlled. Work products of the process

are identifiable

The purpose of the process is generally achieved

Work products proof implementation of base practices

No documented process

No planning or checks of performance of the process

No quality requirements for work product are expressed

Level 2 Managed process

Base practices of the process are planned and tracked. Products are conformed to standards and

requirements

The performance of the process is planned and checked

The responsibility for developing the work products is assigned to a person or team

Requirements for the work products are identified, documented and traced

Work products are put under configuration management and quality assurance

No documented or defined process

Level 3 Established process

The process is managed and performed using a defined process. Projects are using a tailored version of

the standard process.

A documented standard process with tailoring guidelines exists and is used in the project

Historical process performance data is gathered

Experience from the performance of the process is used for process improvement

Resources and needed infrastructure for the performance of the process are identified and made available

The process is not yet quantitatively understood or managed

Process improvement is reactive

Level 4 Predictable process

The process is performed consistently in practice within defined control limits. The quality of work products

is quantitatively known

Level 5 Optimizing process

The process performance is optimized to meet current and future business needs

Process Dimension overview

Primary Life Cycle Processes Category

Supporting Life Cycle

Processes Category

Organizational Life Cycle

Process Category

Process Attributes

Measure capability levels

SPICE Assessments



CAPABILITY MATURITY MODEL CMM CMM tries to capture and describe the differences between high quality organization versus low quality orgs

CMM strives to create software development organizations that are “mature”, or more mature than before

applying CMM.

Describes five levels of SW process maturity.

Includes lots of detail about each level – we will look at some of it

Summary of levels

Level 1 – Initial

o Anything at all. Ad-hoc and chaotic.

o Will have some successes, but will also have failures and badly missed deadlines.

Level 2 – Repeatable

o SW processes are defined, documented, practiced, and people are trained in them.



o Groups across an organization may use different processes

Level 3 – Defined

o SW processes are consistent and known across the whole organization.

Level 4 – Managed

o SW processes and results are measured quantitatively, and processes are evaluated with this data.

Level 5 – Optimizing

o Continuous process improvement.

o Experimenting with new methods and technologies.

o Change processes when find something that works better.

Level 1 – Initial

Team tackles projects in different ways each time

Can have strong successes, but may not repeat

Some time/cost estimates are accurate, many far off

Success comes from smart people doing the right things

Hard to recover from good people leaving

Frequent crises and "firefighting.” (Many believe this is standard for SW development. CMM says NO.)

Most SW development organizations are Level 1.

Estimating curve, process diagram.

Level 2 – Repeatable

Key areas

o Requirements management

o Software project planning

o Project tracking and oversight

o Subcontracts management

o Quality assurance

o Configuration management

Usually takes 18+ months. (Some ask for Level 1.5.)

Estimating curve

Process diagram

Level 3 – Defined

Organization-wide process focus

Organization-wide process definition

Training program in above

Integrated software management (above applied per project)

Software product engineering (coding, etc.)

Inter-group coordination

Peer reviews

Level 4 – Managed

Quantitative process management (data gathering)

Quality management (data-driven quality improvement)

Level 5 – Optimizing

Defect prevention

Technology change management (bring in new methods)

Process change management (improve processes)

Credits

Thanks to the SRM MTech staff, who have provided us the PPTs, which were very helpful in creating this

Comments & Feedback

Thanks to my family members who supported me while I spent hours and hours to prepare this.

Your feedback is welcome at [email protected]

UNIT III SOFTWARE QUALITY ASSURANCE MODELS ... … · CS621 – Software Quality Management Unit -...

Documents

Transcript of UNIT III SOFTWARE QUALITY ASSURANCE MODELS ... … · CS621 – Software Quality Management Unit -...