1

ECE 453 – CS 447 – SE 465 Software Testing & Quality Assurance

InstructorPaulo alencar

2

Overview

Software Quality

Software Quality and Quality Assurance Non-Execution Based Testing Reviews and inspections

3

The State of Software Development

2000 Defense Science Board Study:• 53% of projects were late and over budget, 16%

were on time, 31% were canceled before completion.

• There is tremendous growth in software content in both manned and unmanned systems.

• Software requirements now amount to the bulk of the overall specification requirements (65% for the B-2, 80% for the F-22)

4

Software Quality • The American Heritage Dictionary defines quality

as – “a characteristic or attribute of something.”

• For software, some kinds of quality may be encountered: – Quality of design encompasses requirements,

specifications, and the design of the system. – Quality of conformance is an issue focused

primarily on implementation.– user satisfaction: compliant product + delivery

within budget and schedule, etc.

5

Software Quality

• Quality, simplistically, means that a product should meet its specification.

• This is problematical for software systems– There is a tension between customer quality

requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.);

– Some quality requirements are difficult to specify in an unambiguous way;

– Software specifications are usually incomplete and often inconsistent.

6

Cost of Quality • Prevention costs include

– quality planning– formal technical reviews– test equipment– Training

• Internal failure costs include– rework– repair– failure mode analysis

• External failure costs are– complaint resolution– product return and replacement– help line support– warranty work

7

Software Quality Assurance

FormalTechnicalReviews

Test Planning& ReviewMeasurement

Analysis&

Reporting

ProcessDefinition &Standards

8

Software Quality Team Size

=< 1%

=< 3%

=<4%

=>4%

Software Quality Staff / Development Staff

SAMPLE OF 135 ORGANISATIONS (1983)

9

Role of a Software Quality Team

ReviewApplications

ProvideTechnical Advice

Review and Build a

Quality Environment

Develop Standards and Guidelines

Analyse Development Errors

10

Tasks of a Software Quality Team ROLE CHALLENGE TASKS

ReviewApplications

When to abort a projectExecutive management ignoranceUser ignoranceAudit requirements

Evaluate systems in all phasesProvide management with technical assessmentAscertain user requirements are metAscertain audit requirements are met

ProvideTechnicalAdvice

Changing technologyUse of consultantsAbility to keep current technicallyComplexity of systems

Know current technologyAct as internal consultantAct as technical consultantKnow many systems

Review andBuild aQualityEnvironment

How to evaluate software productsBuild a quality environment

Evaluate software productsCounsel management

DevelopStandardsandGuidelines

Few systems and programming standardsProfessionalism

Help set standardsEvaluate quality of work

AnalyseDevelopmentErrors

Know type of problemsKnow cost of problemsKnow magnitude of problems

Quantify problemsIdentify problemsDetermine cost of problems

11

Role of the SQA Group • Prepares an SQA plan for a project.

– The plan identifies• evaluations to be performed• audits and reviews to be performed• standards that are applicable to the project• procedures for error reporting and tracking• documents to be produced by the SQA group• amount of feedback provided to the software project team

• Participates in the development of the project’s software process description. – The SQA group reviews the process description for compliance with

organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project plan.

12

Role of the SQA Group • Reviews software engineering activities to verify compliance

with the defined software process. – identifies, documents, and tracks deviations from the process and

verifies that corrections have been made.

• Audits designated software work products to verify compliance with those defined as part of the software process. – reviews selected work products; identifies, documents, and tracks

deviations; verifies that corrections have been made– periodically reports the results of its work to the project manager.

• Ensures that deviations in software work and work products are documented and handled according to a documented procedure.

• Records any noncompliance and reports to senior management.– Noncompliance items are tracked until they are resolved.

13

Analyzing Defects

ProductProduct& Process& Process

measurement

... an understanding of how to improve quality ...

Collect information on all defectsFind the causes of the defectsMove to provide fixes for the process

14

Analyzing Defects

Remember, high quality products Must meet all user requirements Must be free of defects

In some processes, the main measure of software product quality is total defect density

Total defect density is measured as defects/KLOC (number of defects

removed in development per thousand LOC)

15

Example Quality Measurement

DRE = E /(E + D)

Defect Removal Efficiency

E is the number of errors found before delivery of the software to the user D is the number of defects found after delivery

16

Why SQA Activities Pay Off?

and fix a defectand fix a defect

100100

1010

loglogscalescale

11

Req.Req.DesignDesign

codecodetesttest

systemsystemtesttest

fieldfielduseuse

0.750.75 1.001.001.501.50

3.003.00

10.0010.00

60.00-100.0060.00-100.00

cost to findcost to find

17

Software Quality Team The major role of the Software Quality Team is to review

applications.

Reviewing Applications– Includes verification and validation– It also includes the following:

1. Software reviews and inspections2. Traceability of software deliverables3. Testing4. Auditing of selected key software items5. Standards and Guidelines

18

1. Software Reviews and Inspections

A software review is an evaluation of a software element to ascertain discrepancies from planned results and to recommend improvements

– e.g. Design Review, Code Review

– Different types, e.g.:

• Walkthrough

• Software Inspection

• Technical Review

19

2. Traceability

– Forwards Traceability

» each input to a phase must be matched to an output of the same phase to show completeness

– Backwards Traceability

» each output is traceable to an input of a phase

20

3. Testing

Testing is the evaluation of a system or component to:• confirm that it satisfies requirements• identify differences between actual and expected

results

Testing may be 50% of project costs;

so use cheaper methods before testing starts,

i.e., walkthroughs and inspections

21

4. Auditing

Audits are independent reviews that assess compliance with specifications, standards and procedures

Perform an audit before software release:• Physical audit check that all items identified as part of the

configuration are present

• Functional audit check that unit, integration and acceptance tests

have been carried out and that records show their success or failure

22

5. Software Quality StandardsStandards and Guidelines

‘A standard is an approved, documented, and available set of criteria used to determine the adequacy of an action (process standard) or object (product standard).’

‘A guideline is a well-defined and documented set of criteria that guides an activity or task.’ (Dorfman & Thayer, 1990)

• differ from standards - allows for judgement and flexibility

Categories of Standards and Assessment– Product Standards– Calibration and Measurement Standards– Quality Management Systems Standards

23

Verification and Validation

24

Verification and Validation

• Validation:

Are we building the right system?

• Verification:

Are we building the system right?

Quality Assurance ensures that verification and validation takes place.

25

Software Reviews and Inspections

A way of using the diversity of a group to:• uncover errors in function, logic, or

implementation • verify that software under review meets

requirements• ensure that the software meets predefined

standards• achieve uniformity of development• make projects more manageable

26

The Review Players

reviewreviewleaderleader

producerproducer

recorderrecorder reviewerreviewer

standards bearer (SQA)standards bearer (SQA)

maintenance maintenance oracleoracle

user repuser rep

27

Software Reviews• A meeting conducted by technical

people for technical people

• A technical assessment of a work product created during the software engineering process

• A software quality assurance mechanism

• A training ground

28

Types of Review

• There are a number of types of review ranging in formality and effect. These include: – Buddy Checking

• having a person other than the author informally review a piece of work.

• generally does not require collection of data• difficult to put under managerial control• generally does not involve the use of checklists to

guide inspection and is therefore not repeatable.

29

Types of Review– Walkthroughs

• generally involve the author of an artifact presenting that document or program to an audience of their peers

• The audience asks questions and makes comments on the artifact in an attempt to identify defects

• often break down into arguments about an issue

• usually involve no prior preparation on behalf of the audience

• usually involve minimal documentation of the process and of the issues found

• process improvement and defect tracking are therefore not easy

30

Types of Review

– Inspection • formally structured and managed peer review

processes • involve a review team with clearly defined roles• specific data is collected during inspections • inspections have quantitative goals set • reviewers check an artifact against an unambiguous

set of inspection criteria for that type of artifact• The required data collection promotes process

improvement, and subsequent improvements in quality.

31

Technical Reviews

The Review Meeting» between 3 to 5 people (reviewers) should be

involved» advance preparation should occur (< 2 hours work)» meeting should be < 2 hours» should be for a small part of the overall software» is attended by the producer, review leader and

reviewers» must have a follow-up procedure to ensure any

corrections are completed

32

Technical Reviews

The Review Meeting» At the end of the review the attendees must

decide whether to:• accept the work product without

modification• reject the work product due to severe

errors (once corrected another review is performed)

• accept the work product provisionally (minor errors to be corrected but no further review)

33

Conducting a Review

be prepared—evaluate be prepared—evaluate product before the reviewproduct before the review

review the product, not review the product, not the producerthe producer

keep your tone mild, ask keep your tone mild, ask questions instead of questions instead of making accusationsmaking accusations

stick to the review agendastick to the review agenda

raise issues, don't resolve themraise issues, don't resolve them

avoid discussions of style—stick to technical avoid discussions of style—stick to technical correctnesscorrectness

schedule reviews as project tasksschedule reviews as project tasks

record and report all review resultsrecord and report all review results

1.1.

2.2.

3.3.

4.4.

5.5.

6.6.

7.7.

8.8.

34

Technical Reviews

The Review Report should contain:- Type of Review, Team Members, Date, Time, Place- Item being reviewed- Agenda- Checklist- Comments per Checklist item

or per Issues list item (Pressman) with problems mentioned Action per Comment - No action, Fix error, Reconsider

 - Decision for the item being reviewed

- Accept, Reject (severe errors), Accept (minor errors)

35

Technical Reviews Review Guidelines

» Review the product not the producer» Set an agenda and maintain it» Limit debate and rebuttal» Enunciate problem areas, but don’t attempt to solve every

problem» Take written notes» Limit the number of participants and insist on advance

preparation» Develop a checklist for each work product» Allocate resources and time schedule» Conduct meaningful training for all reviewers» Review earlier reviews

36

Software Inspection

• The inspection process comprises three broad stages:– preparation

– collection

– repair

• Gilb and Graham [GilbGraham93] expand this three stage process into the inspection steps; Entry, Planning, Kickoff Meeting, Individual Checking, Logging Meeting, Root Cause Analysis Edit, Follow Up, Exit.

37

Benefits of Software Inspection

• 30% to 100% net productivity increases;

• Overall project time saving of 10% to 30%;

• 5 to 10 times reduction in test execution costs and time;

• Reduction in maintenance costs of up to one order of magnitude;

• Improvement in consequent product quality;

• Minimal defect correction backlash at systems integration time.

• In addition to these tangible benefits, less tangible benefits such as a training effect for inspectors are also evident.

38

Disadvantages

• Up front costs (although far outweighed by benefit):– Training– Implementation Support– Ongoing allocation of staff resources

• Not strictly repeatable