T-76.613 Software Testing and Quality Assurance · Runeson and Isacsson, 1998, Software Quality...

HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.3601 Basics of Software Engineering

11.4.2008

Software Quality Assurance (SQA)

Juha ItkonenSoberIT

2Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Contents

Software Quality Assurance

Introduction to quality and quality assurance

SQA in CMM(I)

Agile Viewpoint to SQA

+ ( A Bug’s Life )


Software Quality Assurance


Definition of software quality

Quality: (1) The degree to which a system, component or process meets specified requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations.

(IEEE Standard Glossary of Software Engineering Terminology [IEEE610.12])

Quality is value to some person(s)(Gerald M. Weinberg, 1992, “Quality Software Management”)


Good Enough Quality

Defined by James Bach IEEE Computer, 1997, vol. 30, no. 8, pp. 96-98.

To claim that any given thing is good enough is to agree with all of the following propositions:

It has sufficient benefitsIt has no critical problemsThe benefits sufficiently outweigh the problemsIn the present situation, and all things considered, further improvement would be more harmful than helpful

Aiming to Good Enough Quality does not necessarily allow us to do less quality practices

We must find the critical problems and at least be able to estimate the risks of the remaining (unknown) problems


Process quality and product quality

Quality in process -> quality in product

Project: instantiated process

Quality according to ISO 9126

Process quality contributes to improving product quality, which in turn contributes to improving quality in use

Process

Project


Wikipedia says

Quality Assurance (QA) is the activity of providing evidence needed to establish confidence among all concerned, that quality-related activities are being performed effectively.

QA assures the existence and effectiveness of procedures that attempt to make sure - in advance -that the expected levels of quality will be reached.


Definition Quality Assurance IEEE Standard Glossary of Software Engineering Terminology

1. A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements.

2. A set of activities designed to evaluate the process by which products are developed or manufactured.


Quality Related Concepts

Runeson and Isacsson, 1998, Software Quality Assurance – Concepts and Misconceptions,in proceedings of Euromicro Conference, pp. 853 - 859.

Software Quality Engineering

Quality AssuranceQuality Development

Software Quality

Assurance

Quality Management

Verification and Validation

Requirements Engineering

System and Software Design

Implementation


Defining the ConceptsSoftware Quality Engineering

Quality AssuranceQuality Development

Software Quality

Assurance

Quality Management

Verification and Validation

Requirements Engineering

System and Software Design

Implementation

Software Quality Engineering (SQE) All activities involved in engineering a software product with respect to its quality

Activities to achieve quality (Quality Development) as well as to assure the required software quality (Quality Assurance)

Software Quality Assurance (SQA)Activities for independent assurance of adherence to defined processed (as stated in CMM)

Quality Management (QM) Project’s internal activities to monitor and manage the software quality

E.g., quality planning, process definitions, process monitoring, …Verification and Validation (V&V)

Concrete activities to verify that the system and its components conforms to their specifications

and to validate that the customer needs are fulfilled

E.g., different types of tests and inspectionsRuneson and Isacsson, 1998, Software Quality Assurance –Concepts and Misconceptions, in proceedings of Euromicro Conference, pp. 853 - 859.


Defect Prevention and Detection (quality development and quality assurance)

Both are needed and must be taken into account


Defect Prevention (quality development, quality building)

Quality practices include all activities that contribute to the quality of the final product

Defect prevention is essentialPreventing defects to occur in the first placeDevelopment practices, conventions, techniques, and tools

Many practices are used to prevent bad quality deliverables

Coding conventions and standardsDesign patternsReviewsCommunication practicesEtc.

Reviews and inspectionsDetect defects in current phasePrevent defects in the following phases


Defect Detection (quality assurance, verification & validation)

Earlier detected defects are cheaper to fix

Prevented defects can be even cheaper

Defect detection is always needed

Nobody is perfect, all defects cannot be prevented

Detection of some bearable amount of defects can be cheaper than prevention

Detection for confidence

Testing and measuring gives information of the achieved quality

We try to find defects in order to convince that there are none


The Purpose of Quality Assurance

Independent assurance of adherence to defined processes

Creating sufficient and applicable processes and practices for achieving good enough quality level

Monitor the process and ensure that

the selected processes and practices are followeddeviations are reported and acted uponbad practices are improved or replaced by better practices

Support the delivery of good enough quality deliverables

Providing visibility into the achieved level of qualityProviding valuable feedback for people

For improving their working practices

Measuring and tracking quality metrics and process issues


Example: Unit Testing as a quality practice

Unit testing is a quality practice

A developer writes unit tests for all production code

Each source code class must have unit tests and the tests must be automated

All unit tests are reviewed by another developer

All unit tests are run, and must pass, nightly


Example: Unit Testing as part of QA

The role of quality assurance team is toUnderstand the benefits of the unit testing practice

Train and motivate development teams to use the unit testing practice

Assure that each class have unit testsall unit tests have been peer reviewednightly builds are run and pass

Measure and track trends in relevant metricsAmount of unit testsAmount of effort used to reviewing the unit testsResults of the peer reviews (number, type, severity of found issues)Results of the nightly buildsAmount and type of defects that the unit tests catch Amount and type of defects found later, that should have been caught by the unit tests

Help developers to analyse the collected data and improve their unit testing

Is Quality Assurance really deeded?


Software Quality Assurance in CMM(I)


CMM Quality Assurance Objectives

Supports the delivery of high-quality products and services

providing visibility into, and feedback on, processes and work productsfor the project staff and managers at all levels

Ensure that the planned processes are implemented

Practices in verification process area ensure that the specifiedrequirements are satisfied


CMM Quality Assurance is

Objectively evaluating

performed processes

work products

services

Against the applicable

process descriptions

standards

procedures

Identifying and documenting non-compliance issues

Providing feedback on the results to

project staff

managers

Ensuring that non-compliance issues are addressed


CMM Quality Assurance Organisation

Independence

Independent SQA group

SQA people separate from those directly developing the work product

Reporting

Non-compliance issues primarily solved on a team level

Independent reporting channel to escalate non-compliance issues as necessary

To upper management


CMM Quality Assurance Goals and Practices

Objectively evaluate processes and work products

Clearly stated evaluation criteriaWhat, when, how, and who must be involved

Evaluation and non-compliance reports

Documented corrective actions

Provide objective insight

Communicating the non-compliance issues

If local (team level) resolution of the issues is not possible ensure that appropriate level of management can resolve the issue

Tracking the non-compliance issues to resolution

Tracking and analysing trends of quality issuesCommunicating the quality trends


Agile Viewpoint to Quality Assurance


A Different Approach

In agile development methodologies the independent SQA is not so clearly visible

Not a separate organization or role

In agile world the documented process and practices are not emphasized

Individuals are respected and the team can decide their own practices

Quality management on team or project level is emphasized

The Methodology-Growing Technique is one example of agile approach to quality assurance


Methodology-Growing Technique A. Cockburn, Agile Software Development, 2002

“On-the-fly methodology construction and tuning”

Choosing and documenting the practices

Evaluating the practices

Identifying non-compliance issues and bad practice issues

Finding resolutions and tuning the process

Focusing into the team, what was done and how well it worked out, and how to become better

Instead of documents, independent evaluation, and reporting issues


Methodology Growing Technique

Discover the strengths and weaknesses of your organization through short project interviews

1. Create a base methodology at the start of the project

Adjust it together with the development team

2. Run a small interview in the middle of an increment

“Are we going to make it, working the way we are working?”

3. Hold a team reflection workshop after each increment

4. (Post-project review)


Reflection Workshop

A team meeting lasting typically around half a day

Evaluating the effects of each element of the processAfter delivering a running tested incrementOnly then you can see what was underdone and what was overdone

Main questions“What did we learn?”“What can we do better?”

Results can be Tightened standardsNew practices and conventionsChanged practicesDropping unnecessary things outChanges in physical working environmentChanges in team organisation…


Reflection Workshop - Deliverable

Worked well describes the current practices of the team

Problems Lists the identified issues

Try these presents the agreed resolutions for the issues and other improvement actions

Evaluation criteria is not documented

Worked well

Problems

Try these

Reflection Workshop Poster


Choices for (S)QA

OrganisationIndependent QA teamIntegrated into the project team

EnsuringCompliance to documented processesApplicability and improvement of the current processes and practices

Evaluation CriteriaAgainst predefined criteria Identifying issues and problems

FocusDocuments and processes End products and quality

CommunicationReporting to managementSupporting the team


Summary: The Purpose of Software Quality Assurance

Independent assurance of adherence to defined processes

Creating sufficient and applicable processes and practices for achieving good enough quality level

Monitor the process and ensure that

the selected processes and practices are followeddeviations are reported and acted uponbad practices are improved or replaced by better practices

Support the delivery of good enough quality deliverables

Providing visibility into the achieved level of qualityProviding valuable feedback for people

For improving their working practices

Measuring and tracking quality metrics and process issues


A Bug’s Life


Test Oracles – How do we know it’s broken?

An oracle is the principle or mechanism by which you recognize a problem

Test oracle provides the expected result for a test

Specification document

Formula (mathematical)

Computer program

A person

In many cases it is very hard to find an oracle

Even the customer and end user might not be able to tell (beforehand) which is the correct behaviour

Oracle problem is one of the fundamental issues in test automation:How do we teach an automated test to recognize a defect or failure when it happens? – This is not a trivial problem.


What is a software defect (bug)?

A software defect is present when the program does not do what its end user reasonably expects it to do. (Myers, 1979)

The extent to which a program has bugs is measured by the extent to which it fails to be useful. (Beizer, 1984)

A software defect is a mismatch between the program and its specification.

A defect can be detected if the specification exists and is correct.


Rules for identifying a defect

The software does not do something that the specification says it should do

The software does something that the specification says it should not do

The software does something that the specification does not mention

The software does not do something that the specification does not mention but it should

The software is difficult to understand, hard to use, slow, or will be viewed by the end user as just plain not right

Suorce: Software Testing, Ron Patton


Why defects have to be reported?

To make sure found defects are not lost again (forgotten)

To get found defects fixed

To enable defect managementPrioritizationScheduling for fixingTrade-off decisionsTracking trends

Evaluating qualityEstimating effort and schedules

To support quality assuranceMeasuring defect counts, severity distributions, trends, …Estimating risks


Bugs have a life-cycle

New Assigned

Reopened

Resolved Verified

Closed

Unconfirmed

Open States End States


Reporting the found defects

Report the defects immediately

Don’t leave it until the end of the day

Use some defect management system to report and manage defects (e.g. Bugzilla, Mantis)

Do not manage defect reports in emails, postit’s, or .doc’s

Defect reports are challenging to write

Bearing bad newsExplaining complicated behaviourCommunicating to people with different mindset using as few words as possibleGoal is to make people fix their mess instead of creating some new fancy functionality

In worst case, defect reports are the main communication channel between testing and development


Basic structure of a defect report

1. Defect-report identifier

2. Title: A brief, but descriptive summary

3. Date, time, and who found

4. Test item and environment including version and build numbers

5. Description: A detailed description of the defect

1. Expected results2. Actual results3. Repeatability (how to repeat, if possible)4. Additional information that may help to isolate and

correct the cause of the incident

6. Severity of the defect


Writing down defects

Understand the defect forstSpend some time to understand what is the actual defect and under which conditions it occursFind out how to reproduce the defectWhat were the specific expected and actual behaviour/values

Be brief Write condensed report with all relevant information but nothingmore

Be clear Avoid ambiguous or misleading terms

Be neutralDon’t be judgemental; reports should be written against the product,not the person, and state only the facts

Be concreteCommunicate the severity as clearly as possibleReal consequences for the users and customer

T-76.613 Software Testing and Quality Assurance · Runeson and Isacsson, 1998, Software Quality...

Documents

Transcript of T-76.613 Software Testing and Quality Assurance · Runeson and Isacsson, 1998, Software Quality...