01 Software Testing Introduction

Software TestingIntroduction

Introduction

• An introduction to software testing.

• Serves as a tutorial to formal testing of software

• Topics covered – definitions of testing

– validation and verification

– levels of testing from unit testing through to acceptance testing

– relationship with requirements and design specifications and

– test documentation.

What is Software Testing?

• Many definitions of software testing

• All definitions boil down to: – software testing is the process of executing software in a controlled

manner, in order to answer the question "Does the software behave as

specified?". • Software testing is often used in association with the terms

verification and validation. – Verification is the checking or testing of items, including software, for

conformance and consistency with an associated specification. – Software testing is just one kind of verification, which also uses

techniques such as reviews, analysis, inspections and walkthroughs.


– Validation is the process of checking that what has been specified is what

the user actually wanted. • Validation: Are we doing the right job?

• Verification: Are we doing the job right? • “bug” refers to a problem or fault in a computer. • Software testing should not be confused with debugging.

– Debugging is the process of analyzing and locating bugs when software

does not behave as expected. • A methodical approach to software testing is a much more

thorough means of identifying bugs.


• Debugging supports testing, but cannot replace testing.

• However, no amount of testing can guarantee discovery of all

bugs. • Other activities often associated with software testing are:

– static analysis– dynamic analysis

• Static analysis investigates the source code, looking for

problems and gathering metrics without executing the code. • Dynamic analysis examines the behavior of executing

software.

The Cost of Testing

• It is well known that most of an iceberg is hidden under water.

• The same is true of software.

• The number of lines of code in the product often describes the size of a software project, but that's just the tip of the iceberg.

• The amount of software that needs to be written to test the

product can be staggering. • Often the number of lines of code of test software will

exceed the lines of code in the product software.

Cost

– Suppose you have to write three lines of test software for every new line of software you deliver.

– Then suppose management says that all your test software must be fully tested.

– So for every line of test software you write, you need to write 3 lines of software that tests the software that tests your product.

– If X is the number of lines in your product, you will have to write 3X lines of test software, and 9X lines of code that tests the test software.

– Your job has just increased by a factor of 12!

Cost

• What generally happens is that people write as much test software as they can in the time left over at the end of the project.

• The test software is inadequate, so the product gets shipped with some bugs in it.

• Testing is a big job.

• You have to plan money, people, and most of all, TIME for it. • Don't expect to write and debug all your test code in the last

month before the critical design review.

Cost

• Even if you could, it wouldn't do you much good.

• By that time the design is cast in concrete, so nobody is going to change anything unless your test programs find catastrophic errors.

• If you find moderate errors, people will say, "That's a shame, but we can't do anything about it now. Why didn't you catch these errors sooner?"

• You have to code and test early in the development cycle while there is still time to take corrective action.

Software Specifications and Testing

• Validation and verification activities, such as software testing, cannot be meaningful unless there is a specification for the software.

• Depending on the size of the development, specification of software can range from a single document to a complex

hierarchy of documents. – A hierarchy of software specifications will typically contain three or more

levels of software specification documents. • Requirements Specifications specify what the software is

required to do and may also specify constraints on how this may be achieved.


• Architectural Design Specifications describe the architecture of a design which implements the requirements. Components within the software and the relationship between them.

• Detailed Design Specifications describe how each component in the software, down to individual units, is to be implemented.


• A hierarchy of specifications, makes possible testing software at various stages of the development.

• The levels of testing which correspond to the hierarchy of

software specifications: – Unit Testing: each unit (basic component) of the software is tested

– Integration Testing: progressively larger groups of tested software components corresponding to elements of the architectural design

are integrated and tested until the software works as a whole.

– System Testing, the software is integrated to the overall product

and tested to show that all requirements are met.


– Acceptance Testing: upon which acceptance of the completed software is based. This will often use a subset of the system tests, witnessed by the customers for the software or system.

• Once each level of software specification has been written,

the next step is to design the tests. • Tests should be designed before the software is

implemented. – if the software was implemented first, it would be tempting to test

the software against what it is observed to do rather than against

what it is specified to do.


• Within each level of testing, test results are evaluated.

• If a problem is encountered, either the tests are revised and reapplied, or the software is fixed and the tests reapplied.

• This is repeated until no problems are encountered, at which point development can proceed to the next level of

testing. • Testing does not end at the conclusion of acceptance

testing.

• Software has to be maintained to fix problems which show up during use and to accommodate new requirements.


• Software tests have to be repeated, modified and extended.

• Revision and repetition of tests forms a major part of the overall cost of developing and maintaining software.

• The term regression testing is used to refer to the repetition of earlier successful tests in order to make sure that changes to the software have not introduced side effects.

Test Design Documentation

• Tests design is subject to same engineering principles as the design of software.

• Good design consists of a number of stages which progressively elaborate the design of tests from a high level strategy to detailed test procedures.

• These stages are: – test strategy,

– test planning,

– test case design, and

– test procedure design.

Test Design Documentation

• The design of tests is driven by the specification of the software.

• At the highest level tests will be designed to verify that the software faithfully implements the Requirements Specification.

• At lower levels tests will be designed to verify that items of software implement all design decisions made in the Architectural Design Specification and Detailed Design

Specifications. • As with any design process, each stage of the test design

process should be subject to informal and formal review.

Test Strategy

• A test strategy is a statement of the overall approach to testing, identifying what levels of testing are to be applied and the methods, techniques and tools to be used.

• A test strategy should be applicable to all of an organizations

software developments. • Developing a test strategy which efficiently meets the needs of

an organization is critical to the success of software

development within the organization. • The application of a test strategy to a software development

project should be detailed in the projects software quality plan.

Test Plans

• The next stage of test design is development of a test plan.

• A test plan states the – items to be tested are,

– at what level they will be tested,

– what sequence they are to be tested in,

– how the test strategy will be applied to the testing of each item,

– and describes the test environment. • A test plan may be project wide, or a hierarchy of plans

relating to the various levels of specification and testing:

Plans

– An Acceptance Test Plan: describes the plan for acceptance testing of the software. Usually published as a separate document, but might be published with the system test plan.

– A System Test Plan: describes the plan for system integration and testing. Usually published as a separate document, but might be

published with the acceptance test plan. – A Software Integration Test Plan: describing the plan for

integration of tested software components. May form part of the

Architectural Design Specification. – Unit Test Plan(s): describes the plans for testing of individual units

of software. May form part of the Detailed Design Specifications.

Plans

• Each test plan provides a plan for verification that the software fulfills the requirements or design statements of the software specification.

• In the case of acceptance testing and system testing, this means the Requirements Specification.

Test Plans

• Test Plan Outline (Adapted from IEEE Standard for Software Test Documentation (829) ) 1.BACKGROUND

2.INTRODUCTION

3.ASSUMPTIONS

4.TEST ITEMS

List each of the items (programs) to be tested

Plan

5.FEATURES TO BE TESTED

List each of the features (functions or requirements) which will be tested or demonstrated by the test.

6.FEATURES NOT TO BE TESTED

Explicitly lists each feature, function, or requirement which won't be tested and why not.

7.APPROACH

Describe the data flows and test philosophy.

Simulation or Live execution, Etc.

8.ITEM PASS/FAIL CRITERIA Blanket statement

Itemized list of expected output and tolerances

Plan

9.SUSPENSION/RESUMPTION CRITERIA

Must the test run from start to completion?

Under what circumstances may it be resumed in the middle?

Establish check-points in long tests.

10.TEST DELIVERABLES

What, besides software, will be delivered?

Test report

Test software

11.TESTING TASKS Functional tasks (e.g., equipment set up)

Administrative tasks

Plan

12.ENVIRONMENTAL NEEDS

Security clearance

Office space & equipment

Hardware/software requirements

13.RESPONSIBILITIES

Who does the tasks in Section 10?

What does the user do?

14.STAFFING & TRAINING

15.SCHEDULE

16.RESOURCES

17.RISKS & CONTINGENCIES

18.APPROVALS

Test Case Design

• After the test plan stage, the next stage of test design is to specify a set of test cases for each item to be tested at that level.

• A number of test cases will be identified for each item to be tested at each level of testing.

• Each test case specifies how the implementation of a particular requirement or design decision is to be tested and the criteria

for success of the test. • The test cases may be documented with the test plan, as a

section of a software specification, or in a separate document called a test specification or test description.

Case Design

• The specification of the following test cases may be documented by a separate document or in the appropriate test plan:– acceptance testing – system integration – for each stage of integration of tested software components– testing of individual units of software

• System testing and acceptance testing involve an enormous number of individual test cases.– an index which cross references between requirements and test cases.

Referred to as a Verification Cross Reference Index (VCRI) and is attached to the test specification.

Case Design

– Cross reference indexes may also be used with unit testing and

software integration testing. • Design test cases for both positive testing and negative

testing. – Positive testing checks that the software does what it should.

– Negative testing checks that the software doesn't do what it shouldn't.

Test Procedures

• The final stage of test design implements a set of test cases as a test procedure, specifying the exact process to be followed

to conduct each of the test cases. – For each item to be tested, at each level of testing, a test procedure

specifies the process to be followed in conducting the appropriate test cases.

– A test procedure cannot leave out steps or make assumptions. The level of detail must be such that the test procedure is deterministic and

repeatable. – Test procedures should always be separate items, because they contain

a great deal of detail which is irrelevant to software specifications.

Results Documentation

• The outputs of each test execution should be recorded. – Results are assessed against criteria in the test specification

• Each test execution should also be recorded in a test log. – The test log contains records of

• when each test run,

• outcome of each test execution, and

• may include key observations made during test execution.

– A log may not be exist for unit and software integration tests. • Test reports may be produced during the testing process.

– A test report summarizes the results and documents analysis.

– An acceptance test report often is a contractual document.

Summary

• List of rules to an aid effective software testing. – Always test against a specification.

– Document the testing process: specify tests and record test results.

– Test hierarchically against each level of specification. Finding more errors earlier will ultimately reduce costs.

– Plan verification and validation activities, particularly testing.

– Complement testing with techniques such as static analysis.

– Test positively that the software does what it should, but also negatively that it doesn't do what it shouldn't.

01 Software Testing Introduction

Documents

Transcript of 01 Software Testing Introduction