CS4311 Spring 2011

Verification & Validation

Dr. Guoqiang Hu Department of Computer Science

UTEP

AnnouncementWhat: CS4390 (Cross-listed with CS5390): “Software Testing”

Testing Only, Nothing Else!

When: This SummerPrerequisite: CS4311

* Note: The V&V content covered in this course (CS4311) can be considered as an introduction to the above course

What Are Verification and Validation?Groups of 2Why? Who? What? Against what? When? How?5 minutes

Verification:Evaluating the product (a system or component) of a development phase to determine whether the product satisfies the specification at the start of the phase.

Did we build the system or component right?

Validation:Evaluating the product (mostly the system or subsystem) during or at the end of the development process to determine whether it satisfies the specified requirements.

Did we build the right system?

V&V Activities and the Software Lifecycle Requirements engineering:

Determine general test strategy/plan(techniques, criteria, team)Test requirements specification• Completeness• Consistency• Feasibility (Functional, performance requirements)• Testability (Specific; Unambiguous; Quantitative; Traceable)

Generate acceptance/validation testing data

Design:Determine integration test strategy

Assess/Test the design: Completeness; Consistency; Handling scenarios; Traceability (to and fro). (design walkthrough, inspection)

V&V Activities and the Software Lifecycle Implementation:

Determine unit test strategyTechniques (Static v. Dynamic):• Read/Have it read• Code walkthrough/Formal code inspection• Formal verification/Proof• Manual testing

Tools, and whistles and bells (driver/harness, stub)

Maintenance:Determine regression test strategy

Documentation maintenance (vital)

V&V Activities and the Software Lifecycle The “V” model:

Where Do the Errors Come From?Groups of 2What kinds of errors? Who? 3 minutes

Kinds of errors:Missing information

Wrong information/design/implementationExtra information

Facts about errors:To err is human (but different person has different error rate).Different studies indicate 30 to 85 errors per 1000 lines. After extensive testing, 0.5 to 3 errors per 1000 lines remain.The longer an error goes undetected, the more costly to correct.

Basic ConceptsCorrect specification:

Specification matches the client’s intent.

Correct program:Program matches its specification.

Error:A human activity that leads to the creation of a fault.

Fault/Bug:The physical manifestation of an error. It can cause a failure.

Failure:The state when a fault is encountered during execution.

Basic ConceptsFault identification:

Process of determining what fault caused a failure

Fault correction:Process of changing a system to remove a fault

Debugging:Process of finding and fixing program faults

Testing:Designing and executing tests. And (if bugs present) debugging.

Test case:A particular set of input and the expected output

Basic ConceptsTest set:

A finite set of test cases working together with the same purpose

Test objective:The main goal for a particular test. Ex., finding faults/fault detection, or, demonstrating reliability/confidence building (no/low failure rate in normal use).

Test objective affects test strategy, test criteria, and test selection.

Test criteria:Specifies testing requirements/stopping rule/measurement. It is closely linked to test techniques. Ex., for coverage-based test techniques, 100% statements, or branches, coverage, or both.

The General Approaches of Verification & ValidationGroups of 2How?2 minutes

The Exhaustive Test ExamplesGroups of 23 minutes

How many test cases are required?

How long will it take?

The Exhaustive Test ExamplesGroups of 23 minutes

A

B

C

The Exhaustive Test ExamplesGroups of 23 minutes

The Purpose of TestingThe purpose of testing is NOT to prove the program is correctInstead, it is to find problems in the program so that the found problems can be fixed

Interesting phenomena of testing results:Successful test:

The more faults are found, the more successful is a test case/ set. Quality of the code:

The more faults are found in a unit of code, it normally means the worse is the quality of the code.

Studies found that more faults may still go undetected in the same piece of code.

Although you can say the quality of the code has been improved after the found faults are fixed.

Hierarchy of V&V techniques

V&V

Dynamic Techniques

SymbolicExecutionTesting

InformalAnalysis

FormalAnalysis

Static Techniques

Review

Inspection

Walkthrough

Hierarchy of TestingTesting

Program Testing

Top DownBottom Up

Integration Testing

Unit Testing

System Testing

Big Bang

Sandwich

Black Box

White Box

Function

PerformanceReliabilityAvailability

AcceptanceTesting

Properties

Security

EquivalenceBoundaryDecision TableState TransitionUse CaseDomain Analysis

Control Flow Data Flow

UsabilityDocumentationPortabilityCapacity

Ad hoc

PilotAlpha

Beta

Hierarchy of TestingTesting

Program Testing

Top DownBottom Up

Integration Testing

Unit Testing

System Testing

Big Bang

Sandwich

Black Box

White Box

Function

PerformanceReliabilityAvailability

AcceptanceTesting

Properties

Security

EquivalenceBoundaryDecision TableState TransitionUse CaseDomain Analysis

Control Flow Data Flow

UsabilityDocumentationPortabilityCapacity

Ad hoc

PilotAlphaBeta

Types of System TestingFunction Testing: Integrated system performs specified function• Scenarios• Black-box techniquesProperties Testing: Integrated system tests against non-functional requirements• Performance, Reliability, Security, Usability, etc.• Performance:

Stress test: maximum throughput Overload test: exceed specification Volume test: sustained large throughput Response time Robustness: test things not specified, but quite possible Recovery: crash, or, recoverable?

Acceptance Testing: Customers test system• Pilot (initial, customer), Alpha test (in-house), Beta test (on-site)

Types of Faults (not exhaustive)Algorithmic: algorithm or logic does not produce the proper output for the given inputSyntax: improper use of language constructsComputation (precision): formula’s implementation wrong or result not to correct degree of accuracyDocumentation: documentation does not match what program doesStress (overload): data structures filled past capacityCapacity: system performance unacceptable as activity reaches its specified limitTiming: coordinating of events not correctThroughput: system does not perform at speed requiredRecovery: failure encountered and does not recover (crashed)

Who Are Involved?Professional/Third party testers: organize and run testsAnalysts: involved in system requirements definition and specificationDesigners: involved in the design and understand proposed solution and solution’s constraintsImplementers: involved in the implementation and understand the constraints associated with implementationConfiguration management representative: arranges for changes to be made consistently across all artifacts

The AdviceView testing as part of the development processTesting is the last line of defense: Errors indicate that there is a problem with the development process

Test Plan Objectives of Test Plan:

Facilitate task of testing (strategy):• The scope, approach, resources, and schedule• Test techniques• Test criteria• Test documentation requirements• Avoid repetition• Improve test coverage• Improve test efficiency• Provide structure for final tests

Improve communication about testingProvide structure for:• Organizing• Scheduling• Managing

Test Plan ReportTable of ContentsDOCUMENT CONTROLApprovalDocument Change ControlDistribution ListChange Summary1. INTRODUCTION

1.1. Purpose 1.2. Scope 1.3. System Overview 1.4. Test Approach Overview

2. APPLICABLE REFERENCES3. TESTING APPROACH4. TEST SCHEDULE5. TEST XX

3. TESTING APPROACHSpecify types of tests to be performedList specific testsTest descriptions are in section 5May include• Test Management Requirements: how testing is to be managed• Personnel Requirements• Hardware Requirements• Software Requirements• Cost

Section 5 and laterTest No.Current Status (Passed / Failed / Pending)Test titleTesting approachConcluding RemarksTesting TeamDate Completed:

Assignments Lead: V&V Due Date: Tuesday, April 19, 2011 Due Time: Midnight MDT