1

Agenda for verify activity

1. Objective2. Verification methods3. Test documentation4. Test execution5. Test results6. Reviews7. Special tests

2

1. Objective

Verify activityVerify tasksCompletion criteriaPseudo-completion criteria

1. Objective

3

Verify activity

Verifies that the product complies with all its requirements, and documents these test results

1. Objective

4

Verify tasks

verifyproduct by ins, demo,

test

write test plan

write test spec

verifyproduct

by analysis

writetest

procedure

product spec

test readinessreview (TRR)

verification review (VR)

test results

test plan

product

test spec

test proc

insdemoteststa

ana

product design

1. Objective

5

Completion criteria

Customer agrees testing is complete

1. Objective

6

Pseudo-completion criteria

Testing completeProduct verification review complete

1. Objective

7

2. Verification methods

InspectionAnalysisDemonstrationTestDistinction among methods

2. Verification methods

8

Inspection (INS) (1 of 2)

Definition: Verification without the use of special laboratory appliances or procedures. • Generally non-destructive. • Includes sight, hearing, smell, touch, and taste • May include simple physical manipulation• May include mechanical & electrical

measurement• May include other forms of investigation

2. Verification methods

9

Inspection (INS) (2 of 2)

Done when convenient, normally when no further changes to the configuration are planned

Examples• Weighs less than 10 pounds• Surface free of visible defects

2. Verification methods

10

Analysis (ANA) (1 of 2)

Definition: Verification by technical or mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals

2. Verification methods

11

Analysis (ANA) (2 of 2)

• Verification through the technical evaluation of data

• Data may may include lower-level product test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation

• MTBF and MTBCF analysis may include failures from testing

12

Demonstration (DEM)

Definition: Verification using actual operation, adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored.

Examples• Closed caption text appears when

commanded• Light goes off when safety switch activated

2. Verification methods

13

Test (TST)

Definition: Verification by technical means, including operation, and involves application of scientific principles and procedures.

Many opinions say that test involves exercising the product and collecting data that needs to be reduced

Test examples• Signal to noise ratio > 2• Vibration power spectral density <0.2 g2/Hz at 20

Hz

2. Verification methods

14

Distinctions among methods (1 of 2)

SubjectiveAgreement between customer &contractorCrude distinction

• Analysis -- Verification that can occur without the product being present

• Inspection -- Simple verification without exercising the product

• Demonstration -- Simple verification while exercising the product

• Test -- Everything else

2. Verification methods

15

Distinctions among methods (2 of 2)

Importance of distinguishing among methods• Distinction is not terribly important• Inspection represents less risk • Inspection and demonstration are easier

than test• Many methods change when test spec

written

2. Verification methods

16

3. Test documentation

Test planTest specificationTest procedure

4. Test documentation

17

Test plan (1 of 3)

Generated by the contractorDefines approach to testingEstablishes approach to test methods

• Demonstration and test -- defined in test spec• Analysis -- defined in verification by analysis

plan• Inspection -- defined as inspection directives

4. Test documentation

18

Test plan (2 of 3)

Defines roles of test specs and test procedures

Defines how to capture test resultsDefines reviews

4. Test documentation

19

Test plan (3 of 3)

Test and evaluation master plan (TEMP)• Special case of test plan• Generated by customer• Defines customer plan for evaluating product• Applies to the highest level government

product

3. Test documentation

20

Test specification (1 of 2)

Generated by contractorDefines agreement between customer and

contractor about what each demonstration and test must do

Includes the pass/fail criteriaIs not a substitute for the product specification

3. Test documentation

21

Test specification (2 of 2)

Analysis plan• Generated by contractor• Defines agreement between customer and

contractor about what each analysis must do

• Includes the pass/fail criteria• Is not a substitute for the product

specification

3. Test documentation

22

Test procedure (1 of 3)

Generated by the contractorDefines specific test set upDefines specific test process for running testApplies to demonstration, test, and selected

test and analysis

3. Test documentation

23

Test procedure (2 of 3)

Inspection directive• Generated by contractor• Defines agreement between customer and

contractor about what each inspection must do

• Includes the pass/fail criteria• Usually a single sheet that also documents

results of test• Is not a substitute for the product

specification

3. Test documentation

24

Test procedure (3 of 3)

Requirements Design Build

Test spec Test description Test procedure

Test description• Expands detail between the test spec &

procedures• Analogous to design• Allows people to agree on details of test before

starting to write test procedure

is similar to

3. Test documentation

25

4. Test execution

SimulationConfigurationTestsRegression tests

4. Test execution

26

Simulation (1 of 2)

A test tool that reduces costVerifies extremes that would be hard to

produce in actual testsSupports performance and environmental

testing

4. Test execution

27

Simulation (2 of 2)

Simulation verification• Simulations need to be validated for

accuracy• Customer may not have confidence in

simulation if the phenomena being simulated is complicated; e.g. software throughput

4. Test execution

28

Configuration

Tests should be done on configured hardware and software to ensure repeatability

4. Test execution

29

Tests

May be run in a lab, in the manufacturing environment, when installed, or when used in the field

Important tests may warrant a quick-look report giving test results within 24 hours of the test

4. Test execution

30

Regression tests

Definition: Tests that are run to confirm that a product performance is OK after a change to the product

Run after originally scheduled verification testFor example, if a new device driver is added to

an already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver

4. Test execution

31

5. Test results

Test method

Analysis (ANA)

Inspection (INS)

Demonstration (DEMO)

Test (TEST)

Pass/fail

VDB

VDB

VDB

VDB

Analysis

Study

n/a

n/a

Study

Test

Sheet

Sheet

Sheet

Report

Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports

provide supporting data

Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports

provide supporting data5. Test results

32

6. Reviews

Readiness reviewProduct verification reviewFailure review board

6. Reviews

33

Readiness review

Obtains customer concurrence that contractor is ready to perform test

Confirms that test procedures are satisfactory, that product to be tested will be configured, and that correct people will witness the test

Presents schedule of testing, analysis, and review

6. Reviews

34

Product verification review (1 of 2)

Obtains customer concurrence that testing is satisfactory

Held after testExamines results of tests, including analysis

6. Reviews

35

Product verification review (2 of 2)

List tests that passExplains exceptions, problems, and failures;

addresses deviations and waiversMay be executed as a series of reviews with

each review associated with one or more tests

6. Reviews

36

Failure review board

Examines each hardware failureDefines failure mechanism such as

electrostatic discharge, manufacturing defect, or manufacturing

Defines impact of failureDefines options for preventing future failures

6. Reviews

37

7. Special tests

Reliability testingDesign of experimentsAccelerated life testingMaintainability and testabilityEnvironmental stress screening (ESS)Electromagnetic testingFailure report and corrective action system

7. Special tests