Component Testing: Part I

April 11, 2023 1

Component Testing: Part I

Ednaldo Dilorenzo de Souza [email protected]

mailto:[email protected]

April 11, 2023 2

Summary

Introduction Fundaments of Testing Software Testability Test Case Projects Environment Testing, Architecture and Specialized

Applications Software Test Strategies Component Testing

April 11, 2023 3

Introduction

Testing is an important process to support assurance; As software becomes more pervasive and is used more often to

perform critical tasks, it will be required to be of higher quality; Because testing requires the execution of the software, it is

often called dynamic analysis [Harrold]; Testing consists in compare the outputs from the executions

with expected results to identify those test cases on which the software failed;

Testing cannot show the absence of faults, if can show only their presence.

April 11, 2023 4

Fundaments of Testing

Testing goals: Find faults in a software program; A good test case have a high probability to find errors; A good test process should find errors;

Testing fundamentals: Testing should be connected with client requirements; Testing should be planned before testing execution; Isolation of suspected components; Testing should start in individual components; Complete testing is imposible; Testing shouldn’t be executed for developers;

April 11, 2023 5

Software Testability

“Software Testability means how easy the software is for testing” [PRESS];

There are lots of characteristics of a testable software: Operability; Observables; Controllability; Decouplability; Simplicity; Stability; Understandably;

April 11, 2023 6

How to design for software testability and how to measure the degree to which you have achieve it?

Software testability is the process of investigate where software faults are and how to execute [Voas];

Testability provide some guidance for testing; Testability can give you confidence of correctness in fewer

tests;

Software Testability

April 11, 2023 7

Testability’s goal is to assess software accurately enough to demonstrate whether or not it has high quality;

There are two ways to reduce the number of required tests: Select tests that have a great ability to reveal faults; Design software that has a greater ability to fail when faults do

exist (design for testability); There are several criteria on program design:

More of the code must be exercised for each input; Programs must contain constructs that are likely to cause the

state of the program to become incorrect if the constructs are themselves incorrect;

Programs must be able to propagate incorrect states into software failures;

Software Testability – Design for Testability

April 11, 2023 8

Information Loss; Implicit Information Loss; Explicit Information Loss; Design Heuristics; Specification Decomposition; Minimizations of variation reuse; Analysis;

Software Testability – Design for Testability

April 11, 2023 9

“Sensitive Analysis quantifies behavioral information about the likelihood that faults are hiding” [Voas];

Repeatedly executes the original program and mutations of its source code and data states using two assumptions: Single-fault assumption; Simple-fault assumption;

The specific purpose of sensitivity analysis is to provide information that suggests how small the program’s smallest faults are likely to be;

The strength of sensitivity analysis is that your prediction is based on observed effects from actual faults;

The weakness is that the faults injected and observed are only a small set from what might be an infinite class of faults;

Software Testability – Sensitive Analysis

April 11, 2023 10

Execution Analysis; Infection Analysis; Propagation Analysis; Implementation;

Software Testability – Sensitive Analysis

April 11, 2023 11

Main goal of Test Projects is having a great probability of finding faults in software;

Hitachi Software, has attained such a high quality software that only 0.02 percent of all bugs in software program emerge at user’s site [Yamaura];

Test Case Projects

April 11, 2023 12

Test Case Projects

Documenting Test Cases - Benefits Designing test cases gives you a chance to analyze the

specification from a different angle; You can repeat the same test cases; Somebody else can execute the test cases for you; You can easily validate the quality of the test cases; You can estimate the quality of the target software early on;

Schedule, Cost, and Personnel An average project might apportion 12 months spend 2 months in

testing by quality assurance team; The test-case density; A project with 100,000 LOC needs approximately 10,000 test

cases – 1,000 per programmer;

April 11, 2023 13

Test Case Projects

Steps for Debugging and Testing: Design a set of test cases; Check the test cases; Do code inspection based on

the test cases; Do machine debugging based

on the test cases; Collect quality-related data

during debugging; Analyze the data during

debugging;

April 11, 2023 14

Main ways of testing a software: White-box Testing;

Basic Way Testing; Graph Flow Notation; Cyclomatic Complexity; Test Case Derivation; Graph Matrix;

Control Structure Testing; Condition Testing; Data Flow Testing; Cycle Testing;

Black-box Testing; Graph Based Testing Method; Partition Equivalence; Comparison Testing; Orthogonal Matrix Testing;

Test Case Projects

April 11, 2023 15

Environment Testing, Architecture and Specialized Applications

GUI Testing; Client/Server Architecture Testing; Documentation and Help Devices Testing; Real Time Systems Testing;

April 11, 2023 16

Software Test Strategies

Integration of test case projects methods in well planed steps, resulting in a good software construction;

It should be sufficiently flexible for holding test method and hard for holding planning and management;

A software strategy should execute low and high level tests;

April 11, 2023 17


A Strategic Approach for Testing Software Verification and Validation

Verification means the activities that assure the software implements a function correctly;

Software Test Organization Software developers should test execute only unit tests; Independent test group (ITG) is responsible for execute destructive

tests in the software integration; Software developers should be available for correcting system’s faults;

A Software Testing Strategy Unit Test; Integration Test; Validation Test; System Test;

April 11, 2023 18


A Strategic Approach for Testing Software Testing Completion Criterions

When the test phase is finished? Based on Statistical Criterions [PRESS]:

f(t) = (1/p) ln[l0 pt + 1]

April 11, 2023 19


Strategically Aspects Specify product requirements in a quantifiable way before

test; Show test goals in a clear way; Understand users software and develop a profile for each

user category; Develop a test plan that emphasize a fast cycle test; Build a robust software projected for testing itself; Use effective technical formal revisions as a filter before

testing; Conduct technical formal revisions for evaluate a test

strategy and test cases; Develop an approach for improve the test process;

April 11, 2023 20


Unit Test Used for testing software components separately ; It uses white-box test for exercising code lines; Test cases should be planned with the expected results for

comparison;

April 11, 2023 21


Integration Test If separated components work, why should I make integration

tests? Loosing data through an interface; A module should have a not expected effect;

Top-Down Integration; Bottom-up Integration; Regression Test; Smoke Test;

April 11, 2023 22


Top-Down Integration First-In-Depth Integration; First-In-Width Integration; The main control module is used first for testing; The next modules are being added; Tests are being executed while components are added; Regression tests can be executed for assure new errors are not added;

M1

M2

M7M5 M6

M3

M8

M4

April 11, 2023 23


Bottom-up Integration Atomic modules are added before in the system; Components are being placed when low level components are

tested;

Mc

Ma Mb

D1 D2 D3

April 11, 2023 24


Regression Test All components added in the software may cause errors in

components added before; Regression test strategy test components tested before to assure

new components didn’t affected it; Some tools can be used for execute regression test; It should be separated in modules;

Analyzing Regression Test Selection Techniques [Rothernel] Regression Test Selection for Fault Detection; Framework for Analyzing Regression Test Selection Techniques

Inclusiveness; Precision; Efficiency; Generality; Tradeoffs;

April 11, 2023 25


Analyzing Regression Test Selection Techniques An Analysis of Regression Test Selection Techniques

April 11, 2023 26



April 11, 2023 27



April 11, 2023 28


Smoke Test Used in software products developed fast; Modules built by components are tested; Modules are integrated and tested daily:

Integration risk; Final product quality; Errors diagnose are simplified; Easy progress evaluation;

April 11, 2023 29


Validation Tests Started just after integration tests; Used for verify results based on requirements of the system; An important element in validation test is called configuration

revision; Customer should validate requirements using system; Alfa tests are tests executed in the user environment in the

developer presence; Beta tests are executed by the user without developer’s presence;

Validation, Verification, and Testing: Diversity Rules [Kitchenham] Practical Problems with Operational Testing

It assumes that the most frequently occurring operations will have the highest manifestation of faults;

Transition situations are often the most complete error-prone;

April 11, 2023 30


Validation, Verification, and Testing: Diversity Rules [Kitchenham] Testing Critical Function

Critical functions have extremely severe consequences if they fail; This leads to a problem with identifying the reliability of the system as

a whole; The product reliability can be defined in terms of failure in a given

period of time; For critical functions, you are likely to obtain a measure of reliability

related of failure on demand; The Need For Non-Execution-Based Testing

You can only ignore nonoperational testing if: All faults found by execution testing have exactly the same cost to debug

and correct All faults can be detected, and The relative size of faults is constant across different operations for all time

April 11, 2023 31


Validation, Verification, and Testing: Diversity Rules [Kitchenham] Diminishing Returns

Operational test becomes less and less useful for fault detection when it have been removed faults from the most commonly used functions;

A systems engineer should mix some techniques, including: Additional testing boundary and transition conditions and of critical functions Design inspections and reviews to identify specification and design faults

early in development process Proofs for functions whose correct outcome cannot otherwise be verified,

and Operational testing for those initial tests aimed at identifying the largest

faults and assessing reliability.

April 11, 2023 32


System Test Should exercise the system as a whole for find errors; Recovery Test:

Enforce the system to fail; Verify the system recovery;

Security Test: Verify if hackers can affect the system; Testers should try to obtain secure data from the system;

Stress Test Execute system in a way it should require almost all resources; Sensibility Test

Try to find data that the system fail;

Performance Test For systems need to provide performance requirements; Executed with Stress Test;

April 11, 2023 33


Debugging Process started after the tester find an error and try to remove it; Relation between a fault and its cause; Two possible results:

The cause is found and corrected; The cause is not found;

Why debugging is so difficult? Error and cause geographically remote; Error can disappear when another error was corrected; Error can be caused by non-error; Error can be cause by an human mistake; Error can be a time problem; Error can be difficult to be found; Error can be caused by distributed reasons; Error can be found in a component;

April 11, 2023 34

Component Testing

Testing Component-Based Software: A Cautionary Tale [Weyuker] The Ariane 5 Lesson

In June 1996, during the maiden voyage of the Ariane 5 launch vehicle, the launcher veered off course and exploded less than one minute after take-off;

The explosion resulted from insufficiently tested software reused from Ariane 4 launcher;

Testing Newly Developed Software Unit Testing; Integration Testing; System Testing;

April 11, 2023 35

Component Testing

Testing Component-Based Software: A Cautionary Tale [Weyuker] Problems with systems built from reusable components:

Performance problems; Fitting selected components together; Absence of low-level understanding;

Developing a component for a particular project with no expectation of reuse, testing proceed as usual;

Significant additional testing should be done for changing priority; Modify or not the source code? Facilitating Reusable Component Testing

Associate a person or a team for maintaining each component; Add software specification; Add test suite; Add pointers between the specification and parts of the test suite;

April 11, 2023 36

Component Testing

Third-Party Testing and the Quality of Software Components [Councill] 10 contribuitors devoted 70 person-hour to developing a definition

of component; Ariane Project; Certification business introduced a Maturity Model focused on

large organizations; 99 percent of all US businesses are small businesses; Producers provide the developing and testing procedures,

purchasers or their agents conduct second-party testing and independent testing organizations perform third-party testing;

April 11, 2023 37

References

[PRESS] Pressman, Roger S., Engenharia de Software, 5° Ed., Rio de Janeiro, MacGraw-Hill, 2002.

[Councill] William T. Councill, Third-Part Testing and the Quality of Software Components, IEEE Software, Quality Time, July 1999.

[Adams] Tom Adams, Functionally Effective Functional Testing Primer, IEEE Software, bookshelf, September 1995.

[Rothermel] G. Rothermel and M. J. Harrold, Analyzing Refression Test Selection Techniques, IEEE Transactions of Software Engineering, Vol. 22, N° 8, August 1996.

[Stocks] P. Stocks and D. Carrington, A Framework for Specification-Based Testing, IEEE Transactions on Software Engineering, Vol. 22, N° 11, November 1996.

[Harrold] M. J. Harrold, Testing a Roadmap, College of Computing, Georgia Institute Technology, 2000.

April 11, 2023 38

References

[Franki] P. G. Franki and R. G. Hamlet, Evaluating Testing Methods by Delivered Reliability, IEEE Transactions on Software Engineering, Vol. 24, N° 8, August 1998.

[Kitchenham] B. Kitchenham and S. Linkman, Validation, Verification, and Testing: Diversity Rules, IEEE Software, August 1998.

[Voas] J. M. Voas and K. W. Miller, Software Testability: The New Verification, IEEE Software, May 1995.

[Yamaura] T. Yamaura, How to Design Practical Test Cases, IEEE Software, December 1998.

[Weyuker] E. J. Weyuker, Testing Component-Based Software: A Cautionary Tale, IEEE Software, October 1998.

Component Testing: Part I

Documents

Transcript of Component Testing: Part I