Software Metric; defect removal efficiency, Cyclomate Complexity

Defect SeedingMutation Testing

• Cyclomate Complexity

• Basis Path Testing During Integration• Basis path testing can also be applied to

integration testing when units/components are integrated together

• McCabe’s Design Predicate approach:• Draw a “structure chart”• Calculate integration complexity• Select tests to exercise every “type” of

interaction, not every combination

Types of InteractionsTypes of unit/component interactions:

• Unconditional – Unit/component A always calls unit/component B. A calling tree

always has an integration complexity of at least one. The integration complexity is NOT incremented for each occurrence of an unconditional interaction.

• Conditional – Unit/component A calls unit/component B only if certain conditions

are met. The integration complexity is incremented by one for each occurrence of a conditional interaction in the structure chart.

• Mutually Exclusive both)– Unit/component A calls either unit/component B or

unit/component C (but not based upon certain conditions being met.) The integration complexity is incremented by one for each occurrence of a mutually exclusive conditional interaction in the structure chart.

• Iterative – Unit/component A calls unit/component B one or more times based

upon certain conditions being met. The integration complexity is incremented by one for each occurrence of an iterative interaction in the structure chart.

• Integration Basis Path Test Cases• Basis path testing can also be applied to

integration testing when units/components are integrated together

• Complexity: 5• Basis set of paths:– 1. A,B,C,E & then A calls I– 2. A,B,C,F & then A calls I– 3. A,B,D,G (only once) & then A calls I– 4. A,B,D,G (more than once) & then A calls I– 5. A,B,D,G,H & then A calls I

• Integration Basis Path Testing • Interaction types:– 2 conditional– 1 mutually exclusive conditional– 1 iteration– 1 simply because we have a graph

• Basis set of paths:1. A,C,F (only once)2. A,C,F (more than once)3. A,B,D,G & then B calls E &then A calls C4. A,B,D,H & then B calls E & then A calls C,F5. A,B,D,H & then B calls E, I & then A calls C,F

Defect removal efficiency (contd.)

• Defect removal efficiency is very complicated, as different kinds of defect removal operations have different removal efficiency.

• As illustrated in next example; various defect removal operation at various phases of software development yields different defect removal efficiency.

• Defect Removal Effectiveness DRE= Defects removed during development phase x100%

Defects latent in the productDefects latent in the product = Defects removed during

development phase+ defects found later by user

• The Defect Removal Effectiveness for each of the phases would be as follows:• Requirements DRE = 10 / (10+3+0+2+1) x 100% = 63%• Design DRE = (3+18) / (3+0+2+1+18+4+5+2) x 100% = 60%• Coding DRE = (0+4+26) / (0+2+1+4+5+2+26+8+7) x 100% = 55%• Testing DRE = (2+5+8) / (2+1+5+2+8+7) x 100% = 60%

• Development DRE = (Pre-release Defect) / (Total Defects) x 100% (10+3+2+18+4+5+26+8) / (10+3+2+1+18+4+5+2+26+8+7) x 100 = 88%

Example:

Defect seeding

• The concept of defect seeding is to insert a known errors into a software deliverable prior to review, inspection or testing.

• For example suppose 100 known errors are inserted into the system prior to integration testing and test case detects only 30 errors, it concludes that integration testing is 30% efficient.

Fault injection technique

• In the context of software engineering, fault injection is a technique or methodology meant for improving the test coverage.

• This is usually done by introducing faults in the program source code in order to test the code paths.

• In particular it is done to test the error handling test paths that might otherwise be left un- followed and thus, untested.

Fault injection technique (contd.)• The injected fault goes through a well defined cycle before it

becomes an observable failure. • After the fault is injected, the program or the application is

executed. • Upon execution, the injected fault may cause an error which is to be

considered as an invalid state within the boundaries of the system. • This error may cause many further errors within the boundaries of

the system. • In this way each error acts as an injected fault and propagates to the

system boundaries and comes to observance. • When the states of the errors are observed at the boundaries of the

system they are known as failures. This whole mechanism is known as “fault – error – failure – cycle”.

• The fault injection technique was used to introduce faults at the level of hardware system.

• This type of fault injection method is known as HWIFI or hardware implemented fault injection.

• It tends to simulate failures within a hardware system. • Soon after that it was found that faults could also be introduced

in the software system and it could be helpful in accessing the software system more appropriately.

• These fault injection techniques are collectively known as SWIFI or software implemented fault injection.

Categories of fault injection techniques

• Techniques for software implemented fault injection can be classified into two major categories namely:

• Compile time injection: – This can be defined as an injection technique in which the source code is modified

so that the simulated faults can be injected into the system. – One popular method for doing this is “mutation testing” in which the existing code

is changed so as to induce faults. – the faults produced are very similar to those added unintentionally by the

programmers. – Another technique which is modification of code mutation technique known as code

insertion fault injection adds code rather than modifying the existing piece of code. – This is done with the use of simple functions which take an existing value and

perturb (disturb) it using some logic into another value. Such functions are called perturbation functions.

• Run time injection: – This technique can be defined as the technique which makes use of a

software trigger to inject fault in the system which is executing. – Using this technique faults can be injected in a variety of ways such as

those listed below:– Time based triggers

An interrupt is generated when the timer reaches a specified time. The interrupt handler associated with the timer will inject the fault.

– Interrupt based triggers Software trap mechanisms and hardware exceptions are effectively used to generate a fault in the code of the system at a particular point. This gives instant access to a specific memory location.

Hardware faults• Hardware/Physical Fault that arise during system operation are best classified by their

duration:– Permanent– transient– intermittent.

• Permanent faults: Caused by irreversible component damage, such as a semiconductor junction that has shorted out because of thermal aging, improper manufacture, or misuse. Since it is possible that a chip in a network card that burns causing the card to stop working, recovery can only be accomplished by replacing or repairing the damaged component or subsystem.

• Transient faults: Triggered by environmental conditions such as power-line fluctuation, electromagnetic interference, or radiation. These faults rarely do any lasting damage to the component affected, although they can induce an erroneous state in the system. According to several studies, transient faults occur far more often than permanent ones, and are also far harder to detect.

• Intermittent faults: Caused by unstable hardware or varying hardware states. They can be repaired by replacement or redesign.

Software faults

• Function faults: Incorrect or missing implementation that requires a design change to be corrected.

• Algorithm faults: Incorrect or missing implementation that can be fixed without the need of design change.

• Timing/serialization faults: Missing or incorrect serialization of shared resources.

• Checking fault: Missing or incorrect validation of data, or incorrect loop, or incorrect conditional statement.

• Assignment fault: Values assigned incorrectly or not assigned.

Mutation Testing

• Can trace birth of Mutation Testing back to a student paper written in 1971• To assess the quality of the tests by

performing them on mutated code

• To use these assessments to help construct more adequate tests

• To thereby produce a suite of valid tests which can be used on real programs

Testing Method• Mutant processes are created to try to

mimic typical syntactic errors made by programmers

• Many differing mutants are run against the specified tests to assess the quality of the tests

• The tests are attributed a score between 0 and 1, as to whether they can distinguish between the original and the mutants

Traditional Syntactical Mutation Operators

• Deletion of a statement• Boolean:

• Replacement of a statement with anothereg. == and >=, < and <=• Replacement of boolean expressions with true or false

eg. a || b with true

• Replacement of arithmeticeg. * and +, / and -

• Replacement of a variable

If process is not error-free, fix it

Test Mutants

The Mutation Process

Process

MutationMutationMutation Tests

Test Process

Create Mutants

Yes

Test CompleteNo

Any Live Mutants?

Problem with Tests?

Any Mutations that are caught by tests are killed

New Test Data

limitation• Mutation testing can only target a sub-set of

all the possible faults a programmer could possibly make.

• Therefore it cannot be guaranteed that all possible errors can be detected with the test cases.