Grey Box testing

Tor Stålhane

What is Grey Box testing

Grey Box testing is testing done with limited knowledge of the internal of the system.

Grey Box testers have access to detailed design documents with information beyond requirements documents.

Grey Box tests are generated based on information such as state-based models or architecture diagrams of the target system.

State based testing The tests are derived from a state model of the

system. We can derive the state model in several way, e.g. from

• Expected system behavior• State part of a UML design or requirements

specification.• Other state diagrams

Most system will, however, have a large number of states

Binder’s state control faults – 1

Binder has make a list of common state –related problems in software systems. This list may be used as an input to

• State based testing• State machine or code inspection

Binder’s state control faults – 2 • Missing or incorrect

– Transitions – new state is legal but incorrect– Events – valid message ignored

• Extra, missing or corrupt state – unpredictable behavior

• Sneak path – message accepted when it should not be accepted

• Illegal message failure – unexpected message causes a failure

• Trap door – system accepts undefined message.

State test criteria

We can choose one or more of the following test selection criteria:

• All states – testing passes through all states• All events – testing forces all events to occur

at least once• All actions – testing forces all actions to be

produced at least once

State test strategiesAll round-trip paths• All transition sequences beginning and ending

in the same state• All simple paths from initial to final stateThis strategy will help you to find• All invalid or missing states• Some extra states• All event an action faults

Round-trip path tree – 1 A round-trip path tree • Is built from a state transition diagram• Includes all round-trip paths

– Transition sequences beginning and ending in the same state

– Simple paths for initial to final state. If a loop is present, we use only one iteration

• Is used to – Check conformance to explicit behavioral models– Find sneak paths

Round-trip path tree – 2

A test strategy based on round-trip path trees will reveal:

• All state control faults• All sneak paths – messages are accepted when

they should not • Many corrupt states - unpredictable behavior

Round-trip path tree – 3

It is important to remember that a test based on the round-trip path tree is model-based.

Full coverage from the round-trip path tree implies full model coverage.

Full path coverage – which we will look at later – implies full code coverage.

Challenge for round-trip path testing

In order to test a system based on state transitions via triggers, predicates (guards) and activities, we need to be able to observe and register these entities.

Thus, we may need to include “points of observations” in the code that gives us access to the necessary information.

Round-trip tree – small example

C

BA

C B A

A B A

A

a[p1] / w

b[p2] / ua[p1

] /

w

b[p2

] /

u

Transitions Each transition in a state diagram has the formtrigger-signature [guard] / activity. All parts are

optional• trigger-signature: usually a single event that

triggers a potential change of state.• guard: a Boolean condition that must be true

for the transition to take place.• activity: an action that is performed during the

transition.

Test description – 1

Each test completes one branch of the round-trip tree – from to . The necessary transitions describes the test case.

The table on the next slide shows the test case for -> A -> C -> A C B A

A B A

A

Test description – 2

ID Start state

Event Condition Reaction New state

1 constructor - - A

2 A a p1 w C3 C b p2 u A

C B A

A B A

A

b[p2] / u

a[p1] / w

C

BA

a[p1

] /

w

b[p2

] /

u

Sneak path test casesA sneak path – message accepted when it

should not be accepted – can occur if • There is an unspecified transition• The transition occur even if the guard

predicate is false

Sneak path test description ID Start

stateEvent Condition Reaction New

state1 constructor - - A

2 A c p1Error

messageA

3 A a p1 - falseError

messageA

C B A

A B A

A

b[p2] / u

a[p1] / w

b[p2] / u

a[p1] / w

C

BA

a[p1

] /

w

b[p2

] /

u

State diagram for a sensor - 1

B

E D

C

A

State diagram for a sensor - 2

A B

CE D

A

E B

D

E

E C

[sensor alarm] / sound alarm[no sensor alarm] / test

[false alarm] / test [alarm OK / request reset]

[ACK] / reset

[test OK][test fails] / replace

[test fails] / replace/ test

Sensor round-trip path tree

Acknowledgement

Most of the previous presentation is based on a slide set from the University of Ottawa, Canada

Mutation testing

Tor Stålhane

Type 1 mutation testing – 1 Type 1 mutation testing is done as follows:1. Write a chunk of code2. Write a set of tests 3. Test and correct until the test suite runs

without errors4. Change a random part of the code – e.g. a

“+” to a “-”. This is called a code mutant. We will only consider mutants that compiles without error messages

5. Run the test suite again

Type 1 mutation testing – 2

6. If the tests suite– runs without errors, then we need to extend the

test suite until we discover the defect.– diagnoses the defect and got back to step 4 to

create a new mutant.

The test process stops when all of X new mutants are discovered by the current test suite.

Type 2 mutation testing

Type 2 mutation testing – also called “fuzzing” – has many ideas in common with random testing. The main difference is that:

• Random testing requires us to generate random tests from scratch.

• Type 2 mutation testing starts with an input that works OK and then change part of it in a random way.

Software functions are not continuous

When we discuss mutation testing, it is important to remember that a function implemented in software is not continuous. E.g. x = 2.00 and x = 1.99 can give dramatically different results.

A small changes in input can have a large effect on the output.

Type 2 mutation testing example – 1

SUT – a system for computing F(x) – takes an input consisting of

• F – a three character string identifying a probability distribution function.

• A real number x. The allowed value range will depend on F, e.g. if F = “ exp”, then x must be a positive number, while if F = “nor”, then x may be any number.

Type 2 mutation testing example – 2We can perform type 2 mutation testing as follows:1. Run a test with input <“exp”, 3>2. Check that the result is correct3. Make a mutant by drawing a random integer value 1

(F) or 2 (x). – If we draw a 1, generate a random integer n from 0 to 10 – string size – and

generate a random string of length n– If we draw a 2, generate a random real value x

4. Compute F(x)5. Check the result – especially any error messages6. If we are satisfied then stop, otherwise repeat from

step 3

Mutant testing strategies – 1 The number of possible mutants is large. In

order to have a reasonably sized test set there are several strategies for reducing the number of mutated systems of components.

The following slides will give a short description of these strategies. Those who will use the strategies should consult the paper by Papdakis and Malevris on mutation testing.

Mutant testing strategies – 2 Mutation testing strategies are either of the first

order or the second order.• First order strategy – perform a random

selection of a portion of the generated mutants – e.g. 10% or 20%.

• Second order strategy – combine two mutants to get one component to test. The strategy will differ depending on how we choose the mutants.

Assumptions

The following strategies assume that• Different mutants of the same code can be

combined – i.e. they do not concern the same operator, ID etc.

• Mutants that do not introduce errors – e.g. mutations of comments – are removed from the mutant set.

Second order mutation testing – 1 • Random Mix: combine two mutants selected

at random from the mutant set. Delete after selection to avoid choosing the same mutant twice.

• First2Last: sort all mutants according to code line number. Combine the first with the last, the second with the second-to-last and so on.

• Same Node: same as Random Mix but selected from the same code block

Second order mutation testing – 2 • Same Unit: same as Random Mix but selected

from the same unit – e.g. class or method• DiffOp: same as Random Mix but we select

mutants where different operators are mutated – e.g. one mutation of a “+” and one mutation of a “>”.

The strategies described above can also be combined – e.g. SU_DiffOp consists of using the DiffOp strategy but only on two mutation from the same unit

Effectiveness measuresWe will use two measures for mutant test

effectiveness – test effectiveness and cost effectiveness.

Note that low values implies good results – number of exposed faults is large.

FaultsExposedofNo

testcasesofNoessEffectivenTest

.

.

FaultsExposedofNo

tsMuEquivalentofNotestcasesofNoessEffectivenCost

.

tan..

Comments to First Order mutants

Selecting 10% of all generated mutants is best both with regard to cost effectiveness and test effectiveness.

Strong Mutation – using all generated mutants – is the worst.

Comments to Second Order mutants

• SU_F2Last – Same Unit and First combined with Last – scores highest on test effectiveness

• Random Mix – score highest on cost effectiveness

• No second order strategy is more effective than the Rand(10%) strategy. Here we have that FD = No. of test cases / 1.34