Project FoX: A Tool That Offers

Automated Testing Using a

Formal Approach

Ivo Neskovic

Agenda

• Software Engineering: What could go wrong?

• Formal Methods

• Project FoX

• Case Study: The buffer system

• Conclusions and Future Work

• Bibliography

The Problem of Software Engineering

• Faulty systems are a common notion nowadays.

• SE is an engineering discipline, yet lacking the

engineering formality.

• Subjective and informal testing.

• Impossible to prove that the system:

– Does what it is supposed to do.

– Does not do what it is not supposed to do.

• Needs structured and precise system designs.

Formal Methods • The applied mathematics of computer systems engineering, used to

specify and model the behaviour of a system and mathematically verify

that the system design and implementation satisfy functional and safety

properties.

• Specification Languages:

– Abstract State Machines

– Generalized State Machines

– Communicating Sequential Processes

– Specification and Description Language

– Petri Nets

– Temporal Logic of Actions

– B and event – B method

– Z

Formal Methods at a Trial

• Benefits:

– Specification may be used as a basis for proving the presence or lack

of certain properties in the design and by inference in the developed

system.

– Mathematical proof of correctness (Theorem proving).

– Model checking (Proving desired properties in a design).

– Formal Testing.

• Used mainly for safety critical systems such as aerospace

engineering.

• Criticism:

– Expensive and time consuming approach (though questionable).

– Lack of tooling support.

Incorporating Formal Methods in the

Development Cycle

Project FoX • Produce the complete set of test cases from a formal specification.

• Execute the tests on the systems implementation.

• Locate errors and non-equivalences and report them to the user.

• Developed in Java for Java.

• Compatible with Java Standard Edition, Enterprise Edition, Mobile

Edition.

• Can be extend to work in conjunction with popular Java frameworks.

• Operates on compiled bytecode with the addition of a few specific

annotations.

• Utilizes the test drivers of JUnit.

• FoX provides a bridge between regular Java developers and the benefits

of complete positive and negative testing, proven to find all faults.

Using Project FoX

• Two artefacts necessary:

– Formal specification of the system.

– The system’s implementation.

Buffer Case Study – Description • Simple buffer in a factory.

• Accepts parts, any parts.

• Parts have a name and an ID.

• The buffer has a capacity of 2.

• The buffer can be empty, partially

full or completely full.

• Supports adding and removing

items.

• If the capacity is reached, no

additional items can be placed in

the buffer unless and item is

removed firsts.

Buffer Case Study – Formal

Specification • Modelled as a Generalized State

Machine (stream X-Machine).

• A theoretical model of computing,

pioneered by Samuel Eilenberg

in1974 (X-Machine).

• Separates flow control from

processing.

• Flow control is abstracted to a level

suitable for representation as a finite

state machine.

• Complex data structures are modelled

as an infinite memory.

• Able to model both static (data) and

dynamic (control) parts of a system.

Buffer Case Study – Formal

Specification (cont.) • Simple buffer in a factory.

< xMachine name = " Buffer " >

• The buffer can be empty, partially full or completely full.

< states >

< state initialState = " true " > empty </ state >

< state > non_empty </ state >

< state > full </ state >

</ states >

Buffer Case Study – Formal

Specification (cont.) • Accepts parts, any parts.

< input name = " part " ref = " BufferObject " / >

• The buffer has a capacity of 2.

< types >

< builtInType name = " capacity " type = " integer " / >

< builtInType name = " buffer " type = " set: BufferObject " / >

</ types >

< memory >

< memoryBlock ref = " buffer " initialValue = " null " / >

< memoryBlock ref = " capacity " initialValue = " 2 " / >

</ memory >

Buffer Case Study – Formal

Specification (cont.) • Parts have a name and an ID.

< types >

< complexType name = " ItemType " >

< attributes >

< builtInType name = " type " type = " string " / >

</ attributes >

</ complexType >

< complexType name = " BufferObject " >

< attributes >

< complexType name = " type " ref = " ItemType " / >

< builtInType name = " itemId " type = " integer " / >

</ attributes >

</ complexType >

< /type >

Buffer Case Study – Formal

Specification (cont.) • Supports adding and removing items.

< functions >

< function name = " add_part " >

< guard >

!buffer. contains ( part ) && buffer .

size () + 1 < capacity . value ()

</ guard >

< body > buffer . add ( part ) ; </ body >

< output > Part Added </ output >

</ function >

...

</ functions >

< transitions >

< transition >

< startingState >

empty

</ startingState >

< appliedFunction >

add_part

</ appliedFunction >

< endingState >

non_empty

</ endingState >

</ transition >

...

</ transitions>

Buffer Case Study – Implementation public class BufferObject {

private int itemId;

private ItemType type;

public BufferObject(int itemId,

ItemType type) {

this.itemId = itemId;

this.type = type;

}

}

public class ItemType {

private String type;

public ItemType(String type) {

this.type = type;

}

}

Buffer Case Study – Implementation • @Xmachine - annotating the class representing the system modeled

with the specification.

• XMachineModel – a class representing the model, containing a number

of useful helper methods.

@XMachine(inputType = "BufferObject",

sampleInputs = {

"integer: 10, ItemType: (string:Box)",

"integer: 17, ItemType: (string:HeavyBox)",

"integer: 25, ItemType: (string:ReallyHeavyBox)"

})

public class Buffer extends XMachineModel {

Buffer Case Study – Implementation • @XMMemoryBlock – a field level annotation, associating Java data

structures with their specification equivalents.

@XMMemoryBlock(name = "buffer")

private List<BufferObject> buffer;

@XMMemoryBlock(name = "capacity")

private int capacity;

public Buffer() {

super("Buffer");

buffer = new LinkedList<BufferObject>();

capacity = 2;

}

Buffer Case Study – Implementation

• @XMFunction – a method level annotation, referencing the

modeled functions implementations.

• reportOutcome( outcome: String) – one of the many helper

methods of the XMachineModel class.

@XMFunction(name = "add_part")

public void addPart(BufferObject part) {

if (!buffer.contains(part) && buffer.size() + 1 <

capacity) {

buffer.add(part);

reportOutcome("Part Added");

}

Buffer Case Study – Executing Fox

Buffer Case Study – Executing FoX

(implanted error) if (!buffer.contains(part) && buffer.size() + 1 <

capacity) {

buffer.add(part);

capacity++;

reportOutcome("Part Added");

}

Buffer Case Study – Generated Test

Cases • Tests report the sequence of inputs used for the specific scenario, the

sequence of expected outputs and the actual output.

• Outcome is reported to the user via the usual JUnit red / green

notifications.

<tests>

…

<test testID=”2”>

<input>[ itemId: 17 type: HeavyBox, itemId: 10 type: Box]</input>

<expectedOutput>

[ Part Added, Part Added – Become Full ]

</expectedOutput>

<output>[ Part Added, Part Added – Become Full ]</output>

</test>

…

</tests>

Conclusions and Future Work

• FoX enables developers to leverage the already

proven theories for formal testing.

• Provides a fully automated testing process, ranging

from complete test set generation (satisfying some

design for test conditions), to test preparation and

execution.

• Operates on any Java based software system,

being transparent to it's underlining technologies.

• Provides complete positive and complete negative

testing.

Conclusions and Future Work (cont.)

• Next steps:

– Thorough evaluation.

– An additional tool to make the specification step easier

and closer to the developer, aiming to “hide” the formality

as much as possible.

– NetBeans and Eclipse integration.

– A standalone X-Machine IDE providing additional related

functionalities.

– Branch out to other languages and frameworks (eg. C#

and .NET).

Bibliography • S. Eilenberg, Automate, Languages and Machines, Vol. A. Academic Press,

London, 1974.

• M. Holcombe, “X-Machines as a basis for dynamic system specification,”

Software Engineering Journal, vol. 3(2), pp. 69-76, 1988.

• F. Ipate and M. Holcombe, “Specification and Testing using Generalized

Machines: a Presentation and a Case Study,” Softw. Test. Verif. Reliab, vol. 8,

pp. 61-81, 1998.

• M. Holcombe and F. Ipate, Correct Systems: Building a Business Process

Solution. Springer, Applied Computing Series, November 1998.

• G. Eleftherakis and A. Cowling, “An Agile Formal Development Methodology,” in

1st South Eastern European workshop on Formal Methods (SEEFM 03),

(Thessaloniki), pp. 36-47, Nov. 2002. Agile Formal Methods: Practical, Rigorous

Methods for a changing world.

• P. Kefalas, G. Eleftherakis, and E. Kehris, “Communicating X-Machines: a

practical approach for formal and modular specification of large systems,”

Information and Software Technology, vol. 45, pp. 269-280, Apr. 2003.

Thank you

• Contact:

– ivo.neskovic@gmail.com

– http://twitter.com/trumpets