Project FoX: A Tool That Offers Automated Testing Using a Formal Approach
-
Upload
trumpets -
Category
Technology
-
view
407 -
download
1
description
Transcript of Project FoX: A Tool That Offers Automated Testing Using a Formal Approach
Project FoX: A Tool That Offers
Automated Testing Using a
Formal Approach
Ivo Neskovic
Agenda
• Software Engineering: What could go wrong?
• Formal Methods
• Project FoX
• Case Study: The buffer system
• Conclusions and Future Work
• Bibliography
The Problem of Software Engineering
• Faulty systems are a common notion nowadays.
• SE is an engineering discipline, yet lacking the
engineering formality.
• Subjective and informal testing.
• Impossible to prove that the system:
– Does what it is supposed to do.
– Does not do what it is not supposed to do.
• Needs structured and precise system designs.
Formal Methods • The applied mathematics of computer systems engineering, used to
specify and model the behaviour of a system and mathematically verify
that the system design and implementation satisfy functional and safety
properties.
• Specification Languages:
– Abstract State Machines
– Generalized State Machines
– Communicating Sequential Processes
– Specification and Description Language
– Petri Nets
– Temporal Logic of Actions
– B and event – B method
– Z
Formal Methods at a Trial
• Benefits:
– Specification may be used as a basis for proving the presence or lack
of certain properties in the design and by inference in the developed
system.
– Mathematical proof of correctness (Theorem proving).
– Model checking (Proving desired properties in a design).
– Formal Testing.
• Used mainly for safety critical systems such as aerospace
engineering.
• Criticism:
– Expensive and time consuming approach (though questionable).
– Lack of tooling support.
Incorporating Formal Methods in the
Development Cycle
Project FoX • Produce the complete set of test cases from a formal specification.
• Execute the tests on the systems implementation.
• Locate errors and non-equivalences and report them to the user.
• Developed in Java for Java.
• Compatible with Java Standard Edition, Enterprise Edition, Mobile
Edition.
• Can be extend to work in conjunction with popular Java frameworks.
• Operates on compiled bytecode with the addition of a few specific
annotations.
• Utilizes the test drivers of JUnit.
• FoX provides a bridge between regular Java developers and the benefits
of complete positive and negative testing, proven to find all faults.
Using Project FoX
• Two artefacts necessary:
– Formal specification of the system.
– The system’s implementation.
Buffer Case Study – Description • Simple buffer in a factory.
• Accepts parts, any parts.
• Parts have a name and an ID.
• The buffer has a capacity of 2.
• The buffer can be empty, partially
full or completely full.
• Supports adding and removing
items.
• If the capacity is reached, no
additional items can be placed in
the buffer unless and item is
removed firsts.
Buffer Case Study – Formal
Specification • Modelled as a Generalized State
Machine (stream X-Machine).
• A theoretical model of computing,
pioneered by Samuel Eilenberg
in1974 (X-Machine).
• Separates flow control from
processing.
• Flow control is abstracted to a level
suitable for representation as a finite
state machine.
• Complex data structures are modelled
as an infinite memory.
• Able to model both static (data) and
dynamic (control) parts of a system.
Buffer Case Study – Formal
Specification (cont.) • Simple buffer in a factory.
< xMachine name = " Buffer " >
• The buffer can be empty, partially full or completely full.
< states >
< state initialState = " true " > empty </ state >
< state > non_empty </ state >
< state > full </ state >
</ states >
Buffer Case Study – Formal
Specification (cont.) • Accepts parts, any parts.
< input name = " part " ref = " BufferObject " / >
• The buffer has a capacity of 2.
< types >
< builtInType name = " capacity " type = " integer " / >
< builtInType name = " buffer " type = " set: BufferObject " / >
</ types >
< memory >
< memoryBlock ref = " buffer " initialValue = " null " / >
< memoryBlock ref = " capacity " initialValue = " 2 " / >
</ memory >
Buffer Case Study – Formal
Specification (cont.) • Parts have a name and an ID.
< types >
< complexType name = " ItemType " >
< attributes >
< builtInType name = " type " type = " string " / >
</ attributes >
</ complexType >
< complexType name = " BufferObject " >
< attributes >
< complexType name = " type " ref = " ItemType " / >
< builtInType name = " itemId " type = " integer " / >
</ attributes >
</ complexType >
< /type >
Buffer Case Study – Formal
Specification (cont.) • Supports adding and removing items.
< functions >
< function name = " add_part " >
< guard >
!buffer. contains ( part ) && buffer .
size () + 1 < capacity . value ()
</ guard >
< body > buffer . add ( part ) ; </ body >
< output > Part Added </ output >
</ function >
...
</ functions >
< transitions >
< transition >
< startingState >
empty
</ startingState >
< appliedFunction >
add_part
</ appliedFunction >
< endingState >
non_empty
</ endingState >
</ transition >
...
</ transitions>
Buffer Case Study – Implementation public class BufferObject {
private int itemId;
private ItemType type;
public BufferObject(int itemId,
ItemType type) {
this.itemId = itemId;
this.type = type;
}
}
public class ItemType {
private String type;
public ItemType(String type) {
this.type = type;
}
}
Buffer Case Study – Implementation • @Xmachine - annotating the class representing the system modeled
with the specification.
• XMachineModel – a class representing the model, containing a number
of useful helper methods.
@XMachine(inputType = "BufferObject",
sampleInputs = {
"integer: 10, ItemType: (string:Box)",
"integer: 17, ItemType: (string:HeavyBox)",
"integer: 25, ItemType: (string:ReallyHeavyBox)"
})
public class Buffer extends XMachineModel {
Buffer Case Study – Implementation • @XMMemoryBlock – a field level annotation, associating Java data
structures with their specification equivalents.
@XMMemoryBlock(name = "buffer")
private List<BufferObject> buffer;
@XMMemoryBlock(name = "capacity")
private int capacity;
public Buffer() {
super("Buffer");
buffer = new LinkedList<BufferObject>();
capacity = 2;
}
Buffer Case Study – Implementation
• @XMFunction – a method level annotation, referencing the
modeled functions implementations.
• reportOutcome( outcome: String) – one of the many helper
methods of the XMachineModel class.
@XMFunction(name = "add_part")
public void addPart(BufferObject part) {
if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
reportOutcome("Part Added");
}
Buffer Case Study – Executing Fox
Buffer Case Study – Executing FoX
(implanted error) if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
capacity++;
reportOutcome("Part Added");
}
Buffer Case Study – Generated Test
Cases • Tests report the sequence of inputs used for the specific scenario, the
sequence of expected outputs and the actual output.
• Outcome is reported to the user via the usual JUnit red / green
notifications.
<tests>
…
<test testID=”2”>
<input>[ itemId: 17 type: HeavyBox, itemId: 10 type: Box]</input>
<expectedOutput>
[ Part Added, Part Added – Become Full ]
</expectedOutput>
<output>[ Part Added, Part Added – Become Full ]</output>
</test>
…
</tests>
Conclusions and Future Work
• FoX enables developers to leverage the already
proven theories for formal testing.
• Provides a fully automated testing process, ranging
from complete test set generation (satisfying some
design for test conditions), to test preparation and
execution.
• Operates on any Java based software system,
being transparent to it's underlining technologies.
• Provides complete positive and complete negative
testing.
Conclusions and Future Work (cont.)
• Next steps:
– Thorough evaluation.
– An additional tool to make the specification step easier
and closer to the developer, aiming to “hide” the formality
as much as possible.
– NetBeans and Eclipse integration.
– A standalone X-Machine IDE providing additional related
functionalities.
– Branch out to other languages and frameworks (eg. C#
and .NET).
Bibliography • S. Eilenberg, Automate, Languages and Machines, Vol. A. Academic Press,
London, 1974.
• M. Holcombe, “X-Machines as a basis for dynamic system specification,”
Software Engineering Journal, vol. 3(2), pp. 69-76, 1988.
• F. Ipate and M. Holcombe, “Specification and Testing using Generalized
Machines: a Presentation and a Case Study,” Softw. Test. Verif. Reliab, vol. 8,
pp. 61-81, 1998.
• M. Holcombe and F. Ipate, Correct Systems: Building a Business Process
Solution. Springer, Applied Computing Series, November 1998.
• G. Eleftherakis and A. Cowling, “An Agile Formal Development Methodology,” in
1st South Eastern European workshop on Formal Methods (SEEFM 03),
(Thessaloniki), pp. 36-47, Nov. 2002. Agile Formal Methods: Practical, Rigorous
Methods for a changing world.
• P. Kefalas, G. Eleftherakis, and E. Kehris, “Communicating X-Machines: a
practical approach for formal and modular specification of large systems,”
Information and Software Technology, vol. 45, pp. 269-280, Apr. 2003.
Thank you
• Contact:
– http://twitter.com/trumpets