Test Oracle Generation Procedures and Executable Specifications 4 th Annual International Software...

Test Oracle Generation Procedures and Executable Specifications

4th Annual International Software Testing Conference 2004

Authors: Rupak Das and Sita RamakrishnanPresented by: Dr Sita Ramakrishnan

4th International Software Testing Conference Pune, India 20-21 Feb 2004

2

Overview Oracle – an example A Generic Test Environment Test Oracles and their Importance Problems Assertion-based Implementation Oracles Specification Oracles Real Specification Languages Testability and Diagnosability with industry

standards AsmL Oracles and the Future Conclusions


3

Oracle – an example

The Oracle in Matrix Is a program Predicts choices and

outcomes based on input from the Zion mainframe

Test Oracles …

The Oracle offers Candy to Neo, the following dialogue ensues …Neo: D'you already know if I'm going to take it?The Oracle: Wouldn't be much of an Oracle if I didn't.


4

A Generic Test Environment


5


3 main aspects to the test environment:

1. The test model, which is the application model is augmented to include testing requirements

- By adding application specific constraints

to the component interface protocol

2. The test model (state model) is fed to

the test design software


6


This software tool uses the critical properties of testing requirements built in the test model to generate automated test sequences

3. The test execution phase is the diagnosis phase. Accepts test inputs from SUT,

and test is evaluated as pass or fail.


7

Testing Environment in VISWAS

o Our Testing Environment in VISWAS

supports such an automated process

o Includes the Test Sequence Generation phase and the System under Test


8

Testing Environment in VISWAS

o Diagnosability was not the focus of concern in VISWAS, and test execution phase of test environment was not automated in VISWAS

o Next we look at Test Execution phase and Test Oracle Generation Procedures


9

Test Oracles – What are they?

A method to establish that the system under test has produced expected outputs corresponding to given test inputs.

All testing techniques assume the existence of an oracle, be it a human `eyeball’ oracle, or a software oracle with valid input/output pairs.

Valid input/output pairs (automatically) generated from specifications helps to verify `intended behaviour’ of the system.


10

Active and Passive Oracles

Active Oracles mimic the behaviour of the software under

test mainly built for accuracy rather than

efficiency

Passive Oracles Verifies software behaviour without

replicating it


11

The Problems

Reliable test set problem Whole input domain needs to be taken into

account Otherwise it becomes impossible to create a

reliable exhaustive test set Oracle problem

Grey areas between intended behaviour, expected behaviour and actual behaviour

Some opine that `truly general oracles’ may not be achieved


12

Assertion-based Implementation Oracles

Assertions are ways to specify what a system is supposed to do, and not how it is to do it

Embedded assertions in source code can be used to create implementation oracles

Not a dedicated specification mechanism

Pre-processing of code is often required


13

Assertion-based Implementation Oracles …

Assertions

AsmL specification of a stack interface


14

Assertion-based Implementation Oracles …

Anna (ANNotated Ada) APP (Annotation Pre-Processor) for C Nana (Assertion library for C/C++) Eiffel

Built in assertion support Integrated with exception-handling

Java Multitude of assertion systems iContract, JaWa, Handshake, JMSAssert,

jContractor and JUnit


15

Assertion-based Implementation Oracles

Dependency on control points Assertions are treated as program statements

with pre-defined execution points

Caching of values Pre- and Post-Condition pairs relate the program

states before and after method execution

Expensive for complex/large data structures

Existential and Universal Quantification Translating input to reflect quantification in terms

of `loops of execution’ can be cumbersome


16

Specification Oracles

Translating a specification into an oracle irrespective of implementation

Oracles can be generated automatically from specifications or documentation

Can extract relevant input/output pairs of test data from specification to check intended behaviour


17

Specification Oracles …

Program Testing Assistant Defines test cases interactively Uses layer correspondences to generate test

cases automatically from a library of templates

Test cases execute when appropriate Success criteria defined by specifier Requires at least one successful execution

to generate valid test data sets

Chapman 1982, A Programmer’s Testing Assistant


18


Test Oracle Generator (TOG)Automatically generates test oracles

from a relational specification that uses tabular expressions

Test harness (oracle) takes input/output pair from Program Under Test (PUT)

Relation is evaluatedReturns true/false

Peters and Parnas 1994, Test Oracle Generator


19


Documentation components required by TOG Program Specification Auxiliary Predicates and Functions User Definitions

A relational specification is created If result of TOG execution is present in

the relational specification table, oracle procedure execution is successful


20

Specification Oracles

TOG oracles are only as good as the specification

Access to data structures required to evaluate expressions and predicates

Complexity of test harness reduces readability of the oracle

Developed further by Software Engineering Research Group at McMaster University, Ontario


21

Real Specification Languages

Actual specifications are translated to oracles for validation and verification

Generate test oracles from specifications that are executable in their own right

B, Z, AsmL (Abstract State Machine Language)

Deriving test oracles require an ``executable subset’’ of the original specification


22

Real Specification Languages

Automatic Test Result Evaluation approach – translates Z schema to an evaluation function

Test Template Framework derives ``oracle templates’’ from model based specification written in Z

Multiple oracles from multi-language specifications composed as a unit


23

Testability and Diagnosability with industry standards

o Diagnostic components constructed according to standards facilitate competition in the market place in terms of risks, cost and quality

o Standards imply a maturity in the underlying technology, thus adding to the level of confidence. TTCN-3 is the only standardised language for the specification & implementation of test cases.


24


o UML is a widely used industry standard for software component modelling.

o Graphical format for TTCN-3 (GFT) is based on Message Sequence Charts (MSCs) and UML.

o GFT extends TTCN-3 with test specific concepts such as verdicts & defaults.

o GFT is the basis for the definition of UML 2.0 test profile.


25


o UML 2.0 Test Profile (UTP) has the notion of an arbiter, which is a test component aimed at separating test behaviour from test evaluationo Evaluates test results and assigns

verdicts of a test case

o A verdict is the outcome of a test case being pass, fail, inconc or error as defined in TTCN-3


26


o A Test case may have 4 outcomes:o Pass, Fail, inconclusive or abort

o During the execution of a test case, a test trace is generated and stored in a test log

o UML 2.0 Test Profile, GFT, !TeLa!

have similar diagnostic features


27

AsmL Oracles

Abstract State Machine approach has been proven suitable for large-scale executable specifications

AsmL is a high-level language that implements the ASM paradigm

AsmL specifications can be simulated Microsoft Research - Foundations of

Software Engineering Group


28

AsmL Oracles …

Specification is a non-deterministic ASM program written in AsmL that has a fixed initial state

Individual finite states are grouped together to form `hyperstates’

Results in a finite automaton – used to generate test suite

Extraction of an FSM from an ASM specification by extracting algorithm


29

… and the future

Non-deterministic finite state machines cannot be generated from ASMs by using the extraction algorithm

Combination of the Test Template Framework and Automatic Test Result Evaluation approach for Z specifications could be used

AsmL as a specification and implementation language can be used to generate exhaustive executable oracles

AsmL oracles can be translated to oracles for other .NET aware languages


30

Conclusions

Producing a comprehensive, precise and understandable specification is difficult enough

Adding computability to specifications to generate oracles makes it even more harder

Intrinsic dependence of testing on oracles make them essential

Pros and Cons exist for all approaches Any of the existing approaches is better and less

expensive compared to a human oracle More research and development of automated

oracle generation tools will make the V & V process much easier and productive


31

Contact Details

Authors: Rupak Das and Sita Ramakrishnan Presenter: Dr Sita Ramakrishnan

School Computer Science & Software EngineeringMonash UniversityAustralia

[email protected]: 61-3-99052485 fax: 61-3-99031077

Test Oracle Generation Procedures and Executable Specifications 4 th Annual International Software...

Documents

Transcript of Test Oracle Generation Procedures and Executable Specifications 4 th Annual International Software...