Stefan Mohacsi, Atos (Austria)

15 Years of Evolving Model-based Testing

- An Experience Report

www.eurostarconferences.com

@esconfs #esconfs

2

15 Years of Evolving Model-based Testing -

An Experience Report

Stefan Mohacsi, Atos (Austria)

3

Traditional Test Automation can be Useful, but…

▶ Test cases still have to be designed

manually

▶ Achieving systematic coverage is a

challenging task

▶ Capture & replay of GUI test cases

produces scripts that are hard to

understand & maintain

▶ Maintaining hundreds of test scripts in each

test cycle can require enormous effort

Solution: Model-based Testing!

4

What is Model-based Testing (MBT)?

▶ Tests are generated automatically from a model of the

system under test (SUT)

▶ In case of changes to the SUT, only the model needs to be

maintained, the tests are updated automatically

▶ Benefits: Systematic coverage, early defect detection,

visualisation, significantly reduced maintenance effort

Create/Update

Test Model

Generate

Test Data

Generate

Test Cases Execute

Test Cases

5

Challenges of MBT (1)

Which Model to Use?

Reusing a model created for code generation seems

like the obvious solution but ...

▶ UML models are often complex, not formal enough and

not focused on testing

▶ If code and test cases are generated from the same

source, defects in the source will not be detected!

A separate test model is recommended,

but requires additional effort

6

Challenges of MBT (2)

How to Bridge the Gap between Keyword and Script?

▶ Most test generators only produce abstract keywords

instead of executable scripts

▶ For each keyword, it is necessary to manually implement

and maintain a test script

▶ Additional effort for preparing test data, GUI object

repositories etc. may be required

A generator that can produce executable test scripts

would reduce the creation and maintenance effort

significantly

7

Now Let’s Go on a Journey

through Evolution...

8

▶ Prolog tool written in 1993 by Armin Beer and

Joachim Manz for testing Siemens Hicom software

▶ Test ‟model” had also to be specified in Prolog

▶ Generation algorithm tried to find test sequences

that satisfied all constraints

▶ Suitable for text-based interfaces (not for GUIs)

First Life Forms:

Automated Test case Generator ATG

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

9

IDATG = Integrating Design and

Automated Test Case Generation

▶ Started in 1997 as Cooperation between Siemens and

University of Linz

▶ Model editor and generator written in Visual C++

▶ Layout information for simple Win32 GUIs could be

imported from MFC resource files

▶ Transition-based constraint satisfaction algorithm for

finding valid test sequences

▶ Export of test scripts for Segue QA-Partner® (later

known as Borland SilkTest®)

Cambrian Explosion:

The Research Project IDATG

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

10

Early IDATG Model Editor

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary Lesson Learned: Model creation requires skill and

additional effort, but facilitates early defect detection

State Transitions

Imported

GUI Objects

11

▶ Various types of GUI objects were not included in MFC

resource files and had to be specified by hand

▶ In 1999, a built-in GUI Spy solved the problem:

Complete Win32 GUIs could now be recorded directly

from the screen

Evolutionary Improvements:

The GUI Spy

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

12

▶ Being restricted to only one test execution tool had

several disadvantages

▶ A new interface to Mercury WinRunner® (later HP)

considerably increased the number of potential users

Diversification:

WinRunner® Interface

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

13

▶ Working with transition-based models was arduous

and required advanced skills

▶ In 2001, a more intuitive ‟task flow” concept was

invented based on Activity Diagrams

▶ Building Block Concept: Each step may either

represent an atomic step (blue) or an entire step

sequence (yellow)

Exploring a new Realm:

Task Flow Modeling

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

Lesson Learned: Re-use of parametrizable building

blocks minimizes the effort for test creation and

maintenance

14

Task Flows in IDATG

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

Task flow with

building blocks

Reusable building

block ‟Login”

Recorded

‟Login” dialog

Lesson Learned:

Start modelling

at an abstract

level, then add

more detail step

by step

15

Conquering new Niches:

Support for Java and Web GUIs

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

▶ Being restricted to Win32 GUIs was a major drawback

▶ In 2001, a special GUI Spy for importing Java GUIs

was developed

▶ In 2002, an interface to Internet Explorer for

recording web pages followed

Lesson Learned: Ensuring GUI object recognition is

THE key challenge of automated test execution.

Don’t expect unskilled testers to do this but employ

specialists instead!

16

Evolving new Capabilities:

Requirements Tracing

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

▶ Tracing requirements is essential for

measuring coverage during the system test

▶ An editor for defining a requirement hierarchy and

assigning requirements to tasks was the logical next

step

Lesson Learned: Involve test

experts in early reviews of

requirement and design

specifications in order to avoid

testability problems later on.

17

Mass Extinction & Reanimation:

ICN Disaster and First ESA Project

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

▶ In 2002, the drastic reduction of the Siemens mobile

phone sector ICN left the IDATG project without its

main sponsor

▶ Close to cancellation, the project was revived in 2004

by a new opportunity in space

▶ After adding support for non-GUI interfaces, IDATG

became key part of a generic test framework for

ESA’s Satellite Control and Operation System

18

ESA Intelligent Test Framework

IDATG

Test

Commander

(Tcl/Tk)

GUI Test

Drivers

Satellite

Control and

Operation

System

API Test

Drivers

Manual

Test

Photo: ESA

19

Continual Growth:

Test Data Generation

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

▶ From 2006, it became possible to generate test data

in addition to test sequences

▶ Several systematic methods are supported

(equivalence partitioning, boundary value analysis)

▶ Invention of CECIL (Cause-Effect Coverage

Incorporating Linear boundaries)

in collaboration with the SoftNet Austria competence network

▶ Connection between test data and task flows can be

established by references

20

Chaotic Mutations:

Randomized Testing

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

Random test methods can be applied to

▶ cover a much larger section of the input

domain with relatively low effort

▶ discover faults that are caused by rare input

combinations

▶ obtain statistic measures, e.g., for assessing the

stability of a system

We invented a hybrid algorithm that combines pure

random testing with heuristic search strategies in

interleaving phases to increase the overall coverage

21

Symbiosis:

Test Management Integration

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

▶ A close integration between all components of a test

framework is crucial for an efficient test process

▶ In 2010, IDATG was integrated into the TEMPPO test

framework and renamed to ‟TEMPPO Designer”

▶ Test structures are passed from

TEMPPO Test Manager to TEMPPO

Designer, which returns the

generated test cases

Lesson Learned: The efficiency of MBT depends on the

process maturity and a well integrated tool framework

22

Dawn of a New Age:

Atos and QTP

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

▶ The future of IDATG was once again shrouded in

uncertainty when Siemens IT Solutions and Services

was taken over by Atos in 2011

▶ Fortunately, the new company recognized the

method’s potential and established a Global

Competence Center for Model-based Testing

▶ Finally, an interface to HP QuickTest Professional®

(QTP) was created

▶ In addition to generating QTP scripts, importing

object repositories into IDATG is supported

23

The Summit(s) of Evolution:

Model-based Testing Today

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

▶ In a major ESA project, the ROI of MBT

was measured

▶ It turned out that the additional effort for introducing

MBT paid off after 4 test cycles

▶ See also: Experiences

of Test Automation by

Dorothy Graham and

Mark Fewster

(Addison-Wesley 2012)

0

500

1000

1500

2000

2500

3000

3500

4000

1 4 7 10 13 16 19

Test Repetitions

Eff

ort

(H

ou

rs)

Manual Test

MBT

24

TEMPPO Designer (IDATG)

Model-based Test Process

with TEMPPO Designer V15.0

Test Structure

Definition

Test Data

Generation

Task Flow

Modeling

Low-Level

Specification

Test Case

Generation

Test Execution

Tool TEMPPO Test Manager

Test scripts

Control test run

Test structure data

Test case data

GUI data

25

Practical Hints

▶ Start MBT with test scenarios that have the highest ROI

(many test repetitions, easy to automate)

▶ Pay attention to the reusability and maintainability of

the model components, e.g., use parametrizable

building blocks

▶ If MBT is correctly applied, test maintenance costs can

be significantly reduced!

▶ Don’t forget to plan some experience-based tests. A

test model is a good thing but cannot replace human

experience and intuition!

Precambrian

Carboniferous

Cambrian

Ordovician

Devonian

Silurian

Permian

Triassic

Jurassic

Cretaceous

Paleogene

Neogene

Quaternary

26

Contact Information

Stefan Mohacsi

Atos IT Solutions and Services

Head of CoC Model-based Test

Phone: +43 (0) 50618 / 43417

e-Mail: stefan.mohacsi@atos.net

THANK YOU !