Pavel Drobintsev, Vsevolod Kotlyarov, Ivan Selin, Alexey Tolstoles, Nikita Voinov

Peter the Great Saint Petersburg Polytechnic University

Saint Petersburg, Russia

Scalable testing process based on formal models

Model Oriented Approach

• Multilevel model of application during design process

• Behavioral model is built from requirements

• Model is iteratively specified

• Test cases are generated from this model in an automated way

03.03.2017 2

VRS/TAT

• Software verification and testing toolset

• Requirements are formalized in UCM language

• VRS verifier runs reachability and consistency checks

• Iterative process of model modification until all checks are passed

• Generation of symbolic traces

• Concretization (symbolic traces – traces with specific values)

• Adapting tests to run them on real system

• Testing

03.03.2017 3

Use Case Maps (UCM)

• High-level graphical language for requirements modelling

• ITU Z.151

• Can be used for creating structured behavioral diagrams of interacting agents

• UCM notation doesn’t provide enough information to generate traces

03.03.2017 4

UCM Metadata

• Auxiliary text field linked to a UCM responsibility

• Stores information about variables and signals

• Signal is a way of message transfer between agents

03.03.2017 5

VRS Verifier

• During the initial model development:• Reachability and consistency checks

• During testing process:• Generation of symbolic traces based on UCM model control flow

• Concretization (substitution of left, right and middle values from tolerance range)

03.03.2017 6

VRS/TAT Levels of Abstraction

03.03.2017 7

Automated

Automated

Not automated

Challenges

• Data mapping problem:• Abstract models which need to be extended in order to run against real

system

• Extension may require additional information

• New information must comply with what’s already in the model

• Amount of generated test cases may be too big to run using common methods

03.03.2017 8

Data Structures Conversion Problem

• Structure of concrete test signals does not correspond with real system signals

• Manual creation of detailed test scenarios takes too much time and resources

03.03.2017 9

Data Structures Conversion. Approach

• Process called “Lowering”

• The name comes from descending on lower levels of abstraction

• In general, “Lowering” can be described as creating processing rules for each signal and application of these rules to concrete scenarios

• An editor was made to restrict user from making incorrect structures

03.03.2017 10

Data Structures Conversion. Restrictions

• There are several restrictions to ensure the correctness of test scenarios:• If you separate concrete value into several independent parts, it is prohibited

to change them in a way, when joining them back together will give another result than it was before separation

• Only structures similar to SUT interfaces could be used

• Only constant values from concretization and templates values are allowed

03.03.2017 11

Data Structures Conversion. Results

• Before introduction of “lowering”, test mapping was made by hand after each tests generation

• After automating the tests mapping there are only 2 manual steps left in VRS/TAT process:• UCM model development

• Specifying “lowering” rules

• Both of them only needed to be done once

03.03.2017 12

UPD: VRS/TAT Levels of Abstraction

03.03.2017 13

Automated

Automated

Automated

Huge amount

Execution problem

• Huge test amount

• A lot of time needed for execution

• Costs increase

• Run tools in parallel

03.03.2017 14

VRS Guided Search

• VRS has an option to perform guided search

• Guide is a marked sequence on a model, which must be traversed

• All traces that don’t apply the criteria to travel through guide are cut off by VRS during traces generation

• The idea is to set guides on UCM model and launch several VRS instances with corresponding guides

03.03.2017 15

VRS Guide

03.03.2017 16

Execution process

• User sets guides in the most “thick” places of UCM model

• Several VRS instances run, each on a different compute node with a unique set of guides

• As a result, whole test suite is divided into several parts

• Each part can be processed independently in an isolated container (node, VM, etc.)

03.03.2017 17

Scalability

• Theoretically, scalability is linear

• In real life, it is not achievable because of irregular test distribution and different execution complexity of test cases

03.03.2017 18

Conclusion

• Tests are generated from the verified model and can be ran against real systems

• Presented approach allows to speed up the testing process

• It speeds up both tests generation and tests execution

• Scalable process, performance increase is limited by the model and/or available computational resources

03.03.2017 19

Thanks for your attention!

03.03.2017 20

Backup 1. New features in lowering. TDL

• There is an ability to use SUT interfaces files (*.tdl) in Lowering Editor to obtain parameters signal structure

03.03.2017 21

Backup 2. Few words on VRS

• VRS works with inner model in basic protocols (Hoare triples)

• Each one has:• Pre-condition

• Process

• Post-condition

03.03.2017 22

Backup 2. Few words on VRS

• UCM model is converted into basic protocols model using UCM elements location and UCM metadata (and inverse)

• VRS matches pre and post conditions of different elements and checks tolerance ranges to find all possible traces

• Guided search builds traces from guide forward and backward

03.03.2017 23