Model-based Testing

Introduction and Hands-on session

Stefan EkmanDublin, 2011-01-27

Model Based TestingWhat is Model-based Testing?

What is MBT?

Wikipedia:

“Model-based testing is software testing in which test

cases are derived in whole or in part from a model that

describes some (usually functional) aspects of the system

under test (SUT).”

Advantages

• Model-based testing is easy to understand from both the business and developer communities

• Model-based testing separates (business-) logic from testing code

• Model-based testing increases the test coverage with the same effort as “Classic” test automation

• Model-based testing is the fastest way to get use of automated test

Advantages

• Model-based testing enables us to switch testing tool if needed or support multiple platforms using the same model

• Model-based testing focuses on requirement coverage, not how many test cases you executed last week etc.

• Model-based testing proved to positively affect the maintenance problem, the nemesis of all test automation.

The ModelModelling your tests

Tools / Environment

.GraphML

Model with yEd

GraphWalker

MBT engine

Selenium

Java or SOAP

QTP

Via SOAP

Hudson

Build system

Code behind

public void e_Nav_GoogleImages()

{

log.info("Edge: e_Nav_GoogleImages");

selenium.click(“link=Images");

selenium.waitForPageToLoad(30000);

}

Modelling

What should the model represent?

I suggest that you start with the model representing the end usage/user functionality of the system. That will usually resemble your traditional functional test case approach. There are several other approaches for you to explore later in you new MBT career…

Modelling

To what level do we model?

Modelling

Who creates the model?

• Start out yourself to begin with• Present the model – show it to SME’s and

developers for input• If the system is non-existing, you probably would

consider user stories, product backlog items, use-cases or the equivalent requirement's information.

Modelling

During this process you will collect the first benefits of working with MBT!

GraphWalker

GraphWalker is a tool for generating offline and online test sequences from Finite State Machines and Extended Finite State Machines.

http://www.graphwalker.org/

• Lightweight MBT – No UML• No exit/stop points• Online• Event-driven

GraphWalker is an Open Source MBT engine created by Kristian Karl and are stable and used in large IT testing environment.

GraphWalker - Stop Criteria's

• REACHED_EDGE• REACHED_VERTEX• EDGE_COVERAGE• VERTEX_COVERAGE• TEST_LENGTH• TEST_DURATION• REACHED_REQUIREMENT• REQUIEMENT_COVERAGE• NEVER

GraphWalker – XML configuration

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE MBTINIT SYSTEM "mbt_setup.dtd" ><MBTINIT EXECUTOR="java:com.paddypower.mbt.BetPlacementRegression" EXTENDED="true"

GUI=“true"> <MODEL PATH="model/BetPlacementRegression.graphml"/> <CLASS PATH="Java\PPautomation\bin"/> <CLASS PATH=".\servlet-2.3.jar"/> <CLASS PATH="selenium\selenium-java-client-driver.jar"/> <CLASS PATH="selenium\selenium-server-2.0a6.jar"/><SCRIPT>

PortNumber = 5555; DBconnectionString =

"jdbc:informix-sqli://10.10.10.10/db:informixserver=testdb1;user=user;password=12345";MAX_SELECTIONS = 10;

</SCRIPT> <GENERATOR TYPE="RANDOM"> <CONDITION TYPE="REACHED_VERTEX" VALUE="v_Done"/> </GENERATOR></MBTINIT>

GraphWalker - The Walks

• RANDOM• A_STAR

Will try to generate the shortest possible test

sequence through a model with given stop

condition. The algorithm only works well on

smaller models.

• SHORTEST_NON_OPTIMIZED

GraphWalker - Edge

• An edge is a transition. Could be a click on a button, a timeout, a server API call.

• An edge can have a label, but it is not mandatory.

• The label of an edge will map against a method, function or sub routine during test execution.

• As a rule of thumb, the best practice is to start the name with 'e_'. The reason for this, is that it is easier to recognize the function in the test execution tool, as an edge. For example:e_StartApplicatione_EnterBasicView

• The same name of an edge, can be re-used in the model.

GraphWalker - Vertex

• A vertex is some kind of state that is possible to verify using some kind of oracle.

• A vertex must always have a label.

• The label of a vertex is mapped against a method, function or sub routine during test execution.

• As a rule of thumb, the best practice is to start the name with 'v_'. The reason for this, is that it is easier to recognize the function in the test execution tool, as a vertex. For example:v_ApplicationStartedv_BasicView

• The same name of a vertex, can be re-used in the model.

GraphWalker - Labels

Label [cond. expr.] / Statement1;Statement2;

White spaces not allowed.

Follow the naming convention for the implementing programming

language.

Example:

e_StartApplication

Example:

e_EnterBasicView

GraphWalker - Guards

Label [cond.expr.] / Statement1;Statement2;

Conditional expression that returns a Boolean value.

The guard indicates if the edge is accessible in the current state of the

machine. The language is either Java or JavaScript.

Example:

e_Evaluate [x>0 && y<=10]

Example:

e_Checkout [shoppingBasket.contains("Cerials")]

GraphWalker - Statements

Label [cond.expr.] / Statement1;Statement2;

Any applicable Java or JavaScript statements

Example:

e_Calculate / x=5;y++;

Example:

e_AddCerials / shoppingBasket.add("Cerials");

GraphWalker – Keyword Start

Used by Vertices only.

Each graph must have a start vertex, and that vertex holds this

keyword. There is only one vertex in a graph, holding this keyword.

The Start vertex can only have one out-edge.

If the graph is the main-graph, that edge must have a

label.

If the graph is a sub-graph, that edge cannot have a label.

GraphWalker - REQTAG

Used by Vertices

Used by GraphWalker for keeping track of which

requirements are fulfilled during testing.

In the example above, if v_BookInfomation is passed, it

would mean the requirement with tag UC01 2.2.3 has

passed.

v_BookInfomationREQTAG=UC01 2.2.3

GraphWalker - Weight

Used by Edges only.

Used by MBT during random walks during test sequence generation. It holds a real value between 0 and 1, and represents the probability that a specific edge should be chosen. A value of 0.05, would mean a 5% chance of that edge to be selected during a run.

Note: It only works with the RANDOM generator

v_BookInfomationREQTAG=UC01 2.2.3

e_NavAweight=0.25

e_NavB

GraphWalker – Keyword Blocked

Used by both Vertices and Edges.

A vertex or an edge with the key word BLOCKED, will be

excluded from the model when walked.

v_BookInfomationBLOCKED

Metrics

• Standard test metrics does not apply so well.• Test cases are not meaningful when running online

tests.• MBT tends to drive the requirement work.

GraphWalker – QTP & SOAP

WebService("SoapServicesService").SetTOProperty "WSDL", "http://myComputerName:9090/mbt-services?WSDL"If ( not WebService("SoapServicesService").Reload()  )Then    Reporter.ReportEvent micFail, "MBT failure", "MBT encountered an error. See the MBT log files for information."    ExitTestEnd If

Do until not WebService("SoapServicesService").HasNextStep()    action = WebService("SoapServicesService").GetNextStep()    If len(action) > 0 Then        If not Eval( action ) Then            Reporter.ReportEvent micFail, "Script failure", "The script encountered an error when trying to run function: " & action            ExitTest        End if    End IfLoop

Reporter.ReportEvent micDone, "MBT Statistics", WebService("SoapServicesService").GetStatistics()

5 min break..

I am not in the office at the moment. Please send any work to be translated.

Hands on session

1. Setup the environment

2. Create a model

3. Generate code from model

4. Automate with selenium

5. Create XML configuration file for GraphWalker

6. Run the model!

Files

1. Create directory MBT i.e C:\MBT

2. Create directory Models under in the MBT folder

3. Copy GraphWalker and Selenium jar files to the directoryselenium-java-client-driver.jarselenium-server.jargraphwalker-2.5.1-SNAPSHOT-standalone.jar

Model

Visit www.brickor.com/MBT.htm with Firefox

Record with Selenium IDE by clicking on all elements and as well right clicking the result

Switch off recording.

Options > Format > JUnit4 (Java)

Model

Start yEd to create the model

Generate the Java Code

/** * This method implements the {EDGE_VERTEX} '{LABEL}' */public void {LABEL}(){ log.info( "{EDGE_VERTEX}: {LABEL}" ); throw new RuntimeException( "The {EDGE_VERTEX}: {LABEL} is

not implemented yet!" );}

java -jar graphwalker-2.5.1-SNAPSHOT-standalone.jar source -f Models/mbttest.graphml -t java.template.txt > javacode.txt

Eclipse – Create the Java project

Create new Java Project; MbtTest

Create new Java Package; ie.mbt.test

Create new Class; MBTtest

Copy paste the contents of the generated Java code from the model into the created class.

The java code

Include references to Selenium and GraphWalker in your project. Right click on your project and choose Properties. Under Java Build Path choose Libraries and then click Add External JARs.. Choose the two jar files:

graphwalker-2.5.1-SNAPSHOT-standalone.jarselenium-java-client-driver.jar

private HttpCommandProcessor proc = new HttpCommandProcessor("localhost",4444, "*chrome", “http://www.brickor.com/MBT.htm”);

private Selenium selenium = new DefaultSelenium(proc);

private Logger log = Util.setupLogger(MBTtest.class);// import org.apache.log4j.Logger;

// import org.graphwalker.Util;

Writing the Java Code

In e_init we start the selenium object.

selenium.start();selenium.open("http://www.brickor.com/MBT.htm");

Delete the Throw exception!

Update all e_r11, e_r12 etc

selenium.click("Radio1_1");row1 = Integer.parseInt(selenium.getText(“r11"));

Writing the Java Code

Create the utility GetResult

public boolean GetResult() {int rowTotal = row1 + row2 + row3;int calcTotal =

Integer.parseInt(selenium.getText("totalDiv"));

if (rowTotal==calcTotal)return true;

elsereturn false;

}

IMPORTANT!

Compile you class after any changes, CTRL + B

GraphWalker XML

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE MBTINIT SYSTEM "mbt_setup.dtd" ><MBTINIT EXECUTOR="java:ie.mbt.test.MBTtest" EXTENDED="true"> <MODEL PATH="models/MBTTest.graphml"/> <CLASS PATH="MbtTest\bin"/> <CLASS PATH="selenium-java-client-driver.jar"/> <CLASS PATH="selenium-server.jar"/> <GENERATOR TYPE="RANDOM"> <CONDITION TYPE="TEST_LENGTH" VALUE="1000"/> </GENERATOR></MBTINIT>

Selenium Server

java -jar selenium-server.jar

Will start with default settings on port 4444

Start the scripts..

Command line to start GraphWalker with the correct settings:

java -jar graphwalker-2.5.1-SNAPSHOT-standalone.jar gui xml -f MBTTest.xml

Questions?

More info at:

www.graphwalker.org

mbt.tigris.org

http://en.wikipedia.org/wiki/Model-based_testing

Email: stefan.ekman@gmail.com