EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 1

Model-Based Testing of a Financial Application

Case Study

Prof. Bruno LegeardSmartesting

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 2

Model-based testing to industrialize

the functional testing process

Agenda

Summary and discussion

Financial application – Experience report

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 3

Model-based testing to industrialize

the functional testing process

Agenda

Results and discussion

Financial application – Experience report

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 4

– Business Requirements and Use cases as drivers– Automatic Generation of Test Cases and Executable Scripts

– Implementation of Test Management and Execution tools.– Creation of Competence Centers and Test Services

for functional testing

Test Industrialization – Key issues

“Functional testing industrialization based on a seamless process from Business requirements to Test Assets”

Smartesting®

RequirementsManagement

TestManagement

TestExecution

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 5

Functional testing – manual process

Manual execution or test automation

Test repository

Test Design

Test plan construction

Traceability management

Coverage matrixTest Script

Test resultsRequirements

& specifications

Test Execution Test Execution EnvironmentEnvironment

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 6

Test Architect

Keywords-based testing automation

Automation LayerAutomation Layer

Test Management Test Management EnvironmentEnvironment

Test plan &Test plan &Test casesTest cases

Automatic generation

CoverageCoveragematrixmatrix

ExecutableExecutableTest scriptsTest scripts

Test ResultsTest Resultsmetricsmetrics

Automatic generation

Requirements Test Models

SmartestingTest Publisher

SmartestingTest Designer

Iterative Process

Test Automation

Engineer

Inspection

SubjectMatter Expert

Model-based testing – Smartesting® Process

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 7

Test Generation example

Specifications

- - Tests & Scripts Generation

- Business Requirements Traceability along with Test Process

Behavior modelingwith IBM RSM or Borland

Together

Test Generation withSmartesting Test Designer

Tests Publishing in HP QC/QTP, IBM

Rational…

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 8

Model-based testing to industrialize

the functional testing process

Agenda

Results and discussion

Financial application – Experience report

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 9

Database

SUTMiddle Office

Clearing

BNP-Paribas Case Study

Smartesting on a Business Application of BNPPConfirmation/chasing application – Matching deals, Reporting , Litigation ….

Case study as presented at "atelier BNPP", the external BNPP forum (www.atelier.fr 26/03/08

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 10

Scope of the project :– Migration of technology Borland-Delphi Java - J2EE

Framework– Off-shore Development in India (Internal team - Mumbai)– Functional qualification of the application on site BNPP Paris– Iterative approach (agile project approach)– Functional qualification with Smartesting solution and automation

of the non-regression tests

Testing objectives: Ensure functional qualification and certification of off-shore deliveries

Smartesting on a Business Application of BNPP1/3

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 11

Tooling: • UML modeling : Rational Software Modeler 7.0• Tests generation : Test Designer 3.2• Test Management : HP-Mercury Quality Center 8.0• Test Automation : HP-Mercury Quick Test Professional 8.2

Methodology:• Iterative approach• 6 deliveries to test• Systematic regression

testingRelease Testing

Iteration TestingIT Tests

BA Acceptance

UAT

Unit Tests

System Tests

Smartesting

Smartesting on a Business Application of BNPP2/3

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 12

Project Typology:– Financial application (deal matching, chasing and litigation)– Web application

Project Context:– Migration from Borland-Delphi technologies Java - J2EE

Framework– Development off-shored in India (Internal team)– Functional validation at customer site – Paris

Testing objectives:– Guarantee the non-regression of the new web based application

compare to the Legacy code

Smartesting on a Business Application of BNPP3/3

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 13

Test Environment(HP/Mercury, Borland,Compuware, Rational,

JUnit,…)

TranslationResult analysis

Test Scripts

TestResults

Test Plans &Test cases

AutomaticGeneration

Smartesting Test Designer v3.2

Publisher for HP QC/QTP

Automatic Generation

Requirements UML Specification

Smartesting – Step-by-step

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 14

Starting Point: Requirements and Legacy code

Requirements

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 15

Test execution

EnvironmentHP QTP

Result Analysis

Test Scripts

Test Results

Test Plan &Test cases

Automatic Generation

Smartesting Test Designer v3.2

Publisher for HP TD/QTP

Automatic Generation

Requirements Test Design Test Model

Test modeling

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 16

Model: Class Diagram

- Business key entities- Point of control and observation

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 17

Model: Dynamic behaviour

- Layered UML State Machine- Express the dynamic of the application

Matching (partial)

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 18

Test generation based of self-defining data context

Model - Test Data

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 19

Check the Model

Test Designer™ Plug-In•Type Checking•Export model

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 20

Test environment(HP/Mercury, Borland,Compuware, Rational,

JUnit,…)

Translation

Test Scripts

Test Result

Automatic Generation

Publisher for HP TD / QTP

Automatic Generation

Requirements UML Specification

Test Plan &Test Cases

SmartestingTest Designer

Test Generation

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 21

Test Generation

Test list/Equivalence classes

Test caseTest Data

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 22

Test Design

Automatic Generation

Automatic Generation

Requirements Test model

Test Plan &Test Cases

Test Scripts

Test Result

SmartestingTest Designer

Publisher for HP TD / QTP

Keywords-based testing automation

Automation LayerAutomation Layer

Test Management Test Management EnvironmentEnvironment

Publication into HP QualityCenter & QuickTestPro

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 23

Publication in HP Quality Center

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 24

Automation: Implementing modeled actions

Automated translation of Test cases into test scripts

Implementation of automation libraries

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 25

Project Metrics (1)

Function Number of tests

Number of steps

Number of check-points

Workload (person.day)

Matching 206 2134 7771 30

Pre-Matching

89 584 2462 13

Chasing up 183 2076 2876 10,5

Setup Criteria

88 704 4349 4,5

Warning 180 1945 3142 11

Workload : - Modeling & test generation 48%- Automate key-word and test execution 52%

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 26

Project Metrics (2)

Number of check-point / tests

Number of tests / person.day

Number of check-points / person.day

Matching 37,7 6,8 259

Pre-Matching 27,6 6,8 189,3

Chasing up 15,7 17,4 274

Setup Criteria 49,4 19,6 966

Warning 17,4 16,4 286

Average 29,59 13,4 395

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 27

Test results – 746 test cases generated with Test Designer 3.2 and fully

automated with Quick Test Professional v.8.2– Complete test suite execution time: 4 hours makes it possible

to test each deliverable from India team (each week in release period to ensure non-regression testing)

Savings:– Project delivery on time– Systematic non-regression testing for each release (and after

bug corrections)– Smartesting process covers 95% of functional testing project

needs

Results summary

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 28

Model-based testing to industrialize

the functional testing process

Agenda

Summary and discussion

Financial application – Experience report

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 29

Roles and actions

SystemMatter Expert

Tester and TestAutomation eng.

Test Architect

• Test automation and execution• Test data management• Results analysis

• Test case design• Requirements traceability• Ensure functional coverage

• Create and maintain functional requirements• Manage domain knowledge

Test DesignerTest Designer

Quality Center / Quick Test ProQuality Center / Quick Test Pro

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 30

Quality: test design starts early in the development cycle– The test model is developed in coordination with the

requirements analysis and tests are generated iteratively (agility)

Productivity– ROI from the third test execution– Development of reusable test assets

Improvement of the functional test coverage– Systematic coverage of the application expected behavior

Easier and faster maintenance of the test suite– At every functional change, only the model have to be updated.

Then re-generation of the test suite

Benefits of model-based testing

EuroSTAR 2008 – The Hague – Netherlands – 10-13 November 31

Name: Bruno Legeard

Company: Smartesting

18 rue Alain Savary

25000 Besançon - France

E-Mail: legeard@smartesting.com

Phone: +33 3 81 25 53 63

Web: www.smartesting.com

THANK YOU