Yaxiong Lin

TestOptimal, LLC

Model-based TestingEffective Test Case Design

And Test Automation

Twin Cities Quality Assurance Association (TCQAA)

September 12, 2013

Agenda

What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases

Testptimal LLC

Agenda What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases

Testptimal LLC

Business Impacts of Software Defects

Source: Crosstalk, the Journal of Defense Software Engineering

• U.S. alone: $60 billion per year (NIST 2002)

Testptimal LLC

Where Defects are Found

Testptimal LLC

Source Unknown

Costs of Fixing Software Defects

Testptimal LLC

The Real Cost of Software Defects, LKP Consulting Grouphttp://www.lkpgroup.com/Cost%20of%20Software%20Defects.pdf

Software Testing Challenges Requirement/design defect impact: 64% Non-developer testing catches 69% of

defects Costs 100x to fix defect in production Gain higher confidence

better test coveragerequirement traceability

Testptimal LLC

What is Model-Based Testing?“Model-Based Testing is the Automation of Test Design of Black-box Tests”

Testptimal LLC

Practical Model-base Testing”, Mark Utting and Bruno Legeard

What is Model-Based Testing?“Model-Based Testing is the Automation of Test Design of Black-box Tests”

Testptimal LLC

Practical Model-base Testing”, Mark Utting and Bruno Legeard

Model-based testing is application of Model based design for designing and optionally also executing artifacts to perform software testing. Models can be used to represent the desired behavior of a System Under Test (SUT), or to represent testing strategies and a test environment. wikipedia

Why Model?

Visual representation of complex systems Ask What-If to understand cause-effect Important role in the technology

advancement and software development Software testing is a complex task

Testptimal LLC

MBT Modeling Notations Finite State Machine (FSM)

State Diagram (UML) Extended Finite State Machine (EFSM) Message Sequence Chart (MSC)

Sequence Diagram (UML) Control Flow Graph (CFG)

Activity Diagram (UML) Event Flow Diagram UML Testing Profile – UTP …

Testptimal LLC

Testing Vending Machine

Drink for $1 Accept Quarter and HalfDollar coins Cancel any time for refund

Testptimal LLC

Vending Machine Test Cases

Quarter + Quarter + Quarter + Quarter => Vend Quarter + Quarter + HalfDollar => Vend Quarter + HalfDollar + Quarter => Vend Quarter + HalfDollar + HalfDollar => Vend HalfDollar + Quarter + Quarter => Vend HalfDollar + HalfDollar => Vend HalfDollar + Quarter + HalfDollar => Vend Quarter + Cancel => Refund …

Testptimal LLC

Example Model

Testptimal LLC

Generated Test Sequence

Testptimal LLC

Agenda

What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases

Testptimal LLC

Types of MBT

Offline Testing

Online Testing

Testptimal LLC

Demo:

Test Case Generation for Manual Testing or Offline Testing

Testptimal LLC

Demo:

Login Test Automation – Online Testing

Testptimal LLC

Demo:

WebStore Test Automation – Online Testing

Testptimal LLC

Testing Web App

Testptimal LLC

WebStore MBT Model

© 2012 Testptimal LLC

Generated Test Sequence

Testptimal LLC

Agenda

What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases

Testptimal LLC

Recent MBT User Surveyby Robert Binder, System Verification Associates,2012

MBT reduced escaped bugs by 59%. MBT reduced testing costs by 17%. MBT reduced testing duration by 25%.

Testptimal LLC

Benefits of MBT

Catch requirement / design defects Catch defects earlier Reduce defects in fields Improve efficiency (Agility)

handling changesbetter communication on defects

Gain higher confidencebetter test coveragerequirement traceability

Testptimal LLC

Agenda

What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases

Testptimal LLC

MBT Iterative Development Process

Testptimal LLC

Modeling

Test Generation

AutomationExecution

AnalysisMBT

Models

Requirements

User Stories

MBT Process Input/Output

Testptimal LLC

Input

Requirements User Stories

Test Objectives

AUT/Spec

Environment Configuration

Execution Result

MBT Step

Modeling

Test Generation

Automation

Execution

Analysis

Output

Behavior Models

Test Strategies Test Coverage

Automation Scripts

Test Coverage Defects, Perform. Stat

Model Modification

Model-Based Testing Process

Testptimal LLC

http://journal.thedacs.com/issue/52/145Department of Defense (DoD) Information Analysis Center (IAC)

Agenda

What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases (more demo)

Testptimal LLC

Applications of Model-Based Testing

Conformance Testing Functional Testing System Integration Testing Security Vulnerability Testing

Testptimal LLC

MBT Use Cases (demo)

Finding Temporal Defects Realistic Load Simulation

PerformanceLoad/Stress Testing

Session /Scenario Based Testing Risk Based Testing Quick bug fix verification

Testptimal LLC

Questions?

MBT Papers: http://testoptimal.com/MbtReferences.html

Email: yaxiong.lin@testoptimal.com

Testptimal LLC