Optimized Testing with Bayesian NetworksConclusions and Lessons Learned • Modelling –Consensus...

EUROSTAR 2005

James Waskiel

Government & Enterprise Mobility Solutions

Optimized Testing with Bayesian

Networks

James J. Waskiel

GEMS Copenhagen

Elena Pérez-Miñana

Jean-Jacques Gras

Rishabh Gupta

Motorola Labs

B

AT

EUROSTAR 2005

James Waskiel


Content

• Context

• Problem description

• BBNs Overview

• Modelling development process

• Modelling verification & validation process

• Test case generation

• Conclusions (lessons learned, future work)

EUROSTAR 2005

James Waskiel


Context

• Who are we? – GEMS provides mission-critical and enterprise

communication solutions for work teams in market segments where Motorola has traditionally been strong.

– Major market segments range from public safety and government, to transportation and commercial enterprises, as well as automotive electronic solutions and technology.

• What do we do ?– Development and testing of the VoIP communication

component included in the radio communications and information solutions provided by GEMS

EUROSTAR 2005

James Waskiel


Introduction

• Benefits:

– Identify weak areas of the software delivered to System Test during

development and during SIT testing. This results in more defects

being found earlier in the life cycle, and fewer defects being released

to the customer.

– TWSD CoQ improvements estimated at 0.5-1M $ (~18%) for future

releases

• Problem: understanding software reliability is difficult

– Too much data, and data types.

– Large system: 30% of defects escape release.

• Solution: Manage testing using Defect Models

– Direct testing towards weak areas and high-risk scenarios.

– Model product defects from development factors.

– TWSD CoQ improvements estimated at 0.5-1M $ for future releases

EUROSTAR 2005

James Waskiel


Impl.

Reqts

Design

Impl.

Impl.

Impl.

Impl.

IT

IT

IT IT STDesign

Box Design

Reqts

Design

Impl.

Impl.

Impl.

Impl.

Impl.

IT

IT

IT ITST

Design

REQ

IT

IT

ITIT

ST

What happened?

MSCs

MSCsMSCs

MSCsSW

?

Problem - Box Development

– Too much data and data types: metrics, subjective information

– Decisions based on “best” judgment

– RISK: back to a chaotic process if complexity increases

EUROSTAR 2005

James Waskiel


Bayesian Networks

A: Baseline quality

B: Product Capability

D: Product quality

NODE =

uncertain variable

ARC =

influence relation

NPT=

Conditional probability table

{Good, Bad}

or {1, 2,…, n}

or {intervals}

Based on Bayes theorem: P(B|A) = P(B) * P(A|B) / P(A)

C: Product Challenge

B =

C = High Low High Low

High 0.8 0.4 0.6 0.5

Low 0.2 0.6 0.4 0.5

High Low

D =

EUROSTAR 2005

James Waskiel


5

4

3

2

1

BN Modelling Approach

Project decomposition

Model Inputs

Validation

Data Collection

Observed

ResultsPredictions

Historical

data

BBN model

Netica tool

Fit Impact

Project

Metrics

BTA package Adapt model

$No

Elicitation

with

Experts

BN

Library

Cost of

Quality

Model

Life cycle defect model

Structure

EUROSTAR 2005

James Waskiel


Some Typical Key Factors• Model inputs

– Factors elicited from Motorola software engineers and process experts• Available metrics (e.g. faults, reviews preparation, code size)

• Subjective factors assessed through expert opinion (e.g. team leader, senior eng.)

– Specific to each software activity

• Product Factors– Number of requirements, code size

– Complexity, Interactions

– Age of baseline, existing customer base

• People Factors– Team experience (domain, language, technology, process, tools, etc…)

– Mentoring / coaching availability

– Project staffing, schedule pressure

• Process Capability & Maturity Factors– Requirements quality (stability, clarity, completeness, etc…)

– Supporting Processes (CM, tools, …)

– V&V activities metrics (PSP, reviews preparation time,…)

– Tools capability, availability

EUROSTAR 2005

James Waskiel


Life cycle defect model

Defects

Defects

DefectsUnit Design

Code

Unit

Verification

Code

Verification

Box Design

Box Design

Verification

Box

Specification

Box

Specification

Verification

Box Testing

Integration

Testing

Unit Testing

B

AT

B

AT

B

AT

B

AT

B

AT

B

AT

B

AT

B

AT

B

AT

B

AT

Defects

Defects

Defects

EUROSTAR 2005

James Waskiel


Bayesian networks for the Box testing phase

faults defects

requirements

requirements

technical review

box testing

PCE

scaped

defects

EUROSTAR 2005

James Waskiel


Modelling BNs using the BTA-toolkit

BDE

Toolset

Netica

Web

questionnaire

BTA

EUROSTAR 2005

James Waskiel


Data Collection

EUROSTAR 2005

James Waskiel


Prediction resultsPrediction results after

requirements FTR

REQ_FTR factors

Network

links

EUROSTAR 2005

James Waskiel


Test Suite 2

DevelopmentDevelopment

The Bayesian Test Assistant (BTA)

applied to Testing

System Test

Pre-test

Development

System Test

New Feature Test

TCs

BTA Outputs

Test Suite 1

Component

Defects

Test Selection

Test Selection

Preliminary

Test Strategy

Updated

Test Strategy

Updated data

Preliminary data

Create Test Cases

EUROSTAR 2005

James Waskiel


Box Development Model

• Box models developed with experts in CGISS, GSM, PCS, GEMS

• Predictions of Defects introduced, removed and left

TEST

CODING

REQ

Fault Density

Quality

Quality

DESIGNTEST

Latent Defects

Defects

Faults Found%

%

%Defects

B

AT

B

AT

B

AT

B

AT

B

AT

B

AT

BAT

B

AT

B

AT

B

AT

TEST

FAULTS

FAULTS

FAULTS

EUROSTAR 2005

James Waskiel


Model Development - Test case

• Service can have varying granularity

– System level - One service describing the entire system

– Test case level - One service describing each test case

• We model services at the feature level

• Each Feature is associated with a group of test cases

Test Database

New TCs

Old TCs

New TCs

Old TCs

New TCs

Old TCs

Feature 1

Feature 2

Feature 1{{{

Ser

vic

e 2

Ser

vic

e 3

Box 2

Ser

vic

e 1

Box 1

Box 3

EUROSTAR 2005

James Waskiel


Model Development - Service

• A Service describes the integration of boxes, forming a service

• “Exposure” indicates strength of usage

• Service defects = box_exposure * box_defects

Latent Defects

DefectsTesting

BBNTEST

Faults FoundUnit test

BBNTEST

CODING

BBNCOD

Requirements

BBNREQ

Code Faults

%

%

Fault Density

QualityDesign

BBN

Quality

DESIntegration test

BBNTEST

%

DefectsBox 1

Latent Defects

DefectsTesting

BBNTEST


BBNTEST

CODING

BBNCOD

Requirements

BBNREQ

Code Faults

%

%

Fault Density

QualityDesign

BBN

Quality

DESIntegration test

BBNTEST

%

Defects

Latent Defects

DefectsTesting

BBNTEST


BBNTEST

CODING

BBNCOD

Requirements

BBNREQ

Code Faults

%

%

Fault Density

QualityDesign

BBN

Quality

DESIntegration test

BBNTEST

%

Defects

Box 2

Box 3

Number of Latent

Defects in serviceSer

vic

e

Strong Exposure

Weak Exposure

Medium Exposure

EUROSTAR 2005

James Waskiel


Model Development - Testing

• Integration with New feature and Regression models

• Predictions of:

– Defects found during Regression/New Feature test

– Latent Defects, released to the field per service

Box 2

Feature 1

Ser

vic

e 1

Box 1

Box 3

Old TCs

Regression Test

RegressionFound

Defects

New TCs

New Feature Test

New FeatureFound Defects

Field Defects

EUROSTAR 2005

James Waskiel


Test Case Scoring

• Method– “Test Focus” weight based on test purpose

– Focus weights assigned to test cases as they are written

• Benefit– Score represents potential number of faults exposed by test

– Higher Score => test case is more likely to find a fault

…

10% 30% 40%

0% 10% 90%

0% 0% 100%

+ +

+ +

+ +

= 38

= 69

= 75

SW-FA 1 SW-FA 2 SW-FA 3

Faults left from development

Test Score35 15 75

… … … …

* * *

“Test Focus” weights from Testers

TC 4.1.1

TC 4.1.3

TC 4.1.2

Test cases

EUROSTAR 2005

James Waskiel


Test Case Selection

• Method– Determine number of test cases per service from strategy

– Pick test cases with highest “Score” first

• Benefit– Test strategy spreads more test cases over higher-risk services

– Better than using only the score to order/select cases

Service 4.1

Service 4.2

SW-FA 1 SW-FA 2 SW-FA 3

10% 30% 40%0% 10% 90%0% 0% 100%

+ ++ ++ +

= 38= 69= 75

Test CasesService

… … … …

SelectedSelected

Score

… … … …

80% 50% 30%100% 70% 10%60% 30% 50%

+ ++ ++ +

= 58= 53= 63

Selected

Selected

……TC 4.2.3

TC 4.2.1

TC 4.2.2

TC 4.1.1

TC 4.1.2TC 4.1.3

EUROSTAR 2005

James Waskiel


TETRA Pilot - Results

PCE CODE PHASE

0.62

0.64

0.66

0.68

0.7

0.72

0.74

0.76

0.78

1 2

BOX

PC

E

PCE_PREDICTED PCE_ACTUAL

PCE DESIGN PHASE

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

1 2

BOX

PC

E

PCE_PREDICTED PCE_ACTUAL

EUROSTAR 2005

James Waskiel


Conclusions and Lessons Learned

• Modelling– Consensus on BN expressive capability, factors and structure

– Library of generic models to compose tailored models

• Pilots– Keeping momentum is a challenge, need management support

– The data collection activity is very important

– The use of the web-questionnaire results in a low overhead to provide model inputs and test weights data

• Test selection– It provides a structured, informed, and efficient way of selecting test

cases

– It has provided good results when used in various pilot projects

• Next– Fully automate the test case selection process

– Integration with testing tools, e.g. TRAMS and TMS / Test Central

EUROSTAR 2005

James Waskiel


Q & A

Thank you

Contacts:

[email protected]

[email protected]

[email protected]

[email protected]

mailto:[email protected]








Optimized Testing with Bayesian NetworksConclusions and Lessons Learned • Modelling –Consensus...

Documents

Transcript of Optimized Testing with Bayesian NetworksConclusions and Lessons Learned • Modelling –Consensus...