Software Defect Prediction Techniques in the Automotive Domain: Evaluation, Selection and Adoption

Software Defect Prediction Techniques in the Automotive Domain: Evaluation, Selection and Adoption

Rakesh Rana

Slide | 01

Outline:

Part 1: Introduction

Part 2: The Main Problem

Part 3: Research Questions & Methodology

Part 4: Scientific Contributions

Part 5: Industrial Impact

Part 6: Conclusions

Agenda

Slide | 02

Introduction

The Main Problem

Research Questions & Methodology

Scientific Contributions

Industrial Impact

Conclusions

Software (SW) in Automotive Domain

“It takes dozens of microprocessors running 100 million lines of code to get a premium car out of the driveway, and this software is only going to get more complex”

- IEEE spectrum

Driving Factors:

o Size & Complexity: >1 GB of software distributed over ~100 ECUs.

o Cost of SW & Electronics: for today’s premium carso ~40% of a vehicle’s development costs are

determined by electronics and software, and o 50 - 70% of the development costs for an ECU

(Electronic Control Unit) are related to software.

o Innovation & Trends: today, ~90% of all innovations are driven by electronics and software.

V&VFunctionalityMixed Blessing

Introduction

The Main Problem



Industrial Impact

Conclusions

Slide | 03

S. Fürst, “Challenges in the design of automotive software,” in Proceedings of the Conference on Design, Automation and Test in Europe, 2010, pp. 256–258.R. N. Charette, “This car runs on code,” IEEE Spectrum, vol. 46, no. 3, p. 3, 2009.

Software Defect – Definition

Defect: An imperfection or deficiency in a work product where that work

product does not meet its requirements or specifications and needs to be

either repaired or replaced.

Error: A human action that produces an incorrect result.

Failure: (A) Termination of the ability of a product to perform a required

function or its inability to perform within previously specified limits Or

(B) An event in which a system or system component does not perform a

required function within specified limits.

Fault: A manifestation of an error in software.

Problem: (A) Difficulty or uncertainty experienced by one or more

persons, resulting from an unsatisfactory encounter with a system in use

or (B) a negative situation to overcome.

IEEE standard 1044, Classification for Software Anomalies

Introduction

The Main Problem



Industrial Impact

Conclusions

Slide | 04

Software Defect – an example of defect description

Information type Attribute (example) Example defect description

Basic information Unique ID SWI-1234

Date & time stamp 12-Jul-201X, 16:33:04

Problem StatusOpen/ Resolved/ Closed/

CancelledOpen

Severity Major/ Minor/ Insignificant Minor

Problem TypeRequirement defect/ Design

defect/ Code defect Code defect

Title and Description

Title Misplaced white pixels on the home screen

Description

Precondition

System is up and running.

Action

Enter home screen in default mode (all tiles in standard size).

Note! This error is only applicable first time the user enters the home screen.

Once tiles in the home screen have been expanded, the misplaced pixels

disappear.

Observation

In the middle of the home screen there are some white and grey pixels in line

with the top of the phone tile (between the phone tile and the media tile).

Expected

There should not be any colour deviation of “stains” on the background screen.

Probability

100%

Additional Information Attachments Attached Log file

Comments

Update 201X-08-26:

When entering and exiting the Settings list, the misplaced pixels reappear, even

if tiles in the home screen have previously been expanded (and shrunk back to

default tile size again).

Introduction

The Main Problem



Industrial Impact

Conclusions

Slide | 05

Applying SDP Techniques in the Automotive Domain

Introduction

The Main Problem



Industrial Impact

Conclusions

Main Objective

1. Optimize allocation of test resources:

o Predicting the shape of defect inflow profile, and

o Predicting expected pre-release defect count for an on-going project.

Help experts within the automotive domain to:

2. Assess release readiness of software before release:

Slide | 06

Applying SDP Techniques in the Automotive Domain

Introduction

The Main Problem



Industrial Impact

Conclusions

The Main Problem

1. SW Development Process

o SW sourced through suppliers.

o Model-Based Development, and

2. Evaluation

Slide | 07

Research Questions

RQ1: Which defect prediction techniques are applicable at different stages of software life cycle in the automotive domain?

RQ2: How software reliability growth models can be used for defect predictions in the automotive domain?

RQ3: What are the consequences of mispredicting total number of defects and release readiness?

RQ4: How can correlation based models be used for defect prediction in the automotive domain?

RQ5: How to evaluate reliability characteristics of software at early stages of development using only behavioural models?

RQ6: What are important factors for industrial adoption of machine learning based defect prediction models?

To evaluate how software defect prediction techniques can be effectively applied over the software development life cycle within the automotive domain.

Introduction

The Main Problem



Industrial Impact

Conclusions

Slide | 08

Des

crip

tive

An

alyt

ical

Fun

dam

enta

l

Ap

plie

d

Qu

anti

tati

ve

Qu

alit

ativ

e

Co

nce

ptu

al

Emp

iric

al

Des

crip

tive

-

Rev

iew

of

liter

atu

re

Des

crip

tive

oth

erEv

alu

ativ

e-

ded

uct

ive

Eval

uat

ive-

inte

rpre

tive

Form

ula

tive

-

mo

del

Form

ula

tive

-

fram

ewo

rkFo

rmu

lati

ve-

guid

elin

esC

on

cep

tual

anal

ysis

Lite

ratu

re

revi

ew/a

nal

y

Cas

e st

ud

y

Act

ion

rese

arch

Dat

a an

alys

is

Sim

ula

tio

n

Fiel

d s

tud

y

Paper 1

Paper 2

Paper 3

Paper 4

Paper 5

Paper 6

Paper 7

Paper 8

Paper 9

Paper 10

Paper

No

Research approachResearch type Research methods

Research Methodology

Introduction

The Main Problem



Industrial Impact

Conclusions

Slide | 09

Research approach and methods are classification based on: R. L. Glass, I. Vessey, and V. Ramesh, “Research in software engineering: an analysis of the

literature,” Information and Software technology, vol. 44, no. 8, pp. 491–506, 2002.

Research type classification is based on: C. R. Kothari, Research Methodology: Methods and Techniques. New Age International, 2011.

RQ1: Which defect prediction techniques are applicable at different stages of

software life cycle in the automotive domain?

Introduction

The Main Problem



Industrial Impact

Conclusions

Main contributions:

o Overview of automotive SW development life cycle for large platform projects,

o Seven different SDP techniques were mapped according to the development phase they can be applied,

o Mapped SDP techniques were also classified based on the purpose and application level, and

o Roadmap for increasing the efficiency of defect predictions by using field data.Production Software

Slide | 10

RQ2: How software reliability growth models can be used for defect

predictions in the automotive domain?

Evaluation:

o Two commonly used parameter estimation procedure (MLE, and NLR) were compared,

o Eight SRGMs were evaluated at function level (best: logistic Model),

o Eight SRGMs were further evaluated at Project/Release level using defect data from three different companies from embedded software domain (best: logistic & Gompertz Model),

o How to use historical projects data to make defect count prediction more accurate,

Selection:

o Proposed method for predicting shape of defect inflow profile, and

o The statistical distribution family of defect inflow data was also explored where beta distribution was shown to be prominent family among tested distributions.

Introduction

The Main Problem



Industrial Impact

Conclusions

Slide | 11

Defect inflow intensity until half-way through the project Predicted shape of

defect inflow profileOverall trend Trend after reaching local maximum

Increasing Decreasing S-shape

Decreasing Decreasing Concave

Increasing Increasing Convex

ConcaveS-shape Convex

Empirical Evaluation at Project/Release levelIntroduction

The Problem



Industrial Impact

Conclusions



Slide | 12

Predicted

shape of

defect inflow

profile

Recommended SRGMs

For testing

resource(s)

allocation

For release readiness

assessment using

current project data

S-shape Logistic Logistic

Concave Gompertz Gompertz

Convex Delayed-S Logistic

Recommendation on applying SRGMs for SDP in industry:

o Collect the right data from testing,

o Perform exploratory analysis of data,

o Define goals for applying SRGMs,

o Select appropriate model based on predicted shape/distribution, and

o Use historical information when available

Introduction

The Problem



Industrial Impact

Conclusions



Slide | 13

Introduction

The Problem



Industrial Impact

Conclusions

RQ3: What are the consequences of mispredicting total number of defects and

release readiness?

Consequences of mispredictions:

o Two axis of accuracy of prediction were explicitly identified –

1) the prediction of the asymptote or the total number of defects, and

2) when the total number of defects are discovered or the release readiness.

o Consequences of mispredictions the shape of defect inflow profile were examined, and

o The consequence model was validated at two companies which provided insights into which mispredictions are critical in industry and how these are managed.

Slide | 14

Introduction

The Main Problem



Industrial Impact

Conclusions

Empirical Evaluation at Function (Feature) and ECU level

o Confirmed the Pareto principle of defect distribution (20-60 rule) for automotive software projects,

o Correlation between defects count in consecutive integration points - not found to be strong, and

o Evidence for strong correlation between number of defects found at IP4/IP5 with total pre-release defect count.

RQ4: How can correlation based models be used for defect prediction in the

automotive domain?

Slide | 15

Introduction

The Main Problem



Industrial Impact

Conclusions

RQ5: How to evaluate reliability characteristics of software at early stages of

development using only behavioural models?

Evaluation at Model/Code level

o Behavioural models in MBD offers significant opportunity to do reliability analysis early in the development process,

o Proposed a framework for early identification of design flaws and evaluating the efficacy of test suite to detect potential implementation defects that can potentially violate the safety goals, and

o The framework was validation to provide a proof-of-concept - 2 out of 24 mutants produced unexpected results and exposed the deficiency of the current test protocol.

Slide | 16

Introduction

The Main Problem



Industrial Impact

Conclusions

RQ6: What are important factors for industrial adoption of machine learning

based defect prediction

Adoption:

o Proposed a framework for adoption of machine learning based techniques in industry and its adaptation for software defect prediction,

o Identified nine main factors and various sub-attributes that affect the decisions of adoption of new techniques for SDP in the industry, and

o The framework was validation at two companies and specific challenges for adoption were identified.

Slide | 17

Software Reliability Growth Models (SRGMs)

Analogy Based Predictions

Correlation Based Models

Reliability analysis of Simulink Models

Introduction

The Main Problem



Industrial Impact

Conclusions

Supporting experts with data based model

o Applying different SDP models based on testing data provides empirical evidence and predictions that support experts in making their own forecasts,

o Data based models brings more objectivity in the predictions thus helping experts to potentially avoid some biases.

Slide | 18

Func Increase Calib/Optimization

SRGMs

Correlation

Simulink Model Analysis

ML

Introduction

The Main Problem



Industrial Impact

Conclusions

Efficient resource planning (cost/release timing)

o Automotive SW development is characterised by high time pressure/constraints, and

o Large part of verification & validation at vehicle level is done in late stages of development life cycle, thus:

o Reliability analysis early in development life-cycle can help in developing robust SW, and

o SDP about mid-way can provide useful estimates for efficient resource planning of testing resources.

Slide | 19

Introduction

The Main Problem



Industrial Impact

Conclusions

Conclusions

Slide | 20

Software Defect Prediction Techniques in the Automotive Domain:

Evaluation:

1. SRGMs To support test resource allocation decisions, For assessment of release readiness, and Using historical data to make better predictions.

2. Correlation based models For predicting defect count in next integration point, and For pre-release defect count predictions.

3. Behavioural models For early reliability analysis.

Selection:

1. How appropriate SRGMs could be selected by predicting the shape of defect inflow, and2. Using defect inflow distribution family to select appropriate SRGMs.

Adoption:

1. Explored factors that play important role for adoption (of ML based SDP techniques),2. Adoption framework is expected to help with wider adoption of such techniques.

Software Defect Prediction Techniques in the Automotive Domain: Evaluation, Selection and Adoption

Software

Transcript of Software Defect Prediction Techniques in the Automotive Domain: Evaluation, Selection and Adoption