MTAT.03.094 Software Engineering - ut · Week 10: Verification and Validation I Week 11: Industry...

MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015

MTAT.03.094

Software Engineering

Lecture 11:

Verification and Validation

(Part 2) Dietmar Pfahl

email: [email protected] Fall 2015


Exam Registration – Status as of Nov 19

Exam 1:

• Friday, 08-Jan-2015, 14:15-16:15, room 2-405

• Capacity limit: 90 students

• Registered: 47 – Free seats: 43

Exam 2:

• Friday, 15-Jan-2015, 14:15-16:15, room 2-405

• Capacity limit: 90 students

• Registered: 9 – Free seats: 81

You must register: First-come-first-serve (FIFO) principle


Schedule of Lectures

Week 01: Introduction to SE

Week 02: Requirements Engineering I

Week 03: Requirements Engineering II

Week 04: Analysis

Week 05: Development Infrastructure I

Week 06: Development Infrastructure II

Week 07: Architecture and Design

Week 08: No lecture

Week 09: Refactoring

Week 10: Verification and Validation I

Week 11: Industry Lecture (Testing)

Week 12: Verification and Validation II

Week 13: Agile/Lean Methods

Week 14: Industry Lecture (Agile)

Week 15: Measurement / Course

wrap-up, review and exam

preparation

Week 16: no lecture


Structure of Lectures 09 & 11

• Fundamental Definitions and Concepts

• Test Strategies

• Black-Box

• White-Box

• Assessing the Quality of Test Suites

• Test Coverage

• Mutation Testing

• Static Analysis

• Quality Prediction


How Do We Test This? 34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests

If only 3-way interactions, need only 33 tests

For 4-way interactions, need only 85 tests


Combinatorial Testing

What is it?

• Methods for systematically testing t-way interaction effects of input (or configuration parameter) values.

Why do it?

• The interaction of specific combinations of input values may trigger failures that won’t be triggered if testing input (or configuration parameter) values only in isolation.


Two Scopes of Combinatorial Testing

Pizza Ordering

System under test (SUT)

Test inputs

Test case OS CPU Protocol

1 Windows Intel IPv4

2 Windows AMD IPv6

3 Linux Intel IPv6

4 Linux AMD IPv4

Test Configurations

Test Inputs Size Topp Addr Phone

Sm Custom Val Val

Sm Preset Inv Inv

Med Custom Inv Val

Med Preset Val Inv

Lg Custom Val Inv

Lg Preset Inv Val


Combinatorial Testing – Example 2

Platform configuration parameters:

• OS: Windows XP, Apple OS X, Red Hat Linux

• Browser: Internet Explorer, Firefox

• Protocol: IPv4, IPv6

• CPU: Intel, AMD

• DBMS: MySQL, Sybase, Oracle

Total number of combinations: 3*2*2*2*3 = 72

Do we need 72 test cases?


Pair-Wise Testing (2-Way Interaction)

Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pair-wise interaction)

# of parameter pairs:

# of 2-way interactions N=57:

Test OS Browser Protocol CPU DBMS

1 XP IE IPv4 Intel MySQL

2 XP Firefox IPv6 AMD Sybase

3 XP IE IPv6 Intel Oracle

4 OS X Firefox IPv4 AMD MySQL

5 OS X IE IPv4 Intel Sybase

6 OS X Firefox IPv4 Intel Oracle

7 RHL IE IPv6 AMD MySQL

8 RHL Firefox IPv4 Intel Sybase

9 RHL Firefox IPv4 AMD Oracle

10 OS X Firefox IPv6 AMD Oracle

10)!25(!2

!5

2

5

9032

52

2

540 22

N



# of parameter pairs:

# of 2-way interactions N=57:

a:b 3 x 2 = 6

a:c 3 x 2 = 6 a:d 3 x 2 = 6 a:e 3 x 3 = 9 b:c 2 x 2 = 4 b:d 2 x 2 = 4 b:e 2 x 3 = 6 c:d 2 x 2 = 4 c:e 2 x 3 = 6 d:e 2 x 3 = 6 ------ 57

Test OS

(a)

Browser

(b)

Protocol

(c) CPU

(d)

DBMS

(e)











10)!25(!2

!5

2

5

9032

52

2

540 22

N




Parameter pair 1:

3 x 2 = 6 value combinations















Pair 2:
















Pair 3:














Is Testing 2-Way Interactions Enough?

• Analyses of failure-triggering conditions showed this:

• Medical device (dark blue)

• NASA distrib. DB (light blue)

• Browser (green)

• Web server (magenta)

• Network security (orange)

• TCAS* module (purple)

* Traffic Collision Avoiding System


ACTS Tool (NIST & UT Arlington)


Combinatorial Testing Links

http://cse.unl.edu/~citportal/

http://csrc.nist.gov/groups/SNS/acts/index.html


One problem is left …

Creating test (input) data is the (relatively) easy part!

How do we check that the code worked correctly on the test input?


How to (Automatically) Generate Test Oracles? Creating test (input) data is the (relatively) easy part!

How do we check that the code worked correctly on the test input?

• Using existing test sets (with known oracles) – easy if test sets exist

• Crash testing code to ensure it does not crash for randomly generated test input (‘fuzz testing’) – easy but of limited value

• Embedded assertions – incorporate assertions in code to check critical states at different points in the code – will they cover all possible incorrect states?

• Model-checking using a mathematical model of the system under test and a model checker to generate expected results for each input – expensive but tractable


State-Transition Testing

Create a set of test cases that triggers each state-transition at least once

Input (Event)

Action State-Transition Graph


State Table

State

Input (Event)

Wait for Card

(S1)

Wait for PIN

(S2)

Next

(S3)

Card inserted S2 (Ask for PIN) - -

Invalid PIN - S2 (Beep) -

Valid PIN - S3 (Ask amount) -

Cancel - S1 (Return card) -

... ... ... ...


State Table

State

Input (Event)

Wait for Card

(S1)

Wait for PIN

(S2)

Next

(S3)

Card inserted S2 (Ask for PIN) - -

Invalid PIN - S2 (Beep) -

Valid PIN - S3 (Ask amount) -

Cancel - S1 (Return card) -

... ... ... ...

4 Test Cases: - Input ’Card inserted’ in S1 - Input ’Invalid PIN’ in S2 - Input ’Valid PIN’ in S2 - Input ’Cancel’ in S2


Error Guessing

• Exploratory testing, happy testing, ...

• Always worth including

• Can trigger failures that systematic techniques miss

• Consider

• Past failures

• Intuition

• Experience

• Brain storming

• ”What is the craziest thing we can do?”

• Lists in literature


Exploratory Testing

• Inventors:

• Cem Kaner, James Bach (1990s)

• Definition:

• “Exploratory testing is simultaneous learning, test design, and test execution.”

• Elements / Variants

• Charter: defines mission (and sometimes tactics to use)

• Example: “Check UI against Windows interface standards”

• Session-based test management: Defects + Notes + Interviews of the testers


Testing Strategies

requirements

input

events

output

Black Box Testing White Box Testing


There are many possible paths!

loop < 20x

If-then-else

Selective Testing

White-Box Testing

520 (~1014 ) different paths


a selected path

Control flow testing

Data flow testing

Selective:

Selective Testing

MTAT.03.159 / Lecture 03 / © Dietmar Pfahl 2015

Control Flow Graph (CFG)


empty Blocks (=Nodes): 4

Edges: 4



empty Blocks: 4

Edges: 4

e3 e4

s1

s2

d1 s5

s6

s3

s4

Nodes: 8

Edges: 8

e7 e8

e1

e2

e5 e6



CFG Construction – Example

If (c1) then {

if (c2) then {s1}

s2

while (c3) do {s3}

}

else {

if (c4) then {

repeat {s4} until (c5)

}

}

d1 d2

d3

d5

d4

d6 d7

d8 d9

d10

d11

d12 d13

d14

c1

c4 c2

c5 c3

s1

s2

s4

s3



If (c1) then {

if (c2) then {s1}

s2

while (c3) do {s3}

}

else {

if (c4) then {


}

}

d1 d2

c1

CFG(f) CFG(t)

If (c1) then {

CFG(c1=true)

}

else {

CFG(c1=false)

}

}



If (c1) then {

if (c2) then {s1}

s2

while (c3) do {s3}

}

else {

if (c4) then {


}

}

d1 d2

d4

d6

d9

d10

d11

d14

c1

c4 CFG(if)

s2

If (c1) then {

CFG(if)

s2

CFG(while)

}

else {

if (c4) then {

CFG(repeat)

}

}

CFG (while)

CFG (repeat)



If (c1) then {

if (c2) then {s1}

s2

while (c3) do {s3}

}

else {

if (c4) then {


}

}

d1 d2

d3

d5

d4

d6 d7

d8 d9

d10

d11

d12 d13

d14

c1

c4 c2

c5 c3

s1

s2

s4

s3


Overview of Control Flow Criteria

Statement (or Block) Coverage – all nodes

Decision (or Branch) Coverage – all edges

Condition Coverage

Condition/Decision Coverage

Multiple Condition Coverage

Modified Condition Decision Coverage (MC/DC)

Linearly Independent Paths

Simple Paths

Visit-Each Loop

All Paths

…

d1 d2

d3

d5

d4

d6 d7

d8 d9

d10

d11

d12 d13

d14

c1

c4 c2

c5 c3

s1

s2

s4

s3


Life Insurance Example

boolean AccClient(int age; gtype gender)

if (gender = female && age < 85)

return(TRUE);

if (gender = male && age < 80)

return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4


Statement Coverage

Criterion:

• Execute each statement at least once

• Tools can be used to monitor execution ( coverage)

Possible concern:

• Dead code

• Example: assume that due to some previous calculations, AccClient can only be invoked with parameter value gender = female


Statement Coverage /1



return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:

AccClient(83, female)->accept

AccClient(83, male)->reject

0 %





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:



40 %





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:



80 %





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:



AccClient(25, male)->accept

100 %


Decision (Branch) Coverage /1



return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:



0 %





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:




25 %

T





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:




75 %

T/F

F





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:




100 %

T/F

T/F


Decision (Branch) Coverage /?


if (gender = female){

if (age < 85)

return(TRUE);

return(FALSE);}

if (gender = male){

if (age < 80)

return(TRUE);

return(FALSE);}

return(FALSE);

S

E

5

1

8

1:

2:

3:

4:

5:

6:

7:

8:

9:

d1 = c1

d3 = c3

4 6

Test:




75 %

2

3

7

d2 = c2

9

Dead code!

d4 = c4

T/F

T

T/F

T


Decision (Branch) Coverage /?


if (gender = female){

if (age < 85)

return(TRUE);

return(FALSE);}

if (gender = male){

if (age < 80)

return(TRUE);

return(FALSE);}

return(FALSE);

S

E

5

1 1:

2:

3:

4:

5:

6:

7:

8:

9:

d1 = c1

d3 = c3

6

Test:




100 %

2

3

7

d2 = c2

9

d4 = c4


Condition Coverage

• Test all conditions (in all predicate nodes)

• Each condition must evaluate at least once (or: once to ’true’ and once to ’false’)

• A (simple) condition may contain:

• Relational operators

• Arithmetic expressions

• ...

• A predicate may contain several (simple) conditions connected via Boolean operators

If (A<10 and B>250) then …


Condition Coverage /1



return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:



0 %





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:



50 % or 25 %

T T T





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

Test:



100 % or 62.5 %

T/F T T/F

T F F





return(TRUE);


return(TRUE);

return(FALSE);

S

E

3

1

5

1:

2:

3:

4:

5:

d1 = c1 && c2

d2 = c3 && c4

2 4

100 % or 75 %

T/F T T/F

T F/T F/T

Test:





Advanced Condition Coverage

• Condition/Decision Coverage (C/DC)

• as DC plus: every (simple) condition in each decision is tested in each possible outcome

• Modified Condition/Decision coverage (MC/DC)

• as above plus, every (simple) condition shown to independently affect a decision outcome (by varying that condition only)

• a (simple) condition independently affects a decision when, by flipping that condition and holding all the others fixed, the decision changes

• this criterion was created at Boeing and is required for aviation software according to RCTA/DO-178B

• Multiple-Condition Coverage (M-CC)

• all possible combinations of (simple) conditions within each decision taken


CC, DC, C/DC, M-CC, MC/DC Examples

Condition:

(TF) A = 2; B = 200 (D: False)

[(FT) A = 12; B = 300 (D: False)]

Decision:

(TT) A = 2; B = 300 (D: True)

(FT) A = 12; B = 300 (D: False)

Condition/Decision:

(TT) A = 2; B = 300 (D: True)

(FF) A = 12; B = 200 (D: False)

Multiple Condition:

(TT) A = 2; B = 300 (D: True)

(FT) A = 12; B = 300 (D: False)

(TF) A = 2; B = 200 (D: False)

(FF) A = 12; B = 200 (D: False)

Modified Condition/Decision:

(TT) A = 2; B = 300 (D: True)

(FT) A = 12; B = 300 (D: False)

(TF) A = 2; B = 200 (D: False)

If (A<10 and B>250) then …


Independent Path Coverage

• McCabe cyclomatic complexity estimates number of test cases needed

• The number of independent paths needed to cover all simple paths at least once in a program

• Visualize by drawing a CFG

• CC = #(edges) – #(nodes) + 2

• CC = #(decisions) + 1

if-then-else

while-loop

case-of


Independent Paths Coverage – Example

• Independent Paths Coverage

• Requires that a minimum set of linearly independent paths through the control flow-graph be executed

• This test strategy is the rationale for McCabe’s cyclomatic number (McCabe 1976) …

• … which is equal to the number of test cases required to satisfy the strategy.

1 2

3

5

4

6 7

8 9

10

11 12

13

14

Cyclomatic Complexity = 5 + 1 = 6



Edges: 1-2-3-4-5-6-7-8-9-10-11-12-13-14

Path1: 1-0-0-1-0-1-0-0-1-0---0---0---0---0

Path2: 1-0-1-0-1-1-1-1-1-0---0---0---0---0

Path3: 1-0-0-1-0-1-1-1-1-0---0---0---0---0

Path4: 0-1-0-0-0-0-0-0-0-1---0---1---0---1

Path5: 0-1-0-0-0-0-0-0-0-1---0---1---1---1

Path6: 0-1-0-0-0-0-0-0-0-0---1---0---0---0

1 2

3

5

4

6 7

8 9

10

11 12

13

14



Edges: 1-2-3-4-5-6-7-8-9-10-11-12-13-14

Why no need to cover Path7 ???

Path7: 1-0-1-0-1-1-0-0-1-0---0---0---0---0

Because it equals Path1+Path2-Path3 !!!

Path1: 1-0-0-1-0-1-0-0-1-0---0---0---0---0

Path2: 1-0-1-0-1-1-1-1-1-0---0---0---0---0

P1+P2: 2-0-1-1-1-2-1-1-2-0---0---0---0---0

Path3: 1-0-0-1-0-1-1-1-1-0---0---0---0---0

-P3: 1-0-1-0-1-1-0-0-1-0---0---0---0---0

1 2

3

5

4

6 7

8 9

10

11 12

13

14


Control-Flow Coverage Relationships

Subsumption:

a criterion C1 subsumes another criterion C2, if any test set {T} that satisfies C1 also satisfies C2

Statement

Decision

Complete Path

Linearly Indep. Path

Multiple Condition

MC/DC

Condition

C/DC


Data Flow Testing

• Identifies paths in the program that go

• from the assignment of a value to a variable

• to the use of such variable,

to make sure that the variable is properly used.

X:=14; ….. Y:= X-3;


Data Flow Testing – Definitions

• Def – assigned or changed

• Uses – utilized (not changed)

• C-use (Computation) e.g. right-hand side of an assignment, an index of an array, parameter of a function.

• P-use (Predicate) branching the execution flow, e.g. in an if statement, while statement, for statement.

• Example: All def-use paths (DU) requires that each DU chain is covered at least once

• Why interesting?

• E.g., consider: def-def-def (and no use) or use (without def)


Data Flow Testing – Example

Considering age, there are two DU paths:

(a)[1]-[4]

(b)[1]-[6]

Test case data:

AccClient(*, female)-> *

AccClient(*, male)-> *

[1] bool AccClient(agetype

age; gndrtype gender)

[2] bool accept

[3] if(gender=female)

[4] accept := age < 85;

[5] else

[6] accept := age < 80;

[7] return accept


Data Flow Testing – Example

Considering gender, there is one DU path:

(a)[1]-[3]

Test case data:

AccClient(*, *)-> *

[1] bool AccClient(agetype

age; gndrtype gender)

[2] bool accept

[3] if(gender=female)

[4] accept := age < 85;

[5] else

[6] accept := age < 80;

[7] return accept


Data Flow Criteria

Weaker

Stronger

# tests All uses

All p-uses, some c-uses

All c-uses All p-uses All defs

All def-use paths

All c-uses, some p-uses


Class Testing and State

push(s, elem1)

Show_top(s) -> elem1

pop(s, x) -> x=elem1

pop(s, x) -> x=?

The result of a method call

depends on the state of the

stack


Stack Example

How many test cases

needed to cover all

feasible branches?


Stack Example

How many test cases

needed to cover all

feasible branches?

4 decisions


Stack Example

2 test cases are enough:


Stack Example


D1

D2

D3

D4


Stack Example


D1

D2

D3

D4

D3: false


Stack Example


D1

D2

D3

D4

D1: false

D2: true

D1: false

D2: true

D1: false

D2: true

D1: true

D4: true ... false

D3: true


Stack Example – CFGs

if

if

if

true

true

false

false

true false

for true false


Stack Example – CFG (Class)

push(x)

if

if

if

true

true

false

false

true false

for true false

resize()

pop()

Stack

Stack()


State Diagram for Stack Example

non_exist empty (size = 0)

error

c

Stack() pop()

not_empty (0 < size < length)

[size = 1] pop() push(x)

[size < length - 1 ]

push(x) [size > 1] pop()

[size = length - 1] push(x) / resize()

push(x): size++ / pop(): size—

resize(): length := length * 2




• Test Strategies

• Black-Box

• White-Box


• Test Coverage


• Static Analysis



Code Coverage (Test Coverage)

• Measures the extent to which certain code items related to a defined test adequacy criterion have been executed (covered) by running a set of test cases (= test suites)

• Goal: Define test suites such that they cover as many (disjoint) code items as possible


Code Coverage Measure – Example

• Statement Coverage (CVs)

• Portion of the statements tested by at least one test case.

: number of statements tested

: total number of statemen

100%

ts

ts

p

t

p

SCV

S

S

S


Code Coverage Analysis

• Code coverage analysis is the process of:

• Finding areas of a program not exercised by a set of test cases

• Creating additional test cases to increase coverage

• Determining a quantitative measure of code coverage, which is (believed to be) a predictor of code quality

• Code coverage analyzers automate this process

• Additional aspect of code coverage analysis:

• Identifying redundant test cases that do not increase coverage


Main Classes of Test Adequacy Criteria

• Control Flow Criteria:

• Statement, decision (branch), condition, and path coverage are examples of control flow criteria

• They rely solely on syntactic characteristics of the program (ignoring the semantics of the program computation)

• Data Flow Criteria:

• Require the execution of path segments that connect parts of the code that are intimately connected by the flow of data

• Other Criteria:

• Requirements ’covered’, ...




• Test Strategies

• Black-Box

• White-Box


• Test Coverage


• Static Analysis



Static Analysis

• Document Review (manual)

• Different types

• Static Code Analysis (automatic)

• Structural properties / metrics


Static Code Analysis – Example 1 Bug finding is about pointing out programming mistakes: bad practice, coding errors, unexpected behavior. One interesting example of bugs that static analysis can find is null pointer dereference.

int i=0;

String s = null;

If (i>0) {

s = "positive";

}

If (s.contains("pos")) {

System.out.println(s);

}

This code will compile but at

runtime a null pointer exception

will be thrown, the String s being

null and calling the method

contains.

Static analysis tools, for instance

FindBugs can find this bug and

report it.


Static Code Analysis – Example 2 Bug finding is about pointing out programming mistakes: bad practice, coding errors, unexpected behavior. One interesting example of bugs that static analysis can find is security vulnerabilities.

Static analysis can be applied to find security flaws in code. With dataflow analysis, it is possible to follow the

propagation of input data, and thus detect possible code injection.

Example:

public static void main(String args[]) {

File f = new File(args[0]);

f.open();

//...

}

This program opens a file with an argument entered in command line. The fact that we use this argument to

open a file is just an example, the important fact is that we use directly an input without validation, which

constitutes a serious security vulnerability. Dataflow analysis can detect this kind of flaw by finding the source

of data inputs, following their propagation until their use in a sensitive instruction (like creating a File object).

Source: http://blog.zenika.com/index.php?post/2012/08/23/Static-analysis-(part-1/4)


Reviews - Terminology

• Static testing – testing without software execution

• Review – meeting to evaluate software artifact

• Inspection – formally defined review

• Walkthrough – author guided review


Reviews complement testing


Relative Cost of Faults Maintenance

200

Source: Davis, A.M., “Software Requirements: analysis and specification” (1990)


Reading Techniques

• Ad hoc

• Checklist-based

• Defect-based

• Scenario-based

• Usage-based

• Perspective-based


Perspective-based Reading

• Scenarios

• Purpose • Decrease overlap

(redundancy)

• Improve

effectiveness

Designer

Tester

User




• Test Strategies

• Black-Box

• White-Box


• Test Coverage


• Static Analysis



Quality Prediction

• Based on product and process properties

• Quality = Function(Code Size | Complexity)

• Quality = Function(Code Changes)

• Quality = Function(Detected #Defects)

• Quality = Function(Test Effort)

• Based on detected defects

• Capture-Recapture Models

• Reliability Growth models

Quality def.: Undetected #Defects


Capture-Recapture – Defect Estimation


Capture-Recapture – Defect Estimation

• Situation: Two inspectors are assigned to inspect the same product

• d1: #defects detected by Inspector 1

• d2: #defects detected by Inspector 2

• d12: #defects by both inspectors

• Nt: total #defects (detected and undetected)

• Nr: remaining #defects (undetected)

12

21

d

ddNt )( 1221 dddNN tr


Capture-Recapture – Example

• Situation: Two inspectors are assigned to inspect the same product

• d1: 50 defects detected by Inspector 1

• d2: 40 defects detected by Inspector 2

• d12: 20 defects by both inspectors

• Nt: total defects (detected and undetected)

• Nr: remaining defects (undetected)

10020

4050

12

21

d

ddNt 30)204050(100 rN


Advanced Capture-Recapture Models

• Four basic models used for inspections

• Degree of freedom

• Prerequisites for all models

• All reviewers work independently of each other

• It is not allowed to inject or remove faults during inspection


Advanced Capture-Recapture Models

Model

Probability of defect being

found is equal across ...

Estimator Defect Reviewer

M0 Yes Yes Maximum-likelihood

Mt Yes No Maximum-likelihood

Chao’s estimator

Mh No Yes Jackknife

Chao’s estimator

Mth No No Chao’s estimator


Mt Model

Maximum-likelihood: • Mt = total marked animals

(=faults) at the start of the t'th sampling interval

• Ct = total number of animals (=faults) sampled during interval t

• Rt = number of recaptures in the sample Ct

• An approximation of the maximum likelihood estimate of population size (N) is: SUM(Ct*Mt)/SUM(Rt)

First resampling:

M1=50 (first inspector)

C1=40 (second inspector)

R1=20

N=40*50/20=100

Second resampling:

M2=70 (first and second inspector)

C2=40 (third inspector)

R2=30

N=(40*50+40*70)/(20+30)=4800/50=96

Third resampling:

M3=80

C3=30 (fourth inspector)

R3=30

N=(2000+2800+30*80)/(20+30+30)=7200/80=90


Reliability Growth Models

• To predict the probability of future failure occurrence based on past (observed) failure occurrence

• Can be used to estimate

• the number of residual (remaining) faults or

• the time until the next failure occurs

• the remaining test time until a reliability objective is achieved or

• Application typically during system test


Reliability Growth Models (RGMs)

Purpose:

Stop testing when

a certain percentage (90%, 95%, 99%, 99.9%, …) of estimated total number of failures has been reached

a certain failure rate has been reached

Cumulative #Failures (m)

Test Intensity (t)

(CPU time, test effort, test days, calendar days, …)

100%

95%

(estimated n0)

dtet

0

0

0

0)(n

t

tm


Model Selection

Many different RGMs have been proposed (>100)

To choose a reliability model, perform the following steps:

1. Collect failure data

2. Examine data (failure data vs. test time/effort)

3. Select a set of candidate models

4. Estimate model parameters for each candidate model

Least squares method

Maximum likelihood method

5. Customize model using the estimated parameters

6. Compare models with goodness-of-fit test and select the best

7. Make reliability predictions with selected model(s)


Next Lecture

• Date/Time:

• Friday, 27-Nov, 10:15-12:00

• Topic:

• Agile / Lean Methods

• For you to do:

• Work on Lab Task 6

• Attend assessment session for Lab Task 5

MTAT.03.094 Software Engineering - ut · Week 10: Verification and Validation I Week 11: Industry...

Documents

Transcript of MTAT.03.094 Software Engineering - ut · Week 10: Verification and Validation I Week 11: Industry...