.lusoftware verification & validationVVS

Improving Fault Localization for Simulink Models

using Search-Based Testing and Prediction Models

Bing Liu, Lucia, Shiva Nejati, Lionel BriandSnT Centre, University of Luxembourg

SANER 2017, Klagenfurt, Austria

Simulation

2

Simulink

• Is a data flow-driven block diagram language

• Is widely used in the automotive domain

• Is executable and enables simulation and early testing

• Supports automated code generation

3

Simulink Fault Localization

4

✔ ATestcase

Background: Statistical Debugging

5

b4

b2 b3 b5

b7b6

b8

b10 b11 b12

b14b13

b9

b1

Block Rankings

b1b2b3b4b5b6b7b8b9

b10b11b12b13b14

input1

input2

input3

output

6

Ranking Scoreb1 0b2 0b3 0b4 0b5 0b6 0b7 0b8 0b9 0

b10 0b11 0b12 0b13 0b14 0

input1

input2

input3

output

Execute test case 1b4

b2 b3 b5

b7b6

b8

b10 b11 b12

b14b13

b9

b1

Background: Statistical Debugging

7

Ranking Scoreb6 1b2 0.5b3 0.5b5 0.5b7 0.5b8 0.5b9 0.5

b10 0.5b12 0.5b13 0.5b14 0.5b1 0b4 0

b11 0

input1

input2

input3

output

Execute test case 2b4

b2 b3 b5

b7b6

b8

b10 b11 b12

b14b13

b9

b1

Background: Statistical Debugging

Problem: Performance Limitations

8

Ranking Scoreb6 1b2 0.5b3 0.5b5 0.5b7 0.5b8 0.5b9 0.5

b10 0.5b12 0.5b13 0.5b14 0.5b1 0b4 0

b11 0

• Faulty blocks may not be ranked high

• Many blocks may have the same score

• Engineers may have to inspect many blocks until they find the faulty block(s)

Goal: Improving Statistical Debugging

• Statistical debugging can be improved by using larger test suites

• But, adding test cases is not cost-free in some contexts because

* test oracles need to be developed manually

* running test cases might be expensive

• We need to generate a small but effective set of test cases9

Our Approach

10

Model Test Suite + Test Oracle Ranking

Generate a small but effective set of test cases

No

YesRegenerate rankings

Are existing rankings likely to be improved by adding more test cases?

Static analysis+

Predictor models

By diversifying test cases

Test Case Generation

• We use a Single-State search technique to generate new test cases

• Our search strategy is guided by three alternative test objectives that aim to increase the diversity of the test suite

• Dynamic basic blocks; Baudry et al. [ICSE’06]

• Coverage Density; Campos et al. [ASE’13]

• Coverage dissimilarity; Jiang et al. [ASE’09]

• None of the above test objectives require test oracles11

Stop Test Generation Criteria

• We stop the test generation when adding test cases is unlikely to improve the rankings

• We rely on

* Simulink Super Blocks: based on static analysis of Simulink models

* Predictor models built using a supervised learningtechnique and historical data

12

Decision Trees• Input features:

• Current Round index

• Set Distance

• Ordering Distance

13

Experiment Evaluation

• We applied our approach to 60 single-fault-seeded versions of our three industrial Simulink models

• We compared our three different test objectives

• We evaluated the effectiveness of predictor models

• The experiment results (except for original industry models) are available at https://github.com/Avartar/TCGenForFL/)

14

Results (1)

15

• [Accuracies of different test objectives]ü The generated test cases significantly improved the fault localization accuracy

ü The three test objectives outperform Random test generation

ü No significant difference in the three test objectives

Results (2)

16

• [Impact of adding test cases]ü Adding test cases may not always improve fault localization accuracy

Results (3)

17

• [Effectiveness of predictor model]ü Our approach is able to maintain almost the same fault localization accuracy

while reducing the average number of newly generated test cases by more than half.

Conclusion

• We proposed a new approach to improve fault localization accuracy for Simulink models by extending an existing test suite with a small number of high quality test cases.

• We evaluated our approach on 60 faulty versions of industry models.

• Our approach significantly improves the accuracy of fault localization for small test suite sizes

• Our approach is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half

18