Generating Complex and Faulty Test Data Through Model-Based Mutation Analysis

.lusoftware verification & validationVVS

Generating Complex and Faulty Test Data Through Model-Based Mutation Analysis

University of Luxembourg Interdisciplinary Centre for Security, Reliability and Trust Software Verification and Validation Lab

Daniel Di Nardo, Fabrizio Pastore, Lionel Briand

April 9, 2015

Data Processing

System

Multiple fields Nested structure

Constraints among fields

Huge amount

Invalid data

Accepted & processed

Discarded

Valid data

Data Processing

System

Invalid data


Discarded

Valid data

Real world data

Data Processing

System

Invalid data


Discarded

Valid data

Output in log files

Data Processing

System

Multiple fields Nested structure

Constraints among fields

Huge amount

Invalid data


Discarded

Valid data

Software Engineers manually handcraft erroneous inputs that

cover the fault model of the system

Expensive!

Configurations for Operators Data

Modelling

Generic Mutation Operators

Through

How to generate test data?

How to validate results?

How to satisfy the fault model?

Test Automation

Input Output

Data Modelling using Class Diagrams + OCL Constraints (ASE 2013)

context Vcdu inv: let frameCount : Integer = self.vcFrameCount, previousFrameCount : Integer = self.vcFrameCount in frameCount <> previousFrameCount + 1 implies VcduEvents.allInstances()

->exists(e | e.eventType = COUNTER_JUMP)

Satellite Transmission Header Packet 1 Packet 2 Packet 3 Packet 4...

Packet 1 Packet 2 Packet 3

VCDU 1

VCDU 2

...Packet 4 Packet 5 Packet 6...VCDU 3

...Packet 6...VCDU 4

VCDU 5...Packet 6 Packet 7

VCDU 6

SYN

SYN

SYN

SYN

SYN

SYN

SYN

VCID=1

Header

Header

Header

Header

Header

VCID=2

VCID=1

VCID=1

VCID=1

VCID=0 Idle Packet Pattern

Input Data Model (simplified) Transmission Vcdu

Header

Packet

PacketZone

ActiveZone IdleZone versionNumber : Integer spaceCraftId : Integer checksum : Integer

versionNumber : Integer type : Integer dataLength : Integer data : Bytesequence

data : Bytesequence

1..* 1

1..*

1 1

1

1

Input Data Model (simplified) Transmission Vcdu

Header versionNumber : Integer spaceCraftId : Integer checksum : Integer

Packet versionNumber : Integer type : Integer dataLength : Integer data : Bytesequence

PacketZone

ActiveZone IdleZone data : Bytesequence

VirtualChannel 1 1 1..* 1

1..* 1..*

1 1 1

1

1

Software Under Test Output

Constraints Checking

Test Input

Data Model

Violated Constraints

Model Instance

Field Data Mutation

Based Generation

Mutation & Validation

Output

Data Loading

Data Writing

Data Mutation

Execution

Validation Violated Constraints

Field Data

Input/Output constraints

Mutated Field Data

SUT

Instantiated Objects

Mutated Instantiated Objects

Input/Output structure

Data Model

14

Not useful to test any kind of

(trivially) invalid input

Important to generate test inputs that fit

a Fault Model


Output

Data Modelling

Data Loading

Data Wri3ng

Data Muta3on

Execu3on

Valida3on Violated Constraints

Transmission File Input/Output

structure


Faulty Transmission File

SES-‐DAQ



Fault Model of SES DAQ Fault Description Duplicate VCDU/Packet A VCDU/Packet appears twice in a transmission.

Missing VCDU/Packet A VCDU/Packet is omitted during transmission

Wrong Sequence VCDUs/Packets are sent out of order.

Incorrect Identifier Several transmission data fields have fixed values, e.g. fields identifying the transmitting satellite. Hardware/software errors may assign incorrect identifiers.

Incorrect Checksum Hardware/software errors may result in an incorrect checksum for a Packet or VCDU.

Incorrect Counter Counters are used to track Packet or VCDU ordering. Hardware/software errors may assign incorrect counter values

Flipped Data Bits Physical channel noise may flip one or more bits in the data transmission.

Covering The Fault Model

Generic Mutation Operators

Configurations for the Operators

18

Class Instance Operators

Attribute Instance Operators

Data Mutation Operators Transmission Vcdu



PacketZone



1..* 1..*

1 1 1

1

1

Class Instance Operators Transmission Vcdu



PacketZone



1..* 1..*

1 1 1

1

1

Class Instances Swapping

Class Instance Removal

Class Instance Duplication

Header versionNumber : Integer

spaceCraftId : Integer checksum : Integer

Transmission Vcdu

ActiveZone IdleZone


1

data : Bytesequence

«InputData»

PacketZone 1 1

Selecting Targets


1..* 1..* «InputData»

1

1


Class Instance Removal

Class Instance Duplication

Attribute Operators Transmission Vcdu



PacketZone



1..* 1..*

1 1 1

1

1


Header


PacketZone



1..* 1..*

1 1 1

1

1 Replacement with Random

versionNumber : Integer spaceCraftId : Integer checksum : Integer



PacketZone



1..* 1..*

1 1 1

1


«Identifier» versionNumber : Integer spaceCraftId : Integer checksum : Integer

Header


Header

Packet

PacketZone



1..* 1..*

1 1 1

1


«Identifier» versionNumber : Integer spaceCraftId : Integer «Identifier» checksum : Integer

Replacement with Boundary Condition



Header

Packet

PacketZone



1..* 1..*

1 1 1

1


«Identifier» versionNumber : Integer spaceCraftId : Integer «Identifier» checksum : Integer


versionNumber : Integer type : Integer «Measure» dataLength : Integer data : Bytesequence


Header

Packet

PacketZone



1..* 1..*

1 1 1

1


«Identifier» versionNumber : Integer «Identifier» spaceCraftId : Integer «Identifier» checksum : Integer


«Identifier» versionNumber : Integer «Identifier» type : Integer «Measure» dataLength : Integer data : Bytesequence


Header

Packet

PacketZone

ActiveZone IdleZone


1..* 1..*

1 1 1

1



Replacement with Boundary Condition Bit Flipping

data : Bytesequence


Avoiding Trivial Errors

Keeping Dependencies Transmission Vcdu

Header

Packet

PacketZone

ActiveZone IdleZone


1..* 1..*

1 1 1

1



Replacement with Boundary Condition Bit Flipping

data : Bytesequence


Keeping Dependencies Transmission Vcdu

Header

Packet

PacketZone

ActiveZone IdleZone


1..* 1..*

1 1 1

1


«Identifier» versionNumber : Integer «Identifier» spaceCraftId : Integer «Derived, Identifier» checksum : Integer


data : Bytesequence


Bit Flipping

Refining Target Selection Transmission Vcdu

Header versionNumber : Integer

spaceCraftId : Integer checksum : Integer


PacketZone



1..* 1..*

1 1 1

1

1


«InputData»

«InputData» Packet OCL queries to configure the targets on which an operator may work. E.g. swapping of packets that belong to the same virtual channel

Mapping Between Fault Model and"Mutation Operators

Fault Mutation Operator Configuration Duplicate VCDU Class Instance Duplication «InputData» Duplicate Packet Class Instance Duplication «InputData», «Derived» Missing VCDU Class Instance Removal «InputData» Missing Packet Class Instance Removal «InputData», «Derived» Wrong VCDU Sequence Class Instances Swapping «InputData» Wrong Packet Sequence Class Instances Swapping «InputData», «Derived»,

Query to select packets Incorrect Identifier Attribute Replacement with Random «Identifier», «Derived»

Incorrect Checksum Attribute Replacement with Random «Identifier»

Incorrect Counter Attribute Replacement using Boundary Condition «Measure», «Derived»

Flipped Data Bits Attribute Bit Flipping none



Field Data

Mutated Field Data


Output

Data Modelling

Data Loading

Data Writing

Data Mutation

Execution

Validation Violated Constraints

Input/Output structure


SES-DAQ

Data Mutation Strategies

• Random (RND)

•  Randomly select a mutation operator and randomly applies it to one of the possible targets.

•  Tester specifies number of test cases to generate per test suite.

• All Possible Targets (APT)

•  Each class or attribute of the data model is mutated at least once by each of the mutation operators that can be applied to it.

Empirical Evaluation •  Evaluated the effectiveness of the technique presented by

applying it on the SES-DAQ system.

•  Compare RND/APT to SES testing practice.

•  32 system test cases using synthetic input data.

•  Manually written based on domain expertise.

• GOAL: Determine if our technique can automatically achieve equivalent or better coverage than manual test cases.

Test Suite Generation • Applied both the RND and APT

• APT led to the generation of 43 test cases.

•  To compare the two strategies we generated 43 test cases for RND as well.

• Both APT and RND carry a degree of randomness.

•  Averaged results from 10 automatically generated test suites.

•  JaCoCo used to measure code coverage.

•  Number of bytecode instructions covered.

Experimental Results Technique Coverage (bytecode)

minimum maximum average SES Manual Test Suite -- -- 22820 (70.9%)

RND Test Suite Generation 22550 (70.1%) 23060 (71.7%) 22899 (71.2%)

APT Test Suite Generation 23226 (72.2%) 23374 (72.7%) 23283 (72.4%)

Automatically achieve slightly better coverage than test cases

written manually by domain experts.

Identified one crash condition (unknown fault).

Future Work •  Better evaluate the effectiveness of our test suite generation

strategies with faulty versions of the SES-DAQ software.

•  Devise a Search-Based approach to test suite generation.

•  Fitness based on:

•  Coverage of Model / OCL Constraints

•  Fault model

•  Code Coverage

Complex Test Data Difficult To Handcraft

Automatically achieve slightly better coverage than

manual test cases

Conclusion

Generating Complex and Faulty Test Data Through Model-Based Mutation Analysis

Software

Transcript of Generating Complex and Faulty Test Data Through Model-Based Mutation Analysis