Augmenting Field Data for Testing Systems Subject to Incremental Requirements Changes

.lusoftware verification & validationVVS

Augmenting Field Datafor Testing Systems Subject to

Incremental Requirements ChangesDaniel Di Nardo, Fabrizio Pastore, Lionel Briand

SnT Centre, University of Luxembourg

Journal First - ACM Transactions on Software Engineering and Methodology

Volume 26 Issue 1, July 2017

Data Processing Systems

• Aa

• aa

Data Processing Systems

Data Processing SystemsProcessingWorkflows

Web Crawlers

Data Processing SystemsProcessingWorkflows

Web Crawlers

Solution: Reuse Field Data [ICST’15]

Data Requirements Change

Changes affect Data Fields, Data Types, Constraints

Old Test Cases Cannot be ReusedField Data is Not Available

Test Input for Base Version

Test Input for Updated Version

Automatically Generate

OriginalModel Instance Incomplete Updated

Model Instance

Data LoadingSlice-basedConstraint

Solving

AutomaticModel

TransformationData Writing

Valid UpdatedModel Instance

UpdatedDataField

Data

Data Model

1

Transmission Vcdu

VcduHeader

Packet

PacketZone

ActiveZoneIdleZone

1 11..*1

1

versionNumber : IntspaceCraftId : IntvirtualChanId : Int

apidValue : ApidsequenceCount : IntdataLength : Int

PacketHeader

SarPacketHeadercoarseTime : IntfineTime : Int

GpsrPacketHeadercoarseTime : IntfineTime : Int

ConfigurationspaceCraftId : IntcheckCrc : BoolidleVcid : VcididleApid : Apid

VcduConfigvcid : VcidminPacketSize : IntmaxPacketSize : Intapids : Apid[]

11

1..*

1

prev 1..*

1

1

1

«DataType» Apid

value : Int

1

Transmission Vcdu

VcduHeader

Packet

PacketZone

ActiveZoneIdleZone

1 11..*1

1



PacketHeader





11

1..*

1

prev

context VcduHeader inv: self.vcdu.trans.config.vcduConfigs.vcid

→exists(x | x = self.virtualChanId)

OCL constraints

1..*

1

1

1

«DataType» Apid

value : Int

1

Transmission Vcdu

VcduHeader

Packet

PacketZone

ActiveZoneIdleZone

1 11..*1

1..*

1



PacketHeader





11

1..*

1

prev

context Packet inv: ( self.apidValue.value=1 and self.pHeader.oclIsTypeOf(SarPacketHeader) ) or …

1

1

1



OCL constraints

«DataType» Apid

value : Int

Capture New Data Requirements

inUpdated Data Model

1

Transmission Vcdu

VcduHeader

Packet

PacketZone

ActiveZoneIdleZone

1 11..*1

1..*

1



PacketHeader

«Replacement» MsiPacketHeadercoarseTime : IntfineTime : IntopMode : Intstatus : Int


prev

context Packet inv: ( self.apidValue.value=1 and self.pHeader.oclIsTypeOf(SarPacketHeader) ) or …



11

1..*

1




OCL constraints

«DataType» Apid

value : Int

1

Transmission Vcdu

VcduHeader

Packet

PacketZone

ActiveZoneIdleZone

1 11..*1

1..*

1



PacketHeader

prev

context Packet inv: ( self.apidValue.value=3 or self.apidValue.value=4 and self.pHeader.oclIsTypeOf(MsiPacketHeader) ) or …



11

1..*

1

«Replacement» MsiPacketHeadercoarseTime : IntfineTime : IntopMode : Intstatus : Int




OCL constraints

«DataType» Apid

value : Int


Model Instance


Solving

AutomaticModel



UpdatedDataField

Data

Data Model

t1:Transmissionp1: Packet

apidValue=apid1sequenceCount=1dataLength=200

ph1: SarHeadercoarseTime=10fineTime=30

pd1: PacketDatadata=..

c1: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0

vc1: VcduConfigvcid=vcid1minPacketSize=200maxPacketSize=300apids=<apid1,apid2>

v1:Vcdu a1:ActiveZone

vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…

p2: PacketapidValue=apid1sequenceCount=2dataLength=200






Original Model Instance


Model Instance

Data LoadingSlicing-based

ConstraintSolving

AutomaticModel



UpdatedDataField

Data

Data Model















Replace obsolete classes



ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=












Replace obsolete classes






vc2: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,

apid5,apid6>









Load new configuration







apid5,apid6>









context VcduHeader inv: self.vcdu.trans.con g.vcduCon gs.vcid→exists(x | x = self.virtualChanId)

Updated data may invalidate constraints







apid5,apid6>









context Packet inv: …( (self.apidValue.value=3 or self.apidValue.value=4) and self.pHeader.oclIsTypeOf(MsiPacketHeader) ) or …

Updated constraints may not hold







apid5,apid6>









Incomplete Updated Model Instance


Model Instance

Data LoadingSlicing-based

ConstraintSolving

AutomaticModel



UpdatedDataField

Data

Data Model







apid5,apid6>









Identify all the root-to-leaf paths







apid5,apid6>









Generate a slice for each

a1:ActiveZonet1:Transmissionp1: Packet






apid5,apid6>

v1:Vcdu









All the instances “linked by an OCL”to an element of the path must belong to the slice

context PacketZone inv: self.packets→forAll(x : Packet | self.vcdu.trans.config.vcduConfigs→select(y : VcduConfig |

y.vcid = self.vcdu.vcduHeader.virtualChannelId).apids→exists(z : Apid | z = x.apidValue) …







apid5,apid6>










Elements of collections are linked to the previous element only

Constraint solving

Slice withupdated values

Generateslices


t1:Transmission





apid5,apid6>

v1:Vcdu

a1:ActiveZone


“Solve” a slice//Declare classesabstract sig Vcdu {trans:one Transmission, packetZone:onePacketZone }..

//Declare class instancesone sig vc2 extends VcduConfig{}..

//Facts for associationsfact t1.config=c2..//Facts for config variablesfact { vc2.apids = apid3+apid4..

//Facts for data variablesfact { p1.apidValue = apid1 }fact { p1.sequenceCount = 1 }

//Alloy predicate for OCLspred PacketHeaderApid []{ all e: PacketHeader | …

Alloy Model

UMLt

oAllo

y

OCL constraints…

t1:Transmission





apid5,apid6>

v1:Vcdu

a1:ActiveZone


“Solve” a slice

UMLt

oAllo

y

OCL constraints…

Alloy Analyzer No Solution

Relax Model



Remove facts that prevent

the generation of a solution

Alloy Instance

t1:Transmission


ph1: MsiHeadercoarseTime=10fineTime=30opMode=1status=1



apid5,apid6>

v1:Vcdu

a1:ActiveZone


“Solve” a slice

UMLt

oAllo

y

OCL constraints…

Alloy Analyzer No Solution

Relax Model



Remove facts that prevent

the generation of a solution

Alloy Instance

Solve slice


Copy toIncomplete

Model Instance

Generateslices




t1:Transmission

v1:Vcdu

a1:ActiveZone

vh1

c2:..

vc1…



“Solve” next slice



t1:Transmission

v1:Vcdu

a1:ActiveZone

vh1

c2:..

vc1…



“Solve” next slice

Solving algorithm does not rewrite

variables generatedin previous steps

Solve slice


Copy toIncomplete

Model Instance

Generateslices


If a value “used” in a previous slicehas been modified, restart from first

Solve slice


Copy toIncomplete

Model Instance

Generateslices

Valid Updated Model Instance



Model Instance


Solving

AutomaticModel



UpdatedDataField

Data

Data Model

Empirical Evaluation

• RQ1: How does the proposed approach compare to a non-slicing approach?

• RQ2: Does the proposed approach scale to a practical extent?

• RQ3: Does the proposed approach allow for the effective testing of new data requirements?

• RQ4: Does the proposed approach allow for efficient and effective system testing compared to manual approach?

Case Study• Data Processing System for Satellite Data developed by SES

• Data Model

• 82 classes, 322 attributes, 56 associations, 52 OCL constraints

• Manual testing is expensive

• 130 class instances, 261 attributes, 1152 constrained attribute values

RQ1: How does the proposed approach compare to a non-slicing approach?

• To be justified the proposed approach should provide an advantage over one that does not use slicing

• Created a simplified technique that does not apply slicing

• All attribute values are generated from scratch from a single execution of the Alloy Analyzer

• Generated test inputs from 1 to 40 VCDUs

• 10 inputs for each size

Proposed approach (using slicing) vs a non-slicing approach

Scales betterthan non-slicing

approach

RQ2: Does the proposed approach scale to a practical extent?

• Generation of new data should be fast enough and scale effectively as file size increases

• Generated test inputs from 50 to 500 VCDUs

• 10 inputs for each size

Execution time required to generate inputs vs number of VCDUs

Non-trivial inputsgenerated in reasonable time

RQ3: Does the proposed approach allow for the effective testing of new data requirements?

• Automatically generate new data intended for use in robustness test cases

• Integrate a mutation approach (ICST’15) to generate robustness test cases

• Measure code coverage of the source code that implements the new data requirements

• Compare with test cases written by engineers

Field Data (Version 0)

Updated Data

(Version 1)

RobustnessTest Inputs(Version 1)

Slicing BasedGeneration

Mutation BasedGeneration

RQ3: Does the proposed approach allow for the effective testing of new data requirements?

Test suite

S2 Instructions Covered S2 Branches Covered # of Tests

Avg Min Max Avg Min Max

Auto 74 (77.9%)

74 (77.9%)

74 (77.9%)

9 (81.8%)

9 (81.8%)

9 (81.8%)

103.1

Test suite

S2 Instructions Covered S2 Branches Covered # of Tests


Auto 74 (77.9%)

74 (77.9%)

74 (77.9%)

9 (81.8%)

9 (81.8%)

9 (81.8%)

103.1

Manual 68 (71.6%)

- - 8 (72.7%)

- - 32

More effective than manual testing

RQ4: Does the proposed approach allow for effective system testing?

• In the presence of a new software version engineers are interested in testing the whole system

• Necessary to test unmodified functions when they process a new a data type (e.g. packet data extraction in the presence of a new packet type)

• Same approach of RQ3: robustness testing

• Measure code coverage of the whole system

• Compare with 32 expensive manually crafted test inputs

RQ4: Does the proposed approach allow for effective system testing?

More effective than manual testing

Test suite

Instructions Covered Branches Covered # of Tests


Auto 23,432.1(72.2%)

23,273(71.7%)

23,529(72.5%)

978.7 (51.3%)

957 (50.2%)

987 (51.8%)

103.1

Manual 23,046(71.0%)

- - 950 (49.8%)

- - 32

RQ4: Does the proposed approach allow for efficient and effective system testing?

• Both data modelling and manual testing are expensive

• Data Model: 82 classes, 322 attributes, 56 associations, 52 OCL constraints

• Manual test: 130 class instances, 261 attributes, 1152 constrained attribute values

• Data model useful for other purposes (e.g., discuss with system engineers)

• Manual and automatically generate tests have negligible execution time:

• Manual, 10 mins – Automatically generated, 30 mins

• Automatically generated tests are more effective


Model Instance


Solving

AutomaticModel



UpdatedDataField

Data

Data Model

Augmenting Field Data for Testing Systems Subject to Incremental Requirements Changes

Software

Transcript of Augmenting Field Data for Testing Systems Subject to Incremental Requirements Changes