Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online...
Transcript of Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online...
![Page 1: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/1.jpg)
Test Case Prioritization Using Online Fault
Detection Information
Mohsen Laali
Huai Liu
Margaret Hamilton
Maria Spichkova
Heinz Schmidt
![Page 2: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/2.jpg)
Outline
• Introduction
– Regression Testing
– Test Case Prioritization (TCP)
• Research Questions
• Experimental Methodology & Results
• Conclusions
RMIT University©July 2015 School of Computer Science and IT 2
![Page 3: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/3.jpg)
Introduction
• Regression Testing
• Different Types
– Test Case Minimization
– Test Case Selection
– Test Case Prioritization
RMIT University©Jun 2016 School of Computer Science and IT 3
![Page 4: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/4.jpg)
TCP Techniques
• Coverage Granularity
– Statement
– Branch
• Existing Techniques
– Total
– Additional
RMIT University©Jun 2016 School of Computer Science and IT 4
TS
TB
ABAS
![Page 5: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/5.jpg)
TCP Techniques Illustration
RMIT University©Jun 2016 School of Computer Science and IT 5
• Total Statement TCP: t1, t4, t5, t2, t3
• Additional Branch TCP: t5, t2, t1, t3, t4
![Page 6: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/6.jpg)
Code-level TCP
• Motivation
– Even Distribution of Faults
80-20 rule
– Offline Method
Location of fault
• Can a TCP Technique be improved using the location of previously identified
faults?
• How different coverage criteria could affect the effectiveness of TCP
techniques based on the location of pervious detected faults?
RMIT University©July 2015 School of Computer Science and IT 6
![Page 7: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/7.jpg)
Proposing Online TCP Techniques
RMIT University©Jun 2016 School of Computer Science and IT 7
![Page 8: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/8.jpg)
Online TCP Techniques
RMIT University©Jun 2016 School of Computer Science and IT 8
Online Test Case Prioritization Algorithm
![Page 9: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/9.jpg)
Proposing Online TCP Techniques (Cont.)
RMIT University©Jun 2016 School of Computer Science and IT 9
Weight for each statements
Weight for covered statements
OS 0
OB 0
OB 1OS 1
UCW = 0
UCW = 1
![Page 10: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/10.jpg)
Online TCP – Statement-Based Example
RMIT University©Jun 2016 School of Computer Science and IT 10
si t1 t2 t3 t4 t5
s1 3/3 3/3 3/3 3/3 3/3
s2 3/3 3/3
s3 1 1 1
s4 3/3
s5 1
s6 1
s7 1 1
s8 3/3
s9 1
s10 1
Total 4 3 3 4 4
• Step 1
– Randomly selected t1
– Given 3 detected faults by t1
– Updating weights by 3/3
si t1 t2 t3 t4 t5
s1 1 1 1 1 1
s2 1 1
s3 1 1 1
s4 1
s5 1
s6 1
s7 1 1
s8 1
s9 1
s10 1
Total 4 3 3 4 4
• Step 2
– Randomly selected t5
– Given 2 detected faults by t5
– Updating weights by 2/5
si t1 t2 t3 t4 t5
s1 3/3 2/5+3/5 2/5+3/5 2/5+3/5 2/5+3/5
s2 3/3 3/3
s3 2/5 2/5 2/5
s4 3/3
s5 1
s6 1
s7 2/5 2/5
s8 3/3
s9 1
s10 2/5
Total 4 3 2.4 2.8 2.2• Step 3
– Selecting t2 with maximum weight
– Given 3 detected faults by t2
– Updating weights by 3/8
si t1 t2 t3 t4 t5
s1 3/32/8+3/8+3/8 2/8+3/8+3/8 2/8+3/8+3/8 2/5+3/5
s2 3/3 3/8+3/8
s3 2/8 2/8 2/5
s4 3/3
s5 3/8
s6 1
s7 2/8 2/5
s8 3/3
s9 1
s10 2/5
Total 4 2.12 2.25 2.5 2.2
• Step 4
– Selecting t4 with maximum weight
– Given 4 detected faults by 4
– Updating weights by 4/12
• Final order:
– t1, t5, t2, t4, t3
fi t1 t2 t3 t4 t5
f1
f2
f3
f4
f5
f6
f7
f8
f9
f10
Total 3 3 4 4 2
![Page 11: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/11.jpg)
Experimental Objects
• Siemens Programs
- Consists of 7 programs
- Test case, faulty versions, and oracle version
RMIT University©July 2015 School of Computer Science and IT 11
![Page 12: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/12.jpg)
Experimental Study
RMIT University©Jun 2016 School of Computer Science and IT 12
![Page 13: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/13.jpg)
Summary of Statistical Analyses
RMIT University©Jun 2016 School of Computer Science and IT 13
- Example 1:
OS0 outperforms baseline AS in 7
object programs.
- Example 2:
OB0 outperforms baseline TB in one
object program, while twice has been
outperformed by the TB.
AS AB
AS AB
TS
TS
TB TB
OS0 OS1 OB0 OB1
Higher performing
Lower performing
![Page 14: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/14.jpg)
Summary
• Can a TCP Technique be improved using the location of previously identified
faults?
Our proposed techniques outperform baselines for Siemens
experimental objects using APFD score.
• How different coverage criteria could affect the effectiveness of TCP
techniques based on the location of previously detected faults?
Coverage criterion affects the performance
Comparing our methods with baselines
Both statement & branch-based outperform baselines
Comparing our proposed methods
Branch-based methods more effective than Statement-based
RMIT University©Jun 2016 School of Computer Science and IT 14
![Page 15: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/15.jpg)
![Page 17: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/17.jpg)
Backup Slides
RMIT University©March 2011 International & Development Portfolio 17
![Page 18: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/18.jpg)
APFD
RMIT University©March 2011 International & Development Portfolio 18
![Page 19: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/19.jpg)
RQ 2) Human-level TCP
• RQ 2-1) How often do users repeat the same type of faults in software
developments?
• RQ 2-2) What do affect the frequency of users’ faults in software
developments?
– Volume of contributions to the code
– Time of contributions to the code
– Change rate of the code
– User community of the code language
RMIT University©July 2015 School of Computer Science and IT 19
![Page 20: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/20.jpg)
RQ 2) Human-level TCP
• Data
– Git hub archive
• Methods
– Pre-requisites
Human Error Classification
Error Associations
– RQ 2-1
– RQ 2-2
RMIT University©July 2015 School of Computer Science and IT 20
![Page 21: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/21.jpg)
RQ 2) Human Classification Error
• Method
RMIT University©July 2015 School of Computer Science and IT 21
![Page 22: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/22.jpg)
RQ 2) Bugs Classification
RMIT University©July 2015 School of Computer Science and IT 22
![Page 23: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/23.jpg)
RQ 2) Error Associations
RMIT University©July 2015 School of Computer Science and IT 23
Time
![Page 24: Test Case Prioritization Using Online Fault Detection ... · Test Case Prioritization Using Online Fault Detection Information Mohsen Laali Huai Liu Margaret Hamilton Maria Spichkova](https://reader034.fdocuments.in/reader034/viewer/2022043022/5f3e6fe6dd39a0059b0c472d/html5/thumbnails/24.jpg)
Future Work
• Online TCP Policy
80-20 rule
Captured 20% by the online execution of test cases
• Possible Improvements
Who makes the faults?
Human habits in making faults
Capturing 20% incorporating human factors
RMIT University©Jun 2016 School of Computer Science and IT 24