A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test...
Transcript of A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test...
![Page 1: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/1.jpg)
A Framework for Regression Test Prioritization and SelectionMaster Thesis Final Presentation
December 9th, 2015
Christian Plewnia [email protected]
![Page 2: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/2.jpg)
Regressions, Test Automation, and Regression Testing
Master Thesis Final Presentation | Christian Plewnia2
Source: http://www.seguetech.com
![Page 3: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/3.jpg)
Challenge
Master Thesis Final Presentation | Christian Plewnia3
Development Regression testing
Optimized
regression testing
Code
changes
Testing time
exceeded
![Page 4: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/4.jpg)
Regression Test Optimization
Master Thesis Final Presentation | Christian Plewnia4
All test cases
Regression Test Optimization (RTO)
Regression TestSelection (RTS)
Regression TestPrioritization (RTP)
Many techniques
described in
research
![Page 5: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/5.jpg)
RTO in Practice
• RTO is widely practiced
• techniques do not scale or
• the integration is difficult
Challenges of integration
• Integration into testing process
• Optimization requires data gathering
• Execution of optimized test suite
Master Thesis Final Presentation | Christian Plewnia5
manually by developers
![Page 6: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/6.jpg)
RTS & RTP Implementations
Master Thesis Final Presentation | Christian Plewnia6
Ekstazi RTS
InfiniTest RTS (live testing)
Test Load Balancer RTP
Echelon RTP; internal tool by Microsoft
Dependency Graph Tool* RTS; internal tool by Google
Cleanscape Testwise (ATAC) RTS
DejaVOO RTS
SPYDER RTS
Aristotle Analysis System Library used by some RTO techniques
discontinuedavailable* Real name unknown internal
![Page 7: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/7.jpg)
Master Thesis Final Presentation | Christian Plewnia7
Demo
![Page 8: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/8.jpg)
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia8
![Page 9: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/9.jpg)
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia9
![Page 10: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/10.jpg)
Lazzer Pipeline: Test Discovery
Master Thesis Final Presentation | Christian Plewnia10
• Find all test cases in the giventest suite
• Delegated to the test framework adapter
TestFramework
![Page 11: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/11.jpg)
Lazzer Pipeline: Pre-Optimization Data Collection
Master Thesis Final Presentation | Christian Plewnia11
• Enable data stores to collectinformation
• Example: determine source code changes
Data Store
![Page 12: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/12.jpg)
Lazzer Pipeline: Optimization
Master Thesis Final Presentation | Christian Plewnia12
• Perform regression testoptimization
• Delegated to optimization strategies
• The input and output is a test suite(a collection of test classes containing tests)
OptimizationStrategy
![Page 13: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/13.jpg)
Lazzer Pipeline: Test Run
Master Thesis Final Presentation | Christian Plewnia13
• Run the optimized test suite
• Delegated to the test framework
TestFramework
![Page 14: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/14.jpg)
Lazzer Pipeline: Post-Test-Run Data Collection
Master Thesis Final Presentation | Christian Plewnia14
• Enable data stores to collectinformation
• Example: Save test results in database
Data Store
![Page 15: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/15.jpg)
Lazzer Pipeline: Reporting
Master Thesis Final Presentation | Christian Plewnia15
• Report test results
• Example: write to the log or an XML file
![Page 16: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/16.jpg)
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia16
![Page 17: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/17.jpg)
Evaluation: Running Lazzer via its API
// Configuration
LazzerSettings settings =
new LazzerSettings.Builder(
// Test Framework Adapter
new JUnit4Adapter(),
// Optimization Strategy
Arrays.asList(new FailedFirstPrioritization()),
// Data Stores
Arrays.asList(new TestHistory(…))
.build();
// Instantiate lazzer
Lazzer lazzer = LazzerFactory.createLazzer(settings);
// Execute test optimization & test run
lazzer.run();
Master Thesis Final Presentation | Christian Plewnia17
ClientAdapter
![Page 18: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/18.jpg)
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia18
![Page 19: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/19.jpg)
Developing a Lazzer Optimization Strategy
public interface OptimizationStrategy {
TestSuite optimize(TestSuite testSuite);
}
Master Thesis Final Presentation | Christian Plewnia19
Called by the framework to
optimize the test suite
OptimizationStrategy
![Page 20: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/20.jpg)
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia21
![Page 21: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/21.jpg)
Developing a Lazzer TestRunner
public interface DataStore {
void preOptimizationDataCollection();
void postTestRunDataCollection
(TestRunReport testRunReport);
}
Master Thesis Final Presentation | Christian Plewnia22
Called by the framework before the optimization
Called by the framework after the test run
Data Store
![Page 22: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/22.jpg)
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia24
![Page 23: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/23.jpg)
Developing a Lazzer TestFrameworkAdapter
public interface TestFrameworkAdapter {
TestDiscovery provideTestDiscovery(
Classpath classpathForSystemUnderTest,
Classpath classpathForTests
);
Optional<TestRunReport> runTests(
TestSuite testSuite,
Classpath classpathForSystemUnderTest,
Classpath classpathForTests);
}
Master Thesis Final Presentation | Christian Plewnia25
Executes test run
Finds test cases
TestFramework
![Page 24: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/24.jpg)
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia26
![Page 25: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/25.jpg)
Evaluation
• Software quality assessed according to ISO/IEC 25010
• Aspects:
Master Thesis Final Presentation | Christian Plewnia27
UsabilityPerformance
EfficiencyFunctionality
Maintainability Reliability Portability
Compatibility Security
![Page 26: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/26.jpg)
Evaluation: Performance Efficiency
Master Thesis Final Presentation | Christian Plewnia28
0
10
20
30
40
50
60
70
80
90
100
110
120
130
0 100 200 300 400 500 600 700 800 900 1000
Run T
ime [m
s]
Runs [1]
Lazzer Run Time Evaluation 1,000 runs; 1 outlier (989ms) not considered
0
10
20
30
40
50
60
70
80
90
100
110
120
130
Run T
ime [m
s]
Maximum: 121 msAverage: 6 msMedian: 4 msMinimum: 1 ms
![Page 27: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/27.jpg)
Evaluation: Usability
• Setup
• Command Line: deploy framework JAR + extensions
• Maven: add some lines to project configuration
• Invocation
• Command Line: execution of a Java application
• Maven: run “test” phase
• Inspection of test results
• View log output
Master Thesis Final Presentation | Christian Plewnia29
![Page 28: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/28.jpg)
Evaluation: Functionality
• Working prototype
• Limitations
Configuration of data stores difficult
Tests of JUnit version (<4) not supported
Parameterized tests not recognized
Test results are only logged ⇒ extend API for custom reporting
No prioritization on method level
Not supported order: TestA.x(), TestB.y(), TestA.y()
Master Thesis Final Presentation | Christian Plewnia30
![Page 29: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/29.jpg)
Future Work on the Prototype
Tasks
• Overcoming current limitations
• Support for parallel testing
• More framework configuration options
• Lazzer needs more tests
Master Thesis Final Presentation | Christian Plewnia31
![Page 30: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/30.jpg)
Future Research
Some topics for future research:
• Implementation and evaluation of RTO techniques
• Support for other programming languages
• How to choose the best RTO technique?
• Recommendation for test suite reduction
• Test optimization as a service
Master Thesis Final Presentation | Christian Plewnia32
![Page 31: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/31.jpg)
Conclusion
Contributions
• Overview of state of research on RTO
• Framework for RTO
• Working prototype
Master Thesis Final Presentation | Christian Plewnia33
First step towards an RTO platform
![Page 32: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/32.jpg)
![Page 33: A Framework for Regression Test Prioritization and Selection · A Framework for Regression Test Prioritization and Selection Master Thesis Final Presentation December 9th, 2015 Christian](https://reader030.fdocuments.in/reader030/viewer/2022041017/5ecb28a17886ea396f6713a8/html5/thumbnails/33.jpg)
• Architecture
• Unit Testing
SonarQube Static Code Analysis of Lazzer Framework
Master Thesis Final Presentation | Christian Plewnia35
• Complexity
• Rule Compliance
• Documentation
Version 51 / 7th December 2015 14:25 Uhr