Post on 23-Jan-2018
Mining Performance Regression Testing Repositories for Automated Performance AnalysisKing Chun Foo
Zhen Ming Jiang
Bram Adams
Ahmed E. Hassan
Ying Zou
Queen’s University, Canada
1
Parminder Flora
Research In Motion, Canada
Performance Regression
• Software changes over time– New features, bug fixes– May slow down the system
• Performance regressions describe situations where the performance degrades compared to previous releases
2
Performance Regression Testing
• Part of performance testing effort• A process of putting load on a system to test its
performance– Mimic the field usage which contains a mix of scenarios– Span from a few hours to a few days
• During a test, various data are collected– Execution logs– Performance metrics, e.g., CPU utilizations, response time
3
Current Practice of Performance Regression Testing
• Analysts conduct a performance regression test for a new software release
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
3. Analyze Test Result3. Analyze Test Result
4. Report and Store Analysis
4. Report and Store Analysis
4
Current Practice of Performance Regression Testing
• Tools check the metric averages against pre-defined thresholds
• Generate a plot for each metric
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
3. Analyze Test Result3. Analyze Test Result
4. Report and Store Analysis
4. Report and Store Analysis
5
Current Practice of Performance Regression Testing
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
3. Analyze Test Result3. Analyze Test Result
4. Report and Store Analysis
4. Report and Store Analysis
• Compare plots of the new test to those from a prior test
• look for evidence of performance regressions
6
Current Practice of Performance Regression Testing
• File bug report if necessary
• Archive all data and result in a central repository
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
3. Analyze Test Result3. Analyze Test Result
4. Report and Store Analysis
4. Report and Store Analysis
7
Challenge 1Lots of Metrics
• 2000+ metrics for our industrial case study
• Time consuming to analyze
• Difficult to consider all correlating metrics together
• Hard to compare more than 2 tests at once
8
Challenge 2No Up-to-date Performance Objectives
9
• Metric averages of the most recently passed test are used as thresholds– Miss common problems
Challenge 3 Analysts’ Subjectivity
• Analysts have different perceptions of performance regressions
• Analysis may be influenced by– Analyst’s knowledge– Deadline
10
Research Goal
1. Conduct Performance Regression Test
1. Conduct Performance Regression Test
2. Visualize Performance Metrics
2. Visualize Performance Metrics
4. Report and Store Analysis
4. Report and Store Analysis
11
Automate the analysis of
performance regression test
Automate the analysis of
performance regression test
3. Analyze Test Result3. Analyze Test Result
Performance Signatures
Intuition: Metric correlations are the same across tests
12
RepositoryRepository
Arrival RateMedium
Arrival RateMedium
CPU Utilization Medium
CPU Utilization Medium
Throughput Medium
Throughput Medium
RAM Utilization Medium
RAM Utilization Medium
Job Queue Size Low
Job Queue Size Low
…
Performance Signatures
Performance Signatures
• Performance signatures represents the expected metric correlations– Frequent violations mean abnormal behavior
• We derive performance signatures with Association Rule Mining
13
Throughput Medium
Throughput Medium
RAM Utilization Medium
RAM Utilization Medium
Job Queue Size Low
Job Queue Size Low
Premise Consequent
Performance Signatures
• Confidence– Probability that the premise leads to the
consequent
• Confidence will change if the new release behaves differently from past releases– Big changes represent performance regressions
14
Technique Overview
1. MetricNormalization
1. MetricNormalization
2. Metric Discretization
2. Metric Discretization
3. Performance Signature Derivation and Comparison3. Performance Signature
Derivation and Comparison4. Report
Generation4. Report
Generation
New RunNew Run
Performance Regression
Report
Performance Regression
Report
Deviation flagged?No – add to historical dataset for analysis of
future testsRepositoryRepository
15
Technique Overview
1. Metric Normalization– Remove test dependent information (e.g., lab
name)
1. Metric Discretization– Put metric values into discrete levels for our data-
mining technique
16
Technique Overview
3. We derive the performance signatures and measure the confidence change for each rule in the new test– Rules with large confidence change is reported
17
Technique Overview
4. Generate a report of performance regressions– Contains the metrics that show performance
regressions and violated signatures– Metrics are sorted by level of severity
18
Performance Regression ReportStep 1. Overview of Metrics with
Performance Regressions
19
Performance Regression ReportStep 2. Explanation of Performance
Regressions
20
Performance Regression ReportStep 3. Regressions Over Time
21
Case Studies
• 2 Open Source Systems– Dell DVD Store and JPetStore– Manually injected bugs to the systems to simulate
performance regressions
• Enterprise System– Compare metrics flagged by our technique against
analyst’s reports– Verify with the organization metrics that are not
present in the analyst’s reports
22
Case Studies Result
• Open source systems:– Precision: 75% - 100%– Recall: 52% - 67%
• Enterprise System:– Precision: 93%– Discovered new regressions that were not
included in the analysis reports
23
Conclusion
24