Recommendation Systems for Issue Management
-
date post
13-Sep-2014 -
Category
Science
-
view
161 -
download
1
description
Transcript of Recommendation Systems for Issue Management
Lund University - cs.lth.se/markus_borg/
Recommendation Systems for Issue ManagementMARKUS BORG, SOFTWARE ENGINEERING RESEARCH GROUP
Lund University - cs.lth.se/markus_borg/
• PhD student 2010-• Software engineering• Requirements-test alignment• Member of SWELL
• Supervisor: Per Runeson
Markus Borg
Per Runeson
• MSc CS and Eng. 2007• Developer, ABB (2007-2010)
• Compiler and editor development• Safety-critical systems• Embedded development
Lund University - cs.lth.se/markus_borg/
Agenda
• Background– Bug report inflow in large projects– Recommendation systems
• Overview of some ongoing bug research– Duplicate detection– Team assignment– Effort prediction– Change impact analysis
Lund University - cs.lth.se/markus_borg/
There Will Be Bugs!
Lund University - cs.lth.se/markus_borg/
Issue Reports as Batons in the Development “Relay Race”
Release
Change Board
Project managerTester
Issue handover
Developer
Developer Tester
Developer
Architect
Analyst
Designer
Lund University - cs.lth.se/markus_borg/
Challenges in the Bug Overflow
• 10 000s of bug reports in large projects– Continous inflow of bug reports– Limited resources to deal with it
• Q1. Which issues do we already know about?• Q2. Who should investigate the issue?• Q3. How long time will it take to correct this bug?• Q4. If we make a corrective change, what is the impact?
Lund University - cs.lth.se/markus_borg/
Recommendation System
• General definition
“a software application that aim to support users in their decision-making while interacting with large information spaces”
– Amazon, Netflix, RateBeer, Filmtipset...
• In software engineering”a software application that provides information items estimated to be valuable for a software engineering task in a given context”
Lund University - cs.lth.se/markus_borg/
Two main approaches to recommendations
Bug example
Lund University - cs.lth.se/markus_borg/
Q1. Which issues do we already know about?
Duplicate Detection
Lund University - cs.lth.se/markus_borg/
Issue Duplicate Detection
Lund University - cs.lth.se/markus_borg/
Network Analysis of Bug Reports
Lund University - cs.lth.se/markus_borg/
Subgraph Analysis in Bug Networks
Bug star
One central bug report pointing at several others
Dense ringMost bug reports are connected
Lund University - cs.lth.se/markus_borg/
Textual Similarity Analysis
1. Study natural language content
2. Calculate overlap
3. Recommend duplicates
Lund University - cs.lth.se/markus_borg/
Calculating Textual Similarity
Issue reports
Represent
Calculate angles in vector space
Lund University - cs.lth.se/markus_borg/
Reported Results
• Results typically reported as:– ”% found among top 10 recommendations”
• Sony Ericsson 40%• Firefox 60%, 90%• Eclipse 40%, 70%• Open Office 60%, 65%• Mozilla Foundation 45%, 65%• Android 60% (Jens?)
• Are these good results? Good question.
Lund University - cs.lth.se/markus_borg/
Q2. Who should investigate the issue?
Team assignment
Lund University - cs.lth.se/markus_borg/
Bug Assignment
Bug tossing!
Lund University - cs.lth.se/markus_borg/
Machine Learning to Find Patterns
• Bugs found in India late in testing phase are severe?• Bugs reports containing term ”termination” are costly?• Bugs reported on fridays with attached stack traces tend to
be deferred to later releases?
Lund University - cs.lth.se/markus_borg/
Automated Bug Assignment – Supervised Machine Learning
Machine learning
Train
Recommend team
Lund University - cs.lth.se/markus_borg/
How to Represent a Bug Report?
• Function• Severity• System Version• Submit Date• Submitter Location
etc.• … And the text! Title and description.
Lund University - cs.lth.se/markus_borg/
Experimental Setup
Ericsson
Company A
Machine learning
4 x
Pre-processing & feature selection
25-35 teams to choose between!
Lund University - cs.lth.se/markus_borg/
Results
• Prediction accuracy using state-of-the-art machine learning:– Ericsson 30-70%– Company A 30%
• At Ericsson– In line with accuracy of human assignment– Deploying tool in selected projects
• Textual features improve predictions, but training times increase a lot!– Working on finding a good balance
Lund University - cs.lth.se/markus_borg/
Q3. How long time will it take to correct this bug?
Effort prediction
Lund University - cs.lth.se/markus_borg/
Bug Report Clustering – Unsupervised Machine Learning
Automatically cluster bug reports based on natural language in report
Lund University - cs.lth.se/markus_borg/
Prediction of Bug Resolution Time
1. A new bug is submitted
2. Put it in the right cluster
3. Predict its resolution time
Lund University - cs.lth.se/markus_borg/
Results
• Clustered bug reports based on textual content– Android– Eclipse– Company A
• Average resolution times in bug clusters are different– Statistically significant
• Is it a good way to predict resolution times?– No, at least not yet…
Lund University - cs.lth.se/markus_borg/
Q4. If we make a corrective change, what is the impact?
Change impact analysis
Lund University - cs.lth.se/markus_borg/
Recommendations for Impact Analysis
Mandated by safety standards!
Reqs.Tests
Lund University - cs.lth.se/markus_borg/
Networks of Impacted Artifacts
Data mining in archive of previous impact analysis reports
Lund University - cs.lth.se/markus_borg/
Identification of Key Artifacts
Lund University - cs.lth.se/markus_borg/
Recommend Possible Impact
1. Textual similarity analysis
2. Follow links in network
Lund University - cs.lth.se/markus_borg/
Recommend Possible Impact
3. Rank recommendations– Network statistics– Textual similarity
Lund University - cs.lth.se/markus_borg/
Lund University - cs.lth.se/markus_borg/
Results
• Experiments in a laboratory setting– 30% of previous impact among top 5 recommendations– 40% among top 10 recommendations
• Prototype evaluation in industrial setting ongoing...
Lund University - cs.lth.se/markus_borg/
Summary
Lund University - cs.lth.se/markus_borg/
Summary
• Issue inflow in large projects can be daunting
• On the other hand, issue reports contain important information!
• With the right tools, the information overflow can be tamed to recommendations!