Post on 13-Sep-2014
description
Findability through Traceability
- A Realistic Application of Candidate Trace Links?
Markus Borg
Software Engineering Research GroupDept. of Computer Science
Lund University / Markus.Borg@cs.lth.se
• MSc. Computer science and engineering
• Software engineer (2007 - 2010)• PhD Student Software Engineering (2010- )• Research interests
• Large-scale development• Traceability• Information management
Lund University / Markus.Borg@cs.lth.se
Outline of the Presentation
• Industrial case– Work task in safety-critical development
• Previously proposed solution (>100 papers)– Decision support using semi-automated trace
recovery• Impact on the case
– Automation Analysis– Work flow: Pre auto. vs. post auto.
• Conclusion
Lund University / Markus.Borg@cs.lth.se
Definitions
Lund University / Markus.Borg@cs.lth.se
Definitions
A trace link is a specified association between a pair of software artifacts.
Trace recovery is the process of establishing trace links among existing software artifacts.
Cleland-Huang et al. (2012)
Lund University / Markus.Borg@cs.lth.se
Definitions (2)
Findability is the degree to which a system or environment supports navigation and retrieval.
Morville (2005)
Lund University / Markus.Borg@cs.lth.se
Definitions (3)
Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections.
Manning et al. (2008)
Lund University / Markus.Borg@cs.lth.se
Description of the Case
Lund University / Markus.Borg@cs.lth.se
Case Company
• Large mutli-national company– Industrial automation
• Plan-driven development– Iterative gate-model– ~300 software engineers (Sweden and India)
• Safety-critical– Regulations mandate dev. process– V&V is vital to certify product
Lund University / Markus.Borg@cs.lth.se
Impact analysis
• Software changes should be analyzed before implementation
• Task integrated in issue tracker
Lund University / Markus.Borg@cs.lth.se
• For changes that have impact on safety parts– A question template must be filled in by the
engineer– 6 out of 13 questions require trace links
Impact analysis (2)
”List modified code files”
“Which documents need to be modified?”
“Which requirements and functions need to be retested?”
Lund University / Markus.Borg@cs.lth.se
Document spacePre-study
SRS
Func. spec.
System design
Detailed design
Source code
Design test spec.
Test scripts
Func. test spec.
System test spec.
IA Report
Lund University / Markus.Borg@cs.lth.se
Manual trace link seeking
Search similar reports
Search project docs.
Ask team
member
Q1. ?Q2. ?
.
.Qn. ?
Q1. ! Q2. !
.
.Qn. !
Impact analysis template
Impact analysis template
Lund University / Markus.Borg@cs.lth.se
Trace Recovery based on Information Retrieval
Lund University / Markus.Borg@cs.lth.se
IR-based Trace Recovery
• Information need
To what should this artifact be traced?
• Solution ideaSuggest textually similar artifacts
• AssumptionProvides a useful starting point for tracing
Lund University / Markus.Borg@cs.lth.se
IR-based Trace Recovery (2)
IR-based trace
recovery tool
==
Search tool with
automatically executed queries
Not very different from state-of-practice search solutions…
Lund University / Markus.Borg@cs.lth.se
Semi-automated trace link seeking
Search similar reports
Search project docs.
Ask team
member
Q1. ?Q2. ?
.
.Qn. ?
Q1. ! Q2. !
.
.Qn. !
Impact analysis template
Impact analysis template
Lund University / Markus.Borg@cs.lth.se
Automation Analysis:Method
Lund University / Markus.Borg@cs.lth.se
• Model of types and levels of human interaction with automation– 4 types– 10 levels
Analysis method (Parasuraman et al., 2000)
Lund University / Markus.Borg@cs.lth.se
Types of Automation
Acquire info
Analyze info
Decide action
Implement action
Not meaningful to distinguish for IR-
systems (Parasuraman et al., 2000)
Lund University / Markus.Borg@cs.lth.se
Levels of Automation
• From no assistance at all to a fully autonomous solution
(Parasuraman et al., 2000)Manual work
Decision support
Automated action, informs human
Automated action, not informing human
Level 1
Level 2-5
Level 6-8
Level 9-10
Lund University / Markus.Borg@cs.lth.se
Types and levels of automation
Acquire AnalyzeDecide
Implement
Leve
l
1
2
3
4
5
6
7
8
9
10
Lund University / Markus.Borg@cs.lth.se
Types and levels of automation (2)
Acquire
Analyze DecideImplement
Lev
el
1
2
3
4
5
6
7
8
9
10
Lund University / Markus.Borg@cs.lth.se
Automation Analysis:Results
Lund University / Markus.Borg@cs.lth.se
Type: Acquire + analyze information
• Software artifacts stored in different systems, e.g., – Requirements Databases– Test Management Systems– Source Code Repositories
• Challenges– Distributed information– Poor tool interoperability (i.e., information silos)– Finding relevant information
Lund University / Markus.Borg@cs.lth.se
Acquire + analyze information (2)
Pre-automation• Engineers must access
different information systems
• Engineer manually enters queries in search tools
• Possibly support for Boolean search operations, filtering etc.
Post-automation• Trace recovery tools must
access different information systems
• Tool automatically executes search queries
• Possibly support for filtering
Lund University / Markus.Borg@cs.lth.se
Acquire + analyze information (3)
IA Report
Reqs. DB
Test DB
DMS
Lund University / Markus.Borg@cs.lth.se
Acquire + Analyze information (4)
Acquire
Analyze DecideImplement
Lev
el
1
2
3
4
5
6
7
8
9
10
Post auto
Pre auto
Lund University / Markus.Borg@cs.lth.se
Type: Decide on Action
• Candidate trace links found• Decide which to report in the impact analysis report
Pre-automation• Engineers decide which
trace links to report
Post-automation• Engineers decide which
trace links to report
Lund University / Markus.Borg@cs.lth.se
Decide on action (2)
Acquire
Analyze DecideImplement
Lev
el
1
2
3
4
5
6
7
8
9
10
Post auto
Pre auto
Lund University / Markus.Borg@cs.lth.se
Type: Action implementation
• Report trace links in impact analysis report
Pre-automation• Engineers manually fill in
impact analysis report using free text
Post-automation• Engineers establish trace
links in trace recovery tool• Tool represents artifacts
and trace links as a network
Q1. ! Q2. !
.
.Qn. !
Lund University / Markus.Borg@cs.lth.se
Action implementation (2)
Acquire
Analyze DecideImplement
Lev
el
1
2
3
4
5
6
7
8
9
10
Post auto
Pre auto
Lund University / Markus.Borg@cs.lth.se
Conclusion
Lund University / Markus.Borg@cs.lth.se
Deploying a trace recovery tool
• An IR-based trace recovery tool would increase the level of automation...
Instead, other types of automation are addressed
- but not mainly targeting decision making
Lund University / Markus.Borg@cs.lth.se
Deploying a trace recovery tool (2)
• Consequently– less change involved than has been
acknowledged– risks related to decision making already present– should be combined with manual searching
• An IR-based trace recovery tool should mainly be considered a way to increase findability– A complementary search tool rather than a
semi-automatic trace tool
Lund University / Markus.Borg@cs.lth.se
Thanks for your attention!