Developing Natural Language-based Software Analyses and Tools to Expedite Software Maintenance Lori...
-
Upload
reginald-dickerson -
Category
Documents
-
view
214 -
download
0
Transcript of Developing Natural Language-based Software Analyses and Tools to Expedite Software Maintenance Lori...
Developing Natural Language-based
Software Analyses and Toolsto Expedite Software Maintenance
Lori PollockCollaborators: K. Vijay-Shanker, Emily Hill,
GiriprasadSridharaPast collaborators: David Shepherd, Zachary P.
Fry, KishenMaloor
ProblemModern software is large and complex
object oriented class hierarchy
Softwaremaintenance:
- search/locate
- navigate
- understand
- modify
-> Automated Support
Successesin Software Maintenance Tools
object oriented class hierarchy
Good with local tasks
Good with traditional structure
object oriented class hierarchy
Scattered tasks are difficult
Programmers use more than traditional program structure
Challengesin Software Maintenance Tools
public interface Storable{...
activate tool
save drawing
update drawing
undo action
public void Circle.save()
//Store the fields in a file....
object oriented system
Key Insight:Programmers leave natural language clues that
can benefit softwaremaintenance tools
Observations
Studies on choosing identifiers
Impact of human cognition on names [Liblit et al. PPIG 06] Metaphors, morphology, scope, part of speech hints Hints for understanding code
Analysis of Function identifiers [Caprile and Tonella WCRE 99] Lexical, syntactic, semantic analysis Use for software tools: metrics, traceability, program
understanding
Carla, the compiler writer Pete, the programmer
I don’t care about names.
So, I could use x, y, z. But, no one
will understandmy code.
NLPA
Our Research Focus and Impact
Software Maintenance Tools
SearchSearch UnderstandingUnderstanding ……
Natural Language Analysis
Word relations(synonyms, antonyms, …
Word relations(synonyms, antonyms, …
Part of speech tagging
Part of speech tagging ……AbbreviationsAbbreviations
ExplorationExploration
Our Research Contributions…
FindConceptConcern location tool
Clue Extraction +NL-based Program
Representation
Motivated useof NL clues
during maintenance
Dora theProgram Explorer
AbbreviationExpander
Word Relation ToolComparison Study
[MACS 05, LATE 05] [AOSD 06, IET 08]
[AOSD 07, PASTE 07][ASE 05]
iTimnaAspect Miner
[ASE 07]
[MSR 08] [ICPC 08]
Automatic Natural Language Clue Extraction from Source Code
Key Challenges:Decode name usageDevelop automatic NL clue
extraction process (focused on Java)
Create NL-based program representation
Molly, the Maintainer
What was Pete thinking
when he wrote this code?
Natural Language: Which Clues to Use?
Software MaintenanceTypically focused on actionsObjects are well-modularized
Focus on actions Correspond to verbsVerbs need Direct Object
(DO)
Extract verb-DO pairs
Extracting Verb-DO Pairs
Two types of extractionclass Player{ /** * Play a specified file with specified time interval */ public static boolean play(final File file,final float fPosition,final long length) { fCurrent = file; try { playerImpl = null; //make sure to stop non-fading players stop(false); //Choose the player Class cPlayer = file.getTrack().getType().getPlayerImpl(); …}
Extraction from comments
Extraction from method signatures
public UserList getUserListFromFile( String path ) throws IOException {
try {
File tmpFile = new File( path );
return parseFile(tmpFile);
} catch( java.io.IOException e ) {
thrownew IOrException( ”UserList format issue" + path + " file " + e );
}
}
Extracting Clues from Signatures
1. Part-of-speech tag method name
2. Chunk method name
3. Identify Verb and Direct-Object (DO)
get<verb> User<adj> List<noun>From <prep>File <noun>
get<verb phrase> User List<noun phrase>FromFile <prep phrase>
POS Tag
Chunk
Representing Verb-DO PairsAction-Oriented Identifier Graph
(AOIG)verb1 verb2 verb3 DO1 DO2 DO3
verb1, DO1 verb1, DO2 verb3, DO2 verb2, DO3
source code files
use
use
use
use
use
use
useuse
Action-Oriented Identifier Graph (AOIG)
Example
play add remove file playlist listener
play, file play, playlist remove, playlist add, listener
source code files
use
use
use
use
use
use
useuse
Evaluation of Clue Extraction
Compared automatic vs ideal (human) extraction 300 methods from 6 medium open source programs Annotated by 3 Java developers
Promising Results Precision: 57% Recall: 64%
Context of Results Did not analyze trivial methods On average, at least verb OR direct object obtained
Using AOIG in Concern Location
Find, collect, and understand all source code related to a particular concept
Concerns are often crosscutting
State of the Art for Concern Location
Mining Dynamic Information [Wilde ICSM 00]
Program Structure Navigation [Robillard FSE 05,
FEAT, Schaefer ICSM 05]
Search-Based Approaches RegExp[grep, Aspect Mining Tool 00]
LSA-Based [Marcus 04]
Word-Frequency Based [GES 06]
Reduced to similar problem
Slow
Fast
Fragile
Sensitive
No Semantics
Limitations of Search Techniques
1. Return large result sets2. Return irrelevant
results3. Return hard-to-
interpret result sets
Find-Concept Search Tool
concept
Find-Concept query
Recommendations
Source Code
Method a
Method bMethod c
Method d Method e
NL-basedCode Rep
Result GraphNatural
Language Information
1. More effective search
2. Improved search terms
3. Understandable results
Underlying Program Analysis
Word Recommendation AlgorithmStemmed/Rooted: complete,
completingSynonyms: finish, completeCo-location: completeWord()Uses traversals of Action-oriented
identifier graph (AOIG)
Experimental Evaluation
Research Questions Which search tool is most effective at forming
and executing a query for concern location? Which search tool requires the least human
effort to form an effective query?
Methodology:18 developers completenine concern locationtaskson
medium-sized (>20KLOC) programs
Measures:Precision (quality), Recall (completeness),
F-Measure (combination of both P & R)
Find Concept, GES, ELex
Overall Results
Effectiveness FC > Elex with statistical
significance FC >= GES on 7/9 tasks FC is more consistent than
GES Effort
FC = Elex = GES
FC is more consistent and more effective in experimental study without requiring more effort
Across all tasks
NLPA
Our Research Focus and Impact
Software Maintenance Tools
SearchSearch UnderstandingUnderstanding ……
Natural Language Analysis
Word relations(synonyms, antonyms, …
Word relations(synonyms, antonyms, …
Part of speech tagging
Part of speech tagging ……AbbreviationsAbbreviations
ExplorationExploration
Dora the Program Explorer*
* Dora comes from exploradora, the Spanish word for a female explorer.
DoraDora
Natural Language Query• Maintenance request• Expert knowledge• Query expansion
Natural Language Query• Maintenance request• Expert knowledge• Query expansion
Relevant Neighborhood
Program Structure• Representation
• Current: call graph• Seed starting point
Relevant Neighborhood• Subgraph relevant to query
Query
State of the Art in Exploration
Structural (dependence, inheritance) Slicing Suade [Robillard 2005]
Lexical (identifier names, comments) Regular expressions: grep, Eclipse search Information Retrieval: FindConcept [Shepherd
2007], Google Eclipse Search [Poshyvanyk 2006]
Dora: Using Program Structure + Ids
Program: JBidWatcher, an eBay auction sniping program
Bug: User-triggered add auction event has no effect
Task: Locate code related to ‘add auction’ trigger Seed: DoAction() method, from prior knowledge
Example Scenario:
Key Insight: Automated tools can use program structure and identifier names to
save the developer time and effort
DoNada() DoNada() DoNada() DoNada() DoNada()DoNada() DoNada()DoNada()DoNada() DoNada() DoNada()
DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()
DoNada() DoNada()DoNada() DoNada() DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()
Using only structural information
DoAction() has 38 callees, only 2/38 are relevant Relevant
Methods
Irrelevant Methods
Looking for: ‘add auction’ trigger
DoAction()
DoAdd()
DoPasteFromClipboard()
And what if you wanted to explore more than one edge away?
Locates locally relevant items, but many irrelevant
Using only lexical information
50/1812 methods contain matches to ‘add*auction’ regular expression query
Only 2/50 are relevant
Locates globally relevant items, but many irrelevant
DoNada() DoNada() DoNada() DoNada() DoNada()DoNada() DoNada()DoNada()DoNada() DoNada() DoNada()
DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()
DoNada() DoNada()DoNada() DoNada() DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()
Combining Structural &Lexical Information
Structural: guides exploration from seed
Looking for: ‘add auction’ trigger
RelevantNeighborhood
DoAction()
DoPasteFromClipboard()
DoAdd()
Lexical: prunes irrelevant edges
The Dora Approach
Determine method relevance to queryCalculate lexical-based relevance score
Prune low-scored methods from neighborhood
Recursively explore
Prune irrelevant structural edges from seed
31
Evaluation of the Dora Approach
Evaluated on 9 concerns Lexical + structural >
structural However, success
highly dependent on lexical scoring performance
Structureonly
Structureonly
NLPA
Our Research Focus and Impact
Software Maintenance Tools
SearchSearch UnderstandingUnderstanding ……
Natural Language Analysis
Word relations(synonyms, antonyms, …
Word relations(synonyms, antonyms, …
Part of speech tagging
Part of speech tagging ……AbbreviationsAbbreviations
ExplorationExploration
Automatic Abbreviation Expansion
1.Split Identifiers: Punctuation Camel case No boundary
e.g., strlen
2.Identify non-dictionary words
3.Determine long form
non-dictionary wordnon-dictionary word
no boundaryno boundary
• Don’t want to miss relevant code with abbreviations
• Given a code segment, identify character sequences that are short forms and determine long form
•Approach: Mine expansions from code [MSR 08]
Simple Dictionary Approach
Manually create a lookup table of common abbreviations in code
- Vocabulary evolves over time, must maintain table
- Same abbreviation can have different expansions depending on domain AND context:
cfg?Control Flow Graph
Context-Free Grammar
configuration
configure
Types of Non-Dictionary Words
Single-WordPrefix (attr, obj, param, i)Dropped Letter (src, evt, msg)
Multi-WordAcronyms (ftp, xml, [type names])Combination (println, doctype)
OthersNo boundary (saveas, filesize)Misspelling (instanciation, zzzcatzzzdogzzz)
Long Form Search Patterns
Given short form arg, we search for regular expressions matching long forms in code:Single-Word
PrefixargumentDropped letteraverage
Multi-WordAcronym attribute random groupCombinationaccess rights
Search Pattern Order
Search by abbreviation type:
AcronymAcronymAcronymAcronym
CombinationCombinationCombinationCombination
PrefixPrefixPrefixPrefix
Dropped Dropped LetterLetter
Dropped Dropped LetterLetter
Multi-Word Single Word
Conservative
GreedyGreedy
How do we identify potential long forms for each type?
Inspired by static scoping, start from method containing abbreviation and search increasingly broader “scopes” until clear winner:
1.JavaDoc
2.Type Names of declared variables
3.Method Name
4.Statements5.Referenced identifiers and string literals
6.Method comments
7.Class comments
Context-based Approach through Scope
What if no long form found?
Fall back to Most Frequent Expansion (MFE)
MFE leverages successful local expansions and applies throughout the program1.Program: provides domain knowledge2.Java: more general programming
knowledge
Scope 57% more accurate than state of the art LFB
Scope 30% more accurate than Java MFE
Program MFE acceptable approximation when speed more important than accuracy
Experimental EvaluationN
umbe
r of
Cor
rect
Exp
ans
ions
250
200
150
100
50
0NoExp
LFB JavaMFE
ProgMFE
OurScope
Accuracy: 22% 40% 45% 54% 59%
63%
In Conclusion…
Evaluation studies indicateNatural language analysis has far more potential to improve software maintenance tools than we initially believed
Existing technology falls shortSynonyms, collocations, morphology, word
frequencies, part-of-speech tagging, AOIG
Keys to further successImprove recall Extract additional NL clues