Extracting Examples for API Usage Patterns
Hudson Silva Borges(Msc Student)
Marco Túlio Valente(Advisor)
March 20th, 2014
Agenda
(1) Introduction
(2) Proposed Solution
(3) Evaluation
(4) Conclusions
2March 20th, 2014 Extracting Examplesfor API Usage Patterns
Introduction
3March 20th, 2014 Extracting Examplesfor API Usage Patterns
Introduction
● Systems often reuse functionality provided by APIs
● Reusing APIs generally requires a non-trivial effort
● Learning resources are a major obstacle● Documentation is the most severe obstacle
4March 20th, 2014 Extracting Examplesfor API Usage Patterns
5March 20th, 2014 Extracting Examplesfor API Usage Patterns
Motivation
● Rely on client usage data to improve APIs learning resources
6March 20th, 2014 Extracting Examplesfor API Usage Patterns
Why Examples?
Robillard e DeLine, Empirical Software Engineering – 2011
“code examples are an essential element of API learning”
7March 20th, 2014 Extracting Examplesfor API Usage Patterns
APIMiner 1.0
● Examples for single API methods
8March 20th, 2014 Extracting Examplesfor API Usage Patterns
Problem
Robillard e DeLine, Empirical Software Engineering – 2011
“Small examples that nevertheless demonstrate API usage patterns involving
more than one method call will be more useful than single-call examples”
Proposed Solution
9March 20th, 2014 Extracting Examplesfor API Usage Patterns
APIMiner 2.0 – Overview
10March 20th, 2014 Extracting Examplesfor API Usage Patterns
Extends APIMiner 1.0 to consider usage patterns and readability improvements to provide
better source code examples
Usage Patterns
● Set of API methods frequently called together
● Example:
open() ⇒ close()
11March 20th, 2014 Extracting Examplesfor API Usage Patterns
APIMiner 2.0 – Architecture
12March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
Architecture: Source Code Analyzer
13March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
Architecture: Source Code Analyzer
14March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
Client System Method API Calls
A ma1
m1, m
10
A ma2
...
B mb1
m22
B mb2
m2, m
3, m
11
...
Architecture: Patterns Analyzer
15March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
● API Usage Pattern
● Implications of the form X Y⇒
open() ⇒ close()
Patterns Analyzer – Association Rules
16March 20th, 2014 Extracting Examplesfor API Usage Patterns
Client System Method API Calls
A ma1
m1, m
10
A ma2
...
B mb1
m22
B mb2
m2, m
3, m
11
...
● Usage Data
Patterns Analyzer – Association Rules
17March 20th, 2014 Extracting Examplesfor API Usage Patterns
Transaction Items
t1
m1, m
10
t2
...
tn
m22
tn+1
m2, m
3, m
11
...
● Input to the algorithm
Patterns Analyzer – Association Rules
18March 20th, 2014 Extracting Examplesfor API Usage Patterns
Transaction Items
t1
m1, m
3, m
10
t2
m1, m
10
t3
m2, m
3, m
10
t4
m1
t5
m1, m
5, m
8, m
10
t6
m10
● m1, m10
● m1 m⇒ 10
● Algorithm
Patterns Analyzer – Association Rules
19March 20th, 2014 Extracting Examplesfor API Usage Patterns
Transaction Items
t1
m1, m
3, m
10
t2
m1, m
10
t3
m1, m
3, m
9
t4
m2
t5
m1, m
5, m
8, m
10
t6
m10
● m1, m10
● 3 transactions
● m1 m⇒ 10
● Support = 3● Confidence = ( 3 / 4 ) = 75%
● Algorithm
Architecture: Example Extractor
20March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
● For each client's method:
1. Identifies calls to API methods
2. Extracts examples for single API calls
3. Extracts examples for usage patterns
4. Removes similar examples
Architecture: Example Extractor
21March 20th, 2014 Extracting Examplesfor API Usage Patterns
● For each client's method:
1. Identifies calls to API methods
2. Extracts examples for single API calls
3. Extracts examples for usage patterns
4. Removes similar examples
Slicing algorithm
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
Extracting Examples – Readability improvements
22March 20th, 2014 Extracting Examplesfor API Usage Patterns
(1) Variables not identified
Extracting Examples – Readability improvements
23March 20th, 2014 Extracting Examplesfor API Usage Patterns
(1) Variables not identified
Extracting Examples – Readability improvements
24March 20th, 2014 Extracting Examplesfor API Usage Patterns
(1) Variables not identified
Original Extraction
Improved
Extracting Examples – Readability improvements
25March 20th, 2014 Extracting Examplesfor API Usage Patterns
(2) Empty blocks
Extracting Examples – Readability improvements
26March 20th, 2014 Extracting Examplesfor API Usage Patterns
(2) Empty blocks
?
Extracting Examples – Readability improvements
27March 20th, 2014 Extracting Examplesfor API Usage Patterns
(2) Empty blocks
Architecture: Example Extractor
28March 20th, 2014 Extracting Examplesfor API Usage Patterns
JavaDoc
● For each client's method:
1. Identifies calls to API methods
2. Extracts examples for single API calls
3. Extracts examples for usage patterns
4. Removes similar examples
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
Architecture: Ranking Engine
29March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
● Ranking examples● Completeness● Lines of Code● Users Feedback
● Ranking usage patterns● Lift● Number of methods in the association rule
Architecture: JavaDoc Weaver
30March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
Architecture: JavaDoc Weaver
31March 20th, 2014 Extracting Examplesfor API Usage Patterns
Architecture: JavaDoc Weaver
32March 20th, 2014 Extracting Examplesfor API Usage Patterns
Evaluation
33March 20th, 2014 Extracting Examplesfor API Usage Patterns
Android APIMiner 2.0
34March 20th, 2014 Extracting Examplesfor API Usage Patterns
www.apiminer.org
Android APIMiner 2.0
● Version 4.1.2-r1
● 26,540 methods● 2,968 classes
● 17,303 (65.2%) methods with JavaDoc annotation● 2,575 (86.76%) classes with JavaDoc annotation
35March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Dataset
● Script to automatically search and download in Github
● Criteria:● Java Programming language ● Minimum of 50 commits● Not a fork
● Some projects inserted manually
36March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Numbers:● 396 projects● 57,658 classes ● 450,762 methods● 362,900 calls to API methods.
● Highlight:● CyanogenMod● Astrid● Wordpress
37March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Dataset
● Rigid strategy
● Clients selected based on their popularity among the Android developers community
● Example Dataset Mining Dataset⊆
● Results:● 151 projects● 287,263 examples
38March 20th, 2014 Extracting Examplesfor API Usage Patterns
Example Dataset
Transactions
● Number of method calls in the useful transactions
39March 20th, 2014 Extracting Examplesfor API Usage Patterns
010
2030
API methods calls
% o
f use
ful t
rans
actio
ns
2 4 6 8 10 14 18 22 26 30 34 38
Median of 3
Mean of 4.6
Transactions
● Intra-class and Inter-class calls in useful transactions
40March 20th, 2014 Extracting Examplesfor API Usage Patterns
Association Rules
● Problem: What values to use for support and confidence?
● Our strategy:● A not very large number of rules● Maximize the API methods coverage
● Tests varying the values of support and confidence
41March 20th, 2014 Extracting Examplesfor API Usage Patterns
Association Rules
● Number of rules by varying the minimum support value
42March 20th, 2014 Extracting Examplesfor API Usage Patterns
Association Rules
● API methods coverage by varying the minimum support value
43March 20th, 2014 Extracting Examplesfor API Usage Patterns
Association Rules
● Used values:● Support = 21● Confidence = 70%
● Results:● 38,661 rules● 1,952 rules with antecedent size equal one● Coverage of 624 API methods; 191 API classes
44March 20th, 2014 Extracting Examplesfor API Usage Patterns
Examples
45March 20th, 2014 Extracting Examplesfor API Usage Patterns
● 151 Android client systems
● 287,263 examples● 102,442 examples for single API methods ● 184,821 examples for the usage patterns
● Coverage:● 3,915 Android API methods (14.75%) ● 549 Android API classes (18.5%)
Examples
46March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Size of the examples
● For single API methods● 50% have at most three lines of code
● For usage patterns● 50% have at most 14 lines of code
Ranking
47March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Activity.findViewById(int)
● API method with more examples
● Top 20 examples analyzed
● Examples size equal 1
● No compilation problems
Activity.findViewById(int)
48March 20th, 2014 Extracting Examplesfor API Usage Patterns
● To return a value
● In an assignment
● As a target
● In a casting
Usage Patterns
49March 20th, 2014 Extracting Examplesfor API Usage Patterns
● MediaRecorder.prepare()
● First usage patterns recommended:
MediaRecorder.prepare()
MediaRecorder.start()
MediaRecorder.setAudioSource(int)
MediaRecorder.setOutputFile(String)
MediaRecorder.prepare()
50March 20th, 2014 Extracting Examplesfor API Usage Patterns
MediaRecorder.prepare()
51March 20th, 2014 Extracting Examplesfor API Usage Patterns
MediaRecorder.prepare()
52March 20th, 2014 Extracting Examplesfor API Usage Patterns
MediaRecorder.prepare()
53March 20th, 2014 Extracting Examplesfor API Usage Patterns
MediaRecorder.prepare()
54March 20th, 2014 Extracting Examplesfor API Usage Patterns
● http://developer.android.com/reference/android/media/MediaRecorder.html
MediaRecorder.prepare()
55March 20th, 2014 Extracting Examplesfor API Usage Patterns
● http://developer.android.com/reference/android/media/MediaRecorder.html
Field Study
56March 20th, 2014 Extracting Examplesfor API Usage Patterns
Field Study
57March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Period of 13th May 2013 to 11th October 2013 (five months)
● No promotions
● Total of 32,335 visits
Field Study
58March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Top ten countries in visits
Field Study
59March 20th, 2014 Extracting Examplesfor API Usage Patterns
● 5,545 requests for single API examples
● 2,601 (46.9%) clicks on “Example” button
● 176 requests for usage patterns
● 135 (out of 536) API methods with usage pattern were requested
User Study
60March 20th, 2014 Extracting Examplesfor API Usage Patterns
User Study
61March 20th, 2014 Extracting Examplesfor API Usage Patterns
● 29 subjects
● Two corrective maintenance tasks in a small Android app
Do the recommendations provided by APIMiner 2.0 help developers to implement
maintenance tasks with less effort?
Do the recommendations provided by APIMiner 2.0 help developers to implement
maintenance tasks with less effort?
More Aqui
62March 20th, 2014 Extracting Examplesfor API Usage Patterns
Maintenance Tasks
63March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Designed to represent real maintenance activities
● Tasks with requirements well defined
● Tasks:● Obtaining location (Tutorial)● Inserting new properties● System messages
● Automated tests to check
● Crossover methodology
64March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Number of subjects who completed the tasks
User Study – Results
65March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Number of subjects who completed the tasks
User Study – Results
66March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Number of subjects who completed the tasks
User Study – Results
67March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Time spent in the tasks
User Study – Results
68March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Examples Requests
● 686 examples requests for single API methods● 182 examples requests for usage patterns
● 23 examples for single API methods by subject● 6 examples for usage patterns examples by subject
User Study – Results
Conclusions
69March 20th, 2014 Extracting Examplesfor API Usage Patterns
Contributions
70March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Usage patterns examples
● Readability improvements
● Android APIMiner 2.0● 1,952 usage patterns for 624 API methods● 287,263 examples● www.apiminer.org
● Evaluation● Data produced when creating the instance● Usage data obtained from real Android developers● Controlled experiment with 29 subjects
Future Work
71March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Detect different usage types among the examples
● Ranking criteria based on readability
● Other static slicing implementations
Publications
72March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Montandon J. E. et al. Documenting APIs with Examples: Lessons Learned with the APIMiner Platform. In 20th Working Conference on Reverse Engineering (WCRE), 2013.
● Borges H. S. et al. APIMiner 2.0: Recomendaçao de exemplos de uso de APIs usando regras de associaçao. IV Congresso Brasileiro de Software: Teoria e Prática (Sessão de Ferramentas), 2013.
● Borges H. S.. APIMiner 2.0: Recomendação de Exemplos de Uso de APIs usando Regras de Associação. Poster, ELA-ES, 2013.
Architecture: API and Client systems
75March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
● API Source Code
API’s source code
● API Clients to Mining and Selected API Clients
Repository with API client systems
Architecture: Mining Database
76March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
● Database that stores the data generated by the Source Code Analyzer
Architecture: Examples Database
77March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
● Store examples and usage patterns produced
Architecture: JavaDoc
78March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Database
Patterns Analyzer
APISourceCode
API Clients to
Mining
Souce Code Analyzer
Examples Extractor
Examples Database
JavaDoc
RankingEngine JavaDoc Weaver
Selected API Clients
● Output of the proposed approach
● Ratio of methods with at least an API call among the systems
● Considering all client methods● In median, for each 3 methods are called 4 API methods● 75% of the systems had a ratio between 0.78 and 4.40 API
calls/method
● Considering only client methods with API method calls● In median, for each 3 methods are called 9 API methods● 50% of the projects have between 2.72 and 3.89 API
calls/method
79March 20th, 2014 Extracting Examplesfor API Usage Patterns
Mining Dataset
Transactions
● Transaction = Client method
● Input to the association rules algorithm
● Not all clients' methods call an API method
● Transactions with a single call are not relevant
80March 20th, 2014 Extracting Examplesfor API Usage Patterns
Transactions
● Characterization of the all 512,993 transactions
81March 20th, 2014 Extracting Examplesfor API Usage Patterns
77.42%
9.19%
13.38%
Without calls
With a single calls
With more than one call
Transactions
● Characterization of the all 512,993 transactions
82March 20th, 2014 Extracting Examplesfor API Usage Patterns
77.42%
9.19%
13.38%
Without calls
With a single calls
With more than one call
Transactions
● Percentage of useful transactions per project
83March 20th, 2014 Extracting Examplesfor API Usage Patterns
Average of 30.49%
Median of 26.53%
Transactions
● Top five most common transaction items
84March 20th, 2014 Extracting Examplesfor API Usage Patterns
Association Rules
85March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Top five rules with higher support
Examples
86March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Number of API method calls in the examples
Examples
87March 20th, 2014 Extracting Examplesfor API Usage Patterns
● API methods with more examples
Examples
88March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Classes that most appeared in all examples
Usage Patterns
89March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Usage patterns without examples
● 280 association rules (14% of 1,952)
● 88 API methods out of the 624 distinct API methods
Field Study
90March 20th, 2014 Extracting Examplesfor API Usage Patterns
● 31 evaluations of examples by end-users
● Examples of 26 API methods
● Methods with more evaluations:● Color.rgb(int, int, int)
● ViewPager.OnPageChangeListener.onPageSelected(int)
● Intent.getStringArrayExtra(String)
● NotificationCompat.Builder.setVibrate(long[])
NotificationCompat.Builder.setVibrate(long[])
91March 20th, 2014 Extracting Examplesfor API Usage Patterns
User Study – Setup
92March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Subject characterization
● Crossover methodology● Group A and Group B
● Restrictions:● Subjects should have skills on the Java● Subjects should not be experts in the Android platform
User Study – Execution
93March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Brief introduction
● More Aqui App
● Task Obtaining location as tutorial
● Subjects:
1. Read the description
2. Register the starting time
3. Run the automated test
4. Perform the task
5. Register the ending time
User Study – Results
94March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Knowledge of the subjects in the Java
95March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Knowledge of the subjects in the Android
User Study – Results
96March 20th, 2014 Extracting Examplesfor API Usage Patterns
● Professional experience of the subjects (in years)
User Study – Results
Related Work
97March 20th, 2014 Extracting Examplesfor API Usage Patterns
● IDE-based Recommendation Systems
● Vertical Code Completion (VCC)● Mining API Code Snippets (MACs)
● API Documentation
● APIMiner 1.0● ExoaDocs
● APIMiner 2.0
Top Related