JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons,...
Transcript of JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons,...
![Page 1: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/1.jpg)
Paruj Ratanaworabhan Kasetsart University, Thailand
Ben Livshits and Ben Zorn Microsoft Research, Redmond
JSMeter: Characterizing the Behavior of JavaScript Web Applications
1
in collaboration with
David Simmons, Corneliu Barsan, and Allen Wirfs-Brock
![Page 2: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/2.jpg)
Why Measure JavaScript?
• Standardized, de facto language for the web
– Support in every browser, much existing code
• Browser and JavaScript performance is important
– Are current JavaScript benchmarks representative?
– Limited understanding of JavaScript behavior in real sites
• Who cares?
– Users, web application developers, JavaScript engine developers
3
![Page 3: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/3.jpg)
4
ZDNet 29 May 2008
ghacks.net Dec. 2008
Browser
Wars!
![Page 4: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/4.jpg)
Artificial Benchmarks versus Real World Sites
5
Maps Maps
7 V8 programs:
• richards
• deltablue
• crypto
• raytrace
• earley-boyer
• regexp
• splay
8 SunSpider programs:
• 3-draytrace
• access-nbody
• bitops-nsieve
• controlflow
• crypto-aes
• date-xparb
• math-cordic
• string-tagcloud
JSMeter 11 real sites:
Goals of JSMeter Project
• Instrument JavaScript execution and measure behavior
• Compare behavior of JavaScript benchmarks against real sites
• Consider how benchmarks can mislead design decisions
![Page 5: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/5.jpg)
6
How We Measured JavaScript
\ie\jscript\*.cpp
Source-level instrumentation
custom jscript.dll
custom trace files website visits
Offline analyzers
custom trace files
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Constant
Other Str Ops
Concat Op
![Page 6: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/6.jpg)
Visiting the Real Sites
• Getting past page load performance
• Attempted to use each site in “normal” way:
7
amazon Search a book, add to shopping cart, sign in, and sign out bing Type in a search query and also look for images and news bingmap Search for a direction from one city to another cnn Read front page news ebay Search for a notebook, bid, sing in, and sign out economist Read front page news, view comments facebook Log in, visit a friend pages, browse through photos and comments gmail Sign in, check inbox, delete a mail, and sign out google Type in a search query and also look for images and news googlemap Search for a direction from one city to another hotmail Sign in, check inbox, delete a mail, and sign out
![Page 7: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/7.jpg)
Understanding JavaScript Behavior
Code
Events
JavaScript
Objects
8
![Page 8: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/8.jpg)
Code Behavior
Code
Events
JavaScript
Objects
9
• Function size
• Instructions/call
• Code locality
• Instruction mix
![Page 9: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/9.jpg)
Total Bytes of JavaScript Source
10
0
500
1000
1500
2000
2500
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ablu
e
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp-dna
str
ing-ta
gclo
ud
Sourc
e s
ize (kilobyte
s)
Real Sites V8 SunSpider
Code|Objects|Events
![Page 10: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/10.jpg)
Static Unique Functions Executed
12
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ablu
e
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp-dna
str
ing-ta
gclo
ud
Uniq
ue F
uncti
ons E
xecute
d
Real Sites V8 SunSpider
Code|Objects|Events
![Page 11: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/11.jpg)
Bytecodes / Call
15
0.00
50.00
100.00
150.00
200.00
250.00
300.00
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ablu
e
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp-dna
str
ing-ta
gclo
ud
Byte
codes /
Call
function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) if(elem.nodeType!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a }
Real Sites V8 SunSpider
Code|Objects|Events
![Page 12: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/12.jpg)
Fraction of Code Executed
16
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
sdelt
ablu
e
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp-dna
str
ing-ta
gclo
ud
Fra
cti
on o
f C
ode E
xecute
d
Most code
not executed
Real Sites V8 SunSpider
Code|Objects|Events
![Page 13: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/13.jpg)
Object Allocation Behavior
Code
Events
JavaScript
Objects
17
• Allocation by types
•Live heap composition
• Lifetime distribution
![Page 14: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/14.jpg)
Total Bytes Allocated
18
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ablu
e
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp-dna
str
ing-ta
gclo
ud
Tota
l heap
data
(kilobyte
s)
Real Sites V8 SunSpider
Code|Objects|Events
![Page 15: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/15.jpg)
Heap Data by Type
19
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
amaz
on
bin
gb
ingm
ap cnn
ebay
eco
no
mis
tfa
ceb
oo
kgm
ail
goo
gle
goo
glem
aph
otm
ail
rich
ard
sd
elta
blu
ecr
ypto
rayt
race
earl
eyre
gexp
spla
yV
8 a
ggra
gate
3d
-ray
trac
eac
cess
-nb
od
yb
ito
ps-
nsi
eve
con
tro
lflo
wcr
ypto
-aes
dat
e-xp
arb
mat
h-c
ord
icre
gexp
-dn
ast
rin
g-ta
gclo
ud
aggr
egat
e
He
ap D
ata
by
Typ
e
Objects
Enumarator
RegExp
Math
Error
Date
Native Func
String
Arrays
Script Func
Real Sites V8 SunSpider
Many functions
Rest are strings
Few benchmarks
allocate much data
Code|Objects|Events
![Page 16: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/16.jpg)
Live Heap Over Time (gmail)
20
Functions grow
steadily
Code|Objects|Events
GC reduces
size of heap
Objects grow
steadily too
![Page 17: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/17.jpg)
Live Heap over Time (ebay)
21
Heaps
repeatedly
created,
discarded
Heap contains
mostly functions
Code|Objects|Events
Heap drops to 0
on page load
![Page 18: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/18.jpg)
2 Search Websites, 2 Architectures
Bing Google
Code|Objects|Events
22
You stay on the same page
during your entire visit
Code loaded once
Heap is bigger
Every transition loads a
new page
Code loaded repeatedly
Heap is smaller
![Page 19: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/19.jpg)
Event Handlers in JavaScript
Code
Events
JavaScript
Objects
26
• Number of events • Sizes of handlers
![Page 20: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/20.jpg)
Event-driven Programming Model
• Single-threaded, non-preemptive event handlers
• Example handlers: onabort, onclick, etc.
• Very different from batch processing of benchmarks
• Handler responsiveness critical to user experience
27
Code|Objects|Events
![Page 21: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/21.jpg)
Total Events Handled
28
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000T
ota
l Events
Handle
d
Real Sites V8
Almost no events
Code|Objects|Events
![Page 22: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/22.jpg)
Median Bytecodes / Event Handled
30
0
50
100
150
200
250
300
350
400
450
500M
edia
n B
yte
codes /
Event
Hand
led
Code|Objects|Events
506 2137
![Page 23: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/23.jpg)
Sure, this is all good, but…
• Everyone knows benchmarks are unrepresentative
• How much difference does it make, anyway?
• Wouldn’t any benchmarks have similar issues?
33
![Page 24: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/24.jpg)
Cold-code Experiment
• Observation
– Real web apps have lots of code (much of it cold)
– Benchmarks do not
• Question: What happens if the benchmarks have more code?
– We added extra, unused to code to 7 SunSpider benchmarks
– We measured the impact on the benchmark performance
34
![Page 25: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/25.jpg)
Performance Impact of Cold Code
35
0
100
200
300
400
500
600
700
800
900
Tim
e (
mse
c)
0K
200K
400K
800K
1M
2M
0
100
200
300
400
500
600
700
800
900
Tim
e (
mse
c)
0K
200K
400K
800K
1M
2M
Chrome
3.0.195.38
IE 8
8.0.601.18865 Cold code has
non-uniform impact
on execution time
Cold code makes
SunSpider on Chrome
up to 4.5x slower
![Page 26: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/26.jpg)
Impact of Benchmarks
• What gets emphasis
– Making tight loops fast
– Optimizing small amounts of code
• Important issues ignored
– Garbage collection (especially of strings)
– Managing large amounts of code
– Optimizing event handling
– Considering JavaScript context between page loads
36
![Page 27: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/27.jpg)
Conclusions • JSMeter is an instrumentation framework
– Used to measure and compare JavaScript applications
– High-level views of behavior promote understanding
• Benchmarks differ significantly from real sites
– Misleads designers, skews implementations
• Next steps
– Develop and promote better benchmarks
– Design and evaluate better JavaScript runtimes
– Promote better performance tools for JavaScript developers
37
![Page 28: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/28.jpg)
Additional Resources
• Project: http://research.microsoft.com/en-us/projects/jsmeter/
• Video: Project JSMeter: JavaScript Performance Analysis in the Real World" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn
• Paper:
– “JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web Applications”, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps’10), June 2010.
• Additional measurements: – "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj
Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages), November 2009.
38
![Page 29: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/29.jpg)
Additional Slides
39
![Page 30: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/30.jpg)
Hot Function Distribution
40
Real Sites V8 Benchmarks
80% of time in < 10 functions 80% of time in 100+ functions
80% 80%
Code|Objects|Events
![Page 31: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/31.jpg)
Opcode Distribution
41 Green = SunSpider | Blue= Real Web Apps | Red = V8
Real Apps
Outliers
Code|Objects|Events
![Page 32: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/32.jpg)
Object Type Distribution
42
Real Apps
Code|Objects|Events
economist
is an outlier
(arrays)
Green = SunSpider | Blue= Real Web Apps | Red = V8
![Page 33: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/33.jpg)
Distribution of Time in Handlers
43
Code|Objects|Events
![Page 34: JSMeter: Characterizing the Behavior of JavaScript Web … · 2019-02-25 · David Simmons, Corneliu Barsan, and Allen Wirfs-Brock . Why Measure JavaScript? •Standardized, de facto](https://reader033.fdocuments.in/reader033/viewer/2022041723/5e501f985f7357430f64fd68/html5/thumbnails/34.jpg)
Related Work • JavaScript
– Richards, Lebresne, Burg, and Vitek (PLDI’10)
– Draw similar conclusions
• Java
– Doufour et al. (OOPSLA’03), Dieckmann and U. Hölzle (ECOOP’99)
• Other languages
– C++: Calder et al. (JPL’95)
– Interpreted languages: Romer et al. (ASPLOS’96)
44