mjprof: Monadic approach for JVM profiling
-
Upload
haim-yadid -
Category
Software
-
view
427 -
download
0
description
Transcript of mjprof: Monadic approach for JVM profiling
@lifeyx#Devoxx #Performance #AdoptOpenJDK
mjprof : A Monadic Approach for JVM Profiling
Haim YadidPerformize-IT
@lifeyx#Devoxx #Performance #AdoptOpenJDK
About Me: Haim Yadid
• 21 Years of SW development experience
• independent Performance Expert
• Consulting R&D Groups
• Training: Java Performance Optimization
• Community leader : Java.IL
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Java.IL
• Israeli Java Community and User Group
• Meetup : www.meetup.com/JavaIL/
• Twitter: @java_il
• ~ 400 members
• Meeting every month
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Background &
Motivation
@lifeyx#Devoxx #Performance #AdoptOpenJDK
The Two Most Important Questions
@lifeyx#Devoxx #Performance #AdoptOpenJDK
The Two Most Important Questions
My wife asks me
@lifeyx#Devoxx #Performance #AdoptOpenJDK
The Two Most Important Questions
What are you doing? When will you be back?
My wife asks me
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Profiler
• A Performance analysis tool
• Which answers questions about method calls:
• E.g. : Number of invocation, Average duration,CPU consumptions Contention Memory allocation IO, Cache misses etc…..
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Statistical Profiler
• A.k.a. as Sampling Profiler
• Samples the treads activity every several milliseconds
• Not very accurate for short tasks
• Low/Medium overhead
• Do not provide the number of invocations
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Instrumenting Profiler
• A.k.a Tracing
• Instruments the code
• Knows when a method starts and when it ends
• High overhead (depends on instrumentation)
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Instrumenting Profiler
• A.k.a Tracing
• Instruments the code
• Knows when a method starts and when it ends
• High overhead (depends on instrumentation)
Public void foo(a) {Measure start timeincrease invocation count
bla bla;Measure end time
}
Injected code
@lifeyx#Devoxx #Performance #AdoptOpenJDK
jstack - A Poor Man’s Profiler
• The most trivial sampling profiler
• Part of the JDK
• Connects to a running JVM and samples all the threads
• Only once !
• Writes to standard output
@lifeyx#Devoxx #Performance #AdoptOpenJDK
And it looks like this
• For each thread we have the following block
@lifeyx#Devoxx #Performance #AdoptOpenJDK
And it looks like this
• For each thread we have the following block
"pool-35-thread-1" prio=5 tid=10315b800 nid=0x12c1fe000 waiting on condition [12c1fd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <107ac0140> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos ..... Locked ownable synchronizers: - None
Thread name Priority Id and native id
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Now do this a Million Times!
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Now do this a Million Times!
• Tons of threads
• Deep stacks
@lifeyx#Devoxx #Performance #AdoptOpenJDK
mjprof
@lifeyx#Devoxx #Performance #AdoptOpenJDK
mjprof
• A command line tool
• Acts on jstack output
• Extends it (enriched thread dumps)
• Applies a sequence of operations (monads) to
• Filter
• Aggregate
• Manipulate
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Prerequisites
• Java installed
• Set JAVA_HOME (JDK1.7 or later)
• Add ${JAVA_HOME}/bin to path
• Attaching to processes requires JDK and not JRE
• Can run on a headless machine
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Installing mjprof
• Git repo : https://github.com/AdoptOpenJDK/mjprof
• Download and unzip unzip mjprof1.0-bin.zip
• Or clone and build mvn install assembly:assembly
• Add mjprof directory to your path
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Synopsis and Help (Smoke test)
• Running mjprof without parameters will print help
• It is a good way to see it is functioning
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Synopsis and Help (Smoke test)
• Running mjprof without parameters will print help
• It is a good way to see it is functioning
➜ mjprof git:(master) ✗ mjprof Synopsis A list of the following monads concatenated with . … Data sources: jstack/pid,[count],[sleep]/ -Generate dumps using stack path/path/ -Read thread dump from file stdin -Read thread dumps from standard input visualvm/path/ -Read profiling session from xml export of VisualVM
….
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Monads
• Pipeline Building blocks
• Monads can
• Collect data
• Filter out certain threads
• Transform threads information
• Aggregate threads information
• Snapshot write state to a file or GUI
• Concatenate with a . (period)
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Data Sourcesstdin
file
jstack
jmx
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Data Sources
• cat mythreaddump.txt | mjprof…stdin
file
jstack
jmx
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Data Sources
• cat mythreaddump.txt | mjprof…
• mjprof path/mythreaddump.txt/
stdin
file
jstack
jmx
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Data Sources
• cat mythreaddump.txt | mjprof…
• mjprof path/mythreaddump.txt/
• mjprof jstack/19873/
stdin
file
jstack
jmx
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Data Sources
• cat mythreaddump.txt | mjprof…
• mjprof path/mythreaddump.txt/
• mjprof jstack/19873/
• mjprof jmx/myhost:65327/
stdin
file
jstack
jmx
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Monads
contains -contains group
merge
sort
-sort
frame
count
mergedcalles
-frame
top
bottom
trimbelow
namesuf
-fn
-pkg
-fn-prop
jvm -jvm
blocked
locks
-lockswaiting
running withstack
Filters Aggregations Stack frame level
stdout
gui
snapshot
output
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Extending mjprof
• mjprof is inherently extensible
• Macros
• A text file named macros.properties
• Contains a sequence of macros
• Plugins
• Annotate your extension with @Plugin
• Compile it and place your jar in the directory plugins under mjprof installation
@lifeyx#Devoxx #Performance #AdoptOpenJDK
10 mjprof Recipes
@lifeyx#Devoxx #Performance #AdoptOpenJDK
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How many threads my process has?
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How many threads my process has?
mjprof jstack/1971/path/stack.txt/
jstack/MainClass/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How many threads my process has?
➜ distro mjprof jstack/1971/.count 2014-11-01 10:34:10Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):
Total number of threads is 1045
mjprof jstack/1971/ . countpath/stack.txt/
jstack/MainClass/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How many jetty thread are working ATM?
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How many jetty thread are working ATM?
mjprof jstack/MyApp/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How many jetty thread are working ATM?
mjprof jstack/MyApp/ . contains/name,qtp/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How many jetty thread are working ATM?
➜ 04112014 mjprof jstack/My4tApp/.contains/name,qtp/ 2014-11-04 20:05:03Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):
"qtp817406040-17-selector-ServerConnectorManager@69b87e8d/3" prio=5 ….3 runnable [0x000000011bc5f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000780209080> (a sun.nio.ch.Util$2) - locked <0x0000000780209070> (a java.util.Collections$UnmodifiableSet)
mjprof jstack/MyApp/ . contains/name,qtp/contains/state,RUN/.
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How many jetty thread are working ATM?
➜ 04112014 mjprof jstack/My4tApp/.contains/name,qtp/ 2014-11-04 20:05:03Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):
"qtp817406040-17-selector-ServerConnectorManager@69b87e8d/3" prio=5 ….3 runnable [0x000000011bc5f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000780209080> (a sun.nio.ch.Util$2) - locked <0x0000000780209070> (a java.util.Collections$UnmodifiableSet)
mjprof jstack/MyApp/ . contains/name,qtp/count.contains/state,RUN/.
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Which threads are running my logic ?
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Which threads are running my logic ?
2014-11-04 20:11:55Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):
"Indexing daemon" prio=5 tid=0x00007fb306bf4800 nid=0x6307 waiting on condition [0x000000011c171000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.myapp.IndexManagement$1.run(IndexManagement.java:148)
"main" prio=5 tid=0x00007fb303809000 nid=0x1303 in Object.wait() [0x0000000105500000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at com.myapp.MyAppApp.main(MyAppApp.java:18)
mjprof jstack/MyApp/ . contains/stack,myapp/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Stack-trace is cluttered, how do I shorten it ?
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Stack-trace is cluttered, how do I shorten it ?
mjprof jmx/MyApp/ .
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Stack-trace is cluttered, how do I shorten it ?
mjprof jmx/MyApp/ . frame/mycompany/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Stack-trace is cluttered, how do I shorten it ?
mjprof jmx/MyApp/ . frame/mycompany/-frame/grizzly/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Stack-trace is cluttered, how do I shorten it ?
mjprof jmx/MyApp/ . frame/mycompany/-frame/grizzly/top/3/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How can I group threads by state ?
@lifeyx#Devoxx #Performance #AdoptOpenJDK
mjprof jmx/MyApp/ . group/state/ gui.
Question: How can I group threads by state ?
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How do I aggregate several thread dumps
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How do I aggregate several thread dumps
mjprof jmx/MyApp,10,1000/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How do I aggregate several thread dumps
mjprof jmx/MyApp,10,1000/ .group/state/
merge
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How do I aggregate several thread dumps
mjprof jmx/MyApp,10,1000/ .group/state/
merge .
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How can I combine thread dumps from several processes?
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: How can I combine thread dumps from several processes?
mjprof jmx/myhost1.com:5467/ .
group/state/ gui.jmx/myhost2.com:5467/ .
@lifeyx#Devoxx #Performance #AdoptOpenJDK
@lifeyx#Devoxx #Performance #AdoptOpenJDK
How do I measure thread pool CPU consumption?
@lifeyx#Devoxx #Performance #AdoptOpenJDK
How do I measure thread pool CPU consumption?
mjprof jmxc/MyApp/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
How do I measure thread pool CPU consumption?
mjprof jmxc/MyApp/ . contains/name,COFFEE/
@lifeyx#Devoxx #Performance #AdoptOpenJDK
"*" count=7 tid=* cpu_ns=1164536000 wall_ms=4804 %cpu=24.24 java.lang.Thread.State: TIMED_WAITING100.00% [7/7]\ at Thread.run(Thread.java:744)100.00% [7/7] \ at Sleeper.run(Sleeper.java:51)100.00% [7/7] \ at Sleeper.loopForever(Sleeper.java:37)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:30)100.00% [7/7] \ at Recurse.singleCycle(Recurse.java:16)100.00% [7/7] \ at Recurse.doRecursion(Recurse.java:25)100.00% [7/7] \ at Recurse.doRecursion(Recurse.java:25)
How do I measure thread pool CPU consumption?
mjprof jmxc/MyApp/ . contains/name,COFFEE/-pkg group..
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Which threads are blocked by the lock I hold
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: Which threads are blocked by the lock I hold
mjprof path/threaddump.txt/ .blockedcontains/stack,0x00000007aab51b38/
.
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: What is the overall locking state of my application
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Question: What is the overall locking state of my application
mjprof path/threaddump.txt/ .contains/stack,myapp/locks
.
"CPU3_5" prio=5 tid=0x00007fc4f4889800 nid=0x5703 waiting for monitor… java.lang.Thread.State: BLOCKED (on object monitor) - waiting to lock <0x00000007aad98988> (a java.lang.Object)
"CPU3_4" prio=5 tid=0x00007fc4f404a800 nid=0x5503 waiting for monitor… java.lang.Thread.State: BLOCKED (on object monitor) - waiting to lock <0x00000007aad98988> (a java.lang.Object)
"CPU3_3" prio=5 tid=0x00007fc4f4049800 nid=0x5303 waiting on … java.lang.Thread.State: TIMED_WAITING (sleeping) - locked <0x00000007aad98988> (a java.lang.Object)
@lifeyx#Devoxx #Performance #AdoptOpenJDK
@lifeyx#Devoxx #Performance #AdoptOpenJDK
Thanks / Contact Me
http://il.linkedin.com/in/haimyadid
www.performize-it.com
blog.performize-it.com
https://github.com/lifey
@lifeyx