The Developer Swiss Knife: Troubleshooting Tools for Java EE Applications

Assumptions

•This presentation aims experienced developers with a little (but good) knowledge about how the JVM works•The presentation focus only on the tools, not on the troubleshooting methods•Some of the tools are only for Linux. Windows users, get out! =P•All tools presented in this presentation are Open Source projects and some of them are developed by JBoss

The Application stack

Our primary focus

The Tools

•JDK CLI Tools (jstack, jmap, jps, etc.)•JDK GUI Tools (JVisualVM, JConsole)•Eclipse Memory Analyzer Tool (MAT)•Thermostat•Garbagecat•Samurai•AS7 JDR (JBoss Disaster Report) tool•Tattletale•Byteman*

JDK has its own troubleshooting tools

● jps● jmap● jinfo● jhat● jstack

Some of them are Graphical

● JConsole● JVisualVM

What should I do when my application has a memory problem?

Generating a Heap Dump

● -XX:-HeapDumpOnOutOfMemoryError● jmap● JVisualVM

Ok, I have a Heap Dump. Now can I analyze it?

Meet Eclipse MAT

What should I do if my application have a CPU problem? e.g. High CPU consumption

Generating a Thread Dump

● jstack● kill -3● AS7 CLI command:

– /core-service=platform-mbean/type=threading:dump-all-threads(locked-monitors=true,locked-synchronizers=true)

Now with a Thread Dump, what should I do with it?

Samurai – A Thread Dump viewer

High CPU consumption can be high GC activity

Enabling GC Logging

● Add to java command:

-XX:+PrintGC -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

GC Log content

3.122: [GC 132096K->24592K(504320K), 0.0321910 secs]

4.558: [GC 156688K->26752K(504320K), 0.0340520 secs]

6.609: [GC 158848K->50117K(504320K), 0.0628430 secs]

7.077: [GC 80334K->54921K(504320K), 0.0416730 secs]

7.119: [Full GC 54921K->46716K(504320K), 0.2522170 secs]

8.510: [GC 178812K->60924K(504320K), 0.0190690 secs]

9.683: [GC 193020K->69447K(461696K), 0.0308020 secs]

Garbagecat

java -jar garbagecat-1.0.0.jar --help

usage: garbagecat [OPTION]... [FILE]

-h,--help help

-o,--options <arg> JVM options used during JVM run

-p,--preprocess preprocessing flag

-s,--startdatetime <arg> JVM start datetime (yyyy-MM-dd HH:mm:ss,SSS)

for converting GC logging timestamps to datetime

-t,--threshold <arg> threshold (0-100) for throughput bottleneck

reporting

Garbagecat report========================================

Throughput less than 90% for PARALLEL_SCAVENGE

========================================

18020.492: [GC [PSYoungGen: 161888K->1376K(166016K)] 949359K->789159K(1214592K), 0.0294550 secs]

...

========================================

SUMMARY:

========================================

# GC Events: 46044

GC Event Types: PARALLEL_SCAVENGE, PARALLEL_SERIAL_OLD

Max Heap Space: 1566912K

Max Heap Occupancy: 1562164K

Max Perm Space: 77864K

Max Perm Occupancy: 46645K

Throughput: 99%

Max Pause: 4098 ms

Total Pause: 1703657 ms

First Timestamp: 4616 ms

Last Timestamp: 232465068 ms

========================================

0 UNIDENTIFIED LOG LINE(S):

Applications also might have classloading problems

Tattletale

JBoss AS7 has JDR

Wait a minute! I tried to run jvisualvm in OpenJDK and it says “command not found”

Thermostat – The next heat

Please Remember

● Don't develop applications without make use of testing approaches (unit, integration, functional, ...)

● Although all testing efforts, there is no 100% bug free applications

● For all unpredictable and complex problems, this presentation will help you troubleshoot, if not all, most of the problem

The links

● http://icedtea.classpath.org/wiki/Thermostat

● http://eclipse.org/mat/

● http://yusuke.homeip.net/samurai/en/index.html

● https://code.google.com/a/eclipselabs.org/p/garbagecat/

● http://docs.oracle.com/javase/7/docs/index.html

● http://www.jboss.org/tattletale

● http://jboss.org/byteman

Twitter: @rimolivehttp://aprendendo-cloud-computing.blogspot.com

Thank you!

What about Byteman?

Race Condition Tests with BytemanFlávia Rainone

Don't miss it!