1. Java Application Problems Diagnostics Vitalii TymchyshynTwitter: @tivv00Email: tivv00@gmail.com

2. ContentsProblem typesProblem causesPrepare in advanceDiagnostics checklistStack tracesJava memory 3. Common Problem typesResource underutilizationSlowness in long runSudden haltsDisappearing applicationsProblems under load 4. Common Root CausesDeadlocks / Wrong lock orderingNarrow points / Wide locksLeaksHidden exceptionsUnhandled exceptionsInvalid OS/JVM configuration 5. Prepare: Logging hintsLog stdout/stderr it has JVM outputLog application start/stop with error codeEnable GC loggingLog exception if you dont rethrow or process itLog FATAL in any place you call System.exitLog all thread stacks in multithreading guard code 6. Prepare: Setting up VM-verbose:gc output GC to stderr-XX:+HeapDumpOnOutOfMemoryError-XX:+PrintClassHistogram print classhistogram along with thread stacks on Ctrl- /kill -3-XX:MaxNewSize tame new generation forlarge heaps 7. Diagnostics checklistCheck your host: CPU/IO/Network usage, serious swappingCheck your VM: Heap/GC times, finalizer thread blockDeadlocks/Lock contentionExternal servicesLeaks / Peaks / Working set increase 8. Unix OS metricstop, vmstat, iostatLoad average < Cores * 10System time < 30%Context switches < Cores * 10000Wait time means IOSwap Used < RAM/4Swap IN/OUT < 1mb/1 9. Listing running java VMsjps -vvmFor current user (as most of JVM tools)Shows PIDShows full command line including JVM options, system variables, full class name and program optionsEspecially useful if program was started with a script 10. Getting stack tracesCtrl-Break (windows) / Ctrl- (Unix)Kill -3 jstackjconsoleIDEdebugger 11. What stack can tell youObvious deadlocksNarrow pointsSimplistic CPU profilingExternal resource overutilization/deadlock 12. JVM memory structureJava HeapThread StacksNative memory (UI can eat quite a lot)Permanent generation (classes / interned strings)Byte Buffers (often used as off-heap storage)Mapped files 13. Java HeapHas fixed upper limitStarts to take a lot of time when almost fullCan be tuned a lot (and often should be)GC activity logging should be enabledYou can enable printing overview on kill -3You can enable heap dump on OOM 14. Utilities to get heap informationjstack: some totalsjstat: detailed totalsjmap: get heap overview / dumpjconsole: force GC, collect GUI stats, makedump...Eclipse MAT: my preferred tool to analyze heap dumps