Memory issues in production systems

Production system

Restricted access Application, DB, Application server, log files Debugging, monitoring

Investigation should not affect users Reproducing memory issue is not the best idea

Memory leaks

What is memory leak java.lang.OutOfMemory: heap size other symptoms - extensive garbage collection

Hard to discover Users don't recognize it

Hard to investigate Few information

Sometimes hard to fix Design issues

Investigation of memory issues

Log files application logs, access logs, gc logs

Monitoring behavior of running application JConsole

Heap dump Analysis JHAT, MAT, Netbeans

Log files

Motivation find issues find patterns leading to issues

Problems combining different logs (access log, application

log) bad visibility of root cause – not visible or hidden in

big amount of minor issues memory problems trigger only few types of

messages

Investigating log files

Application logs Log free memory once per minute

Access logs Analysis of traffic and business use cases

GC logs Show problems with GC (stop the world)

Monitoring via JMX

Tools JConsole

Motivation More information than in logs Monitoring tests

Problems Not always applicable on production system

(security reasons) Test might not cover all scenarios

Heap dump analysis

Tools JMAP, JHAT, MAT, Netbeans

Motivation Offline investigation of behavior during specific time

frame Comparison of different heap dumps

Problems Access to heap dumps (security)

Creating Heap dump

jmap -dump:live,format=b,file=heap.dump <pid> Part of JDK Used to create heap dump on demand

JVM parameters -XX:+HeapDumpOnOutOfMemoryError -

XX:HeapDumpPath=/directory/subdirectory Produces file java_pid<pid>.hprof Used for production system, creates heap dump when

OutOfMemory is thrown

Tools for heap dump analysis

JHAT Part of JDK 6 Running as web application

IDE Netbeans, MAT (Eclipse plugin)

Commercial tools

Heap dump analysis - JHAT

Heap Histogram List of classes with number of instances and used

memory Referrers vs Referees

Navigating in object graph Example: Form → String → [c

Heap dump analysis – Netbeans, MAT

Dominator tree Object x dominates object y if every path in object graph

from root to y goes through x Shallow vs Retained heap

Shallow heap is memory used by object Retained set of X is set of objects which would be removed

by GC if X is removed by GC Retained heap is sum of shallow heap in retained set

Other queries Leak suspects Top consumers

Heap dump Analysis - OQL

Object query Language Different dialect for each tool

JHAT Simple SQL combined with javascript callback functions

MAT More SQL like (Subquery, union, distinct) Predefined functions (dominator)

OQL Examples

JHAT select

heap.findClass('org.apache.catalina.util.ServerInfo').statics.serverNumber.toString()

MAT SELECT serverNumber.toString() FROM OBJECTS

org.apache.catalina.util.ServerInfo

Experiences

Bugs Incorrect usage of external libraries or

frameworks Design issues

Bugs

Map <String, String> redirectUrlCache = new LinkedHashMap<String, String>() { @Override protected boolean removeEldestEntry(Map.Entry eldest) { return size() > 10000; }}

String getRedirectUrl(String url) { if(redirectUrlCache.containsKey(url)) { return redirectUrlCache.get(url); } String result = computeRedirectUrl(url); redirectUrlCache.put(url, result); return result;}

Usage of frameworks

Hibernate Short time usage of big amount of memory Too big first level cache

Good understanding of framework is a must

Design Issues

Overused session scope Big pages

Big buffer in page context Page context referenced by tag handlers Heavy load causes big memory usage Fix might require changing use cases

Reusing functionality

Interesting links

http://blogs.sun.com/sundararajan/entry/querying_java_heap_with_oql

OQL help for JHAT

http://www.eclipse.org/mat/downloads.php

Standalone MAT

www.google.com