JavaOne 2014: Java Debugging

© 2014 IBM Corporation

Chris Bailey – IBM Runtime Monitoring and Diagnostics

30th September 2014

Java DebuggingTools and Tricks for Troubleshooting

Document number


Important Disclaimers

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES.

ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.

IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.

IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS

2

© 2014 IBM Corporation3

Introduction to the speaker Chris Bailey

IBM Runtime Monitoring and Diagnostics Architect- 14 years working with Java and JVM technologies

Recent work focus:- Java monitoring, diagnostics and troubleshooting- Java integration into the cloud- JavaScript monitoring, diagnostics and troubleshooting

My contact information:- [email protected] http://www.linkedin.com/in/chrisbaileyibm- http://www.slideshare.net/cnbailey/- @Chris__Bailey

mailto:[email protected]

http://www.linkedin.com/in/chrisbaileyibm

http://www.slideshare.net/cnbailey/


OutOfMemory


Java Memory Usage Maximum memory available to a process is determined by architecture

– 32bit process is limited to 2^32 bytes = 4GB– 64bit process is limited to 2^64 bytes = 16 EiB

● Example for 32bit:

Amount and location of memory used for “OS and C Runtime” determined by OS

0 GB 4 GB

0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000

OS and C-Runtime Java Heap(s)

-Xmx

Native Heap


Layout by OS Windows 32bit:

Windows 32bit with /3GB:

0 GB 4 GB

0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000

OS and C-RuntimeJava Heap(s)

-Xmx

Native Heap

Libraries

0 GB 4 GB

0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000

OS and C-RuntimeJava Heap(s)

-Xmx

Libraries

Native Heap


Layout by OS Linux 32bit:

Linux with hugemem kernel:

0 GB 4 GB

0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000

KernelJava Heap(s)

-Xmx

Native Heap

0 GB 4 GB

0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000

KJava Heap(s)

-Xmx

Native Heap


Native Heap Usage

Java Heap Native Heap


OutOfMemoryErrors An OutOfMemoryError occurs if the Java heap or Native heap is exhausted

– Or the PermArea (for HotSpot before Java 8)

Tools are needed to track the memory usage of each of the pools:– Kernel/OS: OS (malloc) managed– Native Heap: OS (malloc) managed– Java Heap: Garbage Collection managed

Malloc managed memory needs to be tracked at the OS level*

Garbage Collection needs to be tracked at the Java runtime level

* The JVM can grab this information from the OS for you


OS Level Memory Monitoring

#!/bin/shPID=$1INTERVAL=3

# Echo time at start of monitoring.echo timestamp = `date +%s`

# Echo the interval frequency.echo "ps interval = $INTERVAL"

# Run the system command at intervals.while ([ -d /proc/$PID ]) do ps -p $PID -o pid,vsz,rss sleep $INTERVALdone

Linux: “ps” utility Windows: “perfmon” Virtual Bytes counters

Can be collected to CSV file



#!/bin/shPID=$1INTERVAL=3

# Echo time at start of monitoring.echo timestamp = `date +%s`

# Echo the interval frequency.echo "ps interval = $INTERVAL"

# Run the system command at intervals.while ([ -d /proc/$PID ]) do ps -p $PID -o pid,vsz,rss sleep $INTERVALdone

Linux: “ps” utility Windows: “perfmon” Virtual Bytes counters



Visualization of OS level memory monitoring● Garbage Collection and Memory Visualizer (GCMV):

- Available from

● Tool to analyze Java Heap and “Native” Heap memory

● Provides Graphical Display of Data- Allows zooming and cropping- Allows change of axes value and units- Allows comparison of multiple files

● Analysis and Recommendations- Statistical summary and recommendations- Flags errors and warnings

● Supports Linux, Windows, AIX, i5/OS and z/OS


Runtime Monitoring using JVM Tools IBM Health Center (IBM JVMs*)

– -Xhealthcenter

Machine and process memory usage

Native heap breakdown

* limited support exists for HotSpot

Oracle Mission Control (HotSpot JVMs)– -XX:+UnlockCommercialFeatures – -XX:+FlightRecorder

Machine and process memory usage


OS Level Memory Profiling

Windows: UMDH

Takes snapshot of the current native heap

Tracks allocating stack traces

Second snapshot is taken and stack traces for outstanding memory is produced

Detailed on Microsoft website:

http://support.microsoft.com/kb/268343

Also check out “debugdiag”

000000AD bytes in 0x1 allocations (@ 0x00000031 + 0x0000001F) by: BackTrace00468 ntdll!RtlpNtMakeTemporaryKey+000074D0 ntdll!RtlInitializeSListHead+00010D08 ntdll!wcsncat+00000224 leakyjniapp!Java_com_ibm_jtc_demos_LeakyJNIApp_nativeMethod

Linux:Debug Malloc / Malloc Interception

Approach 1:– Intercept calls to malloc and free– Create a stacks allocation table– IBM provides a Linux Memory Tracker

Approach 2: valgrind

valgrind --trace-children=yes --leak-check=full

LEAK SUMMARY: definitely lost: 63,957 bytes in 69 blocks. indirectly lost: 2,168 bytes in 12 blocks. possibly lost: 8,600 bytes in 11 blocks. still reachable: 5,156,340 bytes in 980 blocks. suppressed: 0 bytes in 0 blocks.Reachable blocks (those to which a pointer was found) are not shown.

http://support.microsoft.com/kb/268343


Garbage Collection Monitoring IBM JVMs:

– -verbose:gc – -Xverbosegclog:<name>,X,Y

Visualization available in GCMV for both IBM and Oracle verbose:gc

Oracle JVMs:– -XX:+PrintGCDateStamps– -XX:+PrintGCDetails– -XX:+PrintGCTimeStamps– -Xloggc:<name>


Runtime GC Monitoring IBM Health Center (IBM JVMs*)

– -Xhealthcenter

Used Heap and Pause Times

Allocation Profiling

* limited support exists for HotSpot



Allocation profiling


Java Heap Profiling: Heap Dump Analysis IBM JVMs: System Dump of PHD

– -Xdump:system|heap:events=systhrow,filter=java/lang/OutOfMemoryError

– -Xdump:system|heap:events=user

– -Xtrace:maximal=mt,trigger=method{<name>,sysdump|heapdump}

– com.ibm.jvm.Dump.JavaDump()/HeapDump()

– Health Center

Visualization available in IBM Memory Analyzer– Supports both IBM and HotSpot dumps– Provides leak and footprint analysis– Provides heap and application visualization

Oracle JVMs: HPROF dump– -XX:+HeapDumpOnOutOfMemoryError– jmap -dump:format=b– JConsole


Java Heap Profiling: Heap Dump Analysis


Debugging from Dumps


Debugging from Dumps● System dumps and HPROF dumps contain Object fields and values

● System dumps also have thread stacks and local variables● Provides visibility into code flow● Zero ongoing performance overhead● Dump time is ~2s/GB

● Makes it possible to diagnose a much wider set of issues● Exceptions: ability to look at data not exposed in the message● Functional Issues: possible to look at the data being acted on


Exceptions● java.net.ConnectException

● Stack trace in Memory Analyzer givesaccess to local variables- host Name = Bailey-W530- port = 100

java.net.ConnectException: Connection refused: connect at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:352) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:214) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:201) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:377)


Functional Issues● java.lang.OutOfMemoryError allocating a 1,129,665,376 byte object

● The readTokens method references a ByteArrayInputStream with a pos of 2369

● Content of ByteArrayStream around 2369:

● Which converts to 1,129,665,364

at com/ibm/wsspi/security/token/WSOpaqueTokenHelper.readTokens(WSOpaqueTokenHelper.java:924)at com/ibm/wsspi/security/token/WSOpaqueTokenHelper.createTokenHolderListFromOpaqueToken(WSOpaqueTokenHelper.java:844)at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl.fromBytesInDomain(MessagingEngineIdentityImpl.java:439)at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl.access$200(MessagingEngineIdentityImpl.java:91)at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl$2.run(MessagingEngineIdentityImpl.java:299)at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl$2.run(MessagingEngineIdentityImpl.java:293)at com/ibm/ws/sib/security/impl/BusUtilities.doInBusDomain(BusUtilities.java:115)


Performance


Performance Analysis Need to look at each of the layers of the application:

– Platform: Machine and Operating System resources– Runtime: Java heap resources– Application: Inefficient code, synchronization

Recommended to look at Platform and Runtime first– “Easy to resolve”

Biggest gains are in the application and back end responsiveness– OS and Java Runtime are already highly tuned and self adapting



procs -----------memory---------- -swap- --io-- --system-- ---cpu--- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 17196 679860 1196656 2594884 0 0 1 4 0 0 0 0 100 0 0 0 17196 679868 1196656 2594884 0 0 0 40 1012 43 0 0 100 0 0 0 17196 679992 1196656 2594884 0 0 0 3 1004 43 0 0 100 0

Linux: vmstat {interval} {samples} > vmstat.out

top -H -b -c > top_threads.out

Windows: “perfmon”:Process > Page Faults/secProcess > %Processor TimeNetwork > Output Queue LengthPhysical Disk > Current Disk Queue Length


top - 15:46:52 up 178 days, 4:53, 2 users, load average: 0.31, 0.08, 0.02 Tasks: 77 total, 2 running, 74 sleeping, 1 stopped, 0 zombie Cpu(s): 24.6% us, 0.5% sy, 0.0% ni, 74.9% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 5591016k total, 5416896k used, 174120k free, 1196656k buffers Swap: 2104472k total, 17196k used, 2087276k free, 2594884k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8502 user1 25 0 599m 466m 5212 R 99.9 8.5 0:23.92 java 7547 root 15 0 33472 2916 1144 S 0.3 0.1 778:17.98 X 1 root 16 0 656 228 192 S 0.0 0.0 0:07.43 init


Garbage Collection Monitoring IBM JVMs:

– -verbose:gc – -Xverbosegclog:<name>,X,Y

Visualization available in GCMV for both IBM and Oracle verbose:gc

Oracle JVMs:– -XX:+PrintGCDateStamps– -XX:+PrintGCDetails– -XX:+PrintGCTimeStamps– -Xloggc:<name>


Runtime GC Monitoring IBM Health Center (IBM JVMs*)

– -Xhealthcenter


Allocation Profiling



Allocation profiling


Application method CPU usage IBM Health Center (IBM JVMs*)

– -Xhealthcenter

CPU usage per method

Method call graphs


CPU usage per method

Method call graphs


Application lock usage IBM Health Center (IBM JVMs*)

– -Xhealthcenter

Contention frequency and time

Lock hold time


Contention frequency and time

Lock hold time


Questions?


IBM Developer Kits for Javaibm.biz/javasdk

WebShere Liberty Profilewasdev.net

IBM Bluemixibm.com/bluemix

IBM Developer Kits for Node.jsibm.biz/nodesdk

http://ibm.biz/javasdk

http://wasdev.net/

http://ibm.com/bluemix

http://ibm.biz/nodesdk


ibm.biz/javaone2014Visit Booth 5511 to learn about Cloud, DevOps and Mobile solutions

www.ibm.com/developer

Discovernew technical resources.

Developyour coding skills.

Connectwith developers.

http://ibm.biz/javaone2014


Copyright and Trademarks

© IBM Corporation 2013. All Rights Reserved.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., and registered in many jurisdictions worldwide.

Other product and service names might be trademarks of IBM or other companies.

A current list of IBM trademarks is available on the Web – see the IBM “Copyright and trademark information” page at URL: www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/legal/copytrade.shtml

JavaOne 2014: Java Debugging

Software

Transcript of JavaOne 2014: Java Debugging