Splunking the JVM

Splunking the JVM

Damien DallimoreDeveloper Evangelist

Copyright © 2013, Splunk Inc.

What is this JVM thing ?

2

• Circa 1991, Dr. James Gosling at Sun started developing a technology for next generation smart devices/appliances

• “Green” became “Oak” which became “Java”• Java 1.0 first appeared in January 1996.• The JVM is a virtual machine that runs programs that are compiled into

Java bytecode• Available for many hardware and software platforms• 17 years later , the JVM has evolved from a consumer device

technology, to a browser oriented technology with the explosion of the web , to now becoming deeply rooted in the enterprise software landscape on the server side and in the cloud


17 years later

3

• Oracle took ownership of Java from Sun in January 2010

• The Java Community Process(JCP) is the forum where members develop specifications for Java technology

• Java Specification Requests(JSR) get submitted for new features, are reviewed and then voted on by the JCP Executive committee.

• Editions• Embedded Java, Java ME , Java SE , Java EE

• Current Version is Java 7 (Dolphin)• Java 8 scheduled for 2013

Application Servers Enterprise Service Buses Databases

NoSQL Distributed Big Data Web Servers

Directory Servers Search Engines Build Systems

Gaming Platforms Trading Systems Reservation Systems

Core Banking Messaging Infrastructure Proprietary Systems


JVM Variants

4

• Oracle Hotspot (formerly SUN)– the primary reference JVM implementation

• Oracle JRockit (formerly BEA)– free since May 2011– code base currently being merged with Hotspot, ETA ~JDK 8

• Open JDK– SUN open sourced Hotspot and the Java class library in 2006– Slight differences with Oracle Java still– OpenJDK is the official Java SE7 Reference Implementation

• J9– IBM’s JVM for AIX, Linux, MVS, OS/400, Pocket PC, z/OS

• Azul Systems Zing– based on HotSpot – supports memory heaps up to 512 GB without GC pauses and is able to grow and shrink the heap based on load


The JVM has a healthy future

5

• Hotspot / JRockit code merge creating a best of breed JVM, Oracle to contribute this to OpenJDK

• OpenJDK is thriving, Oracle are contributing and being good stewards of Java (despite initial skepticism)

• Proliferation of alternative JVM languages that can all co-habitate in the JVM and new features in Java 8 to further enhance this multi language platform– Scala– Groovy– Clojure

• The JVM is evolving organically with the shifting tides of Enterprise software, it isn’t about the “J” anymore.

• From the clustered Application Server domination of the 00’s we now see an explosion of Big Data products running in massively distributed environments on commodity hardware or in the cloud– Apache Hadoop family (MapReduce, Hive, Hbase, Cassandra, HDFS)


What is running in JVMs ?

6


JVM “Fanboi”

7

Dr. GoslingFanboiSpeaking of Java as a language as opposed to the JVM platform, James Gosling, the Father of Java, said "Most people talk about Java the language, and this may sound odd coming from me, but I could hardly care less." He went on to explain, "What I really care about is the Java Virtual Machine as a concept, because that is the thing that ties it all together."


JVM Machine Data

8

• The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data• Large scale Application/Web Server clusters• Hadoop & Cassandra Node topologies in the 10’000s !!!

Custom Developed Code

WAR file

Application CodeTomcat

JVMHotspot

Operating SystemLinux

JMX, SNMP, HPROF,GC Logs, Custom Agents(AppDynamics/SplunkJavaAgent)

JMX, Application Logs

JMX, Developer Logs, Splunk Java SDK, SplunkJavaLogging

JVM process OS resource metrics

CORR

ELAT

E


Application & Developer Logs

9

• Application logs• default logs that are part of the product

• Developer logs• any custom code created and deployed

to the application that has it’s own logging

• Written to local disk or a mounted network volume

• Monitor with a Splunk UF


Splunk Java SDK / SplunkJavaLogging

10

• Alternative to writing to log file or needing to deploy a Splunk Universal Forwarder

• Use the Splunk Java SDK to input events directly to Splunk via HTTP Rest.

• Use SplunkJavaLogging to input events directly to Splunk using custom logging appenders.


JVM Process OS Metrics

11

• By JVM Process ID : Process State, Memory, CPU, Disk Usage, Disk I/O, Network I/O, File Descriptor Usage.

• Some OS metrics also exposed via JMX• Splunk for Unix and Linux• Splunk for Windows• Correlate this OS data across your JVM and

Application events ie: your JVM may have hung because of CPU starvation caused by some other process thrashing


Garbage Collection logs

12

• Extended Hotspot JVM options

-verbose:gc-Xloggc:/home/damien/jvm_logs/gc.log -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails

• The log is written to at Garbage Collection time• Useful for tracing full GC cycles• Need to perform field extractions in Splunk• Many GC metrics also available via JMX

54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]


Custom Instrumentation Agents (Advanced)

13

• JVM BCI (byte code instrumentation)• Write custom agents that get injected into

the running JVM• Dynamically inspect the state of

applications running in the JVM• Profiling, debugging, monitoring,

thread/memory analysis• As you write the agent code , the data

output can be file based or over the network

• Check out my SplunkJavaAgent on github• Also AppDynamics have some pretty cool

kung fu in this area, we integrate !


HPROF Profiling Dumps

14

• Binary JVM dumps that allow for deeper JVM resource inspection

• Typical use case is diagnosing memory issues after JVM crashes with java.lang.OutOfMemoryError

• Binary file is usually batch loaded into a third party memory analysis tool like Eclipse MAT

• SplunkJavaAgent can dynamically dump and decode hprof output and send to Splunk

• Awesome source of information for dev/test

Warning : heap dumping is an expensive operation as a full GC gets performed


SNMP

15

• The JVM SNMP Agent provides a single MIB that exposes the JVM’s Management and Monitoring API

http://docs.oracle.com/javase/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib

• Setup the JVM (just the basic settings shown)

Open a UDP Port : -Dcom.sun.management.snmp.port=9004 Configure the ACL : $JAVA_HOME/jre/lib/management/snmp.acl

• Traps can be caught locally to file and monitored• Splunk SNMP Modular Input can poll the JVM SNMP Objects

(coming soon to a theatre near you)

http://docs.oracle.com/javase/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib


JMX (Java Management Extensions)

16

• Manage and Monitor the JVM and Application via exposed MBeans• JVM MBeans (java.lang domain)

• Vendor MBeans (most vendors ship their products with extensive MBean coverage)

• Custom Coded MBeans (whatever your devs wish to code)

• MBeans expose attributes, operations and notifications to give you a powerfully dynamic insight into the runtime state of the JVM and your application.

• Add Splunk to the mix for historical and realtime operational visibility, pro-active issue detection etc..

• Splunk for JMX app on SplunkBase


JMX vs SNMP

17

JMX• Open and easily extensible• Developers can simply create new MBeans• Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBs

SNMP• The built-in SNMP agent of the JVM is not extensible. • You will not be able to use it in order to expose your own custom MIB• If you do want to expose your own MIB, you’d have to create a custom agent


Putting it all together, JVM Splunking Nirvana

18

Copyright © 2013, Splunk Inc.19

Contact me

Email : [email protected] : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : http://www.slideshare.net/damiendallimore Blogs : http://blogs.splunk.com/devWeb : http://dev.splunk.com

http://blogs.splunk.com/dev

http://dev.splunk.com/

http://dev.splunk.com/

Splunking the JVM

Technology

Transcript of Splunking the JVM