CENTRALIZED LOGMANAGEMENT AND JAVA

APPLICATIONMONITORING

Author Evgeniy Khist

AGENDAWhat is wrong about accessing logs the usual way?How to access and search in logs from multiple hosts ofdistributed system?How to monitor Java applications?What is special about application performance monitoring?How collected metrics can be used to find bottlenecks andimprove performance?How to build gomogenous system for centralized logmanagement and Java application monitoring with Logstash,ElasticSearch and Kibana?

WHY DO WE NEED LOGS?To analyze incidents.

WHY DO WE NEED MONITORING?To track current system status to prevent incidents.

To analyze collected statistics to improve system even further.

WHAT DO YOU MEAN BY MONITORING?Infrastructure monitoring - keeping track on available resourcescapacityApplication monitoring - collecting statistics of applicationbehaviour to analyze it later

WHAT ARE THE WAYS TO ACCESS LOGS?SSHFTPNetwork-attached storage (NAS)

WHAT ARE THE WAYS TO SEARCH IN LOGS?grepSearch in text editor

WHAT IS BAD ABOUT DEALING WITH LOGSTHAT WAY?

It becomes hard to access log files on growing number of hostsof distributed systemsIt is not convenient to track some flow in logs from multiplehosts of distributed systems

WHAT IS BAD ABOUT DEALING WITH LOGSTHAT WAY?

It is hard to access logs on multiple hosts

WHAT IS BAD ABOUT DEALING WITH LOGSTHAT WAY?

NAS makes access to logs easier, but not the search

CENTRALIZED LOG MANAGEMENT

Logstash, Elasticsearch and Kibana makes both access to logs andsearch easy

CENTRALIZED LOG MANAGEMENT

We have two classes of server. First, one that ships logs. Second,one that collects and indexes logs.

SIMPLIFIED CENTRALIZED LOG MANAGEMENT

To make things easier, let's use only shipping Logstash agents

CENTRALIZED LOG MANAGEMENTSingle entry point for accessing logs with filtering and searching

support.

is a tool for managing events and logs. You can use itto collect logs, parse them, and store them for later use (like,for searching)

- distributed RESTful search and analytics is a web interface for searching and drilling into all of

your logs

Logstash

ElasticsearchKibana

LOGSTASH CONFIGURATIONinput { file { type => "server_log" path => [ "/opt/jboss/standalone/log/*.log" ] }}

filter { multiline { # merge all lines that does not start with date in any format pattern => "̂(19|20)\d\d[- /.](0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])" negate => true what => "previous" }

grok { # logback pattern "%date{ISO8601} %-5level [%thread] %logger{40} [%file:%line] %msg%n" match => [ "message", "(?m)%{DATA:logdate}%{SPACE}%{LOGLEVEL:level}%{SPACE}\[%{DATA:thread}\]%{SPACE}%{JAVACLASS:classname}%{SPACE}\[%{DATA:file}:%{POSINT:line}\]%{SPACE}%{GREEDYDATA:logmessage}" }

date { match => [ "logdate", "yyyy-MM-dd HH:mm:ss,SSS" ] }}

output { elasticsearch { host => "localhost" cluster => "logs" index => "logstash-%{+YYYY.MM.dd}"

ELASTICSEARCHSchema-free (document-oriented)Support of different data types: string, integer/long,

float/double, boolean, null

Full-Text searchSearch is done withing index

New index can be added periodically (per day, per week,etc)Speed of search doesn't depend on total amount of data

Search is done withing index, new index can be created per dayfor exampleOutdated indexes can be easily deleted (index deletion is fastoperation)Clusterization

KIBANABrowser based analytics and search interface to Logstash and

other timestamped data sets stored in ElasticSearch.

demo.kibana.org

Written entirely in HTML and JavascriptVisualizing data in powerful interfacesPowerful Search SyntaxDashboard customization

KIBANA

Easily create bar, line and scatter plots, or pie charts and maps

KIBANA

Easily create bar, line and scatter plots, or pie charts and maps

KIBANA

Easily create bar, line and scatter plots, or pie charts and maps

KIBANA

Easily create, save, share, and link your visualized data

KIBANA

Powerful Search Syntax

OPERATING SYSTEM AND JVM METRICSSystem loadFree physical memoryOpen file descriptor countHeap memory usageLoaded class countThread count

All these metrics available in JMX

APPLICATION CUSTOM METRICSTotal execution time of each methodNumber of executions of Spring MVC controller methodOther

COMMON TOOLS FOR APPLICATIONMONITORING

ZabbixSplunkNagiosAppDynamics

But it is another standalone tool...

WHAT ABOUT STORING METRICS INELASTICSEARCH?

Almost always metrics can be present as eventsElasticsearch is great tool to store any amount of eventsElasticsearch is great tool to searchKibana is great tool for visualizing data

HOW TO COLLECT METRICS FROM JMX ANDSEND TO ELASTICSEARCH?

jmxtrans-agentjmxtrans-agent-elasticsearch

jmxtrans is effectively the missing connector between speakingto a JVM via JMX on one end and any logging/monitoring

package on the other end.

JMXTRANS-AGENT DECLARATIONJAVA_OPTS="$JAVA_OPTS -javaagent:/path/to/jmxtrans-agent.jar=jmxtrans-agent.xml"

JMXTRANS-AGENT CONFIGURATION<jmxtrans-agent> <queries> <query objectName="java.lang:type=OperatingSystem" attribute="SystemLoadAverage" <query objectName="java.lang:type=OperatingSystem" attribute="FreePhysicalMemorySize" <query objectName="java.lang:type=OperatingSystem" attribute="FreeSwapSpaceSize" <query objectName="java.lang:type=OperatingSystem" attribute="OpenFileDescriptorCount"

<query objectName="java.lang:type=Memory" attribute="HeapMemoryUsage" key="used" <query objectName="java.lang:type=Memory" attribute="HeapMemoryUsage" key="committed" <query objectName="java.lang:type=Memory" attribute="NonHeapMemoryUsage" key="used" <query objectName="java.lang:type=Memory" attribute="NonHeapMemoryUsage" key="committed"

<query objectName="java.lang:type=ClassLoading" attribute="LoadedClassCount" resultAlias <query objectName="java.lang:type=Threading" attribute="ThreadCount" resultAlias= <query objectName="java.lang:type=Threading" attribute="DaemonThreadCount" resultAlias <query objectName="java.lang:type=Threading" attribute="TotalStartedThreadCount" <query objectName="java.lang:type=Threading" attribute="PeakThreadCount" resultAlias </queries> <outputWriter class="org.jmxtrans.agent.ElasticSearchOutputWriter"> <elasticsearchHost>localhost</elasticsearchHost> <elasticsearchPort>9300</elasticsearchPort> <elasticsearchClusterName>elasticsearch</elasticsearchClusterName> <elasticsearchIndex>jmxtrans-%{yyyy.MM.dd}</elasticsearchIndex> <nodeName>nodeName</nodeName><!--empty by default--> <usePrefixAsType>true</usePrefixAsType> </outputWriter> <collectIntervalInSeconds>20</collectIntervalInSeconds></jmxtrans-agent>

JAVAAGENTjava.lang.instrument package provides services that

allow Java programming language agents to instrumentprograms running on the JVM

The mechanism for instrumentation is modification of the byte-codes of methods

JAVAAGENTThe agent class must implement a public static premain method

similar in principle to the main application entry point

public class SimpleJavaAgent implements ClassFileTransformer {

public byte[] transform(ClassLoader loader, String className, Class classBeingRedefined, ProtectionDomain protectionDomain, System.out.println(className); return bytecode; } public static void premain(String args, Instrumentation instrumentation) throws ClassNotFoundException, InstantiationException, IllegalAccessException { instrumentation.addTransformer(this); }}

JAVAAGENTThe manifest of the agent JAR file must contain the attribute

Premain-Class

The value of this attribute is the name of the agent class

META-INF/MANIFEST.MF

Premain-Class: com.example.SimpleJavaAgent

JAVAAGENTAgent is started by adding javaagent option to the command-

linejava -javaagent:/path/to/simplejavaagent.jar -jar jarfile.jar

WHY JAVAAGENT?It can be added to any environment any time, even by DevOpsNo relation to application code at all - zero dependencies

WHAT ABOUT APPLICATION CUSTOMMETRICS?

Often application custom metrics can be present as events too.

So they can be stored in Elasticsearch too.

EVEN APPLICATION PERFORMANCE METRICS?For example, how much time was spent for execution of each

application method?

Yes, easily. Total execution time of each method can be collectedwith sampler and sent to Elasticsearch.

WHAT IS SAMPLER?Sampler gathers performance and memory data by periodicallypolling the monitored application for thread dumps or memory

histograms.

HOW TO COLLECT SAMPLING INFORMATION?sampling-agent

Yes, javaagent once again

SAMPLING-AGENT DECLARATION ANDCONFIGURATION

JAVA_OPTS="$JAVA_OPTS \-javaagent:sampling-agent.jar=samplingInterval=100,outputInterval=20000,outputWriter=org.-Dorg.samplingagent.includePackages=com.example. \-Dorg.samplingagent.excludePackages=java.,javax.,com.sun. \-Dorg.samplingagent.elasticsearchHost=localhost \-Dorg.samplingagent.elasticsearchPort=9300 \-Dorg.samplingagent.elasticsearchClusterName=elasticsearch \ -Dorg.samplingagent.elasticsearchIndex=sampling-%{yyyy.MM.dd} \-Dorg.samplingagent.nodeName=nodeName"

PUTTING ALL TOGETHER

SINGLE ENTRY POINT FOR EVERYTHINGLogsOperating system metricsJVM metricsApplication custom metrics

From all servers of distributed system

SEPARATE KIBANA DASHBOARDShttp://localhost/kibana/#/dashboard/file/logstashs.jsonhttp://localhost/kibana/#/dashboard/file/jmxtrans.jsonhttp://localhost/kibana/#/dashboard/file/sampling.json

OPERATING SYSTEM AND JVM METRICS

Free physical memory and heap memory usage

OPERATING SYSTEM AND JVM METRICS

Average system load and thread count

APPLICATION CUSTOM METRICS

Total execution time of 2 application methods

HOW TO SET UP LOGSTASH, ELASTICSEARCHAND KIBANA FROM SCRATCH?

http://developer-should-know.tumblr.com/post/76434222958/centralized-log-management-and-java-application

More slides