Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

GAINING APPLICATION LIFECYCLE INTELLIGENCE WITH SPLUNK

By Damien Dallimore , Dev Evangelist @ Splunk

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 2

Who am I ?

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 3

From Middle Earth

Make things

JVM background

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 4

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 5

apps.splunk.com

github.com/damiendallimore

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 6

Agenda

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 7

Overview of Splunk and build a simple app

How Splunk can help in the Application Development Lifecycle

Various ways to get data into Splunk and demos

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Data Data Everywhere

8

VOLUME

VARIETYVERACITY

VELOCITY

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

How can Splunk help ?

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Spelunking

10

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Platform for machine data

11

Splunk storage Other Big Data stores

DeveloperPlatform

Data collectionand indexing

Report and

analyze

Custom dashboards

Monitor and alert

Ad hoc search

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

DeveloperPlatform

Report and

analyze

Custom dashboards

Monitor and alert

Ad hoc search

Platform for machine data

12

Splunk storage Other Big Data stores

Data collection

and indexing

Any amount, any location, any source.Schema at read time, not write time

Data in any formatNo RDBMS

Very Extensible

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

What Does Machine Data Look Like?

13

Sources

Twitter

Care IVR

Middleware Error

Order Processing

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Machine Data Contains Critical Insights

14

Customer ID Order ID

Customer’s Tweet

Time Waiting On Hold

Twitter ID

Product ID

Company’s Twitter ID

Sources

Twitter

Care IVR

Middleware Error

Order Processing

Customer IDOrder ID

Customer ID

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Machine Data Contains Critical Insights

15

Order ID

Customer’s Tweet

Time Waiting On Hold

Product ID

Company’s Twitter ID

Sources

Twitter

Care IVR

Middleware Error

Order Processing

Order ID

Customer ID

Twitter ID

Customer ID

Customer ID

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 16

How are we best going wrangle this

data ?

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Release the Developers

17

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Very Extensible Platform for Developers

18

REST API

Build Splunk Apps Extend and Integrate Splunk

Simple XML

JavaScript

Django

Web Framework

JavaJavaScriptPython

RubyC#PHP

Data Models

Search Extensibility

Modular Inputs

SDKs

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 19

Lets build something simple

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 20

Simple Swarm App (ex Foursquare)

Get my actual checkin data in via REST

Search over this data

Visualize

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 21

Application Lifecycle Data

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

BuildUnit Testing

Code

Check-inIntegration

Testing Deploy

Staging

22

Application Development Challenges

Lack of visibility across the product development lifecycle

Pressure to increase velocity and agility with DevOps

Limited insights into behavior and performance from application logs

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Quickly trace and identify errors anywhere in the codebase with real-time search and monitoring

Instrument your app logs to gain application intelligence

Break down dev tool silos with real-time insights from machine data

GAIN END-TO-END VISIBILITY ACROSS THE DEV TOOL CHAIN

FIND AND FIX ISSUES FASTER

PUSH BETTER CODE USING ANALYTICS

Splunk for Application Lifecycle Intelligence

23

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Real-time dashboards show error rate in production and impact of

pushing new builds

Developers can search and visualize web logs, Java logs—

without production access

Alerts notify developers as soon as a problem arises

24

Find and Fix Issues Faster

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Gain end-to-end visibility to make informed decisions

Analytics insights without the need for additional analytics tools

Ask questions while exploring and collecting data

void submitPurchase(purchaseId) {

log.info("action=submitPurchaseStart, purchaseId=%d", purchaseId)//these calls throw an exception on error submitToCreditCard(...) generateInvoice(...) generateFullfillmentOrder(...) log.info("action=submitPurchaseCompleted, purchaseId=%d", purchaseId) }

25

Push Better Code Using Analytics

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

End-To-End Visibility Across The Dev Tool Chain

26

CI / Build Servers

Project and Issue Tracking

Code Repository

QA / Testing Tools

Deployment Servers

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 27

App Development Lifecycle Demo

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 28

Getting your data into Splunk

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 29

Log DataLog Files

Splunk Logging Appenders

CodingSplunk Java SDK

Splunk Spring Integration Adaptors

JMX

MessagingJMS

AMQP w/Rabbit

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 30

Log Data

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 31

Standard Log Files

Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina loadINFO: Initialization processed in 1153 msOct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternalINFO: Starting service Catalina

Application logs that are part of the product

Developer logs for any code that was deployed

Written to local disk or network storage

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 32

Structured and Unstructured Data

Ideally events are in a best practice semantic format

key=value format , JSON

You can perform index time and search time extractions in Splunk

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 33

Logging best practices

Clearly timestamp every event , human readable at beginning of line

Log in text , binary needs decoding

Categorize – Use INFO, WARN, ERROR, DEBUG, Event type etc...

Log unique identifiers

Log anything that can add value when aggregated, charted or further

2012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar"

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 34

SplunkJavaLogging

Sometimes you can’t write to file

Appenders for Java Util Logging , Log4J , Logback

Simply add a logging appender to your logging configuration file

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 35

LogBack Appender Example

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 36

Code

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 37

Better Exception Logging

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 38

Easier to work with in Splunk

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 39

Coding

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 40

Splunk SDK for Java

Use the SDK from any JVM Language , Java / Groovy / Scala etc….

Send log events via REST , UDP or TCP directly to Splunk from your code

Search over data in Splunk

SDK available from dev.splunk.com

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 41

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 42

Spring Integration Adaptors

Inbound AdapterUsed to execute Splunk searches and get data out

Outbound AdapterWrite data to Splunk via REST, TCP , UDPWrite to a named index, submit a REST request, write to a data input bound to a server TCP port

Get the code on Github

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 43

Inbound Adaptor

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 44

Outbound Adaptor

Demo

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 45

JMX

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 46

What is JMX

JMX = Java Management Extensions

Monitor JVM via MBean attributes , operations and notifications

JVM MBeans

Vendor MBeans

Custom Coded MBeans

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 47

Getting this data into Splunk

Runs on all supported Splunk platforms

Works with all main JVM variants

100% Free and Open Source

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 48

Simple to Configure

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 49

Many Connectivity Options

Demo

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 50

Messaging

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 51

JMS

Not a messaging protocol , but a programming interface to many different underlying message providers

WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc…

Demo

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 52

AMQP

Built with Rabbit Java Client Library

AMQP 0.9.1, 0.9, 0.8

Demo

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 53

But wait , there’s more…..

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 54

Poll data from any REST API

Pull data directly off the wire

Capture output from executing any commands

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 55

Closer look at capturing command output

Let’s see what the host Operating System can tell us : top

External programs that provide additional JVM insights : jstat

Index this data in Splunk and correlate

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 56

top

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 57

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 58

jstat

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 59

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 60

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 61

Splunk options galore

Splunk> Enterprise : Free to download and use. Index 500 MB/day.

Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.

Splunk> AMI : BYOL versions for Amazon AWS Cloud.

Splunk> Sandbox : Spin up a cloud instance in minutes.Load in data.

Hunk> : Splunk for data in Hadoop HDFS , MongoDB

10 GB Free

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/ 62

More Info

Splunk Docs , Downloads , Vids : http://www.splunk.com

Download Splunk Apps : http://apps.splunk.com

Ask : http://answers.splunk.com

Watch the App Dev / Devops Video : http://www.splunk.com/goto/appdev

Splunk Developer Platform : http://dev.splunk.com

Splunk on Github : https://github.com/splunk

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Thankyou.

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Questions ?@damiendallimore

ddallimore@splunk.com