Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12!...

31
Copyright © 2013 Splunk Inc. Damien Dallimore Developer Evangelist @ Splunk #splunkconf IntegraBng Splunk Data and FuncBonality Using the Splunk SDK for Java

Transcript of Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12!...

Page 1: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Copyright  ©  2013  Splunk  Inc.  

Damien  Dallimore  Developer  Evangelist  @  Splunk  #splunkconf  

IntegraBng  Splunk  Data  and  FuncBonality  Using  the  Splunk  SDK  for  Java  

Page 2: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Legal  NoBces  During  the  course  of  this  presentaBon,  we  may  make  forward-­‐looking  statements  regarding  future  events  or  the  expected  performance  of  the  company.  We  cauBon  you  that  such  statements  reflect  our  current  expectaBons  and  esBmates  based  on  factors  currently  known  to  us  and  that  actual  events  or  results  could  differ  materially.  For  important  factors  that  may  cause  actual  results  to  differ  from  those  contained  in  our  forward-­‐looking  statements,  please  review  our  filings  with  the  SEC.    The  forward-­‐looking  statements  made  in  this  presentaBon  are  being  made  as  of  the  Bme  and  date  of  its  live  presentaBon.    If  reviewed  aTer  its  live  presentaBon,  this  presentaBon  may  not  contain  current  or  accurate  informaBon.      We  do  not  assume  any  obligaBon  to  update  any  forward-­‐looking  statements  we  may  make.    In  addiBon,  any  informaBon  about  our  roadmap  outlines  our  general  product  direcBon  and  is  subject  to  change  at  any  Bme  without  noBce.    It  is  for  informaBonal  purposes  only  and  shall  not,  be  incorporated  into  any  contract  or  other  commitment.    Splunk  undertakes  no  obligaBon  either  to  develop  the  features  or  funcBonality  described  or  to  include  any  such  feature  or  funcBonality  in  a  future  release.  

 

Splunk,  Splunk>,  Splunk  Storm,  Listen  to  Your  Data,  SPL  and  The  Engine  for  Machine  Data  are  trademarks  and  registered  trademarks  of  Splunk  Inc.  in  the  United  States  and  other  countries.  All  other  brand  names,  product  names,  or  trademarks  belong  to  their  respecCve  

owners.    

©2013  Splunk  Inc.  All  rights  reserved.  

2  

Page 3: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

About  Me  

Page 4: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Developer  Evangelist  at  Splunk  Make   Talk  

4  

Page 5: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Came  from  the  Splunk  Community  

5  

Page 6: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Coder  

6  

Page 7: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

From  Aotearoa  (New  Zealand)  

7  

Page 8: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Agenda  

Page 9: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Agenda  

!   Developer  Pla`orm  Overview  !   The  Basics  !   Digging  a  bit  deeper  !   IntegraBon  Frameworks  !   Tools,  Libraries  and  other  use  cases  !   QuesBons  (if  Bme  allowing  ,  else  find  me  aTerwards)  

9  

Page 10: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Developer  Pla`orm  Overview  

Page 11: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Powerful  Pla`orm  for  Enterprise  Developers  

11  

REST  API  

Build  Splunk  Apps   Extend  and  Integrate  Splunk  

Simple  XML  

JavaScript  

Django  

Web    Framework  

 

Java  JavaScript  Python    

Ruby  C#  PHP  

Data  Models  

Search  Extensibility  

Modular  Inputs  

SDKs  

Page 12: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

The  Splunk  REST  API  

12  

!   Exposes  an  API  method  for  every  feature  in  the  product  –  Run  searches  –  Input  data  –  Manage  Splunk  configuraBons    

!   API  is  RESTful  –  Endpoints  are  served  by  Splunkd  –  Requests  are  GET,  POST,  and  DELETE  HTTP  methods  –  Responses  are  Atom  XML  Feeds    or  JSON    –  Search  results  can  be  output  in  CSV/JSON/XML/Raw  –  AuthenBcaBon  is  header  token  based  

Page 13: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

SDKs  (SoTware  Development  Kits)  

13  

!   Our  SDKs  make  it  easier  to  use  the  REST  API  –  No  need  to  wrangle  with  lower  level  HTTP  REST  semanBcs    

 !   Several  different  language  offerings  now  available  

–  Javascript,  Java,  Python,  PHP,  C#,  Ruby  

!   Now  includes  support  for  more  rapid  development  of  Modular  Inputs    !   Typical  Use  Cases  

–  Accelerate  your  Dev/Test  cycles  –  Integrate  with  exisBng  soluBons  –  Build  new  big  data  applicaBons  

Page 14: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

The  Basics  

Page 15: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Get the Splunk SDK for Java

15  

!   Current  release  v1.2  !   Open  sourced  under  the  Apache  v2.0  license  !   Clone  from  Github  :  git  clone  hnps://github.com/splunk/splunk-­‐sdk-­‐java.git  !   Project  level  support  for  Eclipse  and  Intellij  IDE’s  ,  or  use  the  development  

environment  of  your  choice  !   Pre-­‐requisites  

–  JRE  6+  –  Splunk  installed  –  Ant    

!   Maven/Gradle/Ivy  repository  –  hnp://splunk.arBfactoryonline.com/splunk/ext-­‐releases-­‐local  

Page 16: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

SDK  Class  Design  

16  

Service  

HTTPService   Resource  

ResourceCollecBon   EnBty  

EnBtyCollecBon  

Job  JobCollecBon  

BaseService  

Args  

JobResultsArgs  

Page 17: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Code  &  Demo  

!   ConnecBng  and  authenBcaBng  !   Hirng  a  simple  REST  endpoint  !   Simple  searches  

–  Oneshot  Blocking  Search  –  Simple  Blocking  Search  

!   Indexing  an  event  

17  

Page 18: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Digging  a  Bit  Deeper  

Page 19: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Code  &  Demo  !   Searching  (also  demonstrates  handling  &  displaying  results  )  

–  Asynchronous  with  Job  object    –  RealBme  Search  –  ExporBng  –  Paging  through  large  result  sets  –  Saved  searches  and  dynamic  dispatch  arguments  

!   Indexing  data  –  TCP  –  UDP  –  Index  oneshot  upload  

!   Managing  Splunk    (also  demonstrates  state  management)  –  create  a  Role  enBty  –  create  a  User  enBty  

!   Namespaces          

19  

Page 20: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

IntegraBon  Frameworks  

Page 21: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Let’s  Make  Life  Even  Easier  

! Splunk  SDK  for  Java  makes  using  the  REST  API  easier    !   What  can  we  do  to  make  the  SDK  easier  to  use  ?    !   Integrate  into  standard  enterprise  Java  soTware  frameworks  

–  Spring  IntegraBon  (code  &  demo)  –  Apache  Camel    (code  &  demo)  –  Mule  ESB  

 21  

Page 22: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Tools,  Libraries    &  Other  Use  Cases    

Page 23: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Splunk  Java  Logging  

23  

!   A  logging  framework  to  allow  developers  to  as  seamlessly  as  possible  integrate  Splunk  best  pracBce  logging  semanBcs  into    their  code  

!   Transport  log  events  to  Splunk  directly  from  your  code  !   Custom  handler/appender  implementaBons(REST  and  Raw  TCP)  for  common  Java  logging  frameworks    ̶  LogBack  ̶  Log4j        ̶  java.uBl.logging  

!   UBlity  classes  for  formarng  log  events  !   Configurable  in  memory  buffer  to  handle  network  outages  

Page 24: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Eclipse  Plugin  

24  

!   Adds  extensions  to  the  Eclipse  soTware  development  pla`orm  to  simplify  creaBng  projects  using  the  Splunk  SDK  for  Java  

!   A  new  "Splunk  SDK  for  Java"  project  type  !   A  set  of  code  templates  for  common  tasks  using  the  Splunk  SDK  for  Java,  plus  opBonal  data  parsing  and  logging  capabiliBes    

!   A  launcher  for  Java  applicaBons  that  logs  JVM  acBvity  to  a    Splunk  instance  –  JMX  java.lang  MBeans  –  Byte  Code  InjecBon  method  level  tracing  

Page 25: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Other  JVM  Languages  

Scala Groovy Clojure

Javascript(Rhino) JRuby PHP(Quercus)

Ceylon Kotlin Jython

We don’t need SDK’s for these languages , we can just use the Java SDK !

25  

Page 26: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

HUNK  (Splunk  AnalyBcs  for  Hadoop)  

!   A  new  product  offering  from  Splunk  !   Allows  you  to  use  the  power  and  simplicity  of  Splunk  to  search  over  data  locked  away  in  HDFS  

!   Sits  on  top  of  HDFS  as  if  it  was  a  naBve  Splunk  Index  !   Virtual  Indexes  !   So  you  can  use  the  Splunk  SDK  for  Java  to  develop  your  Hadoop  applicaBons  in  exactly  the  same  way  that  you  currently  use  it.  

26  

Page 27: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Housekeeping  

Page 28: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Where  to  Go  for  More  Info  Twiner  @splunkdev    Blog  hnp://blogs.splunk.com/dev    Demos  hnp://demos.splunk.com    

Email  [email protected]  Portal  hnp://dev.splunk.com    Github  hnps://github.com/splunk      

28  

Page 29: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Links  Github  Gists  for  code  examples  in  this  presentaBon  :  hnps://gist.github.com/damiendallimore    SDK  docs  at  dev.splunk.com  :  hnp://dev.splunk.com/view/SP-­‐CAAAECN    Splunk  SDK  for  Java  Github  repository  :  hnps://github.com/splunk/splunk-­‐sdk-­‐java    Splunk  Spring  IntegraBon  repository  on  Github  :  hnps://github.com/SpringSource/spring-­‐integraBon-­‐extensions/tree/master/spring-­‐integraBon-­‐splunk  Splunk  Spring  IntegraBon  demo  on  Github  :  hnps://github.com/damiendallimore/spring-­‐integraBon-­‐splunk-­‐webex-­‐demo    

Splunk  Apache  Camel  repository  on  Github  :  hnps://github.com/pax95/camel-­‐splunk    Splunk  Apache  Camel  demo  on  Github  :  hnps://github.com/pax95/camel-­‐splunk-­‐example    Splunk  Eclipse  plugin  :  hnp://dev.splunk.com/view/splunk-­‐plugin-­‐eclipse/SP-­‐CAAAEQP    Splunk  Java  Logging  on  Github  :  hnps://github.com/splunk/splunk-­‐library-­‐javalogging    

Splunk  Java  Agent  on  Github  :  hnps://github.com/damiendallimore/SplunkJavaAgent    Splunk  Android  SDK  on  Github  :  hnps://github.com/damiendallimore/splunk-­‐sdk-­‐android    Splunk  REST  API  reference  :  hnp://docs.splunk.com/DocumentaBon/Splunk/latest/RESTAPI/RESTcontents    Free  Splunk  download  :  hnp://www.splunk.com/get?r=header    

Best  pracBce  logging  overview  :  hnp://dev.splunk.com/view/logging-­‐best-­‐pracBces/SP-­‐CAAADP6    Splunk  SDK  for  Java  videos  :  hnp://dev.splunk.com/view/get-­‐started/SP-­‐CAAAECH    HUNK  Beta  video  :  hnp://www.splunk.com/view/SP-­‐CAAAH2F    Maven/Gradle/Ivy  Repository  :  hnp://splunk.arBfactoryonline.com/splunk/ext-­‐releases-­‐local    

29  

Page 30: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Contact  me  

Email  :  [email protected]  Twi6er  :  @damiendallimore  Skype  :  damien.dallimore  Github  :  damiendallimore  Splunkbase  :  damiend  Slideshare  :  hnp://www.slideshare.net/damiendallimore            

I  will  be  around  all  week  ,  feel  free  to  grab  me  for  a  chat  or  a  beer  or  both  J  

30  

Page 31: Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Thank  you  !