SplunkCloud Under*The*Hood* · Disclaimer* 2...

72
Copyright © 2016 Splunk Inc. Splunk Cloud Under The Hood

Transcript of SplunkCloud Under*The*Hood* · Disclaimer* 2...

Page 1: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

Copyright  ©  2016  Splunk  Inc.  

Splunk  Cloud  Under  The  Hood  

Page 2: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

Disclaimer  

2  

During  the  course  of  this  presentaBon,  we  may  make  forward  looking  statements  regarding  future  events  or  the  expected  performance  of  the  company.  We  cauBon  you  that  such  statements  reflect  our  current  expectaBons  and  esBmates  based  on  factors  currently  known  to  us  and  that  actual  events  or  results  could  differ  materially.  For  important  factors  that  may  cause  actual  results  to  differ  from  those  contained  in  our  forward-­‐looking  statements,  please  review  our  filings  with  the  SEC.  The  forward-­‐looking  statements  made  in  the  this  presentaBon  are  being  made  as  of  the  Bme  and  date  of  its  live  presentaBon.  If  reviewed  aOer  its   live   presentaBon,   this   presentaBon   may   not   contain   current   or   accurate   informaBon.   We   do   not  assume  any  obligaBon  to  update  any  forward  looking  statements  we  may  make.      In  addiBon,  any  informaBon  about  our  roadmap  outlines  our  general  product  direcBon  and  is  subject  to  change  at  any  Bme  without  noBce.   It   is   for   informaBonal  purposes  only  and  shall  not,  be   incorporated  into  any  contract  or  other  commitment.  Splunk  undertakes  no  obligaBon  either  to  develop  the  features  or  funcBonality  described  or  to  include  any  such  feature  or  funcBonality  in  a  future  release.  

Page 3: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

3  

Rajiv  BaTula  SoOware  Engineer  

Nikhil  Mungel  Sr.  SoOware  Engineer  

Site  Reliability  Engineering  San  Francisco  

3�  

Page 4: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

4  

What is?

4�  

Page 5: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

5  

Agenda  

  Overview    Design  Principles    Data  IngesBon    Reliability  &  Availability    Security  from  the  Ground  Up    Hybrid    OperaBonal  Excellence  

5�  

Page 6: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

Design  Principles  

Page 7: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

Design  Principles  

7  7�  

Secure   Instant   Reliable   Hybrid  

7�  

Page 8: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

8  

Search  Heads  

Indexers  

Clustering  &  License  Managers  

8�  

Page 9: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

9  

Single  Tenant  

Search  Heads  

Indexers  

Clustering  &  License  Managers  

EC2  Instance  

9�  

Page 10: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

10  10�  

Page 11: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

11  

Secure  By  Default  

Network  IsolaBon  

System  IsolaBon  

EncrypBon  

Non-­‐repudiable  Logging  

Secure  coding  pracBces  

11�  

Page 12: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

12  12�  

Page 13: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

13  13�  

Page 14: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

14  

Reliable  

Replicated  Data  

Replicated  Configs  

Disaster  Recovery  

Redundant  infrastructure  

14�  

Page 15: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

15  

Licenser  Service  

Cert  GeneraBon  

Page 16: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

16  

Instant  

16�  

Page 17: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

17  

Scalable  

17�  

Page 18: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

18  18�  

Page 19: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

19  

Splunked  

19�  

Page 20: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

Data  IngesBon  

Page 21: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

21  

Two  Methods  Of  IngesBng  Data  

S2S   HTTPS  

21�  

Page 22: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

22  

S2S   HTTPS  

Splunk-­‐to-­‐Splunk  with  SSL    

Universal  &  Heavy  Forwarders    

High  performance  

Forward  JSON  events  over  HTTPS    No  Splunk  forwarders  necessary    SDKs  allow  for  applicaBon  integraBon    High  performance  &  scalable  

Page 23: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

23  

S2S  IngesBon  

Indexers  Forwarders  

23�  

Page 24: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

24  

S2S  IngesBon  

Indexers  Forwarders  Round  Robin  DNS  

+  ElasBc  IP  Addresses  

[default outputs.conf] autoLBFrequency=30 dnsResolutionInterval=300

24�  

Page 25: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

25  

HTTP  Event  Collector  

HTTPS  POST  

HWF  

HWF  

HWF  

ELB  

Page 26: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

26  

ELB  

Large  number  of  HTTP  clients  Small  bursts  of  data,  millions  of  3mes  

Page 27: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

27  

ELB  

Small  number  of  HTTP  clients  ConBnuous  streams  of  data  

Page 28: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

Reliable  &  Available  

Page 29: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

29  

High  Availability  

29�  

Page 30: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

30  

High  Availability  

Search  Head  3  Search  Head  2  Search  Head  1  

30�  

Page 31: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

31  

High  Availability  

Search  Head  3  Search  Head  2  Search  Head  1  

31�  

Page 32: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

32  

High  Availability  

Search  Head  3  Search  Head  2  Search  Head  1  

Load  Balancer  

32�  

Page 33: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

33  

High  Availability  

Search  Head  3  Search  Head  2  Search  Head  1  

Load  Balancer  

33�  

Page 34: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

34  

High  Availability  

Search  Head  3  Search  Head  2  Search  Head  1  

Load  Balancer  

34�  

Page 35: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

35  

High  Availability  

Search  Head  3  Search  Head  2  Search  Head  1  

Load  Balancer  

35�  

Page 36: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

36  

Reliability  

3  Indexers  

36�  

Page 37: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

37  

Splunk  Buckets  

Searchable  Bucket  Contains  raw  data  &  Splunk  metadata  

37�  

Page 38: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

38  

Splunk  Buckets  

Searchable  Bucket  Contains  raw  data  &  Splunk  metadata  

Replicated  Bucket  Contains  only  raw  data  

38�  

Page 39: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

39  

Reliability  Indexer  1   Indexer  2   Indexer  3  

39�  

Page 40: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

40  

Reliability  

Searchable  Bucket  

Indexer  1   Indexer  2   Indexer  3  

40�  

Page 41: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

41  

Reliability  

Searchable  Bucket  

Indexer  1   Indexer  2   Indexer  3  

Searchable  Bucket  

41�  

Page 42: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

42  

Reliability  

Searchable  Bucket   Replicated  Bucket  

Indexer  1   Indexer  2   Indexer  3  

Searchable  Bucket  

42�  

Page 43: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

43  

Reliability  

Searchable  Bucket   Replicated  Bucket  

Indexer  1   Indexer  2   Indexer  3  

Searchable  Bucket  

43�  

Page 44: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

44  

Reliability  

Replicated  Bucket  

Indexer  1   Indexer  2   Indexer  3  

Searchable  Bucket  

44�  

Page 45: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

45  

Reliability  

Searchable  Bucket   Replicated  Bucket  

Indexer  1   Indexer  2   Indexer  3  

Searchable  Bucket  

45�  

Page 46: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

46  

High  Availability  

Indexers  

46�  

Page 47: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

47  

High  Availability  

Indexers  

47�  

Page 48: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

48  

High  Availability  

Indexers  

48�  

Page 49: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

49  

High  Availability  

Indexers  

49�  

Page 50: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

50  

High  Availability  

Indexers  

50�  

Page 51: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

51  

High  Availability  

Indexers  

51�  

Page 52: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

52  

High  Availability  

Indexers  

Search  Head  

52�  

Page 53: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

53  

High  Availability  

Indexers  

Search  Head  

53�  

Page 54: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

54  

High  Availability  

Indexers  

Search  Head  

54�  

Page 55: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

55  

High  Availability  

Indexers  

Search  Head  

55�  

Page 56: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

56  

Disaster  Recovery  

Splunk  Buckets  &  ConfiguraBon  

56�  

Page 57: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

57  

Disaster  Recovery  

Amazon  S3  Splunk  Buckets  &  ConfiguraBon  

57�  

Page 58: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

58  

Disaster  Recovery  

Amazon  S3  Splunk  Buckets  &  ConfiguraBon  

58�  

Page 59: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

Security  

Page 60: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

60  

Customer Stack 60�  

Page 61: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

61  

security group

Isolated  by  Security  Groups  

61�  

Page 62: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

62  

security group

Further  Isolated  by  a  Splunk  VPC  

62�  

Page 63: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

63  

AuthenBcaBon  Customer  Forwarders  

Client  SSL  CerBficates  

Server  SSL  CerBficates  

63�  

Page 64: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

64  

AuthenBcaBon  Customer  Forwarders  

Client  SSL  CerBficates  

Server  SSL  CerBficates  

1  

2  

64�  

Page 65: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

65  

AuthenBcaBon  

1  

2  

Rogue  Forwarders  

65�  

Page 66: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

66  

EncrypBon  At  Rest  

Search  Heads  Indexers  Clustering  &  License  Managers  

66�  

Page 67: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

67  

Splunk'd  

Forwarder   Indexer   Search  Head  

encrypt  

67�  

Page 68: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

68  

Splunk'd  

Forwarder   Indexer   Search  Head  

decrypt  

Splunk'd  

decrypt  

68�  

Page 69: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

Hybrid  

Page 70: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

70  

Search Head(s)

Indexer(s)

On Premises Private Cloud Public Cloud

Indexer(s)

On Premises Private Cloud Public Cloud

Hybrid  Search  

Single  Pane  of  Glass  Visibility  

Page 71: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

71  

Wrapping  Up  

  Highly  performant    Highly  available    Built  with  security  from  the  ground  up    OperaBonally  automated    A  single  pane  of  glass  visibility  with  hybrid  search  

71�  

Page 72: SplunkCloud Under*The*Hood* · Disclaimer* 2 During*the*course*of*this*presentaon,*we*may*make*forward*looking*statements*regarding*future*events* or*the*expected*performance*of*the*company.*We

72  

Rajiv  BaTula  SoOware  Engineer  [email protected]  

Nikhil  Mungel  Sr.  SoOware  Engineer  [email protected]  

72�  

Thank  You!  

QuesBons?