HTTP Event Collector, Simplified Developer Logging
-
Upload
glenn-block -
Category
Data & Analytics
-
view
825 -
download
5
Transcript of HTTP Event Collector, Simplified Developer Logging
![Page 1: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/1.jpg)
Copyright © 2015 Splunk Inc.
Glenn Block (@gblock) – Principal Product ManagerJian Lee – Senior Software Engineer Splunk Developer Platform & Core
HTTP Event Collector, Simplified Developer Logging
![Page 2: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/2.jpg)
2
DisclaimerDuring the course of this presentation, we may make forward looking statements regarding future events
or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results
could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking
statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information.
We do not assume any obligation to update any forward looking statements we may make.
In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated
into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
![Page 3: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/3.jpg)
DemoApp Monitoring
![Page 4: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/4.jpg)
4
HTTP Event CollectorA new token-based JSON API for eventsSend events directly from anywhere (servers, mobile devices, IOT)Easy to configure / works out of the box. Easy to secureHighly performant, scalable and available
![Page 5: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/5.jpg)
DemoConfiguring HTTP Event Collector
![Page 6: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/6.jpg)
How you useEnable HTTP Event CollectorCreate/Get a tokenSend events to Splunk using the token– Use HTTP Directly
Create a POST request and set the Auth header with the token
POST JSON in our event format to the collector
– Use logging libraries Support for .NET, Java and JavaScript loggers
6
![Page 7: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/7.jpg)
Sending data//send with curlcurl -k https://localhost:8088/services/collector -H 'Authorization: Splunk 46931F1C-352C-4DF6-820C-F2689CF88494' -d '{"event":"Hello Event Collector"}'
7
![Page 8: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/8.jpg)
./splunk_httpinput/local/inputs.confGlobal Stanza
TokenStanza
Token Name
Enable/Disable the collector
Auth tokenEnable/Disable the token
Default metadata
Default metadata
Default index
Allowed indexes
![Page 9: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/9.jpg)
./splunk_httpinput/defaults/inputs.conf
Default port
SSL Enabled by default
Distributed deployment disabled
![Page 10: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/10.jpg)
10
Event Collector CLI
./bin/splunk http-event-collector help
![Page 11: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/11.jpg)
11
Permissions and delegation
HTTP Event Collector requires the edit_token_http cap.You can delegate token admin to devops / engToken admins can only manage the feature, they do not have any other admin permissions in Splunk
![Page 12: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/12.jpg)
12
A few tipsCreate tokens per app, department, component, service. etc. Not per user or device especially if you are talking about a large number (> 10000)Consider partitioning tokens to different indexes. This will speed up searches and make it easy to archiveConsider delegating token management to devops/engExplicitly set allowed indexes on the token. If not set, the token can send data to any index.Use HTTP over HTTPS when you can. You can get about a 30% performance gain.Ask your devs to batch events. It greatly improves throughtput.
![Page 13: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/13.jpg)
13
![Page 14: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/14.jpg)
14
Scale and High Availability
Indexers
Search Head / Deployment Server
![Page 15: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/15.jpg)
15
Scale and High Availability
Event Collectors Indexers Search Heads
![Page 16: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/16.jpg)
16
Distributed deployment
HTTP Event Collector can scale to meet your needs!
• Build in to splunkd, nothing special to install• Run directly on the indexer• Or run on a dedicated Collector instance and forward to an indexer• Uses Deployment Server to to sync tokens across the Collector
instances
![Page 17: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/17.jpg)
17
How to setup a DS client
splunk set deploy-poll [host]:8088splunk enable deploy serversplunk restart
![Page 18: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/18.jpg)
DemoDistributed deployment
![Page 19: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/19.jpg)
DemoTroubleshooting/Monitoring
![Page 20: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/20.jpg)
20
3rd party integrations
![Page 21: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/21.jpg)
21
Send your container logs DIRECTLY to Splunk
![Page 22: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/22.jpg)
22
How it works
A new log driver capture container’s stdout and pushes to SplunkCurrently it is in development, but should be out of the box soon.We’re contributing to Docker!!!!!!!
docker run --log-driver=splunk --log-opt splunk-token=F81DD289-863D-45EF-B9CE-A7D3514AF2C7 --log-opt splunk-url=https://10.20.17.169:8088 --log-opt splunk-insecureskipverify=true hello-world
![Page 23: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/23.jpg)
Demo:Docker driver
23
![Page 24: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/24.jpg)
24
Next steps?
Breakouts Liberate Your Application Logging
More information docs.splunk.com, see "Getting Data In" dev.splunk.com
Come by the Developer Booth and say hi / ask questions!
Related breakout sessions and activities…
![Page 25: HTTP Event Collector, Simplified Developer Logging](https://reader034.fdocuments.in/reader034/viewer/2022042722/58acbb9e1a28abd3048b47cd/html5/thumbnails/25.jpg)
THANK YOU