Fluent event collector [update]
-
Upload
sadayuki-furuhashi -
Category
Documents
-
view
58.140 -
download
1
Transcript of Fluent event collector [update]
Sadayuki Furuhashi
Fluent
@frsyuki
e Event Collector Service
Treasure Data, Inc.
Structured logging
Pluggable architecture
Reliable forwarding
What’s Fluent?
It's like syslogd, but uses JSON for log messages
What’s Fluent?
“2011-04-01 host1 myapp: cmessage size=12MB user=me”
2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, “user”: “me”}
What’s Fluent?
2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, “user”: “me”}
time tag
record
Fluent vs Scribe
• Deals with structured logs• Easy to install> “gem install fluentd”> also apt-get and yum http://packages.treasure-data.com/
• Easy to customize> add/modify plugins without compile> “gem search -rd fluent-plugin”
Fluent vs Flume
• Easy to setup> “sudo fluentd --setup && fluentd”
• Compact> small engine (2,000 lines) + plugins> JVM-free
• Easy to configure
# receive events via HTTP<source> type http port 8888</source>
# read logs from file<source> type tail path /var/log/httpd.log format apache tag apache.access</source>
# save alerts to file<match alert.**> type file path /var/log/fluent/alerts</match>
# save access logs to MongoDB# and forward to other server<match apache.access> type copy <store> type mongo host 127.0.0.1 </store> <store> type forward <server> host 192.168.0.11 weight 20 </server> <server> host 192.168.0.12 weight 60 </server> </store></match>
Architecture
Input Buffer Output
HTTP+JSONFile tailSyslog...
MemoryFile
FileAmazon S3Fluent...
Pluggable Pluggable Pluggable
Architecture
Input
HTTP+JSONFile tailSyslog...
Pluggable
✓ Receive logs✓ Or pull logs from data sources✓ Non-blocking
Input plugins:
Architecture
Pluggable
✓ Improve performance✓ Provide reliability✓ Provide thread-safety
Buffer plugins:
Buffer
MemoryFile
Architecture
Pluggable
✓ Improve performance✓ Provide reliability✓ Provide thread-safety
Buffer plugins:
chunk
chunk
chunk output
Input
Architecture
Pluggable
✓ Write or send event logs
Output plugins:
Output
FileAmazon S3Fluent...
Log collection
FluentApps
Client library
Apache log
HTTP+JSON(in_http)
TCP+MessagePack(in_tcp)
follow growth of files (in_tail)
Client library
Client library
Ruby (by @frsyuki)Python (by @kzk_mover)PHP (by @chobi_e)Java (by @muga_nishizawa)C++ (by @nobu_k)...
Fluent.open(“myapp”, “localhost”)Fluent.event(“login”, {“user”=>38})#=> 2011-02-03 04:56:01 myapp.login {“user”:38}
Forwarding
Fluent
FluentApps
FluentApps
FluentApps
HA Forwarding
Fluent
Fluent
FluentApps
FluentApps
FluentApps
Active
Active
✓ load balancing✓ continuous heartbeat✓φ accrual failure detector
External filter program
FluentFluent
Externalprogram
・stdin / stdout・TSV
External output program
FluentFluent
Externalprogram
Externalprogram
・TSV file
External input program
Fluent
Externalprogram
Externalprogram
Externalprogram
receiveor pull
・stdout・TSV
Incremental Aggregationby @doryokujin #fluent_meetup2
FluentApps MongoDB
/ Redis
count[“2011-01-01 01h”.”/index”] += 1count[“2011-01-01 01h”.”/index”] += 1count[“2011-01-01 01h”.”/landing”] += 1count[“2011-01-01 02h”.”/index”] += 1...
count[“2011-01-01 01h”.”/index”] == 42count[“2011-01-01 01h”.”/landing”] == 38count[“2011-01-01 02h”.”/index”] == 55count[“2011-01-01 02h”.”/landing”] == 55
FluentApps
MySQL replicator (plan)
FluentMySQL
read binlog continuouslyusing MySQL binlog API
mysql://127.0.0.1
http://www.slideshare.net/mkindahl/binary-log-api-presentation-oscon-2011
3rd Party Plugins
• Scribe input/output (by @kzk_mover)> receive logs sent from scribed
• Amazon S3 output (by @frsyuki)> upload log files to Amazon S3
• MongoDB output (by @repeatedly)> write structured logs to MongoDB
• Redis output (by @yuki24)> write structured logs to Redis
3rd Party Plugins
• Cassandra output (by @railute)> write structured logs to Cassandra
• Growl output (by @goth_writst_cut)> notify messages using Growl(Growl: notification service for Mac OS X)
• Aggregation (by @doryokujin)> incremental aggregation using NoSQL DBs
• ...
http://fluentd.org/plugin/> gem search -dr fluent-plugin
Plugin Developer APIs
• Unit test framework (like “MRUnit”)> Fluent::Test::InputTestDriver> Fluent::Test::OutputTestDriver> Fluent::Test::BufferedOutputTestDriver
• Fluent::BasicTail (base class of “tail” plugin)> text parser is customizable def parse_line(line)
What’s Fluent?
• Event collector service> deals with structured event logs
• Compact> small engine (2,000 lines) + plugins
• Plugin archtecture> install plugins using RubyGems
Demo