Post on 08-May-2015
Pluginsby tagomoris
Fluentd Casual Talks LT2012/05/18
@tagomoris (NHN Japan Corp.)
@tagomoris
NHN Japan Corp.Web Service Business DivisionDevelopment Department 2
Development Platforms
Appendixdiff of 2012/02 and now
• Our Fluentd cluster status
Fluentd Cluster status updates (1)
89 Fluentd processes12 nodes
89 Fluentd processes12 nodes
Fluentd Cluster status updates (2)
ruby 1.9.2 + glibc mallocscribeline
ruby 1.9.3 + jemallocfluent-agent-lite
Fluentd Cluster status updates (3)
from 127 servers146 log streams
from 205 servers246 log streams
Fluentd Cluster status updates (4)
70,000 msgs/sec120Mbps
(at peak time)
92,000 msgs/sec184Mbps
Fluentd Cluster status updates (5)
650GB/day(non-compressed bytes on HDFS)
995GB/day
Fluentd Cluster status updates (6)
Before: 10 pluginsin_forward, out_forward, in_scribe, out_scribe
out_copy, out_roundrobin, out_exec_filterout_hoop
out_flowcounter, out_growthforecast
After: +9 pluginsout_route, in_gc_stat, in_object_space
out_mongo, out_sampling_filterout_amplifier_filter, out_datacounter
out_file_alternative, out_forest
Plugins!
All your plugin are belong to us.
gem search -r fluent-plugin
41 plugins
fluent-plugin-amplifier-filter (0.1.1)fluent-plugin-amqp (0.0.0)
fluent-plugin-cassandra (0.0.2)fluent-plugin-couch (0.5.0)
fluent-plugin-datacalculator (0.0.1)fluent-plugin-datacounter (0.3.0)
fluent-plugin-delayed (0.0.1)fluent-plugin-dstat (0.1.0)
fluent-plugin-file-alternative (0.1.2)fluent-plugin-flowcounter (0.1.4)
fluent-plugin-flume (0.1.0)fluent-plugin-forest (0.1.0)fluent-plugin-growl (0.0.5)
fluent-plugin-growthforecast (0.1.2)fluent-plugin-hipchat (0.1.0)fluent-plugin-hoop (0.1.2)
fluent-plugin-http-enhanced (0.0.3)fluent-plugin-ikachan (0.1.0)fluent-plugin-kestrel (0.1.0)
fluent-plugin-librato-metrics (0.2.3)fluent-plugin-mongo (0.6.7)
fluent-plugin-msgpack-rpc (0.2.1)fluent-plugin-mysql (0.0.2)fluent-plugin-notifier (0.0.1)
fluent-plugin-numeric-monitor (0.0.1)fluent-plugin-parser (0.1.0)
fluent-plugin-pghstore (0.1.2)fluent-plugin-redis (0.2.0)
fluent-plugin-resque (0.2.1)fluent-plugin-s3 (0.2.2)
fluent-plugin-sampling-filter (0.1.1)fluent-plugin-say (0.1.1)
fluent-plugin-scribe (0.10.8)fluent-plugin-sns (0.2.1)fluent-plugin-solr (0.1.1)
fluent-plugin-splunk (0.0.1.1)fluent-plugin-sqs (0.2.1)fluent-plugin-td (0.10.6)fluent-plugin-udp (0.0.1)
fluent-plugin-zabbix (0.0.1)fluent-plugin-zmq (0.0.3)
gem search -r fluent-plugin
13/41 plugins
fluent-plugin-amplifier-filter (0.1.1)fluent-plugin-amqp (0.0.0)
fluent-plugin-cassandra (0.0.2)fluent-plugin-couch (0.5.0)
fluent-plugin-datacalculator (0.0.1)fluent-plugin-datacounter (0.3.0)
fluent-plugin-delayed (0.0.1)fluent-plugin-dstat (0.1.0)
fluent-plugin-file-alternative (0.1.2)fluent-plugin-flowcounter (0.1.4)
fluent-plugin-flume (0.1.0)fluent-plugin-forest (0.1.0)fluent-plugin-growl (0.0.5)
fluent-plugin-growthforecast (0.1.2)fluent-plugin-hipchat (0.1.0)fluent-plugin-hoop (0.1.2)
fluent-plugin-http-enhanced (0.0.3)fluent-plugin-ikachan (0.1.0)fluent-plugin-kestrel (0.1.0)
fluent-plugin-librato-metrics (0.2.3)fluent-plugin-mongo (0.6.7)
fluent-plugin-msgpack-rpc (0.2.1)fluent-plugin-mysql (0.0.2)
fluent-plugin-notifier (0.0.1)fluent-plugin-numeric-monitor (0.0.1)
fluent-plugin-parser (0.1.0)fluent-plugin-pghstore (0.1.2)
fluent-plugin-redis (0.2.0)fluent-plugin-resque (0.2.1)
fluent-plugin-s3 (0.2.2)fluent-plugin-sampling-filter (0.1.1)
fluent-plugin-say (0.1.1)fluent-plugin-scribe (0.10.8)
fluent-plugin-sns (0.2.1)fluent-plugin-solr (0.1.1)
fluent-plugin-splunk (0.0.1.1)fluent-plugin-sqs (0.2.1)fluent-plugin-td (0.10.6)fluent-plugin-udp (0.0.1)
fluent-plugin-zabbix (0.0.1)fluent-plugin-zmq (0.0.3)
file-alternative (1/13)
out_filetime [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }
out_file_alternative
time [TAB] {"key1":"value1", "key2":"value2",... }
time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }
tag [TAB] {"key1":"value1", "key2":"value2",... }
time [TAB] tag [TAB] value1 [TAB] value2 ...
time,tag,value1,value2 ...
hoop (2/13)
HDFSHoopServer
Fluentd
output format compatible with out_file_alternative
mysql (3/13)
MySQLFluentd
JSON into single columnor
values into columns
alpha!
ikachan (4/13)
IRCServer
ikachanServerFluentd
alpha!
growthforecast (5/13)
growthforecast by @kazeburo
flowcounter (6/13)
flowcounter
messages
{"count":300, "bytes":3660, "count_rate":5, "bytes_rate":61}
count_keys *unit minute
datacounter (7/13)
{"google_count":120, "google_rate":2, "google_percentage":20, "unmatched_count":480, ...}
datacounter
messages
unit minuteaggregate allcount_key refererpattern1 google google.com
numeric-monitor (8/13)
{"min":100131,"max":483120, "avg":181343, "percentile_90":283201, "percentile_95":341210}
numeric_monitor
messages
unit minuteaggregate allmonitor_key durationpercentiles 90,95
alpha!
notifier (9/13)
notifierwarn 10crit 30
alpha!
10 Notify msg
12
10
15
13 Notify msg
suppressedin specified interval...
sampling-filter (10/13)
samplingfilterinterval 3
msg msg
msg
msg
msg
msg
msg
msg
msg
msg
amplifier-filter (11/13)
amplifierfilter
ratio 3
10
12
10
15
13
30
36
30
45
39
beautiful real-time monitoring world
Streaming data
Sampling
Counting
Amplifying
GeneratigNotification
Output(Graph, IRC, ...)
parser (12/13) alpha!
numeric_monitor
format /^(?<time>\S+) (?<item>\S+) (?<num>\d+)$/time_format %Y/%m/%d:%H:%M:%Skey_name logreserve_data yes
{"log":"2012/05/18:16:21 DRINK1 1"}
log_based_time message{"log":"2012/05/18:16:21 DRINK1 1", "item":"DRINK1", "num":"1"}
forest (13/13)<match input.A> type foo .... conf_a xxxx.A.yyyy option_x 30 ....</match><match input.B> type foo .... conf_a xxxx.B.yyyy option_x 30 ....</match><match input.C> type foo .... conf_a xxxx.C.yyyy option_x 100 ....</match>
<match input.*> type forest subtype foo remove_prefix input <template> .... conf_a xxxx.__TAG__.yyyy .... </template> <case {C,D}> option_x 100 </case> <case *> option_x 30 </case></match>
NO PLUGINS, NO FLUENTD
Enjoy Fluentd!
One more thing
fluent-plugin-webhdfs
Coming Soon!
Thanks!