Plugins by tagomoris #fluentdcasual

Pluginsby tagomoris

Fluentd Casual Talks LT2012/05/18

@tagomoris (NHN Japan Corp.)

@tagomoris

NHN Japan Corp.Web Service Business DivisionDevelopment Department 2

Development Platforms

Appendixdiff of 2012/02 and now

• Our Fluentd cluster status

Fluentd Cluster status updates (1)

89 Fluentd processes12 nodes

89 Fluentd processes12 nodes


ruby 1.9.2 + glibc mallocscribeline

ruby 1.9.3 + jemallocfluent-agent-lite


from 127 servers146 log streams

from 205 servers246 log streams


70,000 msgs/sec120Mbps

(at peak time)

92,000 msgs/sec184Mbps


650GB/day(non-compressed bytes on HDFS)

995GB/day


Before: 10 pluginsin_forward, out_forward, in_scribe, out_scribe

out_copy, out_roundrobin, out_exec_filterout_hoop

out_flowcounter, out_growthforecast

After: +9 pluginsout_route, in_gc_stat, in_object_space

out_mongo, out_sampling_filterout_amplifier_filter, out_datacounter

out_file_alternative, out_forest

Plugins!

All your plugin are belong to us.

gem search -r fluent-plugin

41 plugins

fluent-plugin-amplifier-filter (0.1.1)fluent-plugin-amqp (0.0.0)

fluent-plugin-cassandra (0.0.2)fluent-plugin-couch (0.5.0)

fluent-plugin-datacalculator (0.0.1)fluent-plugin-datacounter (0.3.0)

fluent-plugin-delayed (0.0.1)fluent-plugin-dstat (0.1.0)

fluent-plugin-file-alternative (0.1.2)fluent-plugin-flowcounter (0.1.4)

fluent-plugin-flume (0.1.0)fluent-plugin-forest (0.1.0)fluent-plugin-growl (0.0.5)

fluent-plugin-growthforecast (0.1.2)fluent-plugin-hipchat (0.1.0)fluent-plugin-hoop (0.1.2)

fluent-plugin-http-enhanced (0.0.3)fluent-plugin-ikachan (0.1.0)fluent-plugin-kestrel (0.1.0)

fluent-plugin-librato-metrics (0.2.3)fluent-plugin-mongo (0.6.7)

fluent-plugin-msgpack-rpc (0.2.1)fluent-plugin-mysql (0.0.2)fluent-plugin-notifier (0.0.1)

fluent-plugin-numeric-monitor (0.0.1)fluent-plugin-parser (0.1.0)

fluent-plugin-pghstore (0.1.2)fluent-plugin-redis (0.2.0)

fluent-plugin-resque (0.2.1)fluent-plugin-s3 (0.2.2)

fluent-plugin-sampling-filter (0.1.1)fluent-plugin-say (0.1.1)

fluent-plugin-scribe (0.10.8)fluent-plugin-sns (0.2.1)fluent-plugin-solr (0.1.1)

fluent-plugin-splunk (0.0.1.1)fluent-plugin-sqs (0.2.1)fluent-plugin-td (0.10.6)fluent-plugin-udp (0.0.1)

fluent-plugin-zabbix (0.0.1)fluent-plugin-zmq (0.0.3)

gem search -r fluent-plugin

13/41 plugins

fluent-plugin-amplifier-filter (0.1.1)fluent-plugin-amqp (0.0.0)

fluent-plugin-cassandra (0.0.2)fluent-plugin-couch (0.5.0)

fluent-plugin-datacalculator (0.0.1)fluent-plugin-datacounter (0.3.0)

fluent-plugin-delayed (0.0.1)fluent-plugin-dstat (0.1.0)

fluent-plugin-file-alternative (0.1.2)fluent-plugin-flowcounter (0.1.4)

fluent-plugin-flume (0.1.0)fluent-plugin-forest (0.1.0)fluent-plugin-growl (0.0.5)

fluent-plugin-growthforecast (0.1.2)fluent-plugin-hipchat (0.1.0)fluent-plugin-hoop (0.1.2)

fluent-plugin-http-enhanced (0.0.3)fluent-plugin-ikachan (0.1.0)fluent-plugin-kestrel (0.1.0)

fluent-plugin-librato-metrics (0.2.3)fluent-plugin-mongo (0.6.7)

fluent-plugin-msgpack-rpc (0.2.1)fluent-plugin-mysql (0.0.2)

fluent-plugin-notifier (0.0.1)fluent-plugin-numeric-monitor (0.0.1)

fluent-plugin-parser (0.1.0)fluent-plugin-pghstore (0.1.2)

fluent-plugin-redis (0.2.0)fluent-plugin-resque (0.2.1)

fluent-plugin-s3 (0.2.2)fluent-plugin-sampling-filter (0.1.1)

fluent-plugin-say (0.1.1)fluent-plugin-scribe (0.10.8)

fluent-plugin-sns (0.2.1)fluent-plugin-solr (0.1.1)

fluent-plugin-splunk (0.0.1.1)fluent-plugin-sqs (0.2.1)fluent-plugin-td (0.10.6)fluent-plugin-udp (0.0.1)

fluent-plugin-zabbix (0.0.1)fluent-plugin-zmq (0.0.3)

file-alternative (1/13)

out_filetime [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }

out_file_alternative

time [TAB] {"key1":"value1", "key2":"value2",... }

time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }

tag [TAB] {"key1":"value1", "key2":"value2",... }

time [TAB] tag [TAB] value1 [TAB] value2 ...

time,tag,value1,value2 ...

hoop (2/13)

HDFSHoopServer

Fluentd

output format compatible with out_file_alternative

mysql (3/13)

MySQLFluentd

JSON into single columnor

values into columns

alpha!

ikachan (4/13)

IRCServer

ikachanServerFluentd

alpha!

growthforecast (5/13)

growthforecast by @kazeburo

flowcounter (6/13)

flowcounter

messages

{"count":300, "bytes":3660, "count_rate":5, "bytes_rate":61}

count_keys *unit minute

datacounter (7/13)

{"google_count":120, "google_rate":2, "google_percentage":20, "unmatched_count":480, ...}

datacounter

messages

unit minuteaggregate allcount_key refererpattern1 google google.com

numeric-monitor (8/13)

{"min":100131,"max":483120, "avg":181343, "percentile_90":283201, "percentile_95":341210}

numeric_monitor

messages

unit minuteaggregate allmonitor_key durationpercentiles 90,95

alpha!

notifier (9/13)

notifierwarn 10crit 30

alpha!

10 Notify msg

12

10

15

13 Notify msg

suppressedin specified interval...

sampling-filter (10/13)

samplingfilterinterval 3

msg msg

msg

msg

msg

msg

msg

msg

msg

msg

amplifier-filter (11/13)

amplifierfilter

ratio 3

10

12

10

15

13

30

36

30

45

39

beautiful real-time monitoring world

Streaming data

Sampling

Counting

Amplifying

GeneratigNotification

Output(Graph, IRC, ...)

parser (12/13) alpha!

numeric_monitor

format /^(?<time>\S+) (?<item>\S+) (?<num>\d+)$/time_format %Y/%m/%d:%H:%M:%Skey_name logreserve_data yes

{"log":"2012/05/18:16:21 DRINK1 1"}

log_based_time message{"log":"2012/05/18:16:21 DRINK1 1", "item":"DRINK1", "num":"1"}

forest (13/13)<match input.A> type foo .... conf_a xxxx.A.yyyy option_x 30 ....</match><match input.B> type foo .... conf_a xxxx.B.yyyy option_x 30 ....</match><match input.C> type foo .... conf_a xxxx.C.yyyy option_x 100 ....</match>

<match input.*> type forest subtype foo remove_prefix input <template> .... conf_a xxxx.__TAG__.yyyy .... </template> <case {C,D}> option_x 100 </case> <case *> option_x 30 </case></match>

NO PLUGINS, NO FLUENTD

Enjoy Fluentd!

One more thing

fluent-plugin-webhdfs

Coming Soon!

Thanks!

Plugins by tagomoris #fluentdcasual

Technology

Transcript of Plugins by tagomoris #fluentdcasual