Norikra Recent Updates

Norikra meetup #2, 2015/06/03

TAGOMORI Satoshi(@tagomoris)

@tagomorisTreasure Data, Inc.

Fluentd, Norikra, Hadoop, ...

Norikra:Open source stream processing server with SQL

• v1.0.0: 2014/05/20• "Stream Processing and Norikra"• http://www.slideshare.net/tagomoris/stream-processing-and-norikra

• "Landscape of Norikra Features"• http://www.slideshare.net/tagomoris/norikra-meetup-features

• (... 16 releases)

• v1.3.1: 2015/05/07

Changes of Norikra

• "Changes" section in README• only for major updates

• "Changes.md"• for details• https://github.com/norikra/norikra/blob/master/Changes.md

• Commit logs

Features newly added

• Suspended queries• NULLABLE fields• Listener plugins• Dynamic plugin reloading

Suspended queries• temporarily stop queries / resume these later

• good for tests, try&errors

• [caution]• suspended queries doesn't saved in stats file

NULLABLE fields

• Queries process events just with all fields• NULLABLE(field)

• queries handle field as NULL if missing• works well for multi data source situation

NULLABLE fields(2)

SELECT a, b,COUNT(*) AS count

FROM s.win:time(1 min)GROUP BY a, b

{"a":1, "b":2} {"a":2}

{"a":1, "b":2, "count":1}{"a":2, "b":2, "count":1}

{"a":2, "b":2}

ignored!

NULLABLE fields(3)

SELECT a, NULLABLE(b),COUNT(*) AS count

FROM s.win:time(1 min)GROUP BY a, b

{"a":1, "b":2} {"a":2}

{"a":1, "b":2, "count":1}{"a":2, "b":2, "count":1}

{"a":2, "b":null, "count":1}

{"a":2, "b":2}

Listener

• Listener• process output data of queries• default listener: memory pool

• Built-in listeners: specified by group• LOOPBACK(target)• STDOUT()

Listener plugin: do anything

• 3rd party listener plugin• syntax: NAME(arg)• JRuby + rubygem

• We can do whatever we want :D• by writing very simple code

• https://github.com/norikra/norikra-lilstener-mock

Listener plugin: patterns

query

push to users

enqueue tokafka

memorypool

filter records

other targetsample records

Listener plugin: sync vs async

Sync listener:> called per every query's output> roughly specified by views (.win:...)> select this if you wanna deliver events ASAP

Async listener:> called per specified interval [0.1sec]> process buffered output data> very easy to write

Dynamic plugin reloading

• SIGHUP• Norikra loads plugins newly installed in

JRuby runtime• doesn't reload plugins already installed

• UDF plugins, Listener plugins• Less restarts!

Other updates:• /engine_statistics, --ui-context-path• GC stats in engine statistics• SO_REUSEADDR• Shutoff mode under high memory pressure• --log4j-properties-path, -javaagent• Esper 5.2, msgpack.gem, ...• and many other bug fixes!

Norikra improvementnever lasting!

Next:• Custom views? Tables?• Query distribution?

Thanks!