Extend Redis with ModulesItamar Haber

2

Who We Are

The open source home and commercial provider of Redis

Open source. The leading in-memory database platform, supporting any high performance OLTP or OLAP use case.

itamar@redislabs.com@itamarhaberChief Developer Advocate at Redis Labs

~10 Things About Redis

4

1.Redis: REmote DIctionary Server2./ rɛdɪs/: “red-iss”3.OSS: http://github.com/antirez/redis4.3-clause BSD-license: http://redis.io5. In-memory: read/write from/to RAM6.A database for: 5 data structures7.And: 4 (+1) more specialized ones

5

8.Developed & maintained: (mostly) Salvatore Sanfilippo (a.k.a. @antirez) and his OSS team at @RedisLabs

9.History: v1.0 August 9th, 2009… v3.2.4 Septmeber 26th, 2016

10.“The Leatherman™ of Databases”:mostly used as a DB, cache & broker

6

11.A couple or so of extra features:(a) atomicity; (b) blocking wait;(c) configurable persistence;(d) data expiration and (e) eviction; as well as transactions, PubSub, Luascripts, high availability & clustering

12.Next version (v4.0): MODULES!

7

Redis 1011. Redis is “NoSQL”0. No (explicit) schema, access by key1. Key -> structure -> data

8

Redis data stratav1.0 Strings

ListsSets

v1.2 SortedSets

v2.0 Hashes

v2.2 Bit arraysv2.8.9 HyperLogLogsv3.2 Geo Sets

Bit fieldsv4 MODULES!

and a brain!

9

How to Redis in 3 steps:1. 150 OSS clients in 50 languages, e.g:

Java, Node.js, .NET, Python, Ruby…2. You make a request, i.e.:

PING3. The server replies, i.e.g:

PONG

10

~$ redis-cli127.0.0.1:6379> SET counter 1OK127.0.0.1:6379> GET counter"1"127.0.0.1:6379> INCRBY counter 1(integer) 2127.0.0.1:6379> APPEND counter b||!2b(integer) 7127.0.0.1:6379> GETSET counter "\x00Hello\xffWorld""2b||!2b"127.0.0.1:6379>

The Evolution of Versatility

12

Flexibility: model (almost) anything with basic “building blocks” and simple rules (v0.0.1)Composability: transactions (v1.2) and server-side Lua scripts (v2.6)Extensibility: modules (v4) for adding custom data structures and commands

MODULES! (a.k.a plugins)

14

Redis before modules:1. Redis is ubiquitous for fast data, fits

lots of cases (Swiss™ Army knife)2. Some use cases need special care3. Open source has its own agenda

So what can you do? FR, PR or fork

15

Redis with modules:1. Core still fits lots of cases2. Module extensions for special cases3. A new community-driven ecosystem4. “Give power to users to go faster”5. … and not only to users …What to expect? Nothing’s impossible!

16

Redis modules are:1. Dynamically (server-)loaded libraries2. Future-compatible3. (are mostly) written in C4. (nearly) as fast as the core5. GA Q4 2016

17

Modules let you:1. Process: where the data is at2. Compose: call core & other modules3. Extend: add new data structures

and commands

18

redis> ECHO "Alpha""Alpha"redis> MODULE LOAD example.soOKredis> EXAMPLE.ECHO "Bravo""Bravo"redis> ^C

~$ wc example.c13 46 520 example.c~$ gcc -fPIC -std=gnu99 -c -o example.o example.c~$ ld -o example.so example.o -shared -Bsymbolic -lc

core command

module library

“new” command

Redis Modules API

20

The API1. Where most of the effort was made2. Abstracts & isolates Redis’ internals3. The server’s (C-) binding contract4. Will not be broken once released5. Exposes three conceptual layers

21

Modules API layers1.Operational: admin, memory, disk,

replication, arguments, replies…2.High-level: client-like access to core

and modules’ commands3.Low-level: (almost) native access to

core data structures memory

22

user app

Redis client

Redis

core

data

GET foo

"bar"

101010

010101

101010

23

user

101010

010101

101010

High

level

API

appmodule

24

user

101010

010101

101010

app

Low

level

API

The Benchmark (Why Bother with Modules?)

26

1.2 1.251.05

0.1

seconds

Time needed

for summing

1,000,000

Sorted Set

scores Python

(local)

Lua API

high low

27

Some exemplary modules:• RediSearch – full text search• countminsketch – streaming counter• redablooms – Bloom filters

• redis-tdigest – rank estimator

28

redismodules.com: Redis Module Hub

29

What Is The Hub

1.Modules developed by: anyone2.Certified by: Redis Labs3.Licenses: Open Source & Commercial4. (will be) Distributed via: Redis Cloud

and Redis Labs Enterprise Cluster5.Where: redismodules.com

30

Neural Redis moduleAlpha

• A brain neural network• Simple to use, will be part of Redis• Online (threaded) training• https://github.com/antirez/neural-

redis

31

redis> NN.CREATE net REGRESSOR 2 3 -> 1 …(integer) 13redis> NN.OBSERVE net 1 2 -> 3redis> NN.OBSERVE net 4 5 -> 9redis> NN.OBSERVE net 3 4 -> 7redis> NN.OBSERVE net 1 1 -> 2… repeat ~6 more times

redis> NN.TRAIN net AUTOSTOPredis> NN.RUN net 1 11) "2.0776522297040843"redis> NN.RUN net 3 21) "5.1765427204933099"

Teaching it

how to add

numbers

32

Build your own (module)• Get Redis (unstable branch)• Read the docs under src/modules• An SDK to jumpstart:

https://github.com/RedisLabs/RedisModulesSDK

• Start coding

33

Hack for Fun and Profit• Online: Oct 17th

San Francisco & Tel-Aviv: Nov 11th

• World fame & cash prizes!• Register at:

https://www.hackerearth.com/sprints/redislabs-hackathon-global/

Thank you

Further Reading

36

1. The Redis Open Source Project Website – http://redis.io2. Redis source code on GitHub – http://github.com/antirez/redis3. Getting started:

1. An introduction to Redis data types and abstractions –http://redis.io/topics/data-types-intro

2. Try Redis (in your browser) – http://try.redis.io3. Karl Seguin’s The Little Redis Book –

http://openmymind.net/2012/1/23/The-Little-Redis-Book/4. Josiah Carlson’s Redis In Action – https://redislabs.com/ebook/redis-in-

action

4. Redis documentation – http://redis.io/documentation5. Redis commands – http://redis.io/commands6. Redis community – http://redis.io/community7. Redis Watch newsletter – https://redislabs.com/redis-watch-archive

37

8. Redis Loadable Modules System – http://antirez.com/news/1069. Introduction to Redis Modules API –

https://github.com/antirez/redis/blob/unstable/src/modules/INTRO.md

10. Redis Modules API reference –https://github.com/antirez/redis/blob/unstable/src/modules/API.md

11. Creating a redis Module in 15 lines of code! –https://gist.github.com/dvirsky/83fc32366d5ad82fc3dca47ed2704377