Redis — memcached on steroids

memcached on steroids

“blazingly fast” – Kirk Haines, Engineyard

Back in the day…

Salvatore Sanfilippo

Back in the day…

LLOOGG

Google Analytics + Realtime

Back in the day…

LLOOGG

Google Analytics + Realtime

“ Redis is working very well…

We never experienced any stability problem

– Salvatore Sanfilippo

Back in the day…

community growth

@simonw Simon Willison

@antirez just posted a redis feature request http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27 - a way of randomly fetching an item from a set without also deleting it 20 Oct 09 via web

Back in the day…

VMWare acquisition

A little server of awesome

http://www.flickr.com/photos/ajc1/4663140532/

man

, th

at’s

to

tally

sto

len

fro

m S

imo

n W

illis

on

Redis Manifesto

1. A DSL for Abstract Data Types

2. Memory storage is #1

3. Fundamental data structures for a fundamental API

4. Code is like a poem

5. We’re against complexity

6. Two levels of API

7. We optimize for joy

a key/value store?

A DSL For Abstract Data Types

redis> SET key hello OK redis> GET key “hello”

a key/value store?



request

a key/value store?



request

awesome, a REPL! redis-cli

on par with


redis> APPEND key world (integer) 10 redis> SETNX key bye (integer) 1 redis> GET key “helloworld” redis> DEL key (integer) 1

memcache

on par with memcache: atomic counters


redis> SET participants 10 OK redis> INCR participants (integer) 11

“It gives Memcached a serious run for its money.” – Mathias Meyer, Peritor GmbH

on par with memcache: cache invalidation


redis> EXPIRE key 60 OK redis> ...



redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis>



redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis>



redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis> ...



redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis> GET key (nil)

“ Memory is the new disk.

–– Jim Gray, Turing Award laureate

Memory storage is #1

Redis key space

http://spotfireblog.tibco.com/wp-content/uploads/in-memory-analytics.jpg


snapshotting mode

http://www.flickr.com/photos/bionicteaching/3212235059/


snapshotting mode


every x seconds every y operations on SAVE / BGSAVE


snapshotting mode


every x seconds every y operations on SAVE / BGSAVE

fork(); // copy on write!

dump(); move();


Append-Only File – Write-Ahead Logging

http://www.flickr.com/photos/generationbass/4827013488/




fsync() on every command





fsync() every second






fsync() per OS






fsync() per OS

BGREWRITEAOF trims AOF


meh, persistence


memory overhead


memory contention

?


memory contention

? Virtual Memory to the rescue


memory contention


because Salvatore is much brighter than all of Microsoft Research


memory contention



…and Redis types do not map to OS pages


memory contention



…and Redis types do not map to OS pages

all keys need to stay in memory values must be swapped in their entirety

swapped by age and size

Fundamental data structures for a fundamental API

lists, we can

redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2


lists, we can


redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql”


lists, we can




Bulk reply

lists, we can




Bulk reply

implemented as Linked Lists

lists, we can




Bulk reply

implemented as Linked Lists

PUSH POP

O(1)

a data structure server

hash tables

sets

sorted sets


lists

RENAME key newkey RENAMENX key newkey RPOP key RPOPLPUSH source destination RPUSH key value RPUSHX key value SADD key member SCARD key SDIFF key [key ...] SDIFFSTORE destination key [key ...] SELECT index SET key value SETBIT key offset value SETEX key seconds value SETNX key value SETRANGE key offset value SINTER key [key ...] SINTERSTORE destination key [key ...] SISMEMBER key member SMEMBERS key SMOVE source destination member SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] SPOP key SRANDMEMBER key SREM key member STRLEN key SUNION key [key ...] SUNIONSTORE destination key [key ...] TTL key TYPE key ZADD key score member ZCARD key ZCOUNT key min max ZINCRBY key increment member ZINTERSTORE destination numkeys key [key ...] ZRANGE key start stop [WITHSCORES] ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] ZRANK key member ZREM key member ZREMRANGEBYRANK key start stop ZREMRANGEBYSCORE key min max ZREVRANGE key start stop [WITHSCORES] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] ZREVRANK key member ZSCORE key member ZUNIONSTORE destination numkeys key [key ...]

APPEND key value BLPOP key [key ...] BRPOP key [key ...] timeout BRPOPLPUSH source destination timeout DECR key DECRBY key decrement DEL key [key ...] EXISTS key EXPIRE key seconds EXPIREAT key timestamp FLUSHALL R FLUSHDB GET key GETBIT key offset GETRANGE key start end GETSET key value HDEL key field HEXISTS key field HGET key field HGETALL key HINCRBY key field increment HKEYS key HLEN key HMGET key field [field ...] HMSET key field value [field value ...] HSET key field value HSETNX key field value HVALS key INCR key INCRBY key increment KEYS pattern LINDEX key index LINSERT key BEFORE|AFTER pivot value LLEN key LPOP key LPUSH key value LPUSHX key value LRANGE key start stop LREM key count value LSET key index value LTRIM key start stop MGET key [key ...] MOVE key db MSET key value [key value ...] MSETNX key value [key value ...] PERSIST key RANDOMKEY

a data processing server






every operation is atomic





every operation is atomic

Mind blown.

CAP & ACID promises

consistency

partition tolerance

availability durability

atomicity

isolation

consistency


“kicks ACID out the door” –Mathias Meyer, Chief Cloud Officer

CAP & ACID promises

consistency

partition tolerance

availability durability

atomicity

isolation

consistency


“kicks ACID out the door” –Mathias Meyer, Chief Cloud Officer

“that’s like saying why aren’t filesystems ACID”

Code is like a poem

C 23.000 lines of code

We’re against complexity

Two levels of APIs

transactions

redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1

Two levels of APIs

transactions


EXEC DISCARD

Two levels of APIs

transactions


atomic

Two levels of APIs

transactions


atomic

there’s always a but:

server crash = partial execution

Two levels of APIs

transactions


atomic



there’s always a but but:

detected on startup => exit

Two levels of APIs

transactions


atomic



there’s always a but but:

detected on startup => exit

there’s always a but but but:

redis-check-aof repairs it

Two levels of APIs

optimistic locking

redis> WATCH key OK redis> MULTI OK

Two levels of APIs

optimistic locking


SET key ciao <redis OK

Two levels of APIs

optimistic locking



redis> SET key bye QUEUED redis> EXEC (nil)

Two levels of APIs

optimistic locking



redis> SET key bye QUEUED redis> EXEC (nil) GET key <redis

“ciao”

We optimize for joy

Pub/Sub

redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1

We optimize for joy

Pub/Sub


type

We optimize for joy

Pub/Sub


channel

We optimize for joy

Pub/Sub

redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 message

We optimize for joy

Pub/Sub


PUBLISH chat “asl?” <redis (integer) 1

We optimize for joy

Pub/Sub



1) “message” 2) “chat” 3) “asl?” ^C

We optimize for joy

Pub/Sub



1) “message” 2) “chat” 3) “asl?” ^C

in other news:

PSUBSCRIBE UNSUBSCRIBE

“ Redis is more than just a key-value store, it’s a lifestyle.

–– Mathias Meyer

Benchmarks

redis-benchmark

$ redis-benchmark

====== SET ======

100000 requests completed in 0.88 seconds 50 parallel clients 3 bytes payload

====== GET ======

100000 requests completed in 1.23 seconds

Linux 2.6, Xeon X3320 2.5GHz, loopback

Benchmarks

Colin Howe

3.7x writes

2.1x reads

speed improvement for over MySQL

Replication

master-slave scenario

http://www.flickr.com/photos/joiearai/4393860397/

Replication



SLAVEOF host port

Replication



SLAVEOF host port

save + re-issuing

Who’s using it

Github

Resque message queue service

Configuration Management

Routing Information

Who’s using it

Digg

Digg Streaming API

personalized news data

real time view

click counts

Who’s using it

wooga

Monster World

Happy Hospital

inventory system

user system

Who’s using it

Peter Noordhuis

# Author: Peter Noordhuis # Description: Simple demo to showcase Redis PubSub with EventMachine

<170 lines of Ruby code> <220 lines of HTML+JS>

Who’s using it

WildlifeNearYou

Vanity

API rate limiting Crowdsourcing taxonomies

A/B testing

The Guardian Crowdsourced analysis of MP expenses

BNP membership list analysis

Hurl HTTP request testing

Who’s using it

WildlifeNearYou

Vanity

API rate limiting Crowdsourcing taxonomies

A/B testing

The Guardian Crowdsourced analysis of MP expenses

BNP membership list analysis

Hurl HTTP request testing

general blogosphere hype storm http://redis.io/topics/using-redis lists about 30 others

“ I think the… problem is… believing that there can be “one true datastore”.

Different technologies excel at different things.

–– Weixi Yen

Check out…

Simon Willison’s tutorial

http://simonwillison.net/static/2010/redis-tutorial

Check out…

http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up

Redis, from the Ground Up

http://try.redis-db.com/

Try Redis

coming August 2011, Salvatore & Pieter, published by O’Reilly

Redis: The Definitive Guide Data modeling, caching, and messaging

also check my Redis bookmarks: http://del.icio.us/lehmannro/redis

http://rediscookbook.org/

The Redis Cookbook

man, that was a hell of a ride.

Thanks. Questions?

Now go use Redis.io!

Redis — memcached on steroids

Technology

Transcript of Redis — memcached on steroids