Post on 06-May-2015
description
memcached on steroids
“blazingly fast” – Kirk Haines, Engineyard
Back in the day…
Salvatore Sanfilippo
Back in the day…
Salvatore Sanfilippo
Back in the day…
LLOOGG
Google Analytics + Realtime
Back in the day…
LLOOGG
Google Analytics + Realtime
Back in the day…
LLOOGG
Google Analytics + Realtime
Back in the day…
LLOOGG
Google Analytics + Realtime
“ Redis is working very well…
We never experienced any stability problem
– Salvatore Sanfilippo
Back in the day…
community growth
@simonw Simon Willison
@antirez just posted a redis feature request http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27 - a way of randomly fetching an item from a set without also deleting it 20 Oct 09 via web
Back in the day…
community growth
@simonw Simon Willison
@antirez just posted a redis feature request http://groups.google.com/group/redis-db/.../cd7e7c65dba53e27 - a way of randomly fetching an item from a set without also deleting it 20 Oct 09 via web
Back in the day…
VMWare acquisition
A little server of awesome
http://www.flickr.com/photos/ajc1/4663140532/
man
, th
at’s
to
tally
sto
len
fro
m S
imo
n W
illis
on
Redis Manifesto
1. A DSL for Abstract Data Types
2. Memory storage is #1
3. Fundamental data structures for a fundamental API
4. Code is like a poem
5. We’re against complexity
6. Two levels of API
7. We optimize for joy
a key/value store?
A DSL For Abstract Data Types
redis> SET key hello OK redis> GET key “hello”
a key/value store?
A DSL For Abstract Data Types
redis> SET key hello OK redis> GET key “hello”
request
a key/value store?
A DSL For Abstract Data Types
redis> SET key hello OK redis> GET key “hello”
request
awesome, a REPL! redis-cli
on par with
A DSL For Abstract Data Types
redis> APPEND key world (integer) 10 redis> SETNX key bye (integer) 1 redis> GET key “helloworld” redis> DEL key (integer) 1
memcache
on par with memcache: atomic counters
A DSL For Abstract Data Types
redis> SET participants 10 OK redis> INCR participants (integer) 11
“It gives Memcached a serious run for its money.” – Mathias Meyer, Peritor GmbH
on par with memcache: cache invalidation
A DSL For Abstract Data Types
redis> EXPIRE key 60 OK redis> ...
on par with memcache: cache invalidation
A DSL For Abstract Data Types
redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis>
on par with memcache: cache invalidation
A DSL For Abstract Data Types
redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis>
on par with memcache: cache invalidation
A DSL For Abstract Data Types
redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis> ...
on par with memcache: cache invalidation
A DSL For Abstract Data Types
redis> EXPIRE key 60 OK redis> TTL key (integer) 53 redis> GET key “helloworld” redis> GET key (nil)
“ Memory is the new disk.
–– Jim Gray, Turing Award laureate
Memory storage is #1
Redis key space
http://spotfireblog.tibco.com/wp-content/uploads/in-memory-analytics.jpg
Memory storage is #1
snapshotting mode
http://www.flickr.com/photos/bionicteaching/3212235059/
Memory storage is #1
snapshotting mode
http://www.flickr.com/photos/bionicteaching/3212235059/
every x seconds every y operations on SAVE / BGSAVE
Memory storage is #1
snapshotting mode
http://www.flickr.com/photos/bionicteaching/3212235059/
every x seconds every y operations on SAVE / BGSAVE
fork(); // copy on write!
dump(); move();
Memory storage is #1
Append-Only File – Write-Ahead Logging
http://www.flickr.com/photos/generationbass/4827013488/
Memory storage is #1
Append-Only File – Write-Ahead Logging
http://www.flickr.com/photos/generationbass/4827013488/
fsync() on every command
Memory storage is #1
Append-Only File – Write-Ahead Logging
http://www.flickr.com/photos/generationbass/4827013488/
fsync() on every command
fsync() every second
Memory storage is #1
Append-Only File – Write-Ahead Logging
http://www.flickr.com/photos/generationbass/4827013488/
fsync() on every command
fsync() every second
fsync() per OS
Memory storage is #1
Append-Only File – Write-Ahead Logging
http://www.flickr.com/photos/generationbass/4827013488/
fsync() on every command
fsync() every second
fsync() per OS
BGREWRITEAOF trims AOF
Memory storage is #1
meh, persistence
http://www.flickr.com/photos/generationbass/4827013488/
memory overhead
Memory storage is #1
memory contention
?
Memory storage is #1
memory contention
?
Memory storage is #1
memory contention
? Virtual Memory to the rescue
Memory storage is #1
memory contention
? Virtual Memory to the rescue
because Salvatore is much brighter than all of Microsoft Research
Memory storage is #1
memory contention
? Virtual Memory to the rescue
because Salvatore is much brighter than all of Microsoft Research
…and Redis types do not map to OS pages
Memory storage is #1
memory contention
? Virtual Memory to the rescue
because Salvatore is much brighter than all of Microsoft Research
…and Redis types do not map to OS pages
all keys need to stay in memory values must be swapped in their entirety
swapped by age and size
Fundamental data structures for a fundamental API
lists, we can
redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2
Fundamental data structures for a fundamental API
lists, we can
redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2
redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql”
Fundamental data structures for a fundamental API
lists, we can
redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2
redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql”
Fundamental data structures for a fundamental API
Bulk reply
lists, we can
redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2
redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql”
Fundamental data structures for a fundamental API
Bulk reply
implemented as Linked Lists
lists, we can
redis> LPUSH databases mysql (integer) 1 redis> LPUSH databases mongodb (integer) 2
redis> LRANGE databases 0 -1 1) “mongodb” 2) “mysql”
Fundamental data structures for a fundamental API
Bulk reply
implemented as Linked Lists
PUSH POP
O(1)
a data structure server
hash tables
sets
sorted sets
Fundamental data structures for a fundamental API
lists
a data structure server
hash tables
sets
sorted sets
Fundamental data structures for a fundamental API
lists
a data structure server
hash tables
sets
sorted sets
Fundamental data structures for a fundamental API
lists
RENAME key newkey RENAMENX key newkey RPOP key RPOPLPUSH source destination RPUSH key value RPUSHX key value SADD key member SCARD key SDIFF key [key ...] SDIFFSTORE destination key [key ...] SELECT index SET key value SETBIT key offset value SETEX key seconds value SETNX key value SETRANGE key offset value SINTER key [key ...] SINTERSTORE destination key [key ...] SISMEMBER key member SMEMBERS key SMOVE source destination member SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] SPOP key SRANDMEMBER key SREM key member STRLEN key SUNION key [key ...] SUNIONSTORE destination key [key ...] TTL key TYPE key ZADD key score member ZCARD key ZCOUNT key min max ZINCRBY key increment member ZINTERSTORE destination numkeys key [key ...] ZRANGE key start stop [WITHSCORES] ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] ZRANK key member ZREM key member ZREMRANGEBYRANK key start stop ZREMRANGEBYSCORE key min max ZREVRANGE key start stop [WITHSCORES] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] ZREVRANK key member ZSCORE key member ZUNIONSTORE destination numkeys key [key ...]
APPEND key value BLPOP key [key ...] BRPOP key [key ...] timeout BRPOPLPUSH source destination timeout DECR key DECRBY key decrement DEL key [key ...] EXISTS key EXPIRE key seconds EXPIREAT key timestamp FLUSHALL R FLUSHDB GET key GETBIT key offset GETRANGE key start end GETSET key value HDEL key field HEXISTS key field HGET key field HGETALL key HINCRBY key field increment HKEYS key HLEN key HMGET key field [field ...] HMSET key field value [field value ...] HSET key field value HSETNX key field value HVALS key INCR key INCRBY key increment KEYS pattern LINDEX key index LINSERT key BEFORE|AFTER pivot value LLEN key LPOP key LPUSH key value LPUSHX key value LRANGE key start stop LREM key count value LSET key index value LTRIM key start stop MGET key [key ...] MOVE key db MSET key value [key value ...] MSETNX key value [key value ...] PERSIST key RANDOMKEY
a data processing server
Fundamental data structures for a fundamental API
RENAME key newkey RENAMENX key newkey RPOP key RPOPLPUSH source destination RPUSH key value RPUSHX key value SADD key member SCARD key SDIFF key [key ...] SDIFFSTORE destination key [key ...] SELECT index SET key value SETBIT key offset value SETEX key seconds value SETNX key value SETRANGE key offset value SINTER key [key ...] SINTERSTORE destination key [key ...] SISMEMBER key member SMEMBERS key SMOVE source destination member SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] SPOP key SRANDMEMBER key SREM key member STRLEN key SUNION key [key ...] SUNIONSTORE destination key [key ...] TTL key TYPE key ZADD key score member ZCARD key ZCOUNT key min max ZINCRBY key increment member ZINTERSTORE destination numkeys key [key ...] ZRANGE key start stop [WITHSCORES] ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] ZRANK key member ZREM key member ZREMRANGEBYRANK key start stop ZREMRANGEBYSCORE key min max ZREVRANGE key start stop [WITHSCORES] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] ZREVRANK key member ZSCORE key member ZUNIONSTORE destination numkeys key [key ...]
APPEND key value BLPOP key [key ...] BRPOP key [key ...] timeout BRPOPLPUSH source destination timeout DECR key DECRBY key decrement DEL key [key ...] EXISTS key EXPIRE key seconds EXPIREAT key timestamp FLUSHALL R FLUSHDB GET key GETBIT key offset GETRANGE key start end GETSET key value HDEL key field HEXISTS key field HGET key field HGETALL key HINCRBY key field increment HKEYS key HLEN key HMGET key field [field ...] HMSET key field value [field value ...] HSET key field value HSETNX key field value HVALS key INCR key INCRBY key increment KEYS pattern LINDEX key index LINSERT key BEFORE|AFTER pivot value LLEN key LPOP key LPUSH key value LPUSHX key value LRANGE key start stop LREM key count value LSET key index value LTRIM key start stop MGET key [key ...] MOVE key db MSET key value [key value ...] MSETNX key value [key value ...] PERSIST key RANDOMKEY
a data processing server
Fundamental data structures for a fundamental API
every operation is atomic
RENAME key newkey RENAMENX key newkey RPOP key RPOPLPUSH source destination RPUSH key value RPUSHX key value SADD key member SCARD key SDIFF key [key ...] SDIFFSTORE destination key [key ...] SELECT index SET key value SETBIT key offset value SETEX key seconds value SETNX key value SETRANGE key offset value SINTER key [key ...] SINTERSTORE destination key [key ...] SISMEMBER key member SMEMBERS key SMOVE source destination member SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] SPOP key SRANDMEMBER key SREM key member STRLEN key SUNION key [key ...] SUNIONSTORE destination key [key ...] TTL key TYPE key ZADD key score member ZCARD key ZCOUNT key min max ZINCRBY key increment member ZINTERSTORE destination numkeys key [key ...] ZRANGE key start stop [WITHSCORES] ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] ZRANK key member ZREM key member ZREMRANGEBYRANK key start stop ZREMRANGEBYSCORE key min max ZREVRANGE key start stop [WITHSCORES] ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] ZREVRANK key member ZSCORE key member ZUNIONSTORE destination numkeys key [key ...]
APPEND key value BLPOP key [key ...] BRPOP key [key ...] timeout BRPOPLPUSH source destination timeout DECR key DECRBY key decrement DEL key [key ...] EXISTS key EXPIRE key seconds EXPIREAT key timestamp FLUSHALL R FLUSHDB GET key GETBIT key offset GETRANGE key start end GETSET key value HDEL key field HEXISTS key field HGET key field HGETALL key HINCRBY key field increment HKEYS key HLEN key HMGET key field [field ...] HMSET key field value [field value ...] HSET key field value HSETNX key field value HVALS key INCR key INCRBY key increment KEYS pattern LINDEX key index LINSERT key BEFORE|AFTER pivot value LLEN key LPOP key LPUSH key value LPUSHX key value LRANGE key start stop LREM key count value LSET key index value LTRIM key start stop MGET key [key ...] MOVE key db MSET key value [key value ...] MSETNX key value [key value ...] PERSIST key RANDOMKEY
a data processing server
Fundamental data structures for a fundamental API
every operation is atomic
Mind blown.
CAP & ACID promises
consistency
partition tolerance
availability durability
atomicity
isolation
consistency
Fundamental data structures for a fundamental API
“kicks ACID out the door” –Mathias Meyer, Chief Cloud Officer
CAP & ACID promises
consistency
partition tolerance
availability durability
atomicity
isolation
consistency
Fundamental data structures for a fundamental API
“kicks ACID out the door” –Mathias Meyer, Chief Cloud Officer
“that’s like saying why aren’t filesystems ACID”
Code is like a poem
C 23.000 lines of code
We’re against complexity
Two levels of APIs
transactions
redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
Two levels of APIs
transactions
redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
EXEC DISCARD
Two levels of APIs
transactions
redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
atomic
Two levels of APIs
transactions
redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
atomic
there’s always a but:
server crash = partial execution
Two levels of APIs
transactions
redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
atomic
there’s always a but:
server crash = partial execution
there’s always a but but:
detected on startup => exit
Two levels of APIs
transactions
redis> MULTI OK redis> GET key QUEUED redis> DEL key QUEUED redis> EXEC 1) “helloworld” 2) (integer) 1
atomic
there’s always a but:
server crash = partial execution
there’s always a but but:
detected on startup => exit
there’s always a but but but:
redis-check-aof repairs it
Two levels of APIs
optimistic locking
redis> WATCH key OK redis> MULTI OK
Two levels of APIs
optimistic locking
redis> WATCH key OK redis> MULTI OK
SET key ciao <redis OK
Two levels of APIs
optimistic locking
redis> WATCH key OK redis> MULTI OK
SET key ciao <redis OK
redis> SET key bye QUEUED redis> EXEC (nil)
Two levels of APIs
optimistic locking
redis> WATCH key OK redis> MULTI OK
SET key ciao <redis OK
redis> SET key bye QUEUED redis> EXEC (nil) GET key <redis
“ciao”
We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1
We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1
type
We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1
channel
We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1 message
We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1
PUBLISH chat “asl?” <redis (integer) 1
We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1
PUBLISH chat “asl?” <redis (integer) 1
1) “message” 2) “chat” 3) “asl?” ^C
We optimize for joy
Pub/Sub
redis> SUBSCRIBE chat Reading messages… (press Ctrl-C to quit) 1) “subscribe” 2) “chat” 3) (integer) 1
PUBLISH chat “asl?” <redis (integer) 1
1) “message” 2) “chat” 3) “asl?” ^C
in other news:
PSUBSCRIBE UNSUBSCRIBE
“ Redis is more than just a key-value store, it’s a lifestyle.
–– Mathias Meyer
Benchmarks
redis-benchmark
$ redis-benchmark
====== SET ======
100000 requests completed in 0.88 seconds 50 parallel clients 3 bytes payload
====== GET ======
100000 requests completed in 1.23 seconds
Linux 2.6, Xeon X3320 2.5GHz, loopback
Benchmarks
Colin Howe
3.7x writes
2.1x reads
speed improvement for over MySQL
Replication
master-slave scenario
http://www.flickr.com/photos/joiearai/4393860397/
Replication
master-slave scenario
http://www.flickr.com/photos/joiearai/4393860397/
Replication
master-slave scenario
http://www.flickr.com/photos/joiearai/4393860397/
Replication
master-slave scenario
http://www.flickr.com/photos/joiearai/4393860397/
SLAVEOF host port
Replication
master-slave scenario
http://www.flickr.com/photos/joiearai/4393860397/
SLAVEOF host port
save + re-issuing
Who’s using it
Github
Resque message queue service
Configuration Management
Routing Information
Who’s using it
Digg
Digg Streaming API
personalized news data
real time view
click counts
Who’s using it
wooga
Monster World
Happy Hospital
inventory system
user system
Who’s using it
Peter Noordhuis
# Author: Peter Noordhuis # Description: Simple demo to showcase Redis PubSub with EventMachine
<170 lines of Ruby code> <220 lines of HTML+JS>
Who’s using it
WildlifeNearYou
Vanity
API rate limiting Crowdsourcing taxonomies
A/B testing
The Guardian Crowdsourced analysis of MP expenses
BNP membership list analysis
Hurl HTTP request testing
Who’s using it
WildlifeNearYou
Vanity
API rate limiting Crowdsourcing taxonomies
A/B testing
The Guardian Crowdsourced analysis of MP expenses
BNP membership list analysis
Hurl HTTP request testing
general blogosphere hype storm http://redis.io/topics/using-redis lists about 30 others
“ I think the… problem is… believing that there can be “one true datastore”.
Different technologies excel at different things.
–– Weixi Yen
Check out…
Simon Willison’s tutorial
http://simonwillison.net/static/2010/redis-tutorial
Check out…
http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up
Redis, from the Ground Up
http://try.redis-db.com/
Try Redis
coming August 2011, Salvatore & Pieter, published by O’Reilly
Redis: The Definitive Guide Data modeling, caching, and messaging
also check my Redis bookmarks: http://del.icio.us/lehmannro/redis
http://rediscookbook.org/
The Redis Cookbook
man, that was a hell of a ride.
Thanks. Questions?
Now go use Redis.io!