Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is...
Transcript of Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is...
![Page 1: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/1.jpg)
Open source in-memory data structure store
![Page 2: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/2.jpg)
Features – At a glance
• Atomicity
• Transactions
• Publish/Subscribe Messaging Paradigm
![Page 3: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/3.jpg)
Atomic Operations
Atomicity stipulates that it should be possible to place every operation at a singular point (linearization point) between its invocation and response1.
![Page 4: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/4.jpg)
Atomic Operations – Explained
Atomicity gives us a guarantee that only one of two things can happen before anything else happens:
1.The operation succeeds and we are left with the expected result
2.The operation fails and no changes are made to the underlying
data
Every command in Redis is performed atomically!
![Page 5: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/5.jpg)
Redis Transactions
The execution of a group of commands in a single step, with two important guarantees:
1.All the commands are serialized and executed sequentially.
2.Either all of the commands or none are processed.
Does this look familiar? Redis Transactions are atomic!
![Page 6: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/6.jpg)
Redis Transactions – Specifics
MULTI: Enters “transaction mode” where we can now list operations
EXEC: Executes the transaction, which now contains multiple operations
DISCARD: Flushes the transaction queue and exits the transaction
![Page 7: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/7.jpg)
PubSub – Redis
Redis allows you to easily implement the Publish/Subscribe Messaging Paradigm.
• SUBSCRIBE
• UNSUBSCRIBE
• PUBLISH
![Page 8: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/8.jpg)
PubSub – Redis
In Redis, messages are organized into channels.
Very robust channel subscribing features:
SUBSCRIBE news.*
SUBSCRIBE news.art.figurative SUBSCRIBE news.music.jazz
![Page 9: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/9.jpg)
Data Model
Key-Value Store
Vs
In-Memory Data Structure Store
![Page 10: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/10.jpg)
Data Model - Basics
• Key-Value pairs
• Value usually string (other datatypes possible)
Key
CIS4930_171F
CIS4301
CEN3031
Value
Advanced Database
Info and Database Sys 1
Intro Software Engr
![Page 11: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/11.jpg)
Data Model – RDBMS vs Key-Value
ID Title Cred
CIS4930_171F Advanced Database 3
CIS4301 Info and Database Sys 1 3
CEN3031 Intro Software Engr 3
Key
Course:CIS4930_171F:Title
Course:CIS4301:Title
Course:CEN3031:Title
Value
Advanced Database
Info and Database Sys 1Intro Software Engr
Key
Course:CIS4930_171F:Cred
Course:CIS4301:Cred
Course:CEN3031:Cred
Value
3
3
3
![Page 12: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/12.jpg)
Data Model – Reverse Index
• Easy way to model relationships
• Reverse index acts as foreign keyKey
Course:CIS4930_171F:Students
Course:CEN3031:Students
Value
[Student:0, Student:10, …]
[Student:10, Student:13, …]
Key
Student:0:Courses
Student:10:Courses
Value
[Course:CIS4930_171F, …]
[Course:CIS4930_171F, Course:CEN3031, …]
![Page 13: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/13.jpg)
Data Structures - Overview
• Strings
• Lists
• Sets
• Sorted Sets
• Hashes
• HyperLogLogs
![Page 14: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/14.jpg)
Data structures - Strings
• Base type
• Maximum length is 512 MB
• Implemented using C-Strings (Append O(n) operation)
• Binary safe
• Can be used as:• Number (Increment, etc.)
• Vector
• Binary Data (Bitwise operations)
![Page 15: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/15.jpg)
Data Structures - Lists
• Implemented as a doubly linked list
• O(1) push and pop operations
• O(n) accessing other nodes• Can be used to build Queues or Stacks• Maximum elements = 2^32-1= 4,294,967,295• Can block retrieval until value is available (BLPOP)
![Page 16: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/16.jpg)
Data Structures - Sets
• Unordered collection of Strings
• Implemented using a hash table• O(1) checking for existence
• Set Operations• Union, Intersection and Difference
• No duplicates allowed
• Maximum Number of Elements = 4,294,967,295
• Useful to represent relationships
• Randomized retrieval of members possible
![Page 17: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/17.jpg)
Data Structures – Sorted Sets
• Ordered collection of Strings
• Implemented using a skip list• O(log(n)) operations
• No duplicates allowed
• Each member has a rank
• Retrieval by range possible
• Can be used to index key-value pairs for range retrieval
![Page 18: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/18.jpg)
Data Structures - Hashes
• Maps between String fields
• Implemented using a hash table• O(1) retrieval/existence operations
• Used to represent objects
• Maximum Number of Elements = 4,294,967,295
![Page 19: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/19.jpg)
Data Structures - HyperLogLogs
• Probabilistic data structure• HyperLogLog is a well known algorithm
• Used to estimate cardinality
• Trades memory for precision
![Page 20: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/20.jpg)
Data Store Comparisons
![Page 21: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/21.jpg)
Persistence
Two kinds of persistence:
• Append only files
• Redis database files
![Page 22: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/22.jpg)
Append Only Files (AOF)
Pros:
Readability
No seeks or corruption problems if there’s an outage
Cons:
Larger than RDB files
Slower than RDB files
![Page 23: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/23.jpg)
Redis Database File (RDB)
Pros:
Great for disaster recovery
Small and compact
Cons:
Poor if you need to minimize the change of data loss in case Redis stops working
Needs to fork() often to persist on disk using child process
![Page 24: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/24.jpg)
Replication
![Page 25: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/25.jpg)
Scaling
![Page 26: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/26.jpg)
Partitioning
Advantages
• Can handle larger data sets
• Can scale to multiple cores and computers
• Not limited to single CPU’s RAM
Disadvantages
• Can’t partition if operations involve multiple keys
• Increased data handling complexity
• Altering capacity becomes difficult
![Page 27: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/27.jpg)
How to Partition
3 Methods:
1. Client Side Partitioning
2. Proxy Assisted Partitioning
3. Query Routing
![Page 28: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/28.jpg)
Partitioning Implementations
• Redis Cluster
• Twemproxy
• Clients supporting consistent hashing
![Page 29: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/29.jpg)
Partitioning Implementations
• Redis Cluster
• Twemproxy
• Clients supporting consistent hashing
Automatically splits dataset among multiple nodes
Will still operating when some of the nodes are failing
![Page 30: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/30.jpg)
Partitioning Implementations
• Redis Cluster
• Twemproxy
• Clients supporting consistent hashing
Is a proxy between the clients and Redis instances
Can automatically shard data among instances
Supports consistent hashing
![Page 31: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/31.jpg)
Partitioning Implementations
• Redis Cluster
• Twemproxy
• Clients supporting consistent hashing
Simply use a client that implements client side partitioning (via consistent hashing)
Examples: Redis-rb, Predis
![Page 32: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/32.jpg)
Redis
Common Operations and Query Language
![Page 33: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/33.jpg)
Popular Storage Commands
• SET
• APPEND
• PUSH
<SET> <Key> <Value>
Sets the value to the corresponding key. If key already exists, it is overwritten.
Returns String “OK” if successful, NULL otherwise.
![Page 34: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/34.jpg)
Popular Storage Commands
• SET
• APPEND
• PUSH
<APPEND> <Key> <Value>
Appends the value to the corresponding key. If key doesn’t exist, it is created.
Returns Integer corresponding to new size of Value.
![Page 35: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/35.jpg)
Popular Storage Commands
• SET
• APPEND
• PUSH
<PUSH> <Key> <Value>
Appends the value to the corresponding key. If key doesn’t exist, it is created. Key must be a list.
Returns Integer corresponding to new size of Value.
![Page 36: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/36.jpg)
Popular Retrieval Commands
• GET
• MGET
<GET> <Key>
Returns the value for the corresponding key.
Returns value of key or NULL if key does not exist.
![Page 37: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/37.jpg)
Popular Retrieval Commands
• GET
• MGET
<MGET> <Key> [Keys…]
Returns all values for all the corresponding keys.
Returns values of keys in an array. Cell in array is NULL if corresponding key doesn’t exist.
![Page 38: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/38.jpg)
Other Popular Commands
• DEL
• RENAME
• EXISTS
• DBSIZE
<DEL> <Key> [Keys…]
Deletes all keys and associated values.
Returns the number of keys deleted.
![Page 39: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/39.jpg)
Other Popular Commands
• DEL
• RENAME
• EXISTS
• DBSIZE
<RENAME> <Key> [newKey]
Renames key to newKey.
Returns error if key does not exist. If newKey already exists it is overwritten.
![Page 40: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/40.jpg)
Other Popular Commands
• DEL
• RENAME
• EXISTS
• DBSIZE
<EXISTS> <Key>
Determines whether or not key exists in database.
Returns True if key exists, False otherwise.
![Page 41: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/41.jpg)
Other Popular Commands
• DEL
• RENAME
• EXISTS
• DBSIZE
<DBSIZE>
Returns number of keys in the database.
![Page 42: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/42.jpg)
Redis supports different languages
![Page 43: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/43.jpg)
Java-specific initialization
• Client to access Redis is called “Jedis”
• Jedis jedis = new Jedis(String host, int port)
![Page 44: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/44.jpg)
Java-specific commands
• jedis.set(“key1”, “abc”)
• jedis.get(“key2”)
String “abc” is now associated with String “key1”.
![Page 45: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/45.jpg)
Java-specific commands
• jedis.set(“key1”, “abc”)
• jedis.get(“key2”)
Returns value associated with key2 if it exists.
![Page 46: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/46.jpg)
Companies that use Redis
• Github
• Craigslist
• Digg
• Amazon (AWS)
Uses Redis to find a user’s route, defined to be “the hostname of the file server on which that user’s repositories are kept.”
![Page 47: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/47.jpg)
Companies that use Redis
• Github
• Craigslist
• Digg
• Amazon (AWS)
Uses Redis to map hostname to port numbers of different users.
![Page 48: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/48.jpg)
Companies that use Redis
• Github
• Craigslist
• Digg
• Amazon (AWS)
Uses Redis to keep track of page views and clicks.
“Redis rocks”
-Digg
![Page 49: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/49.jpg)
Companies that use Redis
• Github
• Craigslist
• Digg
• Amazon (AWS)
Amazon Web Services (AWS) ElastiCache uses Redis.
My experience.
![Page 50: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/50.jpg)
Other Companies that use Redis
• Snapchat
• Uber
• Slack
• Imgur
• Grooveshark
• Airbnb
• Tumblr
![Page 51: Open source in-memory data structure storemschneid/Teaching/CIS... · Every command in Redis is performed atomically! Redis Transactions The execution of a group of commands in a](https://reader035.fdocuments.in/reader035/viewer/2022062914/5e772f4f29db8c1f9b08f209/html5/thumbnails/51.jpg)
Sources1:https://www.cs.cornell.edu/~ie53/publications/icDSN12.pdf
2:http://redis.io/topics/data-types
3:http://stackoverflow.com/questions/9625246/what-are-the-underlying-data-structures-used-for-redis
http://blog.avangardo.com/2013/12/comparison-of-most-popular-nosql-dbmses/
https://www.cl.cam.ac.uk/research/srg/opera/publications/papers/vargasbaconmoody_integrating.pdf