Redis/Lessons learned

Click here to load reader

download Redis/Lessons learned

of 31

Embed Size (px)


An insight into NoSQL solutions implemented at RTV Slovenia and elsewhere, what problems we are trying to solve and an introduction to solving them with Redis. Talk given at #wwwh @ Ljubljana, 30.1.2013 by me, Tit Petric

Transcript of Redis/Lessons learned

  • 1.Lessons learned: NoSQL/RedisTit Petric / Jan 2013Monotek d.o.o.

2. NoSQL: why its hereSQL:- Slow query performance- Concurrency / locking- Hard to scale (even harder for writes, storage)Typical problems- Session storage- Statistics (high write to read ratio)- Modifying schema on large data setsTit Petric / Twitter @titpetric 3. NoSQL: memcacheMemcache (RTV 2008-Present)- Pro: stability, speed- Pro: simple text protocol (added binary to fuck with us)- Love/Hate: Scaling out reads/writes- Con: Persistence- Con: Replication- Con: Key eviction not based on LRU/TTL but slab allocationTit Petric / Twitter @titpetric 4. NoSQL: sharedanceSharedance (2009-2011)- Pro: Persistent KV storage- Pro: Simple text protocol (wrote a client in LUA)- Con: Had to patch daemon to handle eviction load (1 key = 1file, filesystems cant handle this)- Con: Had to use special ReiserFS filesystem on deploymentTit Petric / Twitter @titpetric 5. NoSQL: redisRedis (2011-Present)- Pro: stability, speed- Pro: simple text protocol, documentation, clients- Love: Scaling out reads/writes- Pro: Persistence- Pro: Replication- Con/Feature request: key segmentation / grouping.Tit Petric / Twitter @titpetric 6. NoSQL: uses at RTV SloveniaTo-do (2013-?)- Memcached protocol translator to Redis- Look at twemcached, avoid client based sharding- Implement webdis deployment- Redis scripting with LUATit Petric / Twitter @titpetric 7. NoSQL: Redis data typesRedis is still a Key/Value store!Only we can have different values: - Strings (essentialy the same as memcache) - Hashes (nested k/v pairs) - Lists (simple arrays) - Sets (unique arrays) - Sorted sets (weighted unique arrays)Tit Petric / Twitter @titpetric 8. NoSQL: Redis data types LETS SEE SOME EXAMPLES! Tit Petric / Twitter @titpetric 9. Redis data types: StringsLimiting Google bot crawl rate A 503 (Service Unavailable) error will result in fairlyfrequent retrying. To temporarily suspend crawling, it is recommended to serve a 503 HTTP result code.Tit Petric / Twitter @titpetric 10. Redis data types: StringsLimiting Google bot crawl rateSETNX set a key if it doesnt existEXPIRE expire a key after TTL secondsINCR increment value by oneTit Petric / Twitter @titpetric 11. Redis data types: StringsLimiting Google bot crawl rateTit Petric / Twitter @titpetric 12. Redis data types: HashesNews ratingsHMSET set multiple hash fields / valuesHGETALL get all fields and valuesHINCRBY increment integer value of a hashfieldTit Petric / Twitter @titpetric 13. Redis data types: HashesNews ratingsTit Petric / Twitter @titpetric 14. Redis data types: HashesNews ratingsVote data Why Expire?Race condition.We need HMSETNXCould be better.Tit Petric / Twitter @titpetric 15. Redis data types: HashesOther use cases include:- User data, partial data retrieval- select username, realname, birthday from users where id=?- HMGET users:$id username realname birthday- Using SORT (list|set) BY hash values- Dont use HASHes to store session. Eviction policy works on KEYS not on hash values!Tit Petric / Twitter @titpetric 16. Redis data types: ListsAny kind of information log (statistics,)LPUSH push values to the beginning of the listRPUSH push values to the end of the listLRANGE get a range of valuesLTRIM trim a list to the specified rangeLLEN get the length of the listTit Petric / Twitter @titpetric 17. Redis data types: ListsCollecting statisticsWe can skip the database completelyTit Petric / Twitter @titpetric 18. Redis data types: ListsProcess dataInto SQLdatabaseTit Petric / Twitter @titpetric 19. Redis data types: ListsWell, its a way to scale writes to SQL Processing job can DIE for ages, because:- Back of the envelope calculation for redis memory use:100M keys use 16 GB ram- Logs get processed in small chunks (200 items), avoidingmemory limits. Could increase this by a lot.- We also use sharding so writes are distributed per $tableTit Petric / Twitter @titpetric 20. Redis data types: SetsSet values are UNIQUESADD Add one or more members to a setPerfect use case: set insersection withSINTERSTORE, find duplicates.MySQL is too slow for this, even withindexesTit Petric / Twitter @titpetric 21. Redis data types: SetsSET Intersection in MySQLList1 = first table of dataList2 = second table of dataTit Petric / Twitter @titpetric 22. Redis data types: SetsBulk transfer MySQL data to redisVia: Petric / Twitter @titpetric 23. Redis data types: SetsSET Intersection in RedisMuch faster, without indexes!0.118 seconds vs. mysql 1.35 (+0.36 for index)15x speed increase!Tit Petric / Twitter @titpetric 24. Redis data types: SetsOther possible uses for sets: Common friends between A and B Friend suggestions (You might know) People currently online Tit Petric / Twitter @titpetric 25. Redis data types: Sets vs. Sorted setsOk, typical use case in sql>select title, content from news order bystamp desc limit 0,10#1) Use SORT from redis + HMGET#2) Use sorted sets (ZSET type)Tit Petric / Twitter @titpetric 26. Redis data types: Sorted setsSorted sets by time with a PKauto_increment? NO! Most read news items (sort by views) Order comments by comment rating Friends by most friends in commonTit Petric / Twitter @titpetric 27. Redis data types: Sorted setsOrder comments by ratingZINCRBY increase/decrease score of itemZRANGE return portion of sorted set, ASCZREVRANGE portion of sorted set, DESCTit Petric / Twitter @titpetric 28. Redis data types: Sorted setsSort comments by rating! With pagination!ZRANGE return portion of sorted set, ASCZREVRANGE portion of sorted set, DESCTit Petric / Twitter @titpetric 29. Scaling Redis deploymentSLAVEOF [host] [port]Starts replicating from [host]:[port], making this instance a slaveSLAVEOF NO ONEPromote instance to MASTER roleTit Petric / Twitter @titpetric 30. Scaling Redis deploymentPhpredis client does not implementsharding by itself! But - Master / Multi-slave scaling is easy to do- Failover for reads is easy, node ejection possible- Client deploys still take time twemproxy is an option- Twemproxy also provides sharding support, & Memcached - Want to see what Redis is doing? Issue MONITOR command.- Stale data is better than no data, we still consider Redis volatile- FlushDB = rebuild cache, we tolerate data lossTit Petric / Twitter @titpetric 31. Redis: Q & A sectionQuestions and answers! Follow me on Twitter: @titpetricRead our tech blog: http://foreach.orgTit Petric / Twitter @titpetric