1.Lessons learned: NoSQL/RedisTit Petric / Jan 2013Monotek d.o.o.

2. NoSQL: why its hereSQL:- Slow query performance- Concurrency / locking- Hard to scale (even harder for writes, storage)Typical problems- Session storage- Statistics (high write to read ratio)- Modifying schema on large data setsTit Petric / Twitter @titpetric 3. NoSQL: memcacheMemcache (RTV 2008-Present)- Pro: stability, speed- Pro: simple text protocol (added binary to fuck with us)- Love/Hate: Scaling out reads/writes- Con: Persistence- Con: Replication- Con: Key eviction not based on LRU/TTL but slab allocationTit Petric / Twitter @titpetric 4. NoSQL: sharedanceSharedance (2009-2011)- Pro: Persistent KV storage- Pro: Simple text protocol (wrote a client in LUA)- Con: Had to patch daemon to handle eviction load (1 key = 1file, filesystems cant handle this)- Con: Had to use special ReiserFS filesystem on deploymentTit Petric / Twitter @titpetric 5. NoSQL: redisRedis (2011-Present)- Pro: stability, speed- Pro: simple text protocol, documentation, clients- Love: Scaling out reads/writes- Pro: Persistence- Pro: Replication- Con/Feature request: key segmentation / grouping.Tit Petric / Twitter @titpetric 6. NoSQL: uses at RTV SloveniaTo-do (2013-?)- Memcached protocol translator to Redis- Look at twemcached, avoid client based sharding- Implement webdis deployment- Redis scripting with LUATit Petric / Twitter @titpetric 7. NoSQL: Redis data typesRedis is still a Key/Value store!Only we can have different values: - Strings (essentialy the same as memcache) - Hashes (nested k/v pairs) - Lists (simple arrays) - Sets (unique arrays) - Sorted sets (weighted unique arrays)Tit Petric / Twitter @titpetric 8. NoSQL: Redis data types LETS SEE SOME EXAMPLES! Tit Petric / Twitter @titpetric 9. Redis data types: StringsLimiting Google bot crawl rate A 503 (Service Unavailable) error will result in fairlyfrequent retrying. To temporarily suspend crawling, it is recommended to serve a 503 HTTP result code.Tit Petric / Twitter @titpetric 10. Redis data types: StringsLimiting Google bot crawl rateSETNX set a key if it doesnt existEXPIRE expire a key after TTL secondsINCR increment value by oneTit Petric / Twitter @titpetric 11. Redis data types: StringsLimiting Google bot crawl rateTit Petric / Twitter @titpetric 12. Redis data types: HashesNews ratingsHMSET set multiple hash fields / valuesHGETALL get all fields and valuesHINCRBY increment integer value of a hashfieldTit Petric / Twitter @titpetric 13. Redis data types: HashesNews ratingsTit Petric / Twitter @titpetric 14. Redis data types: HashesNews ratingsVote data Why Expire?Race condition.We need HMSETNXCould be better.Tit Petric / Twitter @titpetric 15. Redis data types: HashesOther use cases include:- User data, partial data retrieval- select username, realname, birthday from users where id=?- HMGET users:$id username realname birthday- Using SORT (list|set) BY hash values- Dont use HASHes to store session. Eviction policy works on KEYS not on hash values!Tit Petric / Twitter @titpetric 16. Redis data types: ListsAny kind of information log (statistics,)LPUSH push values to the beginning of the listRPUSH push values to the end of the listLRANGE get a range of valuesLTRIM trim a list to the specified rangeLLEN get the length of the listTit Petric / Twitter @titpetric 17. Redis data types: ListsCollecting statisticsWe can skip the database completelyTit Petric / Twitter @titpetric 18. Redis data types: ListsProcess dataInto SQLdatabaseTit Petric / Twitter @titpetric 19. Redis data types: ListsWell, its a way to scale writes to SQL Processing job can DIE for ages, because:- Back of the envelope calculation for redis memory use:100M keys use 16 GB ram- Logs get processed in small chunks (200 items), avoidingmemory limits. Could increase this by a lot.- We also use sharding so writes are distributed per $tableTit Petric / Twitter @titpetric 20. Redis data types: SetsSet values are UNIQUESADD Add one or more members to a setPerfect use case: set insersection withSINTERSTORE, find duplicates.MySQL is too slow for this, even withindexesTit Petric / Twitter @titpetric 21. Redis data types: SetsSET Intersection in MySQLList1 = first table of dataList2 = second table of dataTit Petric / Twitter @titpetric 22. Redis data types: SetsBulk transfer MySQL data to redisVia: http://dcw.ca/blog/2013/01/02/mysql-to-redis-in-one-step/Tit Petric / Twitter @titpetric 23. Redis data types: SetsSET Intersection in RedisMuch faster, without indexes!0.118 seconds vs. mysql 1.35 (+0.36 for index)15x speed increase!Tit Petric / Twitter @titpetric 24. Redis data types: SetsOther possible uses for sets: Common friends between A and B Friend suggestions (You might know) People currently online Tit Petric / Twitter @titpetric 25. Redis data types: Sets vs. Sorted setsOk, typical use case in sql>select title, content from news order bystamp desc limit 0,10#1) Use SORT from redis + HMGET#2) Use sorted sets (ZSET type)Tit Petric / Twitter @titpetric 26. Redis data types: Sorted setsSorted sets by time with a PKauto_increment? NO! Most read news items (sort by views) Order comments by comment rating Friends by most friends in commonTit Petric / Twitter @titpetric 27. Redis data types: Sorted setsOrder comments by ratingZINCRBY increase/decrease score of itemZRANGE return portion of sorted set, ASCZREVRANGE portion of sorted set, DESCTit Petric / Twitter @titpetric 28. Redis data types: Sorted setsSort comments by rating! With pagination!ZRANGE return portion of sorted set, ASCZREVRANGE portion of sorted set, DESCTit Petric / Twitter @titpetric 29. Scaling Redis deploymentSLAVEOF [host] [port]Starts replicating from [host]:[port], making this instance a slaveSLAVEOF NO ONEPromote instance to MASTER roleTit Petric / Twitter @titpetric 30. Scaling Redis deploymentPhpredis client does not implementsharding by itself! But - Master / Multi-slave scaling is easy to do- Failover for reads is easy, node ejection possible- Client deploys still take time twemproxy is an option- Twemproxy also provides sharding support, & Memcached - Want to see what Redis is doing? Issue MONITOR command.- Stale data is better than no data, we still consider Redis volatile- FlushDB = rebuild cache, we tolerate data lossTit Petric / Twitter @titpetric 31. Redis: Q & A sectionQuestions and answers! Follow me on Twitter: @titpetricRead our tech blog: http://foreach.orgTit Petric / Twitter @titpetric