Speed up your Symfony2 application and build awesome features with Redis
-
Upload
ricard-clau -
Category
Software
-
view
2.464 -
download
3
description
Transcript of Speed up your Symfony2 application and build awesome features with Redis
![Page 1: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/1.jpg)
SPEED UP YOUR SYMFONY2 APPLICATION AND BUILD
AWESOME FEATURES WITH REDIS
![Page 2: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/2.jpg)
HELLO WORLD
• Ricard Clau, born and grown up in Barcelona
• Server engineer at Another Place Productions
• Symfony2 lover and PHP believer (sometimes…)
• Open-source contributor, sometimes I give talks
• Twitter @ricardclau / Gmail [email protected]
![Page 3: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/3.jpg)
AGENDA
• REDIS introduction, data types and commands
• Getting started with Redis in PHP / Symfony2
• Options out of the box in SncRedisBundle
• Cookbooks implemented in Real World Applications
• Sharding data and the new Redis Cluster
![Page 4: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/4.jpg)
INTRODUCTION TO REDISAnd some demystifications
![Page 5: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/5.jpg)
WHAT IS REDIS?
• REmote DIctionary Server
• Created in 2009 by Salvatore Sanfilipo (@antirez)
• Open source (https://github.com/antirez/redis)
• Advanced in-memory key-value data-structure server
• Part of the NoSQL movement, think back to structures!
![Page 6: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/6.jpg)
TO ME REDIS IS LIKE...
![Page 7: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/7.jpg)
ONLY IN-MEMORY?
• Your data needs to fit in your memory
• It has configurable persistence: RDB snapshots, AOF persistence logs, combine both or no persistence at all
• Think like memcached on steroids with persistence
• http://redis.io/topics/persistence
• “Memory is the new disk, disk is the new tape”
![Page 8: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/8.jpg)
INTERESTING FEATURES
• Master-slave replication
• Pipelining to improve performance
• Transactions (kind of with MULTI ... EXEC)
• Scripting with LUA (~stored procedures)
• Iterators for Keyspaces and SETS and HASHES (>= 2.8)
!
![Page 10: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/10.jpg)
NO QUERIES
• We communicate with Redis via commands
• No indexes, no schemas, just structures under a key
• Very good documentation and sandbox
• All commands have their complexity documented
![Page 11: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/11.jpg)
DATA TYPES
• Strings (binary-safe) up to 512 Mb
• Lists of elements sorted by insertion order (max 2^32 - 1)
• Sets supporting unions, intersections and diffs (max 2^32 - 1)
• Hashes key-value pairs (max 2^32 - 1)
• Sorted sets automatically ordered by score (max 2^32 - 1)
• HyperLogLog to compute cardinality of BIG sets (2^64)
![Page 12: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/12.jpg)
STRINGS COMMANDS
• GET, SET, STRLEN, APPEND
• GETRANGE, SETRANGE
• Bitmap operations: GETBIT, SETBIT, BITCOUNT
• Counter operations: INCR, DECR, INCRBY, DECRBY
• If the value is a number, Redis tries to store it efficiently
![Page 13: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/13.jpg)
LISTS COMMANDS
• Order of insertion matters
• Easy to implement Queues and Stacks
• RPUSH, LPUSH, RPOP, LPOP
• Blocking versions BLPOP, BRPOP, BRPOPLPUSH with timeout
Key S1 S2 S3 SN…
![Page 14: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/14.jpg)
SETS COMMANDS
• Collection of unique unordered elements
• SCARD (Size), SADD, SREM, SISMEMBER, SMEMBERS
• Intersections, Unions and Diffs: SINTER, SUNION, SDIFF, SINTERSTORE, SUNIONSTORE, SDIFFSTORE
• Iterators with SSCAN (Redis >= 2.8)
KeyS1
S5
S3
S2S4
![Page 15: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/15.jpg)
HASHES COMMANDS
• Many key-value pairs under the same key
• HSET, HGET, HGETALL, HLEN, HEXISTS
• Counters HINCR, HINCRBY, HINCRBYFLOAT
• Iterators with HSCAN (Redis >= 2.8)
KeyK1 V1
K2 V2
K3 V3
K4 V4
![Page 16: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/16.jpg)
SORTED SETS COMMANDS
• Collection of unique elements ordered by score
• ZCARD (Size), ZADD, ZREM, ZSCORE
• Ranking: ZRANGE, ZREVRANGE (optional WITHSCORES)
• Iterators with ZSCAN (Redis >= 2.8)
Key S2 - 20S3 - 15
S4 - 1000S1 - 0
![Page 17: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/17.jpg)
HYPERLOGLOG COMMANDS
• Counting unique things in massive data sets using a very small amount of memory
• PFADD, PFCOUNT, PFMERGE
• HyperLogLog has an standard error of 0.81%
• PF prefix in honor of Philippe Flajolet
![Page 18: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/18.jpg)
GETTING STARTEDEasy steps
![Page 19: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/19.jpg)
PHP CLIENTS
• Predis (PHP) vs phpredis (PHP extension)
• Predis is very mature, actively maintained, feature complete, extendable, composer friendly and supports all Redis versions
• Phpredis is faster, but not always backwards compatible
• Network latency is the biggest performance killer
• http://redis.io/clients
![Page 20: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/20.jpg)
EASY STEPS (I)
$ composer require snc/redis-bundle 1.1.*
$ composer require predis/predis 0.8.*
or install the phpredis extension https://github.com/nicolasff/phpredis
![Page 21: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/21.jpg)
EASY STEPS (II)
Add the bundle to the Symfony2 kernel
![Page 22: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/22.jpg)
EASY STEPS (III)
Add some magic to your config.yml
![Page 23: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/23.jpg)
EASY STEPS (IV)
You can now use Redis as a Symfony2 service! !
In your controller extending Symfony\Bundle\FrameworkBundle\Controller\Controller.php
![Page 24: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/24.jpg)
EASY STEPS (V)Or inject it to your services via your services.xml
![Page 25: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/25.jpg)
REDIS-CLI AND MONITOR
• Redis-cli to connect to a Redis instance
• Experiment with http://redis.io/commands
• With Monitor we can watch live activity!
![Page 26: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/26.jpg)
SNCREDISBUNDLESo many options out of the box!
![Page 27: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/27.jpg)
CLIENTS EVERYWHERE
![Page 28: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/28.jpg)
PHP SESSION SUPPORT!
TTL is the session lifetime, Redis expires it for you Session Locking implementation (not optimal)
Added not so many months ago
![Page 29: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/29.jpg)
LOCKING PROBLEM• Problem in heavy-write applications: 2 almost concurrent
requests trying to update same records
• Concurrent AJAX requests for the same user
timeline
R1 R2
R1read
R2save
R2read
R1save
R1 updates are lost!!!!!
![Page 30: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/30.jpg)
DOCTRINE CACHINGImplemented with SET / SETEX / GET
Be careful with expirations!
No expiring Possibly better
use APC
entities data (annotations / xml / yaml)
DQL parsing
RecordSets Setting TTL explicitly
![Page 31: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/31.jpg)
MONOLOG (I)Implemented with LISTS using RPUSH
Be careful with memory!
![Page 32: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/32.jpg)
MONOLOG (II)And add a new monolog handler using the service
Level recommended at least warning
![Page 33: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/33.jpg)
SWIFTMAILER SPOOLImplemented with LISTS using RPUSH
Serialized Swift_Mime_Messages in the list
![Page 34: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/34.jpg)
COOKBOOKSFrom Real World applications
![Page 35: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/35.jpg)
REDIS AS A QUEUE
• Using Lists and LPUSH / RPOP instructions
• We can have different languages accessing data
• I used this to send OpenGraph requests to Facebook (REALLY slow API) with Python daemons
• If you need some advanced message queuing features check RabbitMQ (@cakper last month´s talk) and others
![Page 36: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/36.jpg)
REAL-TIME DASHBOARDS
• We can use hashes to group many stats under one key
• Most of the times we have counters: HINCRBY “stats" <key> <increment>
• HMGET “stats” <key1> <key2> ... to retrieve values and HMSET “stats” “stat1” <value1> “stat2” <value2> to set them
• HGETALL to get the full hash
![Page 37: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/37.jpg)
LEADERBOARDS
• Super-easy with Redis, heavy consuming with other systems
• Each board with a single key, using sorted sets
• ZINCRBY “rankXXX” <increment> <userId>
• Top N ZREVRANGE “rankXXX” 0 N [WITHSCORES]
• You can store several millions of members under 1 key
![Page 38: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/38.jpg)
WHO IS ONLINE? (I)
• We can use SETS to achieve it!
• One set for every minute, SADD <minute> <userId>Time +1m +2m +3m
2
2
7
2
3
1
4 1
1
13
5
SUNION
2
13
5
7
4
![Page 39: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/39.jpg)
WHO IS ONLINE? (II)
• Online users == everyone active in the last N minutes
• SUNION <now> <now-1> <now-2> ...
• SUNIONSTORE “online” <now> <now-1> <now-2> ...
• Number of users: SCARD “online”
• Obtain online users with SMEMBERS “online”
![Page 40: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/40.jpg)
FRIENDS ONLINE?
• If we store user´s friends in a SET with key friends-<user-id>
• Friends online: SINTERSTORE “friends-<userid>-online” “online” “friends-<userid>”
!
!
!
• Imagine how you would do it without Redis!
7
13
5
2
4
15
93
10
7
1
ON
LINE
FRIE
ND
S-12
SINTE
R
FRIENDS-12-ONLINE
![Page 41: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/41.jpg)
SHARDING / CLUSTERINGThis is when it gets tricky…
![Page 42: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/42.jpg)
NEEDS TO FIT IN MEMORY
• Redis Cluster will be available in 3.0 (currently RC1)
• Proxy servers: Twemproxy (also for memcached)
• Client-side partitioning (supported in many clients)
• Sharding strategies (range vs hash partitioning)
• Presharding can help scaling horizontally
![Page 43: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/43.jpg)
CONFIGS WARNING!!
The default behaviour is RandomDistributionStrategy! PHPRedis has a RedisArray class for client-side sharding
but this is not supported yet in the bundle!
![Page 44: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/44.jpg)
RANGE PARTITIONINGMay work in some cases, not a silver bullet
1-99999 100000-199999
200000-299999 300000-399999
id % 4 == 0 id % 4 == 1
id % 4 == 2 id % 4 == 3
Easy to predict scale Load not properly distributed
Load distributed ok Fairly easy to reshard
![Page 45: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/45.jpg)
HASH PARTITIONINGBetter distribution... but still issues If few keys, it also becomes useless
hash(key) % 4 == 0 hash(key) % 4 == 1
hash(key) % 4 == 2 hash(key) % 4 == 3
Distribution also depending on hash algorithm Complicated resharding
HASH CRC16 CRC32 MD5 SHA1
MURMUR3 !
DIST MODULUS KETAMA
![Page 46: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/46.jpg)
PRESHARDINGMaybe overengineered
Many Redis instances in one initial server
!When needed, just split it into more servers and you
don´t need resharding!!! !
Be careful configuring memory limits
![Page 47: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/47.jpg)
REDIS CLUSTER
• Shards the dataset among N nodes
• Has a responsive failover in order to survive certain failures (this was partially covered already with Sentinel)
• Ability to reshard keys internally
• Neither CP nor AP, eventually consistent, not very resistant to network partitions in some scenarios
• Many disagree with some of the decisions and tradeoffs
![Page 48: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/48.jpg)
FINAL THOUGHTSWhen and when not to consider Redis
![Page 49: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/49.jpg)
REDIS IS PERFECT FOR...
• Intensive read-write data applications
• Temporary stored data
• Data that fits in memory
• Problems that fit Redis built-in data types
• Predictability in performance needed
![Page 50: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/50.jpg)
BUT IS NOT SUITABLE WHEN..
• Big data sets, archive data
• Relational data (RDBMS are absolutely fine to scale)
• We don´t know how we will access data
• Reporting applications (no where clauses)
• It gets tricky for distributed applications (replication, clustering)
• ALWAYS choose the right tool for the job!
![Page 51: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/51.jpg)
SPECIAL THANKS
• Ronny López (@ronnylt) & Alonso Vidales (@alonsovidales)
• Salvatore Sanfilippo (@antirez)
• Henrik Westphal (https://github.com/snc)
• Danielle Alessandri (https://github.com/nrk)
• Nicolas Favre-Felix (https://github.com/nicolasff)
• Of course, to all of you for being here today!
![Page 52: Speed up your Symfony2 application and build awesome features with Redis](https://reader034.fdocuments.in/reader034/viewer/2022042601/553a3dfa4a79596f718b4a30/html5/thumbnails/52.jpg)
QUESTIONS?
• Twitter: @ricardclau
• E-mail: [email protected]
• Github: https://github.com/ricardclau
• Blog about PHP and Symfony2: http://www.ricardclau.com