Bitrix24betaversionpresentation 120420085007-phpapp02-120529180752-phpapp02
restmq-100821153418-phpapp02
-
Upload
nithin-k-anil -
Category
Documents
-
view
16 -
download
0
Transcript of restmq-100821153418-phpapp02
RestMQ
Redis based Message Queue
WHAT !?
RestMQ - Message Queue based on Redis
Data manipulation: HTTP GET/POST/DELETE, COMET and WebSockets
No signaling or specific protocol (optional JSON protocol)
http://www.restmq.com/
Gleicon Moraeshttp://github.com/gleiconhttp://zenmachine.wordpress.comhttp://www.7co.cc
Key/Value storage
Think memcached, GET and SET operations
[key] = value
>> SET key meh>> GET keymeh
>> SET fucounter 0>> INCR fucounter1>> GET fucounter1
Redis
Key-Value databaseAtomic operationsSemi-persistent or persistent storagePublish/Subscription channelsDifferent Data types: Sets, Sorted Sets, Lists, Hashes
http://code.google.com/p/redis/http://github.com/antirez/redishttp://rediscookbook.org/
Redis
Data types
StringsListsSets and Ordered SetsHashesPublish/Subscribe channels
Redis
Things you can do:
CacheInverted indexesFast countersThrottle controlCooler stuff at Redis CookbookLoad Balancing
Things you can't do:
Search inside all values for a given string
Redis
Things you can't do:
- Search inside all values for a given key- Search inside all values for a given key- Search inside all values for a given key
Message Queues
Does everything really needs to be tightly coupled ?Order: first come, first servedTransactions: there is no transaction, just trackingJob SchedulingPublish/SubscribeMap/Reduce
Message Queues
SMTP: Oldest Message Queue aroundAvoid hitting your DB real time with MQReal time: Ratings, Voting and CommentsTwitterDo I need NoSQL or I really need to clean my mess ?
RestMQ - Endpoints Diagram
RestMQ - Messages
/q/<queue> : Queue handling (GET/POST/DELETE)
$ curl -X POST -d "value=foobar" http://localhost:8888/q/testqtestq:1
$ curl -X POST -d "value=foobar" http://localhost:8888/q/testqtestq:2
$ curl http://localhost:8888/q/testq { "count": 0, "value": "foobar", "key": "testq:1" }
RestMQ - Messages
/c/<queue> : Comet Endpoint
$ curl http://localhost:8888/c/testq (hangs until there is a message to be received)
/ws/<queue> : WebSocket Endpoint
Needs the proper javascript code
RestMQ - MessagesJSON Protocol inspired by Amazon SQSFirst prototype at http://jsonqueue.appspot.com
add
{ "cmd" : "add", "queue" : "testq", "value" : "abacab" }
del
{ "cmd" : "del", "queue" : "testq", "key" : "testq:31" }
get
{ "cmd" : "get", "queue" : "testq", }
take
{ "cmd" : "take", "queue" : "testq", }
RestMQ - Status
/stats/<queue> : Queue status $ curl http://localhost:8888/stats/test
{ "queue" : "test", "redis": "127.0.0.1:6379 - 10 connection(s)", "len": {"len": 1} }
/stats : Server status $ curl http://localhost:8888/stats/
{ "count": 4, "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"], "redis": "127.0.0.1:6379 - 10 connection(s)" }
RestMQ - Controls
/p/<queue> : Distribution Policy for COMET and WS
Round Robin a message for each consumer Broadcast a message to all consumers
/control/<queue> : Queue start/stop
Pause all COMET consumers (useful for job schedulers)
RestMQ - Data structure Diagram
RestMQ - Algorithm
Algorithm for pushing messages into a queue named 'testq'
SADD testq into a SET called queuesetINCR a per-queue UUID generator called testq:uuidSET testq:<<id>>, messageLPUSH testq:<<id>> in to a LIST named testq:queue
Works for new and existing queues. On-the-fly queue creation.
RestMQ - Algorithm
Algorithm for reading messages from a queue named 'testq'
RPOP a key from a LIST named testq:queueGET the value associated to this key
For non-destructive GET operations:
LINDEX -1 to get a key in testq:queue LISTINCR key:refcount +1GET the value stored for key
RestMQ - Implementation(the core functionality can be implemented on any language)
Main branch 'Enterprise Edition'
PythonTwistedCycloneRedis async client
Embedded version
Sinatra + Ruby Reduced endpoints (GET/POST)
RestMQ - Sinatra Implementationhttp://gist.github.com/524240
GET /q - List all queues
RestMQ - Sinatra Implementationhttp://gist.github.com/524240
GET /q/<queue> - Get a message from <queue>
RestMQ - Sinatra Implementationhttp://gist.github.com/524240
POST /q/<queue> - Insert a message in <queue>
RestMQ - Map/Reduce
Questions ?
Thanks