MongoUK 2011 - Rplacing RabbitMQ with MongoDB
-
Upload
boxed-ice -
Category
Technology
-
view
18.553 -
download
3
Transcript of MongoUK 2011 - Rplacing RabbitMQ with MongoDB
![Page 1: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/1.jpg)
Replacing RabbitMQ with MongoDB
![Page 2: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/2.jpg)
David Mytton
Woop Japan!
![Page 3: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/3.jpg)
![Page 4: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/4.jpg)
www.flickr.com/photos/triplexpresso/496995086/
Queuing: Uses
![Page 5: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/5.jpg)
www.flickr.com/photos/triplexpresso/496995086/
Queuing: Uses
• Background processing
![Page 6: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/6.jpg)
www.flickr.com/photos/triplexpresso/496995086/
Queuing: Uses
• Background processing
• Sending notifications
![Page 7: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/7.jpg)
www.flickr.com/photos/triplexpresso/496995086/
Queuing: Uses
• Background processing
• Sending notifications
• Event streaming
![Page 8: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/8.jpg)
Queuing: Features
![Page 9: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/9.jpg)
Queuing: Features
• Consumers
![Page 10: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/10.jpg)
Queuing: Features
• Atomic
• Consumers
![Page 11: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/11.jpg)
Queuing: Features
• Speed
• Atomic
• Consumers
![Page 12: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/12.jpg)
Queuing: Features
• Speed
• Atomic
• Consumers
• GC
![Page 13: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/13.jpg)
Queuing: Features
•Consumers
![Page 14: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/14.jpg)
Queuing: Features
•Consumers
MongoDB RabbitMQ
Mongo Wire Protocol AMQP
![Page 15: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/15.jpg)
en.wikipedia.org/wiki/State_of_matter
Queuing: Features
•Atomic
![Page 16: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/16.jpg)
en.wikipedia.org/wiki/State_of_matter
Queuing: Features
•Atomic
MongoDB RabbitMQ
findAndModify consume/ack
![Page 17: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/17.jpg)
Queuing: Features
•Speed
![Page 18: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/18.jpg)
Queuing: Features
•GC
![Page 19: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/19.jpg)
Queuing: Features
•GC
MongoDB RabbitMQ
☹ consume/ack
![Page 20: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/20.jpg)
It’s a little different.
![Page 21: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/21.jpg)
Implementation
• Consumers
![Page 22: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/22.jpg)
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
![Page 23: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/23.jpg)
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
{ query: { hats: 5 } }
query: filter (WHERE)
![Page 24: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/24.jpg)
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
{ sort: { added: -1 } }
sort: selects the first one on multi-match
![Page 25: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/25.jpg)
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
{ update: { $set: {inProg: true, start: new Date()} } }
update: modifier object
![Page 26: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/26.jpg)
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
remove: true = deletes on returnnew: true = returns modified objectfields: return specific fieldsupsert: true = create object if !exists()
![Page 27: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/27.jpg)
Implementation
• Consumers
![Page 28: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/28.jpg)
Implementation
• Consumers
def run(self): import time, sched sc = sched.scheduler(time.time, time.sleep) self.poll(sc) sc.run()
def poll(self, sc): doStuff() sc.enter(10, 1, self.poll, (sc,))
![Page 29: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/29.jpg)
Implementation
• Consumers
![Page 30: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/30.jpg)
Implementation
• Consumers
from daemon import Daemon class pantalaimon(Daemon): def run(self): # Do stuff
pineMarten = pantalaimon('/path/to/pid.pid') pineMarten.start()
https://github.com/boxedice/python-daemon
![Page 31: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/31.jpg)
Implementation
• Consumers
https://github.com/boxedice/python-daemon
from daemon import Daemon
class pantalaimon(Daemon): def run(self): import time, sched sc = sched.scheduler(time.time, time.sleep) self.poll(sc) sc.run()
def poll(self, sc): doStuff() sc.enter(10, 1, self.poll, (sc,))
![Page 32: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/32.jpg)
Implementation
• GC
![Page 33: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/33.jpg)
Implementation
• GC
now = datetime.datetime.now()difference = datetime.timedelta(seconds=10)timeout = now - difference
queue.find({'inProg' : True, 'start' : {'$lte' : timeout} })
![Page 34: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/34.jpg)
Fault tolerance
Replica sets
![Page 35: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/35.jpg)
Fault tolerance
Replica sets
Basic automatic failover
![Page 36: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/36.jpg)
Fault tolerance
Replica sets
Basic automatic failover
Writing to n slaves
![Page 37: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/37.jpg)
Fault tolerance
Replica sets
Basic automatic failover
Writing to n slaves
Data centre awareness
![Page 38: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/38.jpg)
Scaling
Capped Collections
![Page 39: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/39.jpg)
Scaling
Capped Collections
Extremely fast
![Page 40: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/40.jpg)
Scaling
Capped Collections
Maintain insertion order (FIFO)
Extremely fast
![Page 41: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/41.jpg)
Scaling
Capped Collections
Don’t have to remove()
Maintain insertion order (FIFO)
Extremely fast
![Page 42: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/42.jpg)
Scaling
Sharding
![Page 43: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/43.jpg)
Scaling
Sharding
findAndModify() locks
![Page 44: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/44.jpg)
Scaling
Sharding
Scaling writes across machines
findAndModify() locks
![Page 45: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/45.jpg)
Scaling
Sharding
![Page 46: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/46.jpg)
Scaling
Sharding
Not with capped collections
![Page 47: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/47.jpg)
Problems / Mongo quirks
![Page 48: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/48.jpg)
Problems / Mongo quirks
Meta
![Page 49: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/49.jpg)
Problems / Mongo quirks
Meta
Durability
![Page 50: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/50.jpg)
Problems / Mongo quirks
Meta
Durability
Global lock
![Page 51: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/51.jpg)
Stick with RabbitMQ?
![Page 52: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/52.jpg)
Stick with RabbitMQ?
QoS
![Page 53: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/53.jpg)
Stick with RabbitMQ?
AMQP
QoS
![Page 54: MongoUK 2011 - Rplacing RabbitMQ with MongoDB](https://reader035.fdocuments.in/reader035/viewer/2022081514/559473b11a28abd8548b4675/html5/thumbnails/54.jpg)
AMQP
Stick with RabbitMQ?
Throttling
QoS