Message Passing and the PubSub Pattern - cs.montana.edu · All computing involves communication -...
Transcript of Message Passing and the PubSub Pattern - cs.montana.edu · All computing involves communication -...
Message Passing and thePubSub Pattern
November 25, 2017
Joe DeBruycker, Rohan Khante
Montana State University
Message Passing
Why is it important?
• All computing involves communication - between processes,machines, users, nodes, etc.
◦ Local: In-process, interprocess (shared memory)◦ Network: Unicast, multicast (protocols)
• Many of our interesting problems involve how to consistentlytransfer information
• 1-to-1 isn’t too hard, but it gets a lot more interesting N-to-M
Message
• An abstraction for any piece of information being transportedfrom one place to another
• Requires a Sender and a Receiver
PubSub DeBruycker-Khante November 25, 2017 2 / 17
Message Passing
PubSub DeBruycker-Khante November 25, 2017 3 / 17
Message Passing
Messages
• Modular - Sender and Receiver are black boxes with acommon defined interface
1 Ease of use/understanding2 Plug-and-play3 Helps debugging
Connectivity
• Spectrum from 1:1 to M:N
• Client-server to Peer-to-peer
PubSub DeBruycker-Khante November 25, 2017 4 / 17
Message Passing
Message Patterns
• Messaging protocols define rules for encoding and decodingmessages
◦ Examples: HTTP, TCP, UDP, WebSub
• Messaging patterns describe how sender and receiverconnect and communicate
◦ Examples: Request-reply, fan-out, task distribution (workqueues, competing consumers),pub-sub
PubSub DeBruycker-Khante November 25, 2017 5 / 17
Publish-Subscribe
Pattern
• One-to-many (or many-to-many)
• One-way communication from publisher to subscriber
PubSub DeBruycker-Khante November 25, 2017 6 / 17
Publish-Subscribe
Applications
• Error logging
• News feeds
• Change APIs
Topologies
• Message Queues - middleware brokers
• Direct connections (i.e. sockets)
PubSub DeBruycker-Khante November 25, 2017 7 / 17
Publish-Subscribe Message Filtering Types
Topic Based
• Subs get all messages published to the channel they sub to
• Publisher is responsible for posting to channels
Content Based• Subs only get those messages which match the filters they set
• Subscriber is responsible for filtering
Hybrid
• Publishers post to topic while subs have a content basedsystem
PubSub DeBruycker-Khante November 25, 2017 8 / 17
Publish-Subscribe
Pros
• Non-blocking (asynchronous)
• Decoupled architecture
• Scalable...to a point
Cons• One way communication
• No message delivery guarantees without extra overhead
• Very large networks increase frequency of instabilities◦ Load-surges: bursty traffic◦ Slowdowns: too many publishers, subscribers can’t keep up
PubSub DeBruycker-Khante November 25, 2017 9 / 17
Database Implementations
Redis• subscribe < key >
• publish < key > < message >
PostgresQL
• Called Asynchronous Notifications
• LISTEN, UNLISTEN
• NOTIFY
Change APIs
• Alternative to polling to get the changes
• CouchDB, RethinkDB
PubSub DeBruycker-Khante November 25, 2017 10 / 17
Cloud Pub-sub Services
Google Cloud Pub/Sub
• Topic based model. Sub to a channel
• Highly scalable with easy addition of servers
• Firebase API
Amazon Simple Notification Service
• Notifications are pushed to clients
• Cloud based middleware
• Added features for mobile apps
PubSub DeBruycker-Khante November 25, 2017 11 / 17
WebSub (AKA PubSubHubbub)
Pub-Sub Protocol• Open source protocol for distributed pub-sub via HTTP
• Goal is real-time change notifications without polling
• Consumers request content from a server, and if it referencesa ”hub” they can subscribe to receive updates as long as theyrun a server
• Supported types - Any HTTP supported type, e.g. text,audio, video, HTML
• Notable ”pushers” - CNN, FOX, Myspace.com, WordPress
PubSub DeBruycker-Khante November 25, 2017 12 / 17
ZeroMQ
Overview• Asynchronous messaging library
• Bindings for multiple languages
• Primary protocol is TCP, also supports various others
Psuedo-sockets• Binding vs connecting to a port
• Socket context - 0MQ’s magic
• No middleware
• Functionality baked into ZMQ Socket Types
PubSub DeBruycker-Khante November 25, 2017 13 / 17
ZeroMQ - PubSub
PubSub DeBruycker-Khante November 25, 2017 14 / 17
0MQ - Server Code
import zmq
from random import randrange
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:5556")
while True:
zipcode = randrange(1, 100000)
temperature = randrange(-80, 135)
relhumidity = randrange(10, 60)
socket.send_string("%i %i %i" % (zipcode, temperature, relhumidity))
PubSub DeBruycker-Khante November 25, 2017 15 / 17
0MQ - Client Code
import sys
import zmq
# Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.SUB)
print("Collecting updates from weather server")
socket.connect("tcp://localhost:5556")
# Subscribe to zipcode, default is NYC, 10001
zip_filter = sys.argv[1] if len(sys.argv) > 1 else "10001"
# Python 2 - ascii bytes to unicode str
if isinstance(zip_filter, bytes):
zip_filter = zip_filter.decode(’ascii’)
socket.setsockopt_string(zmq.SUBSCRIBE, zip_filter)
# Process 5 updates
total_temp = 0
for update_nbr in range(5):
string = socket.recv_string()
zipcode, temperature, relhumidity = string.split()
total_temp += int(temperature)
print("Average temperature for zipcode ’%s’ was %dF" % (
zip_filter, total_temp / (update_nbr+1))
)
PubSub DeBruycker-Khante November 25, 2017 16 / 17
ZeroMQ
Exercises• How many messages does the server send before the client
receives 5 messages from the same topic (in this case, ZipCode)?
◦ Hint: zip filter = ”” subscribes to all messages
• How long does it take on your computer before the subscriberreceives all 5 messages?
◦ Hint: import time
• What order do messages arrive on the client when it issubscribed to multiple publishers?
◦ Hint: A socket can make multiple connect calls
PubSub DeBruycker-Khante November 25, 2017 17 / 17