The ZeroMQ Guide - for Python Developers - MQ/2.2 - The Guide
zeromq
-
Upload
rajan-bhatt -
Category
Documents
-
view
69 -
download
4
Transcript of zeromq
What is ZeroMQ?
Rajan Bhatt1/10/2014
What are we going to cover?
• What is Messaging? • ZeroMQ v/s other Messaging solution• Basics of ZeroMQ• Coding examples • Quiz
What is Enterprise Messaging ?
• Enterprise messaging is a design pattern for loosely coupled system to communicate asynchronously.
• This type of architecture is foundation for “shared-nothing” model.
• Great example of this model in computation is also known as “Actor Model”.
Broad classification of Messaging System
• Broker based Messaging System. – AMQP based system, Implementation RabbitMQ,
ActiveMQ– JMS – Java Message Service.– STOMP – The Simple Text Oriented Messaging
Protocol– MQTT – Light Weight protocol for Sensor networks
• Broker-less Messaging system – ZeroMQ, Nanomsg etc.
AMQP – RabbitMQ
• RabbitMQ is open source message broker software that implements the Advanced Message Queuing protocol. This is written in Erlang language.
• Basic components of AMQP Producer, Exchange, Queue and Consumer
Basic AMQP Model
Publisher
RoutesExchange
Queue
Queue
Consumer n
RoutesPublishes
Consumes Consumer 1
Consumes
In Nutshell - ZeroMQ
• Not MOM ( no broker, no queues, etc..)• “Super” Socket library that acts as concurrency framework• Carries messages across various transport, inproc, IPC, TCP
and multicast.• Connect N-to-N via fan-out, pub-sub, pipeline. Request-reply.• Asynchronous I/O for scalable multicore message-passing
apps.• 40+ languages including C. C++, JAVA, .NET, Python.
In Nutshell - ZeroMQ..
• Is a library that supports certain network communication patterns using socket.
• The “MQ” part comes in because ZeroMQ uses queues internally to buffer messages so that you don’t block your application when sending data.
• When you say, socket.send(…), ZeroMQ actually enqueues a messages to be sent later by a dedicated communication thread.
In nutshell - ZeroMQ
• It does not have overhead of an over-engineered protocol such AMQP
• It can make use of efficient transports such as reliable multicast or Y-suite IPC ytansport
• It makes use of intelligent message batching. This allows ZMQ to efficiently utilize a TCP/IP connection by minimizing not only protocol overhead but system calls.
ZeroMQ – Message Encapsulation
• Encapsulates communication into Messages which may be composed of multiple parts.
• Rather than asking ZeroMQ to receive certain number od f bytes from socket, you ask ZeroMQ to receive a single message.
ZeroMQ - Scalability
• A single ZeroMQ socket can for example connect to multiple end points and automatically load balance messages over them or it can work as some sort of Fan-in, collecting messages from sources through a single socket.
• ZeroMQ follows brokerless design so there is no single point of failure. Combine this with the simplicity and performance and you get something that you can use to make your application distributed.
ZeroMQ Socket – Request- Reply Pattern
• Most basic pattern is client/server model, where a client sends a request and server replies to that request.
• But…• Request type of socket can connect to many
servers and request would be distributes to all connected server.
ZeroMQ Request-Reply Socket Pattern depiction
Zmq client Zmq servers
REQ REP
REP
Request # 1
Request # 2
Reply #1
Reply # 2
Requests are load balanced across server
REP
REP
Demo
• ZMQ Request/Reply Pattern
ZeroMQ – Pub-Sub Pattern
– Publish/Subscribe is classic pattern where senders of messages, called publisher do not program the messages to be sent directly to specific receives, called subscribers. Messages are published without knowledge of what or if any subscribes exists.
Pub-Sub Socket Depiction
ZeroMQ Bus
Pub Socket-1
Sub Socket-3Sub Socket-2Sub Socket-1
Pub Socket-2
Publisher sockets are unaware of number of Subscribers. Subscriber socket can subscribe to multiple of Publisher Socket.
Demo
• ZMQ Publisher/Subscriber Demo
ZeroMQ Socket – Push/Pull Pattern
• Push and Pull sockets let you distribute messages to multiple workers arranged in pipeline. A Push socket will distribute sent messages to its all Pull clients evenly. This is equivalent to producer/consumer model but the results computed by consumers are not sent upstream but downstream to another pull/consumer socket.
ZeroMQ Push-Pull Pattern
Producer Consumers Result Collector
PUSH
PULL
PULL
PULL
PUSH
PUSH
PUSH
PULL
Demo
• ZMQ Push/Pull Socket demo
ZMQ Devices
• ZMQ – sockets are either bind or connect• General pattern is stable part (server) bind
and dynamic part connects to it.
• What do you do when both parts are dynamic ?
• Enter ZMQ device
Types of Devices..
• Queue• Forwarder• Streamer
Client
Client
Client
ZMQ Device
Server
Server
Server
Moving Part Moving PartStable Part
These devises can bind to 2 different ports and forward messages from one end toThe other. The forwarding device can become stable point in your network whereEach component can connect to.
Queue Device
REQ
REQ
REQ
REP
REP
REP
Queue
XREP XREQ
This is the intermediary that sits between clients and servers, forwarding request to server and relaying replies back to client. The ZMQ device takes (ZMQ.QUEUE) and The two sockets bound to well known ports. This is Request/Response broker.
Forwarding Device
PUB
PUB
PUB
SUB
SUB
SUB
Forwarder
XREP XREQ
Just like Queue, Which is like the request-reply broker, For forwarder like the pub-sub Proxy server. It allows both publishers and subscribers to be moving parts and it self becomes stable hub interconnecting them. FORWARDER collects messages from set ofPublishers to and forwards these to a set of subscribers.
Who is using ZeroMQ in production
• Storm platform – communication channel between spouts and bolt.
• Loggly – Cloud based logging platform• Dotcloud – Open source RPC platform called
ZeroRPC based on ZeroMQ messaging.• This is very popular messaging framework for
high frequency financial trading.
Which Messaging is better ? Brokered or Broker-less
• Both messaging patterns have advantages and disadvantages.
• In Enterprise, we see both being used to solve messaging problem which are multidimensional.
• ZeroMQ out-performs other messaging solution in performance, as it is bare-bone framework
Which are other Interesting Open Source Messaging project ?
• Kafka – LinkedIn’s open source message bus, Apache project, used by Tweeter also and many many companies
• NullMQ – Operates over WebSocket, on communication protocol based on STOMP. ZeroMQ semantics in Web Browser.
• NanoMSG-
I know you have questions…
• I may not have answer.. But please ask..• I will find out answers If I don’t know…
Please provide feedback
• [email protected].. • Have Fun with ZeroMQ and all messaging..