Zmq in context of openstack
-
Upload
yatinkumbhare -
Category
Technology
-
view
654 -
download
0
description
Transcript of Zmq in context of openstack
ZMQ / 0MQ / ZeroMQ -Introduction
● Zero Broker, Zero latency, zero cost - culture of minimalism.
● Brokerless - No SPOF
● High Scalability
● Flexible: inproc, ipc, tcp
● Order of bind and connect - Doesn’t matter
● Fast and reliable
REQ-REP# Client
import zmq
context = zmq.Context()
print("Connecting to hello world server…")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
for request in range(10):
print("Sending request %s …" % request)
socket.send("Hello")
# Get the reply.
message = socket.recv()
print("Received reply %s [ %s ]" %
(request, message))
# Server
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to hello world server…")
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
print("waiting for client request")
msg = socket.recv()
print("Message received %s “ % msg)
message = socket.send("World")
PUB-SUBimport zmq
from random import randint
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:5556")
while True:
zipcode = randint(1, 10)
temperature = randint(-20, 50)
humidity = randint(10, 15)
print “Publish data:”,(zipcode,
temperature, humidity)
socket.send("%d %d %d" % (zipcode,
temperature, relhumidity))
import zmq
import sys
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:5556")
zip_filter = sys.argv[1] if len(sys.argv) > 1
else “”
print “Collectin updates from weather service
%s” % zip_filter
socket.setsockopt(zmq.SUBSCRIBE, zip_filter)
#process 5 records
for record in range(5):
data = socket.recv()
print data
PUB-SUB
PUSH-PULL# Ventilator# socket to send messages onsender = context.socket(zmq.PUSH)sender.bind(“tcp://*:5557” )
sink = context.socket(zmq.PUSH)sink.connect(“tcp://localhost:5558” )sink.send(“0”)
# Workers:receiver = context.socket(zmq.PULL)
receiver.connect("tcp://localhost:5557" )
# Socket to send messages to to sink
sender = context.socket(zmq.PUSH)
sender.connect("tcp://localhost:5558" )
# Sink:# Socket to receive messages on
receiver = context.socket(zmq.PULL)
receiver.bind("tcp://*:5558" )
# Wait for start of batch
s = receiver.recv()
Single-Point-Failure
rpc.cast
rpc.call
ZMQ for Openstack
● In openstack, zeromq makes use of two protocols, TCP and IPC.
● TCP Socket, for sending message from nova services to rpc-zmq-receiver service.
● IPC Socket, for forwarding message received by rpc-zmq-receiver to the local nova services, which is listening
on IPC socket.
● ZMQ uses PUB-SUB and PUSH-PULL socket types.
● PUB-SUB - to get reply, in case of rpc.call, this work as direct-consumer/publisher in context of rabbitmq.
● PUSH-PULL - for sending messages to other services, for example rpc.cast.
zmq-receiver service● rpc-zmq-receiver service with different sockets.
● Instantiate ZmqProxy class from oslo.messaging, impl_zmq.
py file.
● Service creates TCP://*:9501, PULL socket, and listen for
messages from other services.
● Topic: fanout~service or zmq_replies.<hostname>, zmq.PUB
socket
● Topic: service.<hostname>, (conductor.node) zmq.PUSH socket
is created.
● The socket protocol will be IPC.
● Here, data is received on TCP://*:9501 and forwarded to IPC
socket.
Nova-* services● Instantiate ZmqReactor from oslo.messaging, impl_zmq.py
file.
● Mostly the nova service manages with zmq.PUSH, zmq.PULL
and zmq.SUB socket type.
● For consuming messages nova service uses IPC sockets with
zmq.PULL, and to cast a message it creates TCP, zmq.PUSH
socket.
● example: when zmq-receiver, publish message on ipc:
///var/run/openstack/zmq_topic_conductor.node, with PUSH
socket, which is consumed by ipc:
///var/run/openstack/zmq_topic_conductor.node with PULL
socket at conductor service.
Nova-compute● topic: fanout~compute, bind to ipc:// , zmq.SUB socket
● topic: compute.<hostname>, bind to ipc:// , zmq.PULL
● nova compute does rpc.call on conductor at the start of the
service.
● The hostname for conductor service is read out from
MatchMakerRing file.
● compute service, creates tcp://<conductor-hostname>:9501,
PUSH socket and send message, which is received by zmq-
receiver, and forwarded to nova-conductor service.
● As, this is rpc.call, to receive reply, nova compute create, ipc:
///var/run/openstack/zmq_topic_zmq_replies.<hostname>
with SUB socket type and subscribe to msg-id (unique uuid),
and await for response from conductor.
MatchMakerRing file{ "conductor": [ "controller" ], "scheduler": [ "controller" ], "compute": [ "controller","computenode" ], "network": [ "controller","computenode" ], "zmq_replies": [ "controller","computenode" ],
"cert": [ "controller" ], "cinder-scheduler": [ "controller" ], "cinder-volume": [ "controller" ], "consoleauth": [ "controller" ] }
Message routing'conductor' {'args': {'service': {u'binary':
u'nova-compute',
u'created_at': u'2014-02-
19T11:02:32.000000',
u'deleted': 0,
u'deleted_at': None,
u'disabled': False,
u'disabled_reason': None,
u'host': u'nodex',
u'id': 2,
u'report_count': 22,
u'topic': u'compute',
u'updated_at': u'2014-02-
19T11:08:10.000000'},
'values': {'report_count': 23}},
'method': 'service_update',
'namespace': None,
'version': '1.34'}
compute make rpc.call to conductor.Sends data on tcp://gravity:9501 with PUSH, which is received by zmq-receiver service of gravity host with topic:conductor.gravity
before rpc.call, include following data● reply-topic as zmq_replies.nodex
● create unique msg_id,
● extra key "method":"-reply"
To receive reply on topic:zmq_replies.nodex, creates socket ipc://...zmq_topic_zmq_replies.nodex with SUB and subscribe to msg_id.
The conductor consume data and takes out reply-topic as
zmq_replies.nodex
Conductor sends back the response by creating tcp://nodex:
9501 with PUSH and send data.
Message routingu'zmq_replies.nodex' {'args': {'msg_id':
u'103b95cc64ab4e39924b6240a8dbaac8',
'response': [[{'binary': u'nova-compute',
'created_at': '2014-02-
19T11:02:32.000000',
'deleted': 0L,
'deleted_at': None,
'disabled': False,
'disabled_reason': None,
'host': u'nodex',
'id': 2L,
'report_count': 23,
'topic': u'compute',
'updated_at': '2014-02-
19T11:08:33.572453'}]]},
'method': '-process_reply'}
● nodex zmq-receiver service, receiver data and
check for topic, based on topic which is
zmq_replies.nodex, create ipc://....
/zmq_topic_zmq_replies.nodex with PUB and
send data.
● Now, the compute already has reply socket ipc://
with SUB, and is already subscribe to msg_id,
which will get back the response of rpc.call.
Devstack with zmq#localrc
● ENABLED_SERVICES+=,-rabbit,-qpid,zeromq #controller
● ENABLED_SERVICES=n-cpu,n-net,zeromq #compute node
Apply Patch: https://review.openstack.org/#/c/59875/3
Due to an issue with zeromq support for notifications in Glance, glance-api fails to start when configured to use ZeroMQ.
Commnet line in lib/glance in devstack repository
#iniset_rpc_backend glance $GLANCE_API_CONF DEFAULT