Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)
-
Upload
james-titcumb -
Category
Software
-
view
226 -
download
1
description
Transcript of Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)
![Page 1: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/1.jpg)
Practical Message Queueing Using
RabbitMQJames Titcumb
PHPem3rd July 2014
![Page 2: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/2.jpg)
James Titcumbwww.jamestitcumb.comwww.protected.co.ukwww.phphants.co.uk@asgrim
Who is this guy?
![Page 3: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/3.jpg)
Who are you?
https://www.flickr.com/photos/akrabat/10168019755/
![Page 4: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/4.jpg)
What is message queueing?
![Page 5: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/5.jpg)
Separation of Concerns
![Page 6: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/6.jpg)
Scaling with Rabbit
RabbitMQApplication
Background processing
![Page 7: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/7.jpg)
Scaling with Rabbit
RabbitMQApplication
Background processing
Background processing
![Page 8: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/8.jpg)
Scaling with Rabbit
RabbitMQApplication
Background processing
Background processing
Background processing
![Page 9: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/9.jpg)
Scaling with Rabbit
RabbitMQApplication
Background processing
Background processing
Background processing
Background processing
![Page 10: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/10.jpg)
Scaling with Rabbit
RabbitMQApplication
Background processing
Background processing
Background processing
Background processing
Background processing
![Page 11: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/11.jpg)
● Fast logging solution
Some real world uses...
![Page 12: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/12.jpg)
● Fast logging solution● Background Processing
Some real world uses...
![Page 13: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/13.jpg)
● Fast logging solution● Background Processing
○ Sending emails
Some real world uses...
![Page 14: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/14.jpg)
● Fast logging solution● Background Processing
○ Sending emails○ Sending SMS
Some real world uses...
![Page 15: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/15.jpg)
● Fast logging solution● Background Processing
○ Sending emails○ Sending SMS○ Analytics, reporting
Some real world uses...
![Page 16: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/16.jpg)
(on precise64, other OSs may vary)
Installing RabbitMQ
![Page 17: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/17.jpg)
● add apt repo○ deb http://www.rabbitmq.com/debian/ testing main
● add signing key○ http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
● apt-get update● apt-get install rabbitmq-server● rabbitmq-plugins enable rabbitmq_management● sudo service rabbitmq-server restart
Using Apt
![Page 18: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/18.jpg)
http://localhost:15672/
![Page 19: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/19.jpg)
Basic Message Queuing
![Page 20: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/20.jpg)
Objective: Basic Queuing
Producer Consumer
test_queue
1 2 3 4 5
![Page 21: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/21.jpg)
composer.json{
"require": {
"videlalvaro/php-amqplib": "2.*"
}
}
then composer install
![Page 22: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/22.jpg)
Please wait, connecting...use PhpAmqpLib\Connection\AMQPConnection;
$connection = new AMQPConnection(
'localhost',
5672,
'guest',
'guest',
'/'
);
$channel = $connection->channel();
![Page 23: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/23.jpg)
basic/producer.phpuse PhpAmqpLib\Message\AMQPMessage;
$channel->queue_declare(
'test_queue',
false,
true,
false, false);
$message = new AMQPMessage('my test message');
$channel->basic_publish($message, '', 'test_queue');
![Page 24: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/24.jpg)
basic/consumer.php$channel->basic_consume(
'test_queue', // Queue to consume
'', // Consumer identifier
false,
true, // No-ack means messages are "auto acknowledged"
false, // Exclusive - no other consumers can use the queue
false,
function(AMQPMessage $message) {
echo $message->body . "\n";
}
);
while (count($channel->callbacks)) {
$channel->wait();
}
![Page 25: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/25.jpg)
What to expect...
![Page 26: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/26.jpg)
Exchanges: Fanout
![Page 27: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/27.jpg)
Objective: Fanout Exchange
test_exchange
amq.KfgPZ3PE
amq.cK5Cp3FC
Consumer
Consumer
Producer
1
1
2
2
3
3
4
4
5
5
![Page 28: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/28.jpg)
fanout/producer.phpuse PhpAmqpLib\Message\AMQPMessage;
$channel->exchange_declare(
'test_exchange',
'fanout',
false, false, false);
$message = new AMQPMessage('my test message #' . $id);
$channel->basic_publish($message, 'test_exchange');
![Page 29: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/29.jpg)
fanout/consumer.php$q = $channel->queue_declare(
'', // Lets RabbitMQ pick a name for queue
false, false, false,
true // Delete this queue
);
$queue_name = $q[0];
$channel->exchange_declare(
'test_exchange', 'fanout', false, false, false);
$channel->queue_bind($queue_name, 'test_exchange');
![Page 30: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/30.jpg)
What to expect...
![Page 31: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/31.jpg)
A word on Temporary Queues
● Only the “current” flow● Specific use cases
test_exchangeProducerMessages
go nowhere
![Page 32: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/32.jpg)
Exchanges: Direct
![Page 33: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/33.jpg)
Objective: Direct Exchange
test_direct
BK = apple
BK = banana, apple
Consumer
Consumer
Producer
3
Message Routing Keys1 = orange2 = banana3 = apple
2 3
BK = orange, banana, apple
Consumer1 2 3
![Page 34: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/34.jpg)
direct/producer.php$channel->exchange_declare(
'test_direct', 'fanout', false, false, false);
$messageContent = 'my test message, key=' . $routingKey;
$message = new AMQPMessage($messageContent);
$channel->basic_publish($message, 'test_direct', $routingKey);
![Page 35: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/35.jpg)
direct/consumer.php$q = $channel->queue_declare('', false, false, false, true);
$queue_name = $q[0];
$channel->exchange_declare(
'test_direct', 'direct', false, false, false);
// Bind for each routing key we want (BINDING KEY)
$channel->queue_bind($queue_name, 'test_direct', 'apple');
$channel->queue_bind($queue_name, 'test_direct', 'orange');
$channel->queue_bind($queue_name, 'test_direct', 'banana');
![Page 36: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/36.jpg)
What to expect...
![Page 37: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/37.jpg)
Exchanges: Topic
![Page 38: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/38.jpg)
Objective:Topic Exchange
test_topic
BK = *.vegetable
BK = #
Consumer
Consumer
Producer
1 2
Message Routing Keys1 = red.vegetable2 = green.vegetable3 = green.fruit4 = red.meat5 = green.grass.long
1 2 3 4 5
BK = green.#
Consumer2 3 5
BK = *.grass.* / *.*.long
Consumer5
![Page 39: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/39.jpg)
Real World Example
![Page 40: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/40.jpg)
![Page 41: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/41.jpg)
Fetch message
Logging Sequence
ApplicationBrowser Log Server
HTTP request
JSON via AMQP
Error!
HTTP response
RabbitMQ
![Page 42: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/42.jpg)
Flexibility!
● Temporary Queues○ e.g. inspect “debug” messages
![Page 43: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/43.jpg)
Flexibility!
● Temporary Queues● Queues to log to DB
![Page 44: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/44.jpg)
Flexibility!
● Temporary Queues● Queues to log to DB● Queues to email “alert/emergency”
![Page 45: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/45.jpg)
Flexibility!
● Temporary Queues● Queues to log to DB● Queues to email “alert/emergency”● Get creative with routing keys
○ RK = app.api.error … BK = #.api.error○ RK = app.form.debug … BK = #.debug
![Page 46: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/46.jpg)
Problem: SPOF
![Page 47: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/47.jpg)
Solution 1: Clustering
![Page 48: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/48.jpg)
Clustering
RabbitMQNode 1
RabbitMQNode 3
RabbitMQNode 2
RabbitMQNode 4
RabbitMQNode 5
RabbitMQNode 6
Load Balance / Floating IP / Low TTL DNS etc.
![Page 49: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/49.jpg)
Clustering
● Everything replicated, except queues
![Page 50: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/50.jpg)
Clustering
● Everything replicated, except queues● Types:
○ RAM○ Disk
![Page 51: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/51.jpg)
Clustering
● Everything replicated, except queues● Types:
○ RAM○ Disk
● Configuration:○ CLI (rabbitmqctl)○ Configuration files
![Page 52: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/52.jpg)
Creating a clusternode1$ rabbitmqctl cluster_status
Cluster status of node rabbit@node1 ...
[{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}]
...done.
node2$ rabbitmqctl cluster_status
Cluster status of node rabbit@node2 ...
[{nodes,[{disc,[rabbit@node2]}]},{running_nodes,[rabbit@node2]}]
...done.
node3$ rabbitmqctl cluster_status
Cluster status of node rabbit@node3 ...
[{nodes,[{disc,[rabbit@node3]}]},{running_nodes,[rabbit@node3]}]
...done.
![Page 53: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/53.jpg)
node2$ rabbitmqctl join_cluster --ram rabbit@node1
node3$ rabbitmqctl join_cluster rabbit@node2
node3$ rabbitmqctl cluster_status
Cluster status of node rabbit@node3 ...
[{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]},
{running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}]
...done.
Creating a cluster
![Page 54: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/54.jpg)
node1$ rabbitmqctl stop_app
node2$ rabbitmqctl forget_cluster_node rabbit@node1
node1$ rabbitmqctl reset
node1$ rabbitmqctl start_app
node2$ rabbitmqctl cluster_status
Cluster status of node rabbit@node2 ...
[{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]},
{running_nodes,[rabbit@node2,rabbit@node3]}]
...done.
Managing Nodes● Stopping/starting nodes● Removing nodes:
![Page 55: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/55.jpg)
Solution 2: HA
![Page 56: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/56.jpg)
HA + Queue Mirroring
RabbitMQNode 1
RabbitMQNode 2
Load Balance / Floating IP / Low TTL DNS etc.
![Page 57: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/57.jpg)
https://github.com/asgrim/rmq-slides
Have a go yourself!
![Page 58: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/58.jpg)
Questions?
![Page 59: Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)](https://reader034.fdocuments.in/reader034/viewer/2022052310/53fe33b28d7f72c4108b45ee/html5/thumbnails/59.jpg)
James Titcumbwww.jamestitcumb.comwww.protected.co.ukwww.phphants.co.uk@asgrim
Thanks for watching!