personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM...
Transcript of personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM...
![Page 1: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/1.jpg)
Dead LettersHandling within Message Broker
Sergej Kurakin
![Page 2: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/2.jpg)
Sergej KurakinAge: 36
Company: Oberlo
Position: Senior Developer
http://careers.oberlo.com/
![Page 3: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/3.jpg)
Dead Letters1. Message Queues2. Poisonpill Message3. Dead Letter Queue4. Alternatives5. What to do with Dead Letters?6. Demos
![Page 4: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/4.jpg)
Message Queues
![Page 5: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/5.jpg)
In computer science, message queues and mailboxes are software-engineering
components used for inter-process communication, or for inter-thread
communication within the same process.
https://en.wikipedia.org/wiki/Message_queue
![Page 6: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/6.jpg)
Simple Explanation● Client (Sender, Producer) - sends message to Message Server.● Message Server (Broker) - Receives message from Client(s) and routes to
proper Worker(s).● Worker (Receiver, Consumer) - receives message and executes task.
![Page 7: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/7.jpg)
![Page 8: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/8.jpg)
Common Usage for WEB● Background Tasks● Parallel Tasks● System Integration● Interlanguage Communication● Scaling● Decoupling● Traffic Shaping
![Page 9: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/9.jpg)
Message Queue Software● Amazon Simple Queue Service● Apache ActiveMQ● RabbitMQ● Gearman● Beanstalk● IBM Integration Bus (WebSphere Message Broker)● Microsoft Message Queuing● Redis*● RDBMS*● Use “Search” for more ;-)
![Page 10: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/10.jpg)
Poisonpill Message
![Page 11: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/11.jpg)
Message that can’t be processed.
![Page 12: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/12.jpg)
Message that can bring down all your Workers.
![Page 13: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/13.jpg)
PHP Fatal error: Uncaught RuntimeException: Fail command received in /home/skurakin/queuenodl/default/consumer.php:27Stack trace:#0 [internal function]: process_message(Object(PhpAmqpLib\Message\AMQPMessage))#1 /home/skurakin/queuenodl/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AMQPChannel.php(986): call_user_func('process_message', Object(PhpAmqpLib\Message\AMQPMessage))#2 [internal function]: PhpAmqpLib\Channel\AMQPChannel->basic_deliver(Object(PhpAmqpLib\Wire\AMQPReader), Object(PhpAmqpLib\Message\AMQPMessage))#3 /home/skurakin/queuenodl/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AbstractChannel.php(202): call_user_func(Array, Object(PhpAmqpLib\Wire\AMQPReader), Object(PhpAmqpLib\Message\AMQPMessage))#4 /home/skurakin/queuenodl/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AbstractChannel.php(340): PhpAmqpLib\Channel\AbstractChannel->dispatch('60,60', '\x07default\x00\x00\x00\x00\x00\x00\x00...', Object(PhpAmqpLib\Message\AMQPMessage))#5 /home/skurakin/queuenodl/default/consumer.php(50): PhpAmqpLib in /home/skurakin/queuenodl/default/consumer.php on line 27
![Page 14: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/14.jpg)
Dead Letter Queue
Image from https://www.drupal.org/sandbox/edwinknol/2413597
![Page 15: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/15.jpg)
Queue where “bad” messages appear.
![Page 16: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/16.jpg)
“Bad” messages● The message is rejected (failed to process)● The TTL for the message expires● The queue length limit is exceeded● Undeliverable message● Other rules may apply
Other rules may apply - consult with your message queue manual!
![Page 17: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/17.jpg)
![Page 18: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/18.jpg)
Who supports Dead Letter Queues● Amazon Simple Queue Service● Apache ActiveMQ● RabbitMQ● IBM Integration Bus (WebSphere Message Broker)● Microsoft Message Queuing● Use “Search” for more ;-)
![Page 19: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/19.jpg)
Alternatives
![Page 20: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/20.jpg)
Alternatives to Dead Letter Queues ● Ignore bad messages● Track statistics● Log exception and store message payload● Implement Dead Letter Queue manually
![Page 21: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/21.jpg)
What to do with Dead Letters?
![Page 22: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/22.jpg)
You decide!● Ignore and clear Dead Letters Queue● Analyze, fix your code and clear Dead Letters Queue● Analyze, fix your code and requeue Dead Letters
![Page 23: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/23.jpg)
Demos
![Page 24: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/24.jpg)
All demos are based on RabbitMQ 3.7 and “php-amqplib”
![Page 25: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/25.jpg)
No error checking
All consumers fail.
Message stays in queue.
System non-operational.On ordinary queue
![Page 26: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/26.jpg)
Reject, no requeue
On ordinary queue
Consumers recover from fail.
Message is lost, unless you handle it.
System operational.
![Page 27: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/27.jpg)
Reject, requeueOn ordinary queue
Consumers recover from fail.
Message stays in same the queue first and gets rejected from all failing consumers.
System semi-operational or non-operational.
![Page 28: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/28.jpg)
Reject, no requeue
On queue with Dead Letter Exchange configured
Consumers recover from fail.
Message is moved into Dead Letter Exchange.
System operational.
![Page 29: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/29.jpg)
Dead Letters - Recap1. Message Queues2. Poisonpill Message3. Dead Letter Queue4. Alternatives5. What to do with Dead Letters?6. Demos
![Page 30: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/30.jpg)
Linkshttps://en.wikipedia.org/wiki/Message_queue
https://www.cloudamqp.com/blog/2014-12-03-what-is-message-queuing.html
https://en.wikipedia.org/wiki/Dead_letter_queue
https://www.rabbitmq.com/dlx.html
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
![Page 31: personal website | Sergej Kurakin aka ZaZa - Dead Letters...RabbitMQ Gearman Beanstalk IBM Integration Bus (WebSphere Message Broker) Microsoft Message Queuing Redis* RDBMS* Use “Search”](https://reader034.fdocuments.in/reader034/viewer/2022051902/5ff2b1de6de3bd777c02828b/html5/thumbnails/31.jpg)
DiscussionSergej Kurakin
Work @mail: [email protected]
Personal @mail: [email protected]
https://www.linkedin.com/in/sergejkurakin