Architecture | The Future of Messaging: RabbitMQ and AMQP | Eberhard Wolff
-
Upload
jax-london -
Category
Technology
-
view
6.860 -
download
5
description
Transcript of Architecture | The Future of Messaging: RabbitMQ and AMQP | Eberhard Wolff
The Future of Messaging: RabbitMQ and AMQP
Eberhard Wolff Architecture and Technology Manager
adesso AG, Germany
Overview
• Why Messaging, AMQP and RabbitMQ • Basic AMQP • Exchanges • More on Spring-AMQP
RPC
• Predominant approach – RMI, SOAP Web Services, CORBA,
HttpInvoker, Burlap, Hessian • Calls remote methods with parameter • …and waits for response
RPC
• Problems: – Explicitly tells the server what to do i.e.
tight coupling – What about network failures? – What about long latencies?
Why Messaging? • Decoupling
– Data, no action i.e. receiver can react arbitrarily
– Asynchronous i.e. decoupled by time
• Reliable – Message can be stored-and-
forwarded – Redelivery until message
processed • Solves typical problems of
distributed systems
Component
Component Messages
Why Messaging?
• But: Requires different architecture • Very different from calling remote
methods • Asynchronous • AJAX has the same model
• See for example “Patterns of Enterprise Integration”
Why AMQP? • Open standard protocol • Standard wire protocol • i.e. just one client library – no matter which
implementation you are using • Less vendor lock in • Efficient
– Binary wire protocol • Support in all major languages • Supported on most OS platforms
What about JMS?
• JMS has been the default for Java messaging system for 10+ years
• But: – Only standardized on the API level – Less flexible than AMQP
• Mapping AMQP/JMS is being defined
Why Rabbit? • Because it has a kewl name • Numerous protocols supported • Most popular choice on EC2 • Foundation for demanding systems e.g.
NASA’s cloud initiative Nebula • Implemented in Erlang • Clustering built in • Currently in 2.6.1 • Supports AMQP 0.8, 0.9, 0.9.1 • 1.0 as a prototype Plug In
Broad Support in RabbitMQ
Broad Support in the JVM Space
• Grails Plug In • Java Client • Scala / Lift support
• We will discuss Spring support in detail • Spring AMQP project 1.0.0 • http://www.springsource.org/spring-
amqp
Why Erlang? • Originally designed for telephone
switches by Ericsson • Much easier to develop scalable and fault
tolerant systems (by factors)
• See Motorola presentation: http://www.slideshare.net/Arbow/comparing-cpp-and-erlang-for-motorola-telecoms-software
• Good tool for reliable and scalable systems
Erlang‘s Model
Light weight
process with state
Monitor
Link to monitor, restart
Light weight
process with state
Light weight
process with state
Messages Messages
Why Erlang?
• Let it crash – If a process fails, it can be easily restarted – Different approach to fault tolerance – Otherwise lots of error handling
• Message Passing in the Core – RabbitMQ is a messaging system…
• Light-weight process model – Scalabiliy
Very Basic AMQP
• Queues: Store messages • Queues might be
– Durable: Survive server restarts – Exclusive: For one connection – autoDelete: Deleted if connection closes
• Queue usually created by consumer • All resources are dynamic • Producer sends a message to a Queue
Code ConnectionFactory conFactory = new CachingConnectionFactory ("localhost");RabbitAdmin admin = new RabbitAdmin(conFactory);admin.declareQueue( new Queue("myQueue", false, true, true));RabbitTemplate template = new RabbitTemplate(conFactory);template.convertAndSend("myQueue", "Hi AMQP!");String receive = (String) template.receiveAndConvert("myQueue");Assert.assertEquals("Hi AMQP!", receive);
Spring’s RabbitTemplate
• Send & receive message • AmqpTemplate:
Generic AMQP interface • RabbitOperations: Rabbit specific
interface: (adds just a callback) • RabbitTemplate: Implementation • Spring might provide support for other
AMQP implementations later
Spring’s MessageConverter
• Messages are binary data • RabbitTemplate uses
MessageConverter to convert between objects and messages
• Can also send binary data if preferred
Spring’s MessageConverter • Default: SimpleMessageConverter
– byte[] directly transferred – String converted with configurable encoding – Serializable are serialized – Content type set accordingly
• JsonMessageConverter converts from / to JSON using Jackson
• MarshallingMessageConverter converts from / to XML using Spring's OXM mapping
• SerializerMessageConverter uses Spring’s Serializer abstraction
Spring‘s AdminTemplate
• Main purpose: Configure the AMQP infrastructure
• E.g. create queues
• AmpqAdmin: Generic AMQP interface • RabbitAdmin: Rabbit specific
Basics of AMQP • Sending messages directly to queues is
not enough • What about e.g. pub / sub?
• Exchange: Route messages (stateless) • Messages are byte-streams • Example used the default exchange
• More dynamic, flexible and cleaner than JMS
AMQP in a nutshell Exchange routes message Stateless Usually created by producer No queue: Message discarded
X
Queues buffer messages Usually created by consumer
Binding binds an Exchange to a Queue
AMQP in a nutshell
AMQP protocol
Producer and Consumer might be written in Java, C#, Python, Ruby …
X
C
C
P
RabbitMQ AMQP protocol
Exchange: Route Messages
• The type of Exchange defined the routing algorithm used
• Binding provides selector for routing • Exchange is addressed by name
• Some standard types • Can provide additional ones
X
Fanout Exchange
• Broadcast to all bound queues • Fast • Simple
• amq.fanout is mandatory
• To broadcast information
X
X
C
C
C
P
Fanout
Fanout Exchange X
Queue fanoutQueue = new Queue("fanoutQueue");admin.declareQueue(fanoutQueue);FanoutExchange fanoutExchange= new FanoutExchange("myFanout");admin.declareExchange(fanoutExchange);admin.declareBinding( BindingBuilder.bind(fanoutQueue). to(fanoutExchange));template.setExchange("myFanout");template.convertAndSend("Hi Fanout!");String receive = (String) template.receiveAndConvert("fanoutQueue");Assert.assertEquals("Hi Fanout!", receive);
Direct Exchange • Routing based on one routing key • amq.direct and the default Exchange (no
name) always exist
• To send work orders to a specific worker
X
X
C
C
C
P
Direct Exchange
express
normal
express normal
Direct Exchange
Queue directQueue = new Queue("direct");admin.declareQueue(directQueue);admin.declareBinding(BindingBuilder .bind(directQueue) .to(new DirectExchange("amq.direct")) .with("helloKey"));template.setExchange("amq.direct");template.convertAndSend("amq.direct","dropMe", "I will be dropped!");template.convertAndSend("amq.direct","helloKey", "Hi Direct!");Assert.assertEquals("Hi Direct!", template.receiveAndConvert("direct"));Assert.assertNull( template.receiveAndConvert("direct"));
Topic Exchange • Routing based on routing pattern • amq.topic is mandatory
• E.g. for public / subscribe scenarios
X
Topic Exchange
X C
C
P
Topic Exchange
order.*
invoice.*
order.DE invoice.USD
Headers Exchange • Routing based on one or more headers and
an expression • amqp.match is mandatory
• Complex routing roles
X
Other Features • Message can be persistent • Request / response using correlations
possible
• Redelivery / acknowledgement possible
• Clustering with e.g. Linux HA possible • ...or send message through multiple
channels and drop duplicates
More about RabbitMQ and Spring
Configuring Rabbit Resources with Spring
• Spring enables decoupling of your application code from the underlying infrastructure
• The container provides the resources
• The application is simply coded against the API
Configuring a ConnectionFactory
• Can easily modify configuration options
<bean id="connectionFactory" class="org.sfw.amqp.rabbit.connection.CachingConnectionFactory"> <property name="username" value="guest"/> <property name="password" value="guest"/> <constructor-arg value="localhost" /> </bean>
Create an object with the given name
and class Call setUsername() with the given value
Parameter for the constructor
<cloud:rabbit-connection-factory id="rabbitConnectionFactory" />
Using a ConnectionFactory from Cloud Foundry
• Will be provided by Cloud Foundry
ConnectionFactory connectionFactory = new RabbitServiceCreator(new CloudEnvironment()) .createSingletonService().service;
<bean id="rabbitTemplate" class="org.springframework.amqp.rabbit.core.RabbitTemplate"> <constructor-arg ref="connectionFactory" /> <property name="routingKey" value=”invoice.USD" /> </bean>
Defining a RabbitTemplate Bean
• Provide a reference to the ConnectionFactory • Optionally provide other references
– MessageConverter – Routing key and exchange to be used if none is
specified
The MessageListener
• So far: Calling receive() on RabbitTemplate
• Needed: Something that is called when a new message appears
• The API defines this interface for asynchronous reception of messages
public interface MessageListener { public void onMessage(Message) { // handle the message } }
Spring’s MessageListener Container
• Spring provides lightweight containers to call MessageListeners
• SimpleMessageListenerContainer • Advanced scheduling and endpoint
management options available • i.e. thread pools, concurrent consumers,
transaction handling
Defining a Message Listener Container
• Every time a new message appears on my.amqp.queue the messageListener is called
<bean class="org.sfw.amqp.rabbit.listener.SimpleMessageListenerContainer"> <property name="connectionFactory" ref="connectionFactory" /> <property name="queueNames" value="my.amqp.queue" /> <property name="messageListener" ref="messageListener" /> </bean>
Spring's message-driven objects
• MessageListener means the receiver depends on Spring API
• Why not just a POJO? • MessageListenerAdapter takes a POJO and
makes it a MessageListener • i.e. calls consume on Bean consumer <bean id="messageListenerAdapter"
class="org.sfw.amqp.rabbit.listener.adapter.MessageListenerAdapter"> <property name="delegate" ref="consumer" /> <property name="defaultListenerMethod" value="consume" /> <property name="messageConverter" ref="jsonMessageConverter" /> </bean>
Easier Using Namespaces
• Results in the same Spring Beans
<rabbit:listener-container connection-factory="connectionFactory“ message-converter="jsonMessageConverter"> <rabbit:listener ref="consumer" method="consume"
queue-names="my.amqp.queue2" /> </rabbit:listener-container>
Consumer code
• No dependency on AMQP! • But: What about the result of the method? • Send to the Reply-To address given in
message properties with same correlationId as original method
@Componentpublic class Consumer { public String consume(String message) { return …; }}
Client Code
• Message sent to destination with routing key • Reply-To set to exclusive, autodelete, non-
durable queue • Response received through Reply-To
converted and returned • Easy request-response! • Beware of potential latency
String response = (String) rabbitTemplate.convertSendAndReceive( "my.fanout", "", "test");
Create Environment using Namespaces
• ...if you don‘t like API calls <rabbit:fanout-exchange name="my.fanout2"> <rabbit:bindings> <rabbit:binding queue="my.amqp.queue2" /> </rabbit:bindings> </rabbit:fanout-exchange> <rabbit:queue name="my.amqp.queue2" /> <rabbit:admin connection-factory="rabbitConnectionFactory" />
A SHORT GLIMPSE ON AMQP 1.0
EVERYTHING YOU KNOW IS WRONG
New Elements
• Links: unidirectional transport between source and target – Flow control – Settling transfers for different semantics (at
most once etc) • Processing nodes have distribution
modes – Copy: Copy message to each link – Move: Move it to just one link
AMQP: Point to Point
Broker
Node mode : move
Sender Receiver Outgoing Link
Incoming Link
Receiver Incoming Link
AMQP: Publish / Subscribe
Broker
Node mode :
copy Sender Receiver
Outgoing Link
Incoming Link
Receiver Incoming Link
Conclusion: AMQP
• Ubiquitous Messaging • AMQP: Protocol standard • Better scalability • Dynamic resources
Conclusion: Spring AMQP
• Easy to use • Flexible (e.g. message encoding) • Allows scalable message handling • Full support for AMQP and RabbitMQ
More
• http://springsource.org/spring-amqp • Also a .NET version available • …and support Spring Integration • http://blog.springsource.com/
2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitm
• Transaction support
• …and there is very similar JMS support J
Questions? @ewolff