JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel
-
Upload
naveen-raj -
Category
Technology
-
view
234 -
download
4
description
Transcript of JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel
![Page 1: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/1.jpg)
1
![Page 2: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/2.jpg)
Scalable messaging with JBoss A-MQ and Camel
Bhavani Ananth@bhavani_anin.linkedin.com/in/bhavaniananth/
2
Naveen Raj [email protected]/in/naveenraj/
![Page 3: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/3.jpg)
Messaging And Integration
JBoss A-MQ
JBoss A-MQ – Protocols
JBoss A-MQ – Message Stores
Vertical and Horizontal Scalability
3
Camel in brief!
Agenda
![Page 4: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/4.jpg)
Messaging And Integration
4
![Page 5: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/5.jpg)
Scalable Solution
5
Large Retail Industry
Billing Go Online Vendor
Inventory Warehouse Logistics
Integration Infrastructure
MessagingMessaging
Messaging
Messaging
MessagingMessaging
![Page 6: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/6.jpg)
Data Is Growing Every Second –
Data Deluge
More Data Due to Emergence of
Internet Of Things
Need For A Scalable Solution
![Page 7: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/7.jpg)
What do we have!
Red Hat JBoss A-MQ
7
![Page 8: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/8.jpg)
JBoss A-MQ
8
![Page 9: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/9.jpg)
What is JBoss A-MQ
9
JMS 1.1-compliant messaging system based on Apache Active MQ.
Consists of a broker and client-side libraries enabling remote communication among distributed client applications.
![Page 10: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/10.jpg)
Architecture and Features
10
JBoss-AMQ
Apache Karaf + Fuse Fabric
Monitoring and
Management
Apache Active MQ
STOMP MQTT
Open Wire AMQP NMS
JMS
![Page 11: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/11.jpg)
JBoss A-MQ – Protocols
11
![Page 12: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/12.jpg)
Protocols Supported
12
Client API’s
JMS
NMS
CMS
Rest/Ajax
Wire Protocols
MQTT
AMQP
STOMP
Open Wire
Transports
TCP/SSL
HTTP/HTTPS
WS/WSS
![Page 13: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/13.jpg)
MQTTMQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. URI- mqtt://Host:[Port]?transportOptions
13
Facebook Messenger Smart Santander Features
Open
Simple
Works well in constrained networks
![Page 14: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/14.jpg)
AMQP
Features
Multi-Vendor interoperability
Highly reliable
Feature rich
AMQP (Advanced Message Queueing Protocol) is an openly published wire specification for asynchronous messaging
URI: “amqp:// Host:[Port]?transportOptions
14
![Page 15: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/15.jpg)
STOMPSTOMP is the Simple (or Streaming) Text Orientated Messaging Protocol.
URI: “stomp://localhost:61613”
15
Broker
Producer
STOMP
Consumer(Ruby,PHP)
TCP
Features
Simple
Used in scripting languages
![Page 16: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/16.jpg)
Openwire
Features
Default wire format used by ActiveMQ.
Efficient
Supports auto reconnect
Supports client failover
OpenWire is the cross language Wire Protocol to allow native access to ActiveMQ from a number of different languages and platforms
16
![Page 17: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/17.jpg)
JBoss A-MQ – Message Stores
17
![Page 18: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/18.jpg)
Message Stores
KahaDB message store
Distributed KahaDB message store
LevelDB message store – the best in terms of scalability and performance
Journaled JDBC adapter
Non-journaled JDBC adapter
18
![Page 19: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/19.jpg)
Message Broker
Message CursorsMessage Store and Message cursors in an Retail
Infrastructure
19
Vendor Systems
System A
System B
Inventory System
Message Store
App Queue 2
Dispatch queue Pending Cursor
App Queue
1
App Queue 3
Queue impl details
![Page 20: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/20.jpg)
Message Broker
Message Cursors – Increase Scalability of Message Stores
Store Based cursors for fast Consumers (Retail Scenario)
20
Dispatch Queue
Pending Cursor
Message Store
Inbound Messages
Vendor Systems
System A
System B
![Page 21: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/21.jpg)
Store Based cursors for slow Consumers
Message Cursors – Increase Scalability of Message Stores Contd…
21
Message Broker
Dispatch Queue
Pending Cursor
Message Store
Inbound Messages
VendorSystems
System A
System B
![Page 22: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/22.jpg)
VMCursors
Message Cursors – Increase Scalability of Message Stores Contd…
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"><pendingQueuePolicy> <vmQueueCursor/> </pendingQueuePolicy></policyEntry>
22
Message Broker
Dispatch Queue
Pending Cursor
Message Store
Inbound Messages
VendorSystems
System A
System B
![Page 23: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/23.jpg)
File Based Cursors
• Does not scale well if the consumers are behind by a large margin
• Used for non persistent messages
Message Cursors – Increase Scalability of Message Stores Contd…
23
Message Broker
Dispatch Queue
Pending Cursor
Message Store
Inbound Messages
File Based Cursors
Temporary Files
VendorSystems
System A
System B
![Page 24: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/24.jpg)
Vertical and Horizontal Scalability
24
![Page 25: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/25.jpg)
Vertical Scalability
Capacity of a single broker to support large numbers of connections from consumers and producers.
To Consider Thread Count CPU Usage Memory Usage
25
![Page 26: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/26.jpg)
Vertical Scalability – Reducing Thread Count
NIO protocol
<transportConnectors><transportConnector name="nio" uri="nio://0.0.0.0:61616"/></<transportConnectors>
26
![Page 27: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/27.jpg)
JBoss A-MQ – Vertical Scalability – Reducing Thread Count
Disable Dedicated Task Runner
ACTIVEMQ_OPTS="-org.apache.activemq.UseDedicatedTaskRunner=false“
Use optimized dispatch for queues<destinationPolicy>
<policyMap><policyEntries>
<policyEntry topic=">" optimizedDispatch="true">
...</policyEntry>
</policyEntries></policyMap>
</destinationPolicy>
27
![Page 28: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/28.jpg)
Vertical Scalability – CPU
Enable tightEncodingEnabled
uri = "failover://(tcp://localhost:61616wireFormat.tightEncodingEnabled=false)";
Enabling compression
tcp://hostA:61617?jms.useCompression=true
28
![Page 29: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/29.jpg)
Vertical Scalability – Memory
Allocate more memory to broker • pass the –Xmx option to the JVM.
ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=false“
Configure big enough memory usage
<systemUsage><memoryUsage>
<memoryUsage limit="1024 mb" /></memoryUsage>
</systemUsage>
29
![Page 30: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/30.jpg)
Horizontal Scalability – Network of Brokers
Concepts
Network Connector
Transport Connector
Store and Forward
30
Broker-1
Broker-2
Network Connector
Producer
Transport Connector
Consumer
Transport Connector
![Page 31: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/31.jpg)
Retail Example – Hub and Spoke Topology
31
![Page 32: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/32.jpg)
Store and Forward- Configuration
Store Store
RetailOutlet
Warehouse
Firewall
<networkConnectors> <networkConnector uri="static:(tcp://warehouse:61617)"
userName="admin" password="admin“duplex=“true”conduitSubscriptions=“true”decreaseNetworkConsumerPriority=“false” />
</networkConnectors>
![Page 33: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/33.jpg)
When there are a lot of producers
Brokers, A, B, and C, accept connections from message producers.
Brokers, X and Y, accept connections from message consumers.
Concentrator Topology - Producer Load Balancing
33
A B C
YX
C1 C2 C3 C4
![Page 34: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/34.jpg)
Some Other Topologies
34
Tree Topology
Graph Topology
Mesh Topology
![Page 35: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/35.jpg)
Retail Example-High Availability
Shared Store
Retail Master Broker
Retail Slave Broker
Warehouse Slave Broker
Warehouse
Master Broker
Shared Store
Network Connection on Failure
Netw
ork C
onnection
<networkConnectors> <networkConnector uri="static:(tcp://retail-master:61617, tcp://retail-slave:61617)" </networkConnectors>
<services> <masterConnector remoteURI=“tcp://retail-slave:62001” userName=“admin” password=“admin”/></services>
![Page 36: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/36.jpg)
Other Master Slave Topologies
Master Broker
Slave Broker
Store Store
Message
Shared Nothing Master Slave
Reliable
Only one slave possible
Downtime when master fails
Client
36
![Page 37: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/37.jpg)
Shared Storage Master-Slave
Master Broker with
file lockSlave Broker
Shared File System or
SAN
Shared File system Master Slave
Recommended if SAN is available
Easy to configure
No restriction on the slaves
Client
Slave Broker
Slave polls for lock
37
![Page 38: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/38.jpg)
Hybrid Scaling
38
Client
Acc Payable Master
Acc Payable
Slave
Accounts Payable Information
Acc Receivable Master
Acc Receivable Slave
Accounts Receivable Information
Client Complexity
Less broker cross
talk
![Page 39: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/39.jpg)
Apache CamelA ride to solve your integration problems
39
![Page 40: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/40.jpg)
Camel-What is it
Open source integration framework
Based on Enterprise Integration Patterns(EIPs)
Supports more than 120+ components
Supports DSL programming with Spring, Java, Scala and groovy
Flexible deployment
Can run standalone with maven
40
CBR Wire Tap TransformationEndpoint Translator
![Page 41: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/41.jpg)
Camel-Architecture
41
![Page 42: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/42.jpg)
Camel Coding Structure
import org.apache.camel.builder.RouteBuilder; public class MyRouteBuilder extends RouteBuilder { public void configure() { from("file:SourceFolder?noop=true")
.to("file:TargetFolder"); } }
<camelContext xmlns="http://camel.apache.org/schema/spring"> <route id=“FileToFile"> <from uri="file:SourceFolder?noop=true""/> <to uri="mock:inside"/> </route></camelContext>
Java DSL
Spring DSL
42
Read a file from SourceFolder and write it to TargetFolder
![Page 43: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/43.jpg)
43
<route> <from uri="activemq:JUDCon?destination.consumer.exclusive=true&destination.consumer.prefetchSize=50"/> <to uri="mock:2014"/> </route>
Routes
Camel and Messaging
Configuring the Destination Options at the Endpoint in the Routes
![Page 44: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/44.jpg)
Camel and Messaging Contd…
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory"> <property name="brokerURL" value="tcp://localhost:61616" /></bean>
Beans
44
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent"> <property name="configuration" ref="jmsConfig"/> <property name="transacted" value="true"/> <property name="cacheLevelName" value="CACHE_CONSUMER" /> </bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop"> <property name= "maxConnections" value="8" /> <property name="connectionFactory" ref="jmsConnectionFactory" /></bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration"> <property name="connectionFactory" ref="pooledConnectionFactory"/> <property name="concurrentConsumers" value="10“ /></bean>
![Page 45: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/45.jpg)
Camel – Scalability Techniques
45
ThrottleLoad
Balancing
from("activemq:queueA").throttle(10).to("activemq:queueB")
<from uri=" activemq:queueA " /><throttle maximumRequestsPerPeriod="3“> <to uri=" activemq:queueB " />
from(" activemq:queueA ").loadBalance().roundRobin().to(" activemq:queueB ", " activemq:queueC", " activemq:queueD");
<from uri=" activemq:queueA "/><loadBalance> <roundRobin/> <to uri=" activemq:queueB"/> <to uri=" activemq:queueC "/> <to uri=" activemq:queueD "/> </loadBalance>
![Page 46: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/46.jpg)
46
Embedding Camel into Your Broker
•Reduces serialization cost•Reduced network Why?•Using Interceptors to your brokerWhen?
•Add the camel route to activemq.xmlHow?
![Page 47: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/47.jpg)
Summary
• Messaging and Integration• Significance of Asynchronous messaging• How scalability is important to solve large flow of data• Scalability – Vertical and Horizontal• What's camel and Integration
47
![Page 48: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/48.jpg)
Useful Links
References• http://camel.apache.org/architecture.html • http://camel.apache.org/jms.html• http://activemq.apache.org/broker-camel-component.html
48
![Page 49: JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel](https://reader037.fdocuments.in/reader037/viewer/2022102721/54bc9bb44a7959f6568b4603/html5/thumbnails/49.jpg)
Thank You
Scale Your Enterprise ECO System
49
@bhavani_an @Naveenrajbala