London JBUG - Connecting Applications Everywhere with JBoss A-MQ

1 redhat Rob Davies

Connec(ng Applica(ons Everywhere with JBoss A-‐MQ

Rob Davies, Technical Director, Fuse Engineering, Red Hat July 2013

2 redhat Rob Davies 2

Rob Davies Technical Director, Red Hat -

h"p://www.redhat.com ■  Software projects:

■  Apache ActiveMQ, ■  Apache Camel ■  Apache ServiceMix

■  On Expert Group for JSR 343: JMS 2.0 ■  Co-‐author of AcCveMQ in AcCon:

3 redhat Rob Davies

History of Fuse open source Development

2004 2005 2006 2007 2008 2009 2010 2011 2012

ActiveMQ created May 2004

The CodeHaus

ServiceMix created May 2005

Camel created March 2007

IONA acquire April 2007

Karaf created

Progress acquire October 2008

FuseSource created October 2010

The Apache Software Foundation

LogicBlaze Formed May 2005

CXF August 2006

Fabric created February 2011

Fuse Forge And ASF

Red Hat acquire Sept 2012

4 redhat Rob Davies

Why use Message-‐Oriented Middleware? •  Robustness to change •  Time Independence •  LocaCon Independence •  Hide Latency •  Scalability •  Event driven •  Simplicity •  Configurable Quality of

Service •  PlaRorm and Language

IntegraCon •  Fault tolerant

5 redhat Rob Davies 5

What is Apache ActiveMQ ?

•  Top Level Apache Software Foundation Project

•  Wildly popular, high performance, reliable message broker

•  Connects to nearly everything

•  Native Java, C/C++, .Net,

•  AMQP 1.0, MQTT 3.1, STOMP (1.0-1.2) and OpenWire

•  STOMP enables Ruby, JS, Perl, Python, PHP, ActionScript …

•  Embedded and standalone deployment options

•  The key component of JBoss A-MQ!

6 redhat Rob Davies

Messaging: The Basics

7 redhat Rob Davies

Message Channels and RouIng

•  Message Channels •  Named communica(on between interested par(es •  JMS calls them ‘Des(na(ons’

•  Can “tune-‐in” to mul(ple channels using wildcards •  Can fine-‐tune message consump(on with selectors •  Can route a message based on content

8 redhat Rob Davies

Message Channels = JMS DesInaIons

Producer

Broker Consumer

Consumer

Consumer

Destination WIDGET

Destination ORDER

9 redhat Rob Davies

Point-‐to-‐Point Channel: JMS Queues

Producer

Consumer

Consumer

Consumer

Queue

10 redhat Rob Davies

Publish/Subscribe Channel: JMS Topics

Producer

Consumer

Consumer

Consumer

Topic


Message RouIng : Selectors

DestinationProducer

Consumercolor='blue'

Consumercolor='red'


Message RouIng : DesInaIon Wildcards

Topic:BAR.BEER

Consumertopic:BAR.>

Topic:BAR.WINE

Producer


Messaging: Features every Message broker should have

But some don’t…


Exclusive Consumers

Producer

Consumer

Consumer

Consumer

Queue


Message Groups

•  Like Exclusive Consumers – but in parallel •  Guaranteed ordering of related messages across a

Queue •  But – load balancing of messages across mul(ple

consumers •  All messages with the same JMSXGroupID go to the

same consumer •  How you group messages is down to the applica(on’s

producer •  To explicitly close a group, set the JMSXGroupSeq to

-‐1

15


JBoss A-‐MQ – Synchronous sends

Message Broker

Message Producer

Consumer

Consumer

Consumer


JBoss A-‐MQ – Synchronous sends

Set alwaysSyncSend on the ActiveMQConnectionFactory

You can set: sendTimeout on the ActiveMQMessageProducer

Synchronous send: JMS Client

producer.send(session.createTextMessage("Hello"), new AsyncCallback() { public void onSuccess() {} public void onException(JMSException exception) { exception.printStackTrace(); }});

Performance Tip – use callbacks …


Deploying JBoss A-‐MQ

•  AcIveMQ Can run standalone or embedded •  As a standalone, or part of a highly available message

broker cluster •  Embedded – easy to use an en(re broker in JUnit tests (no

need to Mock) •  In Tomcat, deployed as a war •  In JEE Server – either co-‐locate – or use client with JCA

•  Ac(veMQ distribu(ons contain a rar for this purpose

18


JBoss A-MQ: Message Storage


Message Delivery Mode

  Messages can be persisted – allowing for decoupled applicaIons

  For Queues, any message delivered as PERSISTENT must be stored on long term storage before being delivered to a consumer

  For Topics, a message delivered as PERSISTENT will only be stored if there exists a durable subscriber

  NON-‐PERSISTENT messages may also be stored on disk if the memory limits of the broker have been reached

20


Message stores supported by JBoss A-‐MQ

■  SQL (JDBC) ■  SQL (JDBC) with journal ■  AMQ Message Store ■  Kaha ■  KahaDB ■  LevelDB

21

Slow but popular

Don’t use (deprecated) The “current” default

The “best yet”


LevelDB Store vs KahaDB

•  Fewer index entries per message than KahaDB •  Faster recovery when a broker restarts •  LevelDB index out-‐perform Btree index at sequen(al access . •  LevelDB indexes support concurrent read access. •  Pauseless data log file garbage collec(on cycles. •  Fewer IOPS to load stored messages. •  It exposes it's status via JMX for monitoring


AcIveMQ: LevelDB Store


JBoss A-MQ: Protocols


OpenWire

•  Advantages: •  Fast – op(mized for

Ac(veMQ •  client failover •  automa(c reconnect •  Client load balancing •  Flow control •  Many advanced features

25

Disadvantages:

•  Not a recognized standard

•  Only Java, C/C++/.Net

http://activemq.apache.org/openwire.html


MQTT h`p://mq`.org

 Advantages: •  M2M/”Internet of Things”

transport •  Proposed as an OASIS

standard •  Extremely light weight •  Growing support from

vendors and OS products •  WebSphereMQ •  Ac(veMQ + Apollo •  Mosquilo •  RabbitMQ

26

Disadvantages:

•  3.1 does not support Queues

•  Advanced features not standard

•  Flow control

•  Failover etc.


AMQP – see www.amqp.org

 Advantages: •  AMQP 1.0 OASIS standard •  Proposed as an OASIS

standard •  Commodi(zes the Broker

27

Disadvantages:

•  One size doesn’t really fit all

•  Currently no plans for IBM or Tibco to adopt it


STOMP – h`p://stomp.github.com

 Advantages: •  Easy to use text based

protocol •  Can use telnet as a client •  Defacto standard:

•  Ac(veMQ + Apollo •  HornetQ •  RabbitMQ •  PocoMQ •  StompServer •  OpenMQ •  Many more …

28

Disadvantages:

•  Not as fast as binary formats


But there’s more: WebSockets   STOMP is a natural wire protocol for WebSockets   And so is MQTT!

29

ActiveMQ

Client(Browser)

WebSocketConnector

StompConverter

ActiveMQ(Backend)

TCP(WebSocket)

HTTP(S)Request/Response


JBoss A-MQ: Enterprise Features – Failover …


High Availability

ActiveMQ Cluster

ActiveMQ Client

Supported by both Java and C++ OpenWire clients


JBoss A-MQ: Enterprise Features – High Availability …


•  JDBC Master/Slave •  Shared File System Master/Slave •  Replicated LevelDB Master/Slave

Master Slave Configurations


Broker Slave

JDBC Master/Slave

Broker Master

ActiveMQ Client

Broker Slave

Database


•  Extreme reliability – but not as fast •  Recommended if already using an enterprise database •  No restric(on on number of slaves •  Simple configura(on •  Configurable lockKeepAlivePeriod

JDBC Master Slave


Broker Slave

Shared File System Master/Slave

Broker Master

ActiveMQ Client

Broker Slave

File System

Larry’s Removals


•  Recommended if you have a SAN, or DRDB •  No restric(on on number of slaves •  Simple configura(on •  Ensure file locking works – and (mes out – NFSv4 good!

Shared File System M/S


Local File System Local File System Local File System

Broker Slave

Broker Master

Broker Slave

ZooKeeper Cluster

Replicated LevelDB Master Slave


ZooKeeper Cluster

Local File System Local File System Local File System

Broker Slave

Broker Master

Client

Broker Slave


Broker Master


•  Requires a HA ZooKeeper Cluster •  No Single Point of Failure •  Dynamic number of slaves •  Simple configura(on •  Sync or Async Replica(on



•  Link Brokers together •  Use Store and Forward •  Are uni-‐direc(onal by default •  All Des(na(ons are global

Network of Brokers


Apache AcIveMQ – Broker Topologies

Store and Forward

ActiveMQ Local broker

ActiveMQ Remote broker


Apache AcIveMQ – Broker Topologies

Bi-directional network

ActiveMQ Local broker

ActiveMQ Remote broker


<networkConnectors> <networkConnector name="backoffice" uri="static://(tcp://backoffice:61617)" duplex="true"> </networkConnector> </networkConnectors>

Bi-directional Config


<networkConnectors> <networkConnector uri="static:(tcp://remote:61617) "/> <dynamicallyIncludedDestinations> <queue physicalName=”free.food.>"/> <queue physicalName=”free.beer.>"/> <topic physicalName=”cricket.scores.>"/> </dynamicallyIncludedDestinations> </networkConnectors>

Reducing Crosstalk


Broker Networks (Store and Forward)


Network of Brokers : Geographically Dispersed


Network of Brokers : Network with Master/Slave


Scaling Networks of Brokers

•  Brokers share rou(ng informa(on across networks •  All des(na(ons are considered Global •  This is really convenient •  Though it starts to get problema(c with 1000’s of

brokers •  However – we can do filtering to shape the traffic

across networks •  But this involves a lot of manual configura(on

49


•  Brokers share rou(ng informa(on across networks •  All des(na(ons are considered Global

•  This is really convenient

•  Though it starts to get problema(c with 1000’s of brokers

•  However – we can do filtering to shape the traffic across networks

•  But this involves a lot of manual configura(on

50

Time to talk about Apache Camel …

Scaling Networks of Brokers


AcIveMQ with embedded Camel

Service

Consumer

ConsumerActiveMQ Broker

Flexible and is faster J


Scaling Networks – use Apache Camel •  Allows for non-‐chaly networks to be established

•  Rou(ng informa(on can be externalized to the broker

•  Successfully used in produc(on for 1000’s of brokers today

52


More problems with Large Deployments


Problems – Deploying and maintenance

  Main problems •  Installing brokers on mul(ple hosts

o  ssh, untar, set directories and environment

•  Sesng configura(on manually for every broker o  copying xml config, tweaking, tes(ng

•  Upda(ng configura(on across cluster •  Upgrading brokers

It’s a tedious and error-prone process


Problems – TradiIonal best-‐pracIce Ips

•  Keep XML as a template and configure node-‐specific details through proper(es

•  Keep configura(on in SVC system (git, svn, …)

•  Keep configura(on separate from installa(on for easier upgrades


Problems -‐ Clients

•  Topology is very “sta(c” •  Clients need to be aware of topology •  Clients need to know broker loca(ons •  Changes are not easy as clients need to be updated •  Adding new resources (brokers) requires client updates •  Not suitable for “cloud” deployments

Fuse Fabric makes deployments more “elastic”


Fuse Fabric to the rescue!


Fuse Fabric – Key Features

Fuse IDE

FMC

JBoss A-MQ Cluster

Camel Endpoints JBoss Fuse

Runtime Registry

•  Supports Hybrid deployments •  Distributed Configura(on •  Run(me registry •  Centralized Management


FuseMQ features

•  mq-‐base profile •  Defines OSGi features and bundles to be installed •  Defines basic broker sesngs

•  mq-‐create command •  Helper command for crea(ng brokers •  Creates an new profile based on mq-‐base •  Op(onally creates new containers •  Assigns the profile to containers (essen(ally starts the broker)

FuseMQ = ActiveMQ deployed in Apache Karaf +


JBoss A-‐MQ A small-footprint, high-performance, open source messaging platform


JBoss A-‐MQ – Value ProposiIon

Value Proposition: –  Easier to configure, monitor,

manage and maintain: Small IT footprint

–  Multiple deployment options – centralized, distributed, embedded: Small IT footprint

–  Proven and robust platform: Based on popular Apache ActiveMQ, Standards-based reliable messaging & High-performance

–  Multiple connectivity options: Multi-standards and multi-platform support

–  Real-time and reliable integration: Supports messaging paradigms: Pub/Sub, Point-2-Point, Store and forward

New in 6.0: AMQP 1.0 support (Tech Preview)

JBoss Developer Studio to include Fuse IDE

JBoss Operations Network + Fabric Management Console

Rebranding, repackaging


JBoss A-‐MQ -‐ Benefits

Breadth of connecIvity – cross-‐language client support and mul(-‐protocol support provides wide range of connec(vity from browser-‐based clients to mobile devices

Proven reliability and support – proven track record of suppor(ng mission-‐cri(cal applica(ons, and backed by enterprise-‐class services and tools

Small IT footprint – high-‐performance and reliable messaging with small footprint and minimal maintenance

Multiple deployment options – Flexible configuration allows multiple deployment options

No license fees – try before you buy, and deploy widely without incurring exorbitant costs


So where is JBoss A-MQ used ?


Smart Grids …

64


Next GeneraIon Air Traffic Control …

65

NextGen Applications

FTI IP Backbone

En Route Controllers

Terminal Controllers

Non-FAA Users (e.g., Airlines, DoD DHS, etc.) FAA

Command Center

SWIM Enterprise Infrastructure


Well not exactly in space – but close

66


And helping run the biggest science experiment – ever!

67


Btw -‐ Why is IntegraIon and Messaging Important ?

Head Office Enterprises Need to know Everything!


There are more things to Integrate with!


The Internet of Things

•  Uniquely iden(fiable objects and their virtual representa(ons in an internet-‐like structure

•  Originally defined by Kevin Ashton, co-‐founder of Auto-‐ID center at MIT

•  Today its meaning has evolved to encompass a world where physical objects are integrated into the informa(on network, and where physical objects par(cipate in business processes.


Machine to Machine

•  M2M involves collec(on and transmission of event level data (used to be called telemetry) from intelligent devices

•  Machines can interact over the net – and/or alached networks – by wired and wireless networks

•  Payloads are typically minimal (temperature, pressure, loca(on, metering etc.)


Internet of Things

Time

Connected Devices on the internet

2010 2000 2020

1 Billion

50 Billion

10 Billion

Ubiquitous posiConing – locaCng people and everyday objects

RFID tags for Inventory control

Advanced sensors, Enterprises fully connected: integraCon of everything

Geo-‐located devices Microcontrollers internet enabled

Vending machines, lone workers, POS, vehicles, planes, tolls, power systems, transport, telemedicine, home appliances, power systems, etc.


M2M Examples   Smart Energy –

•  Smart Grid uses smart meters for monitoring energy uses by premises for billing, substaCon and transmission line monitoring, energy producCon (renewables)

  Inventory Control –

•  smart labels and RFID tags – connected scanners pass informaCon upstream to servers for monitoring

  In-‐Vehicle Systems

• Maintenance informaCon, global posiConing, ‘black-‐box’ (speed and driving habits), wireless payments for tolls/gas

  Environmental Monitoring

•  Weather sta(ons (temperature, air-‐pressure), ocean monitors, earth movements, volcanic ac(vity crop yields etc.

  Livestock monitoring

• Cow herds, monitored for opCmum Cme for milking, sheep herds for locaCon


M2M Topology

Gateway

Hadoop

Servers

Analytics

Gateway

Internet

Mesh Network

Mobile wireless


Power ConsumpIon! •  Balery life for mobiles, power consump(on for smart

devices is an important considera(on.

•  Facebook use MQTT for real-‐(me updates – efficiency was one of the things considered

•  MQTT is more efficient at establishing connec(ons, sends informa(on more reliably, and consumes less power to transmit data than HTTP.


Deluge of Data •  Billions of devices send informa(on will require storage

and processing of terabytes or even petabytes of data.

•  Big Data Inges(on: Processing of vast amounts of data business generates to make good decisions is only half the problem. Inges(ng that data from all areas of the enterprise will require TBs of data to be ingested in highly reliable and scalable way.


Need an Eco system of technologies

Dev

•  Integra(on is gesng harder •  Tradi(onal message brokers are key

to connect “everything” •  Mul(ple protocols need to be

supported

•  Camel can provide integra(on at the edges – and the data center

•  Technologies like Fuse Fabric provide centralized provisioning and management

•  Need to mix protocols, brokers and “message routers” to scale

FuseMQ Clusters

Arrival Airport 1


Demo Time!


Messaging through the Internet of Things

ActiveMQ

AMQP/MQTT routers

AMQP/MQTT routers

AMQP/MQTT routers

MQTT

MQTT-‐S

AMQP

AMQP

AMQP

MQTT

MQTT

websockets


Simplified demo

Java MQTT Client

AMQP MQTT

JBoss A-MQ MQTT/AMQP Gateway

JBoss A-MQ AMQP/OpenWire Broker JMS OpenWire Client


What is Arduino ?

•  Open Source Hardware microcontroller •  Cheap and easily available. •  Open Source Souware. •  Widespread: many projects. •  Extra HW (shields) available (e.g. Ethernet shield)


USB

7-12 v

3 v GRD 5 v Analog Input Pins

Digital Input/Output Pins Pins with ~ are PWM [Analog Output]

GRD Transmitter/Receiver Serial Connection

Microcontroller ATmega328 Operating Voltage 5V Input Voltage (recommended)7-12V Input Voltage (limits)6-20V Digital I/O Pins 14 (of which 6 provide PWM output) Analog Input Pins 6 DC Current per I/O Pin 40 mA DC Current for 3.3V Pin 50 mA


Arduino Programming

•  3 types of memory on Arduino •  Flash memory (program space) – 32 Kb •  SRAM – used by sketches – 2 Kb •  EPROM – long term memory – 1 Kb

•  Programming language based on Wiring: C/C++ library designed to make input/output easier. Programs are called sketches.

•  Arduino has a simple IDE, available on Windows, Linux and Mac


Arduino MQTT to WebSockets

Arduino

AMQP MQTT

JBoss A-MQ MQTT/AMQP Gateway

JBoss A-MQ AMQP/WebSockets Broker HTML 5/WebSockets


Useful Resources:

  hlp://ac(vemq.apache.org   hlp://camel.apache.org   hlp://fuse.fusesource.org/fabric/docs/overview.html   hlp://fuse.fusesource.org/mq/docs/mq-‐fabric.html   hlps://github.com/fusesource/fuseide   hlp://hawt.io   hlp://www.arduino.cc/

Messaging and Integration�

Developer tooling �

Management�

Configuration and provisioning �

Arduino �


Any Ques-ons ?

London JBUG - Connecting Applications Everywhere with JBoss A-MQ

Technology

Transcript of London JBUG - Connecting Applications Everywhere with JBoss A-MQ