Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears

June 2016

Bilgin Ibryam

@bibryam

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears2

Bilgin Ibryam

● Senior Middleware Architect at Red Hat UK

● Apache Camel Committer and PMC member

● Apache OFBiz Committer and PMC member

● Author of Camel Design Patterns (new)

● Author of Apache Camel Message Routing

● Twitter: @bibryam

● Email: bibryam@gmail.com

● Blog: http://ofbizian.com

● LinkedIn: http://www.linkedin.com/in/bibryam

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears3

Apache Camel Project StatusIt has all necessary ingredients for a successful open source project.

● Community: 52 committers, 903 users

● Support by large vendors (Red Hat)

● Connectors (256), DataFormats (40)

● Enterprise Integration Patterns++

● Domain Specific Language

● Ecosystem: Karaf, ActiveMQ, CXF, Fabric,Hawtio, Spring and others

● Monolith, SOA, Microservices, Serverless

Source https://www.openhub.net/p/camel/

Stats Date: 06/06/2016

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears4

Application Integration with CamelWhat do you need to know to create great Camel applications?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears5

Happy Path ScenariosHow Pipes and Filters Pattern looks like in Camel?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears6

VETROWhat is a typical processing flow for a Camel route?

● Validate: validation, schematron, MSV, Jing, bean validation components

● Enrich: enrich and pollEnrich EIPs, custom beans

● Transform: Data formats, auto type conversion, templating components

● Route: Message routing EIPs

● Operate: this is the essence of the processing flow

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears7

Edge ComponentLet's start with a simple Camel route that consumes files

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears8

Edge ComponentHow to expose the same business functionality to multiple consumers?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears9

Edge ComponentEncapsulate endpoint-specific details and prevent them from leaking intothe business logic of an integration flow.

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears10

Read vs Write OperationsHow to evolve Read and Write operations independently?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears11

CQRSThis decouples read from write operations to allow them to evolveindependently.

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears12

Unhappy Path ScenariosHappy paths are the easy ones. More work is required for designing andimplementing the unhappy paths.

● Data Integrity Pattern ● Saga Pattern● Retry Pattern● Idempotent Filter Pattern ● Circuit Breaker Pattern● Error Channels Pattern● Throttling Pattern

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears13

Data IntegrityHow hard can it be to copy files from one location to another?

Download Data Integrity Chapter: http://bit.ly/came-design-patterns-sample

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears14

Data IntegrityTransactional systems

Local transaction manager

Global transaction manager

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears15

SagaHow to avoid distributed transactions and ensure data consistency?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears16

SagaEnsures that each step of the business process has a compensating actionto undo the work completed in the case of partial failures.

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears17

RetryTo enable applications handle anticipated transient failures by transparentlyretrying a failed operation with expectation it to be successful.

● Which failures to retry? ● How often to retry? ● Idempotency ● Monitoring ● Timeouts and SLAs

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears18

RetryCamel RedeliveryPolicy

● The most well known retry mechanism in Camel● Retries only the failing endpoint ● Fully in-memory● Thread blocking behavior by default● Can be asynchronous● Good for small number of quick retries (in milliseconds)

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears19

RetryActiveMQ consumer RedeliveryPolicy

● Retries the message from the beginning of the Camel route● Not used very often, but enabled by default● Fully in-memory● Thread blocking by default● Good for small number of quick retries (in milliseconds)

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears20

RetryActiveMQ Broker Redelivery

● ActiveMQ specific and requires custom logic● It will consume the message again from a queue● Persisted at the broker rather than application memory● Can be clustered and use fail over, load balancing, etc● Good for long persisted retries (in minutes or hours)

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears21

Circuit BreakerHow to guard a system by cascading failures and slow responses fromother systems?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears22

Circuit BreakerImproves the stability and the resilience of a system by guarding integrationpoints from cascading failures and slow responses.

Closed state

Open state

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears23

Circuit BreakerImproves the stability and the resilience of a system by guarding integrationpoints from cascading failures and slow responses.

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears24

Circuit BreakerTwo Circuit Breaker Implementations in Camel 2.18

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears25

BulkheadHow to enforce resource partitioning and damage containment in order topreserve partial functionality in the case of a failure?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears26

BulkheadEnforces resource partitioning and damage containment in order topreserve partial functionality in the case of a failure.

● Multi-threaded EIPs: Delayer, Multicast, Recipient List, Splitter, Threads,Throttler, Wire Tap, Polling Consumer, ProducerTemplate, and OnCompletion.

● Async Error Handler ● Circuit Breaker EIP

Possible Camel bulkhead points:

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears27

Scalability ScenariosVertical scaling (performance tuning)

● Endpoints: messaging client buffers, DB client batching, template caching choices● Concurrent consumers option: Seda, VM, JMS, RabbitMQ, Disruptor, AWS-SQS● Data types choice: affects content based router, splitter, filter, aggregator● Multithreading: parallel processing EIPs, threads DSL construct, Seda component,

asynchronous redelivery/retry● Micro optimizations: log tuning, camel sampler EIP, disable JMX, disable message

history, disable original message record● Startup/Shutdown: Use lazyLoadTypeConverters for a faster application startup, or

configure the shutdownStrategy for a faster shutdown● Tune: JVM options, networking and operating system

Camel performance tuning blog post: http://bit.ly/camel-tuningCamel performance tuning blog post: http://bit.ly/camel-tuning

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears28

Horizontal ScalingService Instance Pattern for accommodating increasing workloads.

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears29

Service InstanceAreas to consider before horizontally scaling a Camel application.

● Service state: load balancer, circuit breaker, resequencer, sampler,throttler, idempotent consumer and aggregator are stateful EIPs!

● Request dispatcher: Messaging, HTTP, file (what about locking?)

● Message ordering: exclusive consumer, message groups, consumerpriority, message priority, virtual topics

● Singleton service requirements: for batch jobs, and concurrent polling

● Other resource contention and coupling considerations

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears30

What did we cover so far?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears31

How patterns are changing?What is happening in the IT industry today?

● Canonical Data Model● Edge Component● Reusable Route● Runtime Reconfiguration● Singleton Service● Batch jobs in JVM

● Bounded Context● Standalone services● Favor code duplication● Less configuration, more redeployment● Container managed singleton● Container scheduling ● Circuit Breaker, Bulkhead, Health checks..

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears32

Win a print copy of Camel Design Patterns

When was the first commit to Apache Camel project done?

Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears33

Win a print copy of Camel Design Patterns

When was the first commit to Apache Camel project done?

More Information

Learn more about Apache Camel: http://camel.apache.org

Check out Camel Design Patterns: https://leanpub.com/camel-design-patterns

Develop Apache Camel based integrations using Red Hat JBoss Fuse: red.ht/FuseDev