Russell 2012 introduction to spring integration and spring batch

Introduction toSpring Integration and Spring Batch

© 2012 SpringOne 2GX. All rights reserved. Do not distribute without permission.

Gary Russell, Staff Engineer, SpringSource; @gprussell

Tuesday, December 18, 12

What we will cover...

• Spring Integration• Spring Batch• Using Spring Batch and Spring Integration together

2Tuesday, December 18, 12

Spring Integration


Integration Styles

• Business to Business Integration (B2B)• Inter Application Integration (EAI)• Intra Application Integration

4

JVM JVM

EAI

External Business Partner

B2B

Core Messaging


Integration Styles

• File Transfer• Shared Database• Remoting• Messaging


Common Patterns

6

Retrieve Parse Transform Transmit


Enterprise Integration Patterns

• By Gregor Hohpe & Bobby Woolf• Published 2003• Collection of well-known patterns• http://www.eaipatterns.com/eaipatterns.html


http://www.eaipatterns.com/eaipatterns.html

http://www.eaipatterns.com/eaipatterns.html

8

“Spring Integration provides an extension of the Spring programming model

to support the well-known enterprise integration patterns.


What is Spring Integration?

• Light-weight messaging framework• Provides an adapter-based platform for communicating with external

systems

• Pipes and Filters at the core of Spring Integration’s architecture–Endpoint (Filter)–Channel (Pipe)–Message


Advantages

• Provides building blocks to implement systems that are:– are loosely Coupled (Logically or Physically)– are Event Driven (EDA)– have a staged event-driven architecture (SEDA)

• Sophisticated support for synchronous / asynchronous messaging


Configuration

• XML Namespace Support• Annotation Support (e.g. @Transformer, @Router, @ServiceActivator)

11

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation="..">... <int:channel id="orders"/> <int:splitter input-channel="orders" expression="payload.items" output-channel="drinks"/> <int:channel id="drinks"/> <int:router input-channel="drinks" expression="payload.iced ? 'coldDrinks' : 'hotDrinks'"/>...</beans>


Spring Integration DSLs

• Scala

• Groovy

12

val messageFlow = filter {payload: String => payload == "World"} --> transform { payload: String => "Hello " + payload} --> handle { payload: String => println(payload) } messageFlow.send("World")

def builder = new IntegrationBuilder()

def flow = builder.messageFlow { transform {payload->payload.toUpperCase()} filter {payload-> payload =="HELLO"} handle {payload->payload}}

assert flow.sendAndReceive("hello") == "HELLO"assert flow.sendAndReceive("world") == null


What is in a Message?

• Unit of information• Encapsulates data• Passed between endpoints• Consists of headers

– contains data relevant to the messaging system• and a payload

– actual data for the receiver– depending on use-cases: POJO instances or serialized data


What is in a Message?

14

package org.springframework.integration;

public interface Message<T> {

MessageHeaders getHeaders();

T getPayload();

}


Message Headers

• Message ID (automatically generated UUID)• Timestamp• Correlation Id• Reply Channel• Error Channel• Expiration Date• Priority• ...

• Add your own headers using a Header Enricher


Function of a Message

• Command Message

• Event Message

• Document Message

16

E

D

C


What is a Channel?

• Channels connect producers and consumers (decoupling)• MessageChannel Interface:

–PollableChannel (Polling Consumer)–SubscribableChannel (Event Driven)

• Implementations:–DirectChannel–PublishSubscribeChannel–QueueChannel–PriorityChannel–RendezvousChannel–ExecutorChannel

17

<int:channel id="input"> <int:queue capacity="10"/></int:channel>


What is an Endpoint?

• Polling or event-driven• Inbound or outbound• Unidirectional (Channel Adapter) or bidirectional (Gateway)• Internal or external (application context)

18

<inbound-channel-adapter/><outbound-channel-adapter/><inbound-gateway/><outbound-gateway/><gateway/><service-activator/>


Router

• Message Router• Content-based router• Recipient list router (with selectors)• Payload type router• Header value router• Exception type router


Transformer

• Delegating via ref/method• Spring Expression Language• Groovy, JRuby, Jython, JavaScript• Object-to-JSON / JSON-to-Object• Payload serializing/deserializing• File-to-bytes, File-to-String• JAXB, JibX, Castor, XMLBeans, Xstream• XPath, XSLT• Object XML Marshalling/Unmarshalling (Spring OXM)• ...


Spring Integration Components

21

• Claim Check (In/Out)• Content Enricher

–Header Enricher–Payload Enricher

• Control Bus• Delayer• JMX Support• Message Handler Chain• Messaging Bridge• Resequencer

• Service Activator• Scripting support (JSR 223)

– Ruby/JRuby, Javascript ...• Groovy• Message History• Message Store

– JDBC, Redis, MongoDB, Gemfire

• Wire Tap• ...


Adapters

22

• AMQP/RabbitMQ• AWS*• File/Resource• FTP/FTPS/SFTP• GemFire• HTTP (REST)• JDBC• JMS• JMX• JPA

• MongoDB• POP3/IMAP/SMTP• Print*• Redis• RMI• RSS/Atom• SMB*• Splunk*• Spring Application

Events

• Stored Procedures• TCP/UDP• Twitter• Web Services

(SOAP or POX)• XMPP• XPath• XQuery*

• ...* Spring Integration Extensions Project


Tooling - Spring Tool Suite (STS)

• Namespace Support• Visualization• 4 Spring Integration specific STS Templates

–Simple Template (Core Components only)–File Polling Template (File Adapter)–War Template (Uses Twitter Adapter)–Adapter Template (Create your own components)


Samples

• https://github.com/SpringSource/spring-integration-samples

• Contains 50 Samples and Applications• Several Categories:

–Basic–Intermediate–Advanced–Applications


https://github.com/SpringSource/spring-integration-samples


Books

• Just Spring Integration• Pro Spring Integration• Spring Integration in Action


What’s new in Spring Integration 2.2

• JPA support– http://blog.springsource.org/2012/10/05/whats-new-in-spring-integration-2-2-part-3-jpa-support/

• MongoDB adapters– http://blog.springsource.org/2012/09/24/whats-new-in-spring-integration-2-2-rc1-part-1-mongodb/

• Transaction synchronization– http://blog.springsource.org/2012/09/26/whats-new-in-spring-integration-2-2-part-2-transaction-

synchronization/

• Retry, Circuit Breaker and ExpressionEvaluatingRequestHandlerAdvice– http://blog.springsource.org/2012/10/09/spring-integration-2-2-retry-and-more/

• Redis Store Adapters


http://blog.springsource.org/2012/10/05/whats-new-in-spring-integration-2-2-part-3-jpa-support/

http://blog.springsource.org/2012/10/05/whats-new-in-spring-integration-2-2-part-3-jpa-support/

http://blog.springsource.org/2012/09/24/whats-new-in-spring-integration-2-2-rc1-part-1-mongodb/

http://blog.springsource.org/2012/09/24/whats-new-in-spring-integration-2-2-rc1-part-1-mongodb/

http://blog.springsource.org/2012/09/26/whats-new-in-spring-integration-2-2-part-2-transaction-synchronization/




http://blog.springsource.org/2012/10/09/spring-integration-2-2-retry-and-more/




Source Code

• https://github.com/SpringSource/spring-integration• https://github.com/SpringSource/spring-integration-samples• https://github.com/SpringSource/spring-integration-extensions• https://github.com/SpringSource/spring-integration-templates• https://github.com/SpringSource/spring-integration-dsl-groovy• https://github.com/SpringSource/spring-integration-dsl-scala


https://github.com/SpringSource/spring-integration

https://github.com/SpringSource/spring-integration



https://github.com/SpringSource/spring-integration-extensions

https://github.com/SpringSource/spring-integration-extensions

https://github.com/SpringSource/spring-integration-templates

https://github.com/SpringSource/spring-integration-templates

https://github.com/SpringSource/spring-integration-dsl-groovy

https://github.com/SpringSource/spring-integration-dsl-groovy

https://github.com/SpringSource/spring-integration-dsl-scala

https://github.com/SpringSource/spring-integration-dsl-scala

Contribute

• Post Question and Answers the Forums–http://forum.springsource.org/forumdisplay.php?42-Integration

• Create Jiras–https://jira.springsource.org/browse/INT

• Submit Pull Requests - Contributor Guidelines:– github.com/SpringSource/spring-integration/wiki/Contributor-Guidelines

• New Spring Integration Extensions Repository


http://forum.springsource.org/forumdisplay.php?42-Integration

http://forum.springsource.org/forumdisplay.php?42-Integration

https://jira.springsource.org/browse/INT

https://jira.springsource.org/browse/INT

https://github.com/SpringSource/spring-integration/wiki/Contributor-Guidelines




Spring Batch


30

Batch JobsDiffer from online/real-time processing applications:

• Long-running– Often outside office hours

• Non-interactive– Often include logic for handling errors or restarts

• Process large volumes of data– More than fits in memory or a single transaction


Batch and offline processing• Close of business processing

– Order processing– Business reporting– Account reconciliation

• Import/export handling– a.k.a. ETL jobs (Extract-Transform-Load)– Instrument/position import– Data warehouse synchronization

• Large-scale output jobs– Loyalty scheme emails– Bank statements

• 31Tuesday, December 18, 12

Job and Step


Chunk-Oriented Processing• Input-output can be grouped together• Input collects Items before outputting:Chunk-Oriented Processing• Optional ItemProcessor


JobLauncher


Simple File Load Job


More Complex Use Cases• It's very common to use an off-the-shelf reader

and writer• More complex jobs often require custom readers

or writers• ItemProcessor is often used if there's a need to

delegate to existing business logic• Use a writer if it's more efficient to process a

complete chunk


Job and Step in Context


JobRepository and Batch Metadata


ExecutionContext• We need to know where a failure occurred to restart a batch

process• Job Repository metadata is used to determine the step at

which the failure occurred• Application Code (in reader/writer) needs to maintain state

within a step (e.g. current chunk)• Spring Batch can supply that data during restart to facilitate

repositioning


Common Batch Idioms• Batch jobs typically process large amounts of homogeneous

input• Makes iteration a common concern: Repeat• Transient errors during processing may require a Retry of

an input item• Some input may not be valid, may want to Skip it without

failing• Some errors should fail the job execution, allowing one to fix

the problem and Restart the job instance where it left off


Spring Batch• Spring Batch supports these common concerns

• Abstracts them in the framework– Job business logic doesn't need to care about details

• Allows for simple configuration with pluggable strategies


Business Logic Delegation – Spring Application


Spring Batch Admin• Sub project of Spring Batch• Provides Web UI and ReSTFul interface to manage batch

processes

• Manager, Resources, Sample WAR– Deployed with batch job(s) as single app to be able to control &

monitor jobs– Or monitors external jobs only via shared database


Scaling and Parallel Processing• First Rule:

– Use the simplest technique to get the job done in the required time

– Do not optimize/parallelize unnecessarily

• Options:– Multi-threaded Step (single process)– Parallel Steps (single process) – Remote Chunking of Step (multi process)– Partitioning a Step (single or multi process)


Using Spring Batch and

Spring Integration Together


Launching batch jobs through messages

• Event-Driven execution of the JobLauncher• Spring Integration retrieves the data (e.g. file system, FTP, ...)• Easy to support separate input sources simultaneously

46

D C

FTP

Inbound Channel Adapter

JobLauncher

Transformer

FileJobLaunchRequest


Providing feedback with informational messages

• Spring Batch provides support for listeners:– StepListener– ChunkListener– JobExecutionListener

47

<batch:job id="importPayments"> ... <batch:listeners> <batch:listener ref="notificationExecutionsListener"/> </batch:listeners></batch:job>

<int:gateway id="notificationExecutionsListener" service-‐interface="o.s.batch.core.JobExecutionListener" default-‐request-‐channel="jobExecutions"/>


Externalizing batch process execution

• Use Spring Integration inside of Batch jobs, e.g.:– ItemProcessor– ItemWriter

• Offload complex processing• Asynchronous processing support:

– AsyncItemProcessor– AsyncItemWriter

• Externalize chunk processing using ChunkMessageChannelItemWriter


Business Logic Delegation – Spring Integration


•

Questions

50

Questions?

Thank You!!


Russell 2012 introduction to spring integration and spring batch

Documents

Transcript of Russell 2012 introduction to spring integration and spring batch