Spring-batch Groovy y Gradle
-
Upload
antonio-mas -
Category
Software
-
view
66 -
download
6
Transcript of Spring-batch Groovy y Gradle
![Page 1: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/1.jpg)
Spring Batch Groovy & Gradle integrationSpring Batch Groovy & Gradle integration
Because we like it…Because we like it… because we have fun with itbecause we have fun with it
![Page 2: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/2.jpg)
About me:About me:
Born with C and Java
Github: http://github.com/amr390 (mostly drops, and tests)
Working with Java and Javascript, take any opportunity to use groovy and python
(I love for both of them)
Grown up with Javasince 2000
Twitter: 390amr (not much into it)
![Page 3: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/3.jpg)
Spring Batch. Groovy and GradleSpring Batch. Groovy and Gradle
What is Spring BatchWhat is Spring Batch
Why integrating spring-batch with Groovy and GradleWhy integrating spring-batch with Groovy and Gradle
Requirements Requirements ➢ DependenciesDependencies
➢ Gradle tasksGradle tasks
What is Spring BatchWhat is Spring Batch
➢Configuring Jobs, Tasks, Steps and stufConfiguring Jobs, Tasks, Steps and stuf
➢Running and Handling JobsRunning and Handling Jobs
Improving Spring Batch performanceImproving Spring Batch performance
➢Multi-threadingMulti-threading
➢Master-Slave approachMaster-Slave approach
➢Parallelizing Step executionsParallelizing Step executions
![Page 4: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/4.jpg)
What is Spring Batch I
Batch Framework
Technical services
Functionality to process large volumes of records
ItemReader, ItemWriter, ItemProcessor
Stateless or Stateful step executions
High-volumeHigh-performanceHigh-scalability
Trough optimization Partitioning techniques
![Page 5: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/5.jpg)
What is Spring Batch IIWhat is Spring Batch II
![Page 6: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/6.jpg)
What is Spring Batch IIIWhat is Spring Batch III
![Page 7: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/7.jpg)
What is Spring Batch IVWhat is Spring Batch IV
STEP:➔ Custom tasklet➔ Restartable➔ Skippable➔ Retry➔ Rollback➔ Listeners Before/After
✗ Reader✗ Processor✗ Writer✗ Chunk✗ Step
➔ Step flow (on status)
![Page 8: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/8.jpg)
What is Spring Batch: exampleWhat is Spring Batch: example
![Page 9: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/9.jpg)
Philae Lander dataPhilae Lander data
Philae Lander is a spacecraft that sends analysis data from comet 67P/Churyumov–Gerasimenko (https://en.wikipedia.org/wiki/Philae_(spacecraft))
Sends tones data in files encrypted in Klingon or Murciano.
Files are zipped, and organized with a naming rule of file1, file2 etc ...
![Page 10: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/10.jpg)
It is always fun using groovy
Can be nicely coupled to other platforms Grifon, Grails...
Power of Gradle
Homogeneous Architecture
Why integrating Spring-batch with Why integrating Spring-batch with groovy and Gradlegroovy and Gradle
![Page 11: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/11.jpg)
Plug insPlug insRepositoriesRepositories
DependenciesDependencies TasksTasks
![Page 12: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/12.jpg)
Requeriments IIRequeriments II
Repositories
Plugins
![Page 13: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/13.jpg)
Dependencies IIIDependencies III
Dependencies
![Page 14: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/14.jpg)
Requirements IVRequirements IV
Gradle tasks
MainClassName: Sets the project main class (because it is a global definition)
run: sets the required arguments for run task, in this case the job configuration class and the Job to be launched.
![Page 15: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/15.jpg)
@Configuration@EnableBatchProcessing
Injects
a JobRepository a JobLauncher a JobRegistry a PlatformTransactionManagera JobBuilderFactorya StepBuilderFactory
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Job definition configurations, steps, tasklets and stuf
![Page 16: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/16.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Running and Handling Jobs
Spring batch admin
![Page 17: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/17.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Running and Handling Jobs
Job Launchers
org.springframework.batch.core.launch
Interface JobLauncher
All Known Implementing Classes:
SimpleJobLauncher
JobExecution run(Job job, JobParameters jobParameters) throws JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException
![Page 18: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/18.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Running and Handling Jobs
Job Launchers
org.springframework.batch.core.launch.support
Class CommandLineJobRunner
![Page 19: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/19.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Performance improvement
There different approached in order to improve the job execution performance
Multi-threading: Using a ThreadPoolTaskExecutor
Remote chunking:Master executes read phase of step slaves are distributed in nodes and execute processing and writing from a chunkProvider.
Partitioning: Clone Job execution step and run steps in parallel
Remote Partitioning:
Instead of running the steps clone locally steps are
distributed
Parallel steps:step execution is splitted to
execute several steps in parallel flows.
![Page 20: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/20.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Multi
threading
![Page 21: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/21.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Multi
threading
Using task namespace <task:executor id="taskExecutor" pool-size="10" />
By java configuration method:ThreadPoolTaskExecutor taskExecutor ...
Steps must be thread safe
![Page 22: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/22.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Master-Slave
approach
![Page 23: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/23.jpg)
PartitionHandler: Interface defining the responsabilities of controlling the execution of partitioned step
Partitioner: Interface to determine the way of creating partitions (key-range, file-chunks...)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Master-Slave
approach
![Page 24: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/24.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Parallelizing Step executions
![Page 25: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/25.jpg)
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
Parallelizing Step executions
@Beanpublic SimpleFlow mainFlow() { SimpleFlow splitFlow = new FlowBuilder<SimpleFlow>("Split Flow") .split(new SimpleAsyncTaskExecutor()) .add(flow2(), flow3()) .build(); return new FlowBuilder<SimpleFlow>("Main Flow") .start(flow1()) .next(splitFlow) .end();}
<batch:step id="step1" next="">...</batch:step><batch:split id="splitSteps" next="step3">
<batch:flow> <batch:step … /></batch:flow><batch:flow> <batch:step … /></batch:flow>
</batch:split><batch:step id="step3">
<batch:tasklet ref="tasklet" /></batch:step>
![Page 26: Spring-batch Groovy y Gradle](https://reader034.fdocuments.in/reader034/viewer/2022050719/55cc527fbb61eb47638b483f/html5/thumbnails/26.jpg)
Demo Github: https://github.com/amr390/groovy-springbatch-demo
Spring batch reference page: http://docs.spring.io/spring-batch/trunk/reference/Spring batch in action: http://www.manning.com/templier/
Spring batch horizontal-vertical scaling:http://www.ontheserverside.com//blog/2014/07/23/horizontal-and-vertical-scaling-strategies-for-batch-applications
Spring batch admin: http://docs.spring.io/spring-batch-admin/
Spring Integration reference page: http://docs.spring.io/spring-integration/reference/
Groovy: http://www.groovy-lang.org/documentation.html
Gradle: https://docs.gradle.org/current/release-notes
Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration
References