Designing Batch Applications

1

© 2008 IBM Corporation

Snehal S. [email protected] XD Technical LeadSOA Technology Practice, ISSW, SWG, IBMhttp://snehalantani.googlepages.com

Designing Batch Applications

2

2© 2008 IBM Corporation

Agenda• Application Design Goals

– Agile Applications to improve time to market– Ensure High Performance

• Agile Applications– Foundational patterns to use when building applications– Shifting the burden of parallel processing to the infrastructure– Designing Shared Services– An example application architecture

• High Performance Applications– Bulk I/O optimizations– Managing referential data– Parallelizing validations, readers, processors, and writers– Performance analysis and identifying performance regressions

• Development Approaches– Domain-driven development and business-driven assembly– Generating service wrappers– Deployment Strategies

• Batch Data Stream Framework

• Conclusions

3


Goal

Discuss how to build agile, high performance batch applications

High Performance: Exploit parallel processing; leverage bulk IOoptimizations; continuous performance regression testing;

Agile: Share business services across batch & OLTP; Improve time tomarket through function reuse; leveraging container-managed services &frameworks

Designing batch applications can be a challenging task. Two important design facts must be kept inmind: massive volume of records to be processed exacerbates any inefficiencies in the implementation;and agility is important for improving time-to-market, reducing maintenance costs, and improvingapplication stability, where business services can be easily composed into new applications. By followinga set of design principles, we can achieve both performance and agility.

4


Batch Platform + OLTP (WLM Integration, Job Pacing, Job Throttling,..)

Batch Platform (Parallel Processing, Job Mgmt, Scheduler Integration

Batch Container (Checkpoint/Restart, Execution Metrics, …)

EJB 3 EJB 2 SpringBatch

C.G.API

BDSFramework

ibatis

JDBC

JPA

Hibernate

WXS

Pure Q

uery

Data Fabric

BDSFramework

+ Spring

Pojo – Annotation –

AOP

5


Components of a Batch Application

Input

BDS

Step 1

Output

BDS

-Where does the data come from?

- How should the businesslogic process a record?

- Where should the data be written to?

- How should the Step be: - Check pointed? - Results processed? Etc…

… Step N

Input

BDSInput

BDS

Output

BDSOutput

BDS

CompleteBatch Job

Start Batch Job

- Execute Step N if Step N-1 rc = 0

6


Developing Batch Applications

7


Foundational Patterns

Strategy Pattern: allows the isolated replacement of algorithms

Input

Fixed Block DatasetVariable Block

DatasetJDBC

FileIBATIS

DomainObject

Separation of functional concerns: compliments the strategy pattern, where functionalboundaries are decoupled through interfaces

InputStream RecordProcessor

Output

Fixed Block DatasetVariable BlockDatasetJDBCJDBC w/ BatchingFileIBATIS

DomainObject

Strategies forreading data

Strategies forprocessing data

Strategies forwriting dataOutputStream

The first design principle to follow is the strategy pattern, where components within theapplication are decoupled through interfaces. Each component is tasked with solving aspecific problem: reading data, processing data, or writing data. This decouplingenables different algorithms to be plugged into the application with zero impact to theother components.

8


How to think about batch jobs

InputStream RecordProcessorMap data toan inputdomain object

Apply validations andtransformations

OutputStream Map processeddomain object tobytes

Reader Algorithm

Processor Algorithm

Writer Algorithm

- Algorithms should be written to operate on a single record:

- Reader Algorithm is a domain object factory, producing a single record for processing

- Processor Algorithm operates on a single record: is unaware how the record was obtained

- Writer Algorithm maps a single record to bytes: expects a valid record that can be persisted

- Batch Container & Batch Step Wrapper are responsible for invoking each functional component

The input component is a domain-object factory, which maps raw bytes into a validdomain object. The Batch Job Step component contains the business logic, includingvalidations, transformations, and business rules that execute against a valid domainobject. This component doesn’t know, nor should it care, how the domain object wasobtained, it simply expects a complete (valid) domain object. Finally the outputcomponent’s only task is to map a processed domain object to raw bytes. The outputcomponent doesn’t care what processing took place, only that the processed domainobject is valid and can be persisted. Each component operates on a single record, whichis important when discussing parallel processing strategies.

Strict enforcement of this encapsulation ensures agility and service reuse. For example,we can change the source of records from a file to a database by changing the inputimplementation. This change will be in complete isolation of the Batch Job Step andthe Output components. Likewise, we can change the output implementation to persistdata to a fixed-block dataset on z/OS instead of a text file, and this change will beunknown to the other modules.

9


Batch Application Design - The Kernel

Input DS Processor

Validation 1 Validation 2

Validation… Validation N

Output DS

InputDom. Obj

OutputDom. Obj

1. Batch Input Data Stream (Input DS) manages acquiring data and creating domain objects

2. Record processor applies business validations, transformations, and logic on the object

3. Batch Output Data Stream (Output DS) persists the output domain object

4. Processor and OutputDS are not dependent on Input method (file, db, etc)

5. Processor and OutputDS only operate on discrete business records

6. Customers can use favorite IOC container to assemble the Kernel, and use xJCL to wirethe batch flow (input -> process -> output) together.

Kernel

select account … from t1 insert balance into …

the input data stream (Input DS) produces valid input domain objects. One record ispassed to the kernel for processing, where the kernel knows nothing about how thatdomain object was obtained. The kernel, also using the strategy pattern, will apply a setof validations and other business logic to the record. As new requirements are dictated,additional validations can be plugged into the kernel.Finally, the kernel returns the output domain object to the output data stream (OutputDS) for persistence.

Validations and business rules that are applied to a batch application must be carefullydesigned. Validations typically make use of referential data, for example, in order tovalidate whether a bank account is active or not, the account number will get looked upin some database. In the context of batch, the cost of a validation is magnified becauseof the number of records that must be processed. If a validation takes 100 ms, and youmust process 10 million records, the absolute fastest time the job can execute in is 1million seconds. Caching, data-aware routing, and complex SQL queries are a fewstrategies that can be applied to reduce the complexity of the validations. For example,a more efficient way to process valid bank account records is to add a clause in thepredicate of the SQL query, select accounts from accountTable where account.status =‘active’. Each record processed by the kernel will be active; therefore the validation canbe removed.

10


Batch Application Design: Parallel Processing

- Writing multi-threaded code is fun, but hard!

- Multi-threaded code can be difficult to debug, and expensive to maintain

- The degree of parallelism and where parallel jobs are executed should be a point-in-timedecision, where the infrastructure and operations team(s) decide based on:

- Impact on SLA’s & deadlines

- Current data partitioning schemes

- Available capacity of endpoints

InputStream RecordProcessorMap data toan inputdomain object

Apply validations andtransformations

OutputStream Map processeddomain object tobytes

Reader Algorithm

Processor Algorithm

Writer Algorithm

- Algorithms should be written as stateless, single-threaded code that processes 1 record

There are two benefits to designing the kernel to operate on a single record whoseorigin is unknown: first, by writing each module to only operate on one record at atime, we can shift the burden of parallel processing outside of the application code andto the infrastructure; and second, the kernel can evolve to become a service that isshared across multiple execution styles – batch, online transactional processing, etc –without sacrificing performance optimizations that each paradigm provides.

Writing multi-threaded code can be difficult, but can significantly reduce the elapsedtime of a batch job. Maintaining and debugging multi-threaded code though can beexpensive. By designing a kernel that is agnostic to the source of valid domain objects,parallel processing can be handled by the infrastructure.

11


Input DS Processor


Output DS

InputDom. Obj

OutputDom. Obj

Kernel

select account … from t1where account > X and account < Y

insert balance into …

Multi-threadedValidation

CachedValidation

Compute GridParallel Job Manager

X=700Y=999

X=300Y=699

X=000Y=299

Job Instance 1

Job Instance 2

Job Instance 3

Parallelism: An Infrastructure Problem- Compute Grid Parallel Job Manager creates anddispatches multiple instances of the batch job

- Each instance operates on its own data segment usingconstrained SQL queries (for example)

the Input DS can be designed to accept parameters that dictate the range of data thatshould be retrieved. In this example, a constraint in the predicate of the SQL queryis used, but byte ranges in files, record ranges in fixed-block datasets, etc are alsopossible. Some external parallel processing engine can create multiple instances ofa batch job, in this case, the Parallel Job Manager component of WebSphereCompute Grid applies a partitioning algorithm that determines the range of data thateach job instance will operate on. To maximize throughput, the partitioning strategyfor a batch job should match how the data has been partitioned. More over, thepartitioning strategy can independent of the application, where the applicationdeveloper is not responsible for determining how many instances (or threads) mustbe executed, nor on what data each thread will operate. By shifting this burden tothe infrastructure, workload management data, capacity metrics, service-levelagreements, and database optimizations can be used to determine the idealconcurrency setting at a specific moment. The infrastructure optimizations, as wellas the details of how the Parallel Job Manager operates, are outside of the scope ofthis chapter.

12


Sharing Services Across Batch and OLTP

Challenge: share services across different execution styles, without sacrificing theperformance optimizations each style offers

Online Transaction Processing (OLTP): Random data access w/ conversational state.Data grid technologies (ie. WebSphere eXtreme Scale) to improve performance

Batch Processing: Sequential data access. Efficiencies when processing data in bulk

Real-time (Queue-based) Processing: Random data access. Data grid technologies &data-aware routing to improve performance

Future? Map/reduce, cascading, etc

13


Designing a Shared ServiceData Injection versus Data Acquisition

- Data Acquisition: The service is responsible for obtaining the record to process

- Data Injection: The service doesn’t know (nor care) how the record was obtained

- Data Acquisition is the typical approach in OLTP

- Data Injection enables services to be shared across multiple execution styles

- Using ORM wrappers (Hibernate, etc) recklessly can limit the ability to share services!

Public AccountObject AccountReconciliationService(String key) { accountObject = DAL.getAccountObject(key); // code to reconcile account data return accountObject;}

Public AccountObject AccountReconciliationService(String key) { accountObject = DAL.getAccountObject(key); accountObject = reconciliationSharedService.processRecord(accountObject) return accountObject;}

Data Acquisition

Data Injection

14


Designing a Shared Service- Domain-specific wrappers are used to retrieve data

- Wrappers acquire data and pass it to the shared service

Public AccountObject AccountReconciliationService(String key) { accountObject = DAL.getAccountObject(key); accountObject = reconciliationSharedService.processRecord(accountObject) return accountObject;}

OLTP

Public void AccountReconciliationBatchService(String key) { while (batchReader.hasNext()) { accountObject = batchReader.next(); // batchReader optimized for bulk reading accountObject = reconciliationSharedService.processRecord(accountObject) batchWriter.writeAccount(accountObject); // batchWriter optimized for bulk writing }}

Batch

Public void onMessage(Message msg) { accountKey = getAccountKey(msg); accountObject = DAL.getAccountObject(accountKey); accountObject = reconciliationSharedService.processRecord(accountObject) return accountObject;}

JMS

Separating the roles of input, process, and output as outlined in the previous section canhelp ensure that agile, high performance batch applications are built. The next step is tobuild services that can run in batch and OLTP infrastructures without sacrificing theperformance optimizations available in each domain. Batch and OLTP workloads arefundamentally different; therefore there are very different strategies for maximizing thethroughput of each workload type. Traditionally there are two fundamental types ofbatch workloads to consider: first, asynchronous random-access batch jobs; and second,asynchronous, sequential-access batch jobs.

Asynchronous, random-access batch jobs are similar to OLTP applications – data isaccessed randomly and therefore the optimizations available include caching and data-aware routing. Random user-requests are passed to the infrastructure using a queue-based infrastructure such as WebSphere MQ. The message-driven beans (MDB’s)waiting for new work in the queues will execute 1 message (ie. 1 record) at a time.

Batch workloads typically read and write data sequentially whereas OLTP workloadsread and write data randomly. Batch workloads access DB2 directly via JDBC whereasOLTP applications will use an object-relational mapping technology like JavaPersistence Architecture (JPA) or Hibernate.

15


Sharing Services Across Batch and OLTP -The role of Containers

OLTP

EJBContainer

WrapperServices

WrapperServices

CG BatchContainer

Batch

SharedServices

Data Access Layer (DAL)

kernel

Transaction, SecurityDemarcation

DB

Pure Query JDBC SQLJ

- Execution Containers (EJB Container,Batch Container) are responsible formanaging transactions, security, etc.

A key component to modern batch processing is the use of container-managed services;where the batch application only contains business-specific code. J2EE applicationservers provide several containers: Web Container for managing servlets, EJBcontainer for managing Enterprise Java Beans, etc. These containers demarcatetransactions, enforce security roles, integrate with application server services such asconnection and thread pooling, etc. The role of containers, and maintaining theirresponsibilities is important in terms of sharing services across batch and OLTP.WebSphere Compute Grid delivers a batch container, which is a peer to the EJB andWeb containers within an application server. The batch container provides services likecheckpoint/restart, where batch jobs can be restarted at some point in time withcomplete transactional and data integrity, as well as enforce security identities, amongmany other functions. Figure 8 illustrates the role of OLTP containers and the batchcontainer when sharing business services across both of those domains.

16


Developing High Performance Batch Applications

17


Types of Batch Processing - Queue Based

RecordProducer

RecordConsumers

DB

Queue-based batch processing

1. Asynchronously process 1 record at a time(X records = X transactions, X select statements, X RPC’s for insert statements)

2. Used when Random access to data is required

3. Solved using message-driven beans and JMS

4. Resource consumption managed by manipulating number of MDB threads

5. Parallelism provided by running multiple MDB’s across a cluster of servers

6. Limited options for operational control

(Out of Compute Grid’s Scope for now…sort of)

JMS data readers and writers can be used within Compute Grid today. We’reinvestigating ways to improve the management of Queue-based batch, for nowthis pattern is out of the scope of Compute Grid though.

18


Types of Batch Processing -Bulk Record Processing

Select * from table1While (data in cursor) { Tran.begin() { process M records; } Tran.begin() { process M records; } …}

Insert M records, M/N RPC’s

1. Asynchronously process chunks of records at a time(X records = X/M transactions, 1 select, X/N RPC’s for inserts)

2. Should be used for sequential data access

3. Execute a single select against the database using holdable cursors

4. Leverage JDBC Batching for writes

5. Periodically commit M records in a given Checkpoint Interval (aka Syncpoint)

6. Bulk Record Processing significantly reduces infrastructure overhead!

Insert M records, M/N RPC’s

There are four typical performance optimizations that are applied for batch workloads.1. A single select query is executed to retrieve the rows to be processed; the database

cursor is marked to remain open across transactions (a.k.a. Cursor Holdability). Theanti-performance approach is to issue many smaller select queries, where eachquery is demarcated by the global transaction managed by the application server’stransaction manager; upon a global transaction commit, the cursor is closed by thedatabase.

2. Tuning the DB2 z/OS buffer pools to hold larger blocks of data. Since the batchworkloads are processing data sequentially, larger buffer pools enable more data tobe pre-fetched from the database.

3. Data pre-fetching threads within the application server, where background threadsare continuously pre-fetching data to ensure the thread of execution is not blockedand waiting on IO.

4. Leveraging the JDBC batch features of prepared statements in Java. Since data willbe written in bulk, N prepared statements can be grouped and executed together asopposed to executing one statement at a time. There are a number of advantageshere: first, there are M/N remote calls to the database when using JDBC batchversus M remote calls, where M is the total number of statements to be executedand N is the size of each group of statements batched together; second, the databaseis able to optimize the execution of the statements. Database products such as DB2for z/OS, DB2 UDB, and Oracle 10g have different optimal batch values

19


Dealing with Validations, Rules, Referential Data

Processor

Validation 1 Validation 2


InputDom. Obj

OutputDom. Obj

KernelDB

DB

RulesEngine

- Validations often utilize referential data and/or business rules to evaluate a record

- Validations can significantly impact performance

- if 1 validation takes 250 ms, then 1,000,000 records could take ~70 hours to process!

1. Can the validation be removed by adding a predicate to the SQL query in the bulk reader?

2. Can we cache some of the referential data?

3. Can we cache all referential data using a data grid (WebSphere eXtreme Scale)?

4. Can we improve the probability of a cache hit using data-aware routing or larger JVM’s?

20


Parallelizing validations, readers, processors, and writers

Thread

Read record (5 ms)

Process record(20 ms)

Validation 1 (5 ms)

Write record (5ms)

time

Validation 2 (5 ms)

Validation 3 (5 ms)

-1 job runs on 1 thread in Compute Grid

- Discrete tasks can be executed in parallel

- Next N records are fetched on abackground thread

- Validations are executed in parallel

- Discrete records can be processedconcurrently

- Optimizations can be applied transparentlyto the application code (if you listen to us)!

21


Identifying Performance Regressions Early- Inefficiencies in your code are magnified in batch processing!

- Bottom-up performance analysis & performance regression tests are important in batch!

- Simple tools and methodologies can be used to identify performance regressions

InputData

Reader Algorithm v1

“Echo”Processor

System.outWriter

OutputData Elapsed time X

InputData

Reader Algorithm v2

“Echo”Processor

System.outWriter

OutputData Elapsed time Y

- Input Data, Echo Processor, and System.out Writer are constant across tests

- Reader Algorithm v2 is tested for a performance regression

- If Y is z% more than X, there is possibly a performance regression in the reader

- Larger the input data, more accurate the elapsed time becomes.

Test 1

Test 2

- By holding various components constant, we can identify potential performance regressionsin any of the algorithms, validations, etc

The performance analysis of components could be incorporated into a continuous-integration platform as well. Since the input, process, and output modules areindependent from one another, each can be executed separately. For example, a systemcan be built for each module where the input test data is constant, each new version ofthe module is executed with the constant test data, and the elapsed time for eachversion is tracked, where any degradation in elapsed time indicates a performanceregression. This bottom-up performance analysis can help ensure each module is highlyoptimized

22


Identifying Bottlenecks at Run Time

- Functional separation of input - process - output allows us to identify bottlenecks at runtime

- Taking measurements before and after each functional call can identify where to focus

- The larger the input data, the more accurate the measurements become

- Finer-grained time calculations can be enabled within the input, process, and output components

for each record { startTime = System.getCurrentTimeMillis(); record = readRecord(); endTime = System.getCurrentTimeMillis(); recordCount++; totalReadTime = totalReadTime + (endTime – startTime);

startTime = System.getCurrentTimeMillis(); processedRecord = processRecord(record); endTime = System.getCurrentTimeMillis(); totalProcessTime = totalProcessTime + (endTime – startTime);

startTime = System.getCurrentTimeMillis(); writeRecord(processedRecord); endTime = System.getCurrentTimeMillis(); totalWriteTime = totalWriteTime + (endTime – startTime); } // print average & total read/process/write times.

Performance of batch applications tends to be very important. Not only must batchapplications complete within a specific (and often demanding) deadline, but also anyinefficiency in the application is magnified. By leveraging the strategy pattern andclearly separating out the roles of input, process, and output bottom-up performanceanalysis can help monitor and debug performance degradations. For example, by takingstart and end times before and after each call to the input component, we’re able to geta rough idea of how much time is spent fetching records. Using theSystem.currentTimeMillis call can be highly inaccurate when processing one record,but when processing hundreds of thousands of records, this time becomes moreaccurate.

A simple performance test could be to determine the amount of time each componentspent completing their respective tasks, and identifying (relatively) where most of thetime was spent in the batch job. For example, when executing a job and gathering theseperformance metrics, one can determine that 60% of the time was spent reading recordswhile the remaining 40% were split evenly in processing and writing records. The firstplace to further analyze for performance improvements could be the input component.

23


An Approach to Evaluating Performance1. Determine the time to process 1 record

- Cost of the Reader algorithm

- Cost of the processor (and its validations) algorithm

- Cost of the Writer algorithm

2. Calculate the theoretical elapsed time for the job

3. Calculate the actual elapsed time for the job- Run the job sequentially in Compute Grid

- Compare actual elapsed time to theoretical elapsed time

- Vary checkpoint frequency to understand its impacts

4. Prove efficient parallel processing- Run N parallel jobs

- Compare elapsed time to theoretical time / N

- Compare elapsed time to actual sequential time / N

5. Stress & Tune a single JVM

Some common performance issues:

1. Processor is expensive due to inefficient validations2. Some resource is over-synchronized, limiting parallel processing efficiencies3. Log files are very large, and slow down processing due to aggregation (ftp, etc)

24


Batch Data Stream Framework

25


XD Compute Grid makes it easy for developers to encapsulate input/output datastreams using POJOs that optionally support checkpoint/restart semantics.

Checkpoint & Restart with Batch Data Streams

Job Start

BatchContainer

open()

positionAtInitialCheckpoint()

externalizeCheckpoint()

close()

1

2

3

4

Job Restart

BatchContainer

open()

internalizeCheckpoint()

positionAtCurrentCheckpoint()

externalizeCheckpoint()

close()

1

2

3

5

4

26


Batch Data Stream Framework (BDSFW)- Recognized that most of the Batch Data Stream implementations were common acrosscustomer engagements

- ie. How many different ways can you really checkpoint a file?

- Implemented a Template/Callback framework

- Templates: Implement as much of the Batch Data Stream interface as possible

- Callbacks: Interfaces that complete the templates

positionAtInitialCheckpoint() { // get connection // get SQL query // create prepared statement}

AccountReader implements JDBCReaderPattern { //…. public String getSQLQuery() { return “select accounts from accountTable”; } // …

BDSFW JDBCReader Template JDBCReaderPattern Callback

- Best practices can be hardened in the Templates

27


Batch Data Stream Framework (BDSFW)

<bds> <logical-name>inputStream</logical-name> <props> <prop name=”ds_jndi_name" value="${accounts.jndiname}”> <prop name="PATTERN_IMPL_CLASS" value="com.ibm.websphere.batch.samples.accounts.AccountReader”> </props> <impl-class> com.ibm.websphere.batch.devframework.datastreams.patterns.LocalJDBCReader </impl-class></bds>

- xJCL snippet that wires together the Template (LocalJDBCReader) and the Callback(AccountReader)

28


Batch Datastream Framework

Input BDSTemplate RecordProcessor Output BDS

Template

ReaderCallback

Processor Algorithm

Writer Callback

- Selector pattern, applied via xJCL, for choosing which input and output BDS template to use

-GenericXDBatch Step implements the various patterns described, including:

- Performance metrics (coarse-grained & fine-grained) for functional components

- Strategy pattern enabling algorithms & call-backs to be plugged in

- Threshold policies, transactional error streams, etc

- Futures: parallel readers/processors/writers can be built into GenericXDBatchStep, andtransparently applied to customer code.

GenericXDBatchStep

29


Compute GridPojo-based App

End-to-end Development tooling

CG BDS Framework

Eclipse-basedCG Batch Simulator

Single ServerCG Batch Unit & Function

Test Environment

CG Batch Packager

CommonDeployment Process

Compute GridInfrastructure

-Customer developsbusiness service POJO’s

-Applications areassembled via IOCContainer

-XD BDS Frameworkacts as bridge betweenjob business logic andXD Compute Gridprogramming model

-XD Batch Simulator fordevelopment

-XD Batch Unit testenvironment for unittesting

-XD batch packager for.ear creation

BusinessServices

Java IDE

Business ServicesTesting Infrastructure

Single Server Unit-testingfor OLTP

30


Development Approaches

31


A Batch Application Development Approach (1)Design Patterns, Domain-Driven Development, Business-Driven Assembly

BDSFW Steps

Pojo Input Streams(Dom. Obj. Factories)

Shared LibrariesShared Libraries Shared Libraries

Pojo Business Logic(Dom. Obj. Processors)

Pojo Output Streams(Dom. Obj. Persisters)

xJCL TemplatexJCL TemplatexJCL Template

App.ear

�XML-based App Development

Business Analysts

Rules Validation & Transformation Tools

Developers

�XML-based App Development�XML-based

App Development�XML-based

App Development�XML-based

App Development

By focusing on building high-performance components, the assembly of thesecomponents into new batch jobs helps ensure that these new jobs are already welloptimized. By building highly optimized, loosely integrated components, it ispossible to build a library of modules that can be easily assembled into new batchapplications.

WebSphere Compute Grid xJCL definitions can be used to assemble the various input,processing, and output components into batch jobs. By building a bottom-upperformance analysis process, each module will ideally be optimized, helping toensure that the new jobs also perform well. Further processes can be built aroundthis library of functions, where application developers’ focus on building morefunctions into the libraries, and business analysts can assemble new functionswithout having to develop or understand the underlying code.

32


A Batch Application Approach (2)

Dependency Injection & AOP Container

Data Fabric for Referential Data

public oDTO sharedService (iDTO)OLTP Client

iDTO

oDTO

BDSFW Service Wrapper

Bulk Data Reader

Bulk Data Writer

iDTO oDTOBatch Container

Web Service, EJB, etc…

OLTP Container

-Application developers are exposed to Pojo’s and the DI/AOP Container

- OLTP and Batch Wrappers can be generated

- DBA’s can focus on writing highly-tuned SQL statements for bulk reading/writing

The shared service is data injected, where the domain object (iDTO, which stands forinput data transfer object) is passed into the service by some external mechanism. Theprocessed domain object (oDTO, which stands for output data transfer object) isreturned by the shared service to the caller. When this service is executed in OLTPmode, an OLTP wrapper (Web Service, EJB, etc) will obtain the iDTO, typicallythrough some OLTP-oriented data access layer, and invoke the shared service. Whenexecuting in batch mode, bulk data readers and writers, managed by some batchcontainer, will feed records into the shared service. A more concrete example of thisshared-services architecture is available in the backup.

33


Deployment Strategies

- 1 ear per batch step?

- 1 ear per “domain” (related business function, owning department, etc), shared libraries forBDSFW implementation classes

- Only 1 ear, shared libraries for BDSFW implementation classes

- Need to consider lifecycle management issues (versioning, continuous availability, versionaffinity, etc)

34


Conclusions

35


Cool ideas we’re looking at:

1. Custom GC policies and heap management for batch

2. Transparently applying multi-threaded readers/processors/writers

3. Container manages the number of threads to maximize throughputand reduce memory

36


Why Maverick Batch is BAD…

• Customers are not in the business ofbuilding/owning/maintaining infrastructure code

– Developers love writing infrastructure code– IT Managers avoid owning and maintaining infrastructure code– IT Executives hate paying for code that doesn’t support the core business

• Learn from history…… “Maverick” OLTP in the mid-1990’s… WebSphere emerged to stamp out “Maverick” OLTP… OLTP has evolved…… It’s time for Compute Grid to stamp out “Maverick” Batch

37


Business Benefit… Cut IT development, operations, and maintenance costs by pursuingthe “Unified Batch Architecture” strategy with Compute Grid

An important challenge customers face is in the area of OLTP + batchinterleave, where batch jobs are no longer run within a batch window, but rather24x7 in "micro-batches". A number of challenges emerge that can be solved bya smart workload manager and tight integration between the batch runtime andthe database. System Z and z/OS are really the only technologies positioned todeliver this required integration (though we're not quite there yet today).

Consolidated Scheduling: z/OS operations teams understand how toschedule. They have a tremendous amount of experience withTWS/OPC/Control-M/CA7 and other such enterprise schedulers. They've builtup quite a bit of infrastructure for auditing and archiving logs w/ Beta92,security, etc. Unfortunately, due to technical limitations of enterprise schedulersover the past few years, multiple scheduling domains have emerged where ascheduling environment is created just for distributed batch workloads andhaphazardly integrated with the z/OS scheduler. We propose that theenterprise scheduling tier be consolidated to z/OS, where all operational controlis managed centrally; WebSphere XD Compute Grid then becomes thecommon batch execution platform to which enterprise schedulers dispatchwork. Compute Grid can then manage the execution of transactional batchworkloads, which run whereever our Grid Endpoints run (and that should be*everywhere*). The goal though is to centralize operational control andmanagement for batch workloads (old and new) across the entire enterprise.

Common Application Architecture for new batch: As customers build up theirnew batch infrastructures, which probably will be written in Java, these newapplications will most likely be independently developed. This result is manyindependent application architectures and home-grown plumbing code thatcustomers have to develop and maintain. Chris and I call this "maverick batch".Numerous operational issues come about because many of these maverickbatch systems are poorly designed and fail to take into account things likeOLTP+Batch interleave. More over, it is not in the best interest of customers todevelop and maintain plumbing code. We don't encourage them to go writeapplication servers, why encourage them to write batch containers?

Unified Batch Architecture: we should standardize the batch platform itselfacross the entire enterprise. Where customers have a single applicationarchitecture (for new batch), standardized hosting infrastructure, and singlescheduling domain for all batch in their enterprise. The new batch runtimeshould transcend platform, where batch applications are platform neutral, andthe location of the data the application requires dictates the placement of theapplication. Since most enterprise data is on z/OS, the batch applications canbe deployed and executed on z/OS, taking advantage of its IO optimizations.More over, platform-neutral batch applications enable new workloads to bemoved to z/OS with little fear about platform incompatibility.

Data Consolidation as the next step: The next wave of consolidation mayvery well be focused on data; where various disparate databases and filestorage are consolidated to a single tier and perhaps a single database andstorage technology. Given the IO optimizations of z/OS, DB2 for z/OS and ZFSbecome a viable target platform on which to consolidate. A platform neutralbatch platform enables customers to pursue data consolidation without fear ofrewriting their applications.

WebSphere XD Compute Grid, as a platform neutral, java-based batchexecution infrastructure, becomes an integral part of the Unified BatchArchitecture strategy. I've written about this as part of our overal productstrategy and vision in an internal blog post:https://w3.tap.ibm.com/weblogs/javabatchinitiative/entry/pinkie_and_the_brain_world. I also have written about how to steer the conversation in competitivesituations back to z/OS:https://w3.tap.ibm.com/weblogs/javabatchinitiative/entry/fighting_off_the_vultures_bv

38


The Batch Vision

1. Batch Containers should run everywhere

2. Portable Batch applications across platforms and J2EE vendors

3. Location of the data dictates the placement of the batch application

4. Centrally managed by your enterprise scheduler

5. Integrating with existing: Disaster Recovery, Auditing, Logging, Archiving

Enterprise Scheduler

Existing Business Processes

Common Batch Application Architecture and RuntimeApp 1 App 2 App 3 App 4 …. App N

WAS DistrWAS CEJEE Server WAS z/OS

DB2 z/OSDB2 UDBWXSNon-DB2

39


WebSphere XD Compute Grid Summary• IBM WebSphere XD Compute Grid delivers a complete batch platform

– End-to-end Application Development tools– Application Container with Batch QoS (checkpoint/restart/etc)– Features for Parallel Processing, Job Management, Disaster Recovery, High Availability– Scalable, secure runtime infrastructure that integrates with WebSphere Virtual Enterprise and WLM on z/OS– Designed to integrate with existing batch assets (Tivoli Workload Scheduler, etc)– Supports all platforms that run WebSphere, including z/OS.– Experienced Services and Technical Sales resources available to bring the customer to production

• Is ready for “prime time”. Several customers in production on Distributed and z/OS today1.Swiss Reinsurance, Public Reference, Production 4/2008 on z/OS2.German Auto Insurer, Production 7/2008 on Distributed3.Turkish Bank, Production on Distributed4.Japanese Bank, Production on Distributed5.Danish Bank, Pre-production on z/OS6.Wall Street Bank (two different projects, globally deployed), Production on Distributed7.South African Bank, Pre-production on Distributed8.Danish business partner selling a core-banking solution built on Compute Grid.9.German Life Insurance Company. Pre-production on z/OS10.Spanish Bank. Pre-production on Distributed.– Numerous other customers in pre-production & PoC

•Vibrant Customer Community–Annual customer conference held in Zurich, hosted by Swiss Re.–User group established for sharing best practices and collecting product requirements–Over 400,000 reads in the Compute Grid developers forum since January 22nd, 2008. (~5k reads per week)

40


Backup

41


Execution Agnostic Application “Kernel”

The following example demonstrates how to share services across batch and OLTP.This example shares an account reconciliation service across batch and OLTP, wherethe BDS Framework is used to integrate the service with the Compute Grid batchcontainer. To share the service, the application architecture is divided into severalsections. The application kernel is composed of the domain object, the record-processorthat is the shared business service, and the several business validations. The kernelapplication is then wrapped by code specific to each execution paradigm: a batch stepwrapper that connects batch data-streams to the kernel service; and an OLTP wrapperthat interacts with the data-access layer to retrieve a single business record to beprocessed.

The shared-service, AccountReconcilliationSharedService in this case, is designed tobe indifferent of the source of the data to be processed; the service accepts a businessrecord (a domain object), applies business validations to that business object, and eitherexecutes the business logic or signals to the caller that the record was not processed.

42


Compute Grid (Batch) Wrapper to the Shared Service

When processing in batch-mode, the Batch Data Stream (BDS) Framework connectsthe data reader and writer batch-data streams to the batch record processor.

The classes DataReader and DataWriter implement the relevant reader and writerpatterns from the BDS Framework.-DataReader: The role of this class is to map the raw data to theAccountDomainObject. The input can be from one of two types of sources: a file and adatabase. Therefore this data reader must implement both the FileReaderPattern andJDBCReaderPattern interfaces from the BDS Framework.

- AccountReconciliationSharedService: the role of this class is to process each batchrecord which are AccountDomainObject’s. Each batch record must be validated;AccountReconcilliationSharedService therefore passes the AccountDomainObject toeach validator specified. If any validations fail, NULL is returned to the BDSFramework. If all validations pass, the account balance of the domain object isreconciled and processed record returned to the BDS Framework, which will write theoutput to the outputStream.

-DataWriter: the role of this class is to map the domain object to raw data and write tothe appropriate location. In the case of this batch step, the output type can either be adatabase or a file, therefore SaldosDataWriter implements both the JDBCWriterPatternand FileWriterPattern interfaces in the BDS Framework.

43


OLTP Wrapper to the Shared Service

When operating in OLTP mode, the shared-service is wrapped with codewhose purpose is to interface with the OLTP data-access layer (DAL) to retrieve therelevant business record to be processed. The OLTP wrapper then invokes the sharedservice, passing the acquired business record as a parameter, similar to how the servicewould be invoked in batch-mode. The important difference here is that when in batch-mode, the input and output streams are optimized to read and write sequential data inbulk. When in OLTP mode, the DAL is optimized to read and write a single, randombusiness record.

As we can see in this diagram, the AccountReconciliationSharedServiceOLTP wrapper interacts with the data-access Layer (DAL) and the AccountObjectDAOdata-access object (DAO). Using the common factory/impl design pattern, we’re ableto interchange the various persistence technologies (JDBC DAO, Hibernate DAO, PureQuery DAO, etc) in a way that is transparent to the rest of the application.

The service, upon initialization, gets a handle to the DAL and theconfigured DAO. The reconcileAccount() method is then invoked in an OLTP context;where the service retrieves the business record to reconcile from the DAO using theprimary key passed to it. That business record, which is the shared domain object, isthen passed to the shared business service in the same way the batch wrapper wouldhand a business record. As previously stated, the shared service was written to beindifferent about the source of the business record to be processed, therefore the samebusiness validations and logic are used from both the OLTP and batch contexts.

44


Batch Container

BatchD

ataStream

Interface

initialize(props)

open()

externalizeChecpoint()


close()

JobStepInterface

destroyJobStep()

processJobStep()

createJobStep

setProperties()

BatchRecord

Processor

initialize()

processRecord()

completeR

ecord()

FileReaderP

attern

initialize(props)

fetchRecord()

processHeader()

BatchD

ataStream

Interfaceinitialize(props)

open()

externalizeChecpoint()


close()

FileWriterP

atterninitialize(props)

writeRecord()

writeHeader()

ByteR

eaderPatternA

dapter

GenericXDBatchStep ByteW

riterPatternA

dapter

The BDS Framework -Implements checkpoint/restart for common IO types

Designing Batch Applications

Documents

Transcript of Designing Batch Applications