Using MRG and Infinispan
for
Large Scale Integration
Prajod Vettiyattil
2
What this session is about
Challenges in Large Scale integration
Use cases for Large Scale integration
How to solve the challenges Solution Products to implement the
solution
The Open Source difference
3
Challenges
Use Cases
Solutions
Key Phrases
4
Phrases: 1
• Large Scale Integration – Integration of 10s or 100s of systems, and
exchange GBs of messages in a day
• Big Data – A changing threshold – Data in the Terabytes, Petabytes, Exabytes…
• Asynchronous Messaging – Message oriented middleware
• Real time systems – Systems that are built to respond to requests in
real time, with predicable, consistent response times
5
Phrases: 2
• Grid – A set of interconnected computers that work in
parallel to solve a computing problem
• Cloud Computing – Computing as a service – Client of the cloud is isolated from the details of
the implementation of the service
6
7
Challenges
Challenges
Use Cases
Solutions
The Key Challenges Large number of systems
8
The Key Challenges Complexity of connection between these systems
9
The Key Challenges
Constraints on the systems and on the connections
10
soap /
http csv /
ftp
Rest /
http csv /
file Adapter
Adapter soap
/ http csv /
ftp
Rest
/ http csv /
file
Adap
ter
Rest /
http csv /
file Adapter
Text/tcp soap /
http csv /
ftp
soap /
http csv /
ftp
Rest /
http csv /
file Adapter
csv /
file
Adap
ter
soap
/ http csv /
ftp
Rest
/ http csv /
file
Adap
ter
csv /
file
Adap
ter Adap
ter
11
Use Cases and Solutions
Challenges
Use Cases
Solutions
Architecture
Wireless Telco BSS Integration
BSS
Applications
Mediation Provisioning CRM Workforce
Management
Number Inventory
Management
Interconnect
Infrastructure
Services for
Middleware
Transport
Services
Caching
Services
Load Balancing
Services
Recovery
Services
Failover
Services
Element
Management
Fault
Management
Revenue
Management
Process Automation/ Business Process Management
Ad
min
istratio
n a
nd
Mo
nito
ring S
ervices
Secu
rity S
ervices
Portals, Front End System, Partner Gateways
Telco Network Systems
Middleware Message Broker Enterprise Service Bus JEE Server
Billing
12
Telco scalability
Some requirements
• 75+ million customers • Plan for Terabytes of CDRs and other
messages per day • Performance is critical to customer
experience and retention • CRM, Billing, Mediation, Middleware, Data
warehouse
13
High volume use case 1
CRM to Billing Integration
Middleware Mediation Network
Switch Billing
Data
Warehouse
Fraud
Management CRM
Other BSS
Apps
14
Point to Point Connection
Technical Requirements • Memory
• Threads
• Sockets
• Sender performance
• Receiver performance
High volume use case 1
CRM to Billing Integration: with Middleware Infrastructure
Middleware Mediation Network
Switch Billing
Data
Warehouse
Fraud
Management
Middleware
Infrastructure
• Caching
• Load Balancing
• Failover
• Recovery
CRM
Other BSS
Apps
15
Middleware Mediatio
n
Networ
k
Switch
Billing
Data
Warehouse
Fraud
Management CRM
Other BSS
Apps
Middleware Infrastructure expanded
Middlewar
e
Medi
ation
Ne
tw
ork
Sw
itc
h
Billing
Data
Warehous
e
Fraud
Managem
ent
Middlewar
e
Infrastruct
ure
CRM
Other BSS
Apps
Middleware
Middleware Infrastructure
High Speed,
Reliable
Massaging
Compute Node
Scaling
Resource
Management
Work Load
Management
Failover Recovery Distributed
Caching Cluster Toolkit
16
Middleware Infrastructure
Products
Middlewar
e
Medi
ation
Ne
tw
ork
Sw
itc
h
Billing
Data
Warehous
e
Fraud
Managem
ent
Middlewar
e
Infrastruct
ure
CRM
Other BSS
Apps Middleware
Middleware Infrastructure High Speed,
Reliable
Massaging
Compute Node
Scaling
Resource
Management
Work Load
Management
Failover Recovery Distributed
Caching Cluster Toolkit
• Any ESB
• Any JEE Server
• Any Message Broker
MRG Messaging
MRG Grid
Infinispan
1
2 3 4 5
6
8
7
17
Middleware Infrastructure: Products
MRG Messaging
Middlewar
e
Medi
ation
Ne
tw
ork
Sw
itc
h
Billing
Data
Warehous
e
Fraud
Managem
ent
Middlewar
e
Infrastruct
ure
CRM
Other BSS
Apps
Middleware Infrastructure
High Speed,
Reliable
Massaging
Compute Node
Scaling
Resource
Management
Work Load
Management
Failover Recovery Distributed
Caching Cluster Toolkit
MRG Messaging
• AMQP support
• Native RDMA,
Infiniband
• Can use MRG Realtime
• Large message
support(> GB)
• Clustering and Failover
• High speed, journal
based persistence
• Java and C++ brokers
• Based on Apache Qpid
18
Middleware Infrastructure: Products
MRG Grid
Middlewar
e
Medi
ation
Ne
tw
ork
Sw
itc
h
Billing
Data
Warehous
e
Fraud
Managem
ent
Middlewar
e
Infrastruct
ure
CRM
Other BSS
Apps
Middleware Infrastructure
High Speed,
Reliable
Massaging
Compute
Node Scaling
Resource
Management
Work Load
Management
Failover Recovery Distributed
Caching Cluster Toolkit
MRG Grid
• Scalable Grid Scheduler
• Resource variety:
Desktop to Cloud
schedulers
• Low latency results:
Using MRG Messaging
• Dynamic provisioning
• High Availability
• Grid Federation
• Is based on the Condor
Grid project
19
Middleware Infrastructure: Products
Infinispan
Middlewar
e
Medi
ation
Ne
tw
ork
Sw
itc
h
Billing
Data
Warehous
e
Fraud
Managem
ent
Middlewar
e
Infrastruct
ure
CRM
Other BSS
Apps
Middleware Infrastructure
High Speed,
Reliable
Massaging
Compute Node
Scaling
Resource
Management
Work Load
Management
Failover Recovery Distributed
Caching
Cluster
Toolkit
Infinispan
• In memory Data Grid
• Distributed cache
• Peer to peer
communication between
nodes
• Flexible persistence:
JDBC, File, Amazon S3
• Map reduce: node local
computing
• Implementation for
performance
20
High volume use case 2
Post Trade Securities Processing
Processing Nodes
Aggregator
Node1 Node2
Node3 Node4
File Splitter +
Load
Allocator
Post Trade
Files
Trading
Applications
Trading
Applications
Accounting
Solution
Risk
Management
Solution
Sources Targets Data Processing Solution
Output
Channels
21
Data Services
Ref Data Solution Customer Master Src 3 Src 4
Post Trade Securities Processing
Technical Requirements
22
Processing Nodes
Aggregator
Node1 Node2
Node3 Node4
File Splitter
+
Load
Allocator
Input
Channels
Output
Channels
Data Services Technical Requirements • File Streaming
• Multiple Data views, Data sources
• Data Aggregation
• Reliable Messaging
Post Trade Securities Processing
Products
Processing Nodes
Aggregator
Node1 Node2
Node3 Node4
File Splitter
+
Load
Allocator
MRG Messaging
MRG Grid
Infinispan
1
5
3
2
4
Input
Channels
Output
Channels
6
7
23
Data Services JBoss Data Services
MRG Realtime
• Consistent, predictable response
• Websphere Realtime: RTSJ
24
Messages/microsecond
Re
sp
on
se
Tim
e
Source:
Red Hat
A Recap of the Solution
25
Challenges
Use Cases
Solutions
Challenges, Solutions and Products Challenge Solution Products
1 Small data elements,
high volume
Distribution, load balancing and
partitioning
MRG Messaging, MRG Grid
2 Large data elements File splitting, distribution, in
place processing
MRG Messaging, MRG Grid
3 Data views, Many data
sources
Data Services JBoss Data Services
3 Predictability Real time kernels, real time
JVMs
MRG Realtime, RTSJ
4 Availability Load balancing, clustering,
failover
MRG Grid, Infinispan
5 Reliability File based caches, DB
persistence
MRG Messaging, Grid,
Infinispan
6 Scalability Compute grids, Data grids,
Asynchronous messaging
MRG Messaging, Grid,
Infinispan
26
Solution Alternatives
27
The Map Reduce method
• Split data, process in parallel, aggregate results
Split
Data
Map Phase Reduce Phase
Task Tracker Task Tracker
Task Tracker Task Tracker
Output
Data
Job Tracker
Data
Name
Node
28
Client
Input
Data
GT3 and Condor
• Globus Toolkit
– Open source toolkit for Compute Grids
– Architecture, Service Model and Implementation
– Job Tracking, Management, Monitoring, Resource
Management
– Data Management: Movement, Location Registry
• Condor
– Grid Framework from University of Wisconsin
– Compute Node Scaling
– Job Scheduling
– Idle time utilization
29
Commercial Tools
• IBM
– IBM Cloudburst
– Websphere Virtual Enterprise
– Websphere Realtime
• Oracle
– Oracle Exalogic
– Oracle Coherence
– Oracle Grid Engine(Sun Grid Engine)
• Terracotta
– Quartz
– Big Memory
30
The Open Source difference
31
The Advantages
• Smaller adoption steps to reduce risk • Flexible Cost Model
– Subscription based pricing – Incident based pricing
• Cloud alignment – Elastic pricing model – Cloud Software platforms use open source
• Innovation from wider community • Custom enhancements
32
Conclusion
33
Key Points Discussed
• Large Scale Integration • Impact of Big Data on Integration • Use cases
– Telecom – Securities
• Solutions – MRG – Infinispan – Data Services
• Open source differentiators
34
Questions
35
Top Related