Debs Presentation 2009 July62009
-
Upload
opher-etzion -
Category
Technology
-
view
949 -
download
1
description
Transcript of Debs Presentation 2009 July62009
A Stratified Approach for Supporting High Throughput Event Processing Applications
July 2009
Geetika T. Lakshmanan Yuri G. Rabinovich Opher EtzionIBM T. J. Watson Research Center IBM Haifa Research Lab IBM Haifa Research Lab
2
Outline Why scalable event processing is an important problem?
Some terms (EPN, EPA)
What has been done already?
Overview of our solution– Credit-Card Scenario
– Profiling and initial assignment of nodes to strata
– Stratification
– Load Distribution Algorithm
– Algorithm optimizations and support for dynamic changes in event processing graph
Implementation and Results
Conclusion
3
Our GoalDevise a generic framework to maximize the overall input (and thus output) throughput of an event processing application which is represented as an EPN, given a specific set of resources (cluster of nodes with varying computational power) and a traffic model. The framework should be adaptive to changes either in the configuration or in the traffic model.
EPNEPNEvent
ProducerEvent
Consumer
Event
Producer
Event
Producer
Event
Consumer
Event
Consumer
Engine
EPAEPA
EPAEPA
Engine
EPAEPA
EPAEPA
Engine
EPAEPA
EPAEPA
Repository
4
Why is this an important problem? Quantity of events that a single application needs to process is constantly increasing (E.g. RFID events, Massive Online Multiplayer Games, financial transactions).
Manual partitioning is difficult (due to semantic dependencies between event processing agents) particularly when it is required to be adaptive and dynamic.
5
Event Processing Agent An event processing agent has input and output event channels.
In general it receives a collection of events as input, derives one or more events as output and emits them on one or more output channels.
The input channels are partitioned according to a context which partitions the space of events according to semantic partitions of relevance
Event Processing Agent
InputChannel
Context Agent SpecDerived Event
Definition
OutputChannel
Event Processing Agent
Filter Transform Detect Pattern
Translate
Route
Aggregate Split Enrich
6
Related Work Scalability in event processing
– Scalable event processing infrastructure (E.g. Astrolobe (2003), PIER (2003), Sienna (2000.))
– Controlled input load shedding (Kulkarni et al. (2008)).– CEP over streams (Wu et al. (2006)).– More work needs to be done.
Numerous centralized implementations arising due to interdependencies among event processing agents.
Synergy between stream processing and event processing.– Distributed stream processing techniques:
• Mehta et al., 1995• Shah et al., 2003• Balazinska et al., 2004• Kumar et al., 2005• Xing et al., 2005, 2006• Zhou et al., 2006• Pietzuch et al., 2006• Gu et al., 2007• Lakshmanan et al., 2008
7
Is this a solved problem?
Scalable event processing
implementations
(Astrolobe, PIER, Sienna)
Centralized
event processing
ImplementationsLoad distribution algorithms
for scalable stream processingShah et al., Mehta et al., Gu et al., Xing et al.
Zhou et al., Liu et al. ………
Event-at-a-time Implementations Set-at-a-time Implementations
Centralized stream processing
Implementations
8
Overview of Our Solution1. Profiling
– Used to assign agents to nodes in order to maximize throughput
2. Stratification of EPN– Splitting the EPN into strata layers
– Based on semantic dependencies between agents
– Distributed implementation design with event proxy to relay events between strata
3. Load Distribution– Distribute load among agents dynamically during runtime and
respect statistical load relationships between nodes
9
Distributed Event Processing Network Architecture Input: Specification of an Event Processing Application
Output: Stratified EPN (event processing operations event processing agents)
EP Node
Stratum 1
EPProxy
EP Node
EP Node
DB
EP Node
Stratum 1
EPProxy
EP Node
EP Node
DB
EP Node
Stratum 2
EPProxy
EP Node
EP Node
DB
EP Node
Stratum 2
EPProxy
EP Node
EP Node
DB
EP Node
Stratum 3
EPProxy
EP Node
EP Node
DB
EP Node
Stratum 3
EPProxy
EP Node
EP Node
DB
Events Events
Event Proxy receives input events and routes them to nodes in a stratum according to the event context.
Event proxy periodically collects performance statistics per node in a stratum.
10
Stratified Event Processing Graph1. Define the event processing application in the form of an Event
Processing Network Dependency Graph G=(V,E) (directed edges from event source to event target)
2. Overview of Stratification Algorithm Create partitions by finding sub graphs that are independent in
the dependency graph. For each sub graph, construct a network of EPAs. Push filters to the beginning of the network to filter out irrelevant
events. Iterate through graph and identify areas of strict interdependence.
(i.e. sub graphs with no connecting edges). For each sub graph define stratum levels.
Amount > 100
More Than5 OccurrencesWithin 1 Hour
Amount > 100
More Than3 OccurrencesWithin 1 Hour
Cancel FollowsDiscount
High VolumePurchase
Purchase
Cancel High VolumeCancel
Give Discount to Company
Discount Canceled
Cancel Discount to Company
Amount > 100
More Than5 OccurrencesWithin 1 Hour
Amount > 100
More Than3 OccurrencesWithin 1 Hour
Cancel FollowsDiscount
High VolumePurchase
Purchase
Cancel High VolumeCancel
Give Discount to Company
Discount Canceled
Cancel Discount to Company
Amount > 100
More Than5 OccurrencesWithin 1 Hour
Amount > 100
More Than3 OccurrencesWithin 1 Hour
Cancel FollowsDiscount
High VolumePurchase
Purchase
Cancel High VolumeCancel
Give Discount to Company
Discount Canceled
Cancel Discount to Company
11
Example: Credit Card Scenario
Amount > 100
More Than5 OccurrencesWithin 1 Hour
Amount > 100
More Than3 OccurrencesWithin 1 Hour
Cancel FollowsDiscount
High VolumePurchase
Purchase
Cancel High VolumeCancel
Give Discount to Company
Discount Canceled
Cancel Discount to Company
Amount > 100
More Than5 OccurrencesWithin 1 Hour
Amount > 100
More Than3 OccurrencesWithin 1 Hour
Cancel FollowsDiscount
High VolumePurchasePurchase
CancelHigh Volume
Cancel
Give Discountto Company
Discount Canceled
Cancel Discount to Company
Stratum 1 Stratum 2 Stratum 3
Stratification algorithm
Event Processing Dependency Graph
Stratified Event Processing Graph
12
Assigning Nodes to Each Stratum
Goal: Executing at a user set percentage of their capacity, nodes in a stratum can process all of the incoming events in their stratum level in parallel under peak event traffic conditions.
– Assume agents in a single stratum are replicated on all nodes in that stratum.
Overall strategy:1. Profiling nodes. Determine maximum event processing capability of available
nodes by observing performance under synthetic workload.
2. Compute ratio of events split between nodes for first stratum.
3. Determine number of nodes to assign to stratum.
4. Repeat for next stratum, and next, until done.
13
Assigning Nodes to Each Stratum
– ti : User set percentage of node capacity
– mi : All the incoming events
– ri : Maximum possible event processing rate (events/sec)
– di : Maximum possible derived event production rate (events/sec)Formulas
Example: Incoming event rate: 200,000 ev/sec. Processing Capacity of node i: 36,000 events/sec. t i=0.95.
((ti*ri)/mi)*100 mi*(di/ri) Stratum ID: n+1
Stratum ID: n
Percentage of event stream directed to node i
Derived event production rate of nodes in stratum n
((0.95*36,000)/200,000)*100 = 17.1%
Percentage of event stream directed to node i
If (di/ri)=0.5, total rate of derived events created by the stratum n nodes is 200,000*0.5=100,000 events/sec
Thus, 6 nodes will be needed in this stratum
Stratum ID: n
Stratum ID: n+1
14
Overview of Dynamic Load Distribution Algorithm
Statistics collected by event Proxy:
– Number of input events processed by execution of agents in a particular context
– Number of derived events produced by the execution of agents in this context
– Number of different agent executions evaluated in this context
– Total amount of latency to evaluate all agents executed in this context
For these statistics, event proxy maintains a time series, and computes statistics such as mean, standard deviation, covariance and correlation coefficient (between agents on the same node, and between contexts for the same agent).
These statistics dictate the choice of load donor and recipient nodes.
Definition of load is purposely generic to incorporate priorities and preferences of application priorities.
15
Overview of Dynamic Load Distribution Algorithm
Event Proxy collects statistics and maintains a time series and makes the following decisions:1. Identify most heavily loaded node in a stratum (donor node).2. Identify a heavy context to migrate from the donor node. (Also use load
correlation as a guiding factor).3. Identify recipient node for migrated load.4. Estimate post migration utilization of donor and recipient nodes. If post
migration utilization of recipient node is unsatisfactory, go back to step 3 and identify new recipient node. If post migration utilization of donor node is unsatisfactory, go back to step 2 and identify new context to migrate.
5. Execute migration and wait for x length time interval. Go to step 1.
Engine Queue AMIT
Engine Queue AMIT
Engine Queue AMIT
Engine Queue AMIT
Engine Queue AMIT
Engine Queue AMIT
EPProxy
Stratum n Stratum n+1
16
Post Migration Utilization Calculation We need to determine whether this migration will lead to overload. If it
triggers other migrations then the system will become unstable. Therefore compute the post migration utilization of the donor and recipient machines.
If the average event arrival rate in time period t for context c is (c) and the average latency to evaluate context c is (c), then the load of this context in time period t can be defined as (c1)(c1).
Thus the post migration utilization, Ud, of the donor machine and Ur of the recipient machine after migrating a context c1, and where nd and nr are the total number of contexts on the donor and recipient respectively, is:
Post migration utilization of donor and recipient nodes must be less than preset quality thresholds.
)
)()(
)1()1(1('
1
dni
i
dd
cipci
cpcUU
)
)()(
)1()1(1('
1
rni
i
rr
cipci
cpcUU
17
Implementation
Used nodes running IBM Active Middleware Technology (AMiT), a CEP engine that serves as a container for event processing agents.
Event processing scenario: credit card scenario
Node hardware characteristics:
– Type 1: Dual Core AMD Opteron 280 2.4 GHz and 1GB memory.
– Type 2: Intel Pentium D 3.6 Ghz and 2GB memory.
– Type 3: Intel Xeon 2.6 Ghz and 2 GB memory.
Amount > 100
More Than5 OccurrencesWithin 1 Hour
Amount > 100
More Than3 OccurrencesWithin 1 Hour
Cancel FollowsDiscount
High VolumePurchasePurchase
CancelHigh Volume
Cancel
Give Discountto Company
Discount Canceled
Cancel Discount to Company
Stratum 1 Stratum 2 Stratum 3
18
Results
398000
90000
300000
81000
162000150000
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
1:1:1 = 3 Machines 2:2:1 = 5 Machines 5:4:1 = 10 Machines
System Type
Eve
nts
/Sec
Stratified Input Rate Partitioned Input RateCentralized = 30,000
Input events processing rate by stratified versus
partitioned event processing networks
Synthetic workload: consists strictly of all events that trigger the generation of derived events. Number of nodes: 3, 5, 10. Heterogeneous mix of Type 1, 2 and 3. Ratios are selected to be “optimal.”
Y-axis: Maximum number of input event processing rate is computed as the sum of the average input event processing rate of all nodes in the network.
Illustrates the maximum performance that can be achieved by the event processing network when it is overloaded
19
15000
21000
9000
4,500
7500
4500
0
5,000
10,000
15,000
20,000
25,000
1:1:1 = 3 Machines 2:2:1 = 5 Machines 5:4:1 = 10 Machines
System Type
Eve
nts
/Sec
Stratified Derived Rate Partitioned Derived RateCentralized = 1,500
Results
Derived events production rate by stratified versus partitioned event
processing networks.
Synthetic workload: consists strictly of all events that trigger the generation of derived events. Number of nodes: 3, 5, 10. Heterogeneous mix of Type 1, 2 and 3. Ratios are selected to be “optimal.”
Y-axis: Maximum number of derived event processing rate is computed as the sum of the average derived event processing rate of all nodes in the network.
20
Results
Percentage of improvement in performance of
the stratified network relative to
a partitioned network
40.0032.67
-32.08 -31.75
-40.00
-30.00
-20.00
-10.00
0.00
10.00
20.00
30.00
40.00
50.00
100% - 5:4:1 12.5.% - 5:4:1
Percentage of Events Participating in Derived Events Production
Imp
rove
men
t (P
erce
nta
ge)
Improvement In Event Processing Rate Improvement In Derived Events Rate
Stratified network of ten nodes, where the proportion of nodes in three strata is 5:4:1
with ten nodes in a partitioned network.
All nodes used for this experiment are of Type 1.
Illustrates how changing the proportion of input events that participate in derived events production in the first stratum level impacts the input event processing rate and derived events production rate of the entire system.
21
Results
Average input events processing rate per node in a stratified
network with different configurations
44100
49083
52800
3443834813
3737539800 39800
0
10000
20000
30000
40000
50000
60000
100% - 5:4:1 50% - 6:3:1 25% - 8:3:1 12.5.% - 11:3:1
Percenatage of Events Participating in Derived Events Productions
Eve
nts
/Sec
5:4:1Ratio Optimal Ratio for Percentage
Compares the average input events processing rate per processing node of a stratified network of ten nodes where the distribution of nodes among the three strata is 5:4:1 to an optimal configuration of nodes to strata.
Demonstrates that reconfiguration of the system with optimal ratio of nodes per stratum can improve performance and reacts effectively to changes in the proportion of input events that participate in derived events production in the first stratum level
22
0
10000
20000
30000
40000
50000
60000
70000
5 10 15 20 25 30
Number of Nodes
Mea
n T
hro
ug
hp
ut
Dynamic Load Distribution (Ours)
Largest Load First
Random
No Load Distribution
Results
Total throughput since the system started event processing
0
50000
100000
150000
200000
250000
300000
0 200 400 600 800 1000 1200
Time (sec)
To
tal T
hro
ug
hp
ut
Dynamic Load Distribution (Ours)
No Load Distribution
Mean throughput of load distribution
compared with LLF, Random and None for different number of nodes
Load is defined as the total number of agents' executions for a particular context.
0.5: load threshold for a node to initiate load distribution.
0.1: load threshold of a context’s contribution to the percentage of the total load on a node, where this context has the highest load correlation coefficient with respect to the remaining contexts on the same node.
0.85: acceptable post-migration utilization of a recipient node. 0.1 is the threshold for percentage decrease in utilization of a donor node to warrant a migration.
Periodically fluctuating workload.
23
Support for Dynamic Changes in EP Graph
Our algorithm supports:– Addition of a new connected subgraph to the existing EPN.
– Addition of an agent to the graph in the EPN.
– Deletion of agents from the graph
– Failure of one or more nodes in a stratum level.
Algorithm is also amenable to agent-level optimizations (E.g. coalescing of neighboring agents).
24
Conclusion and Future Work
We demonstrate a novel architecture for distributed event processing that maximizes the throughput of the event processing system, and stratification algorithm to partition an event processing application on to a distributed set of nodes.
The experimental results illustrate the effectiveness of the stratification technique for achieving an initial partitioning of the event processing graph in a distributed event processing system that anticipates a high volume of agent triggering events.
Performance of a stratified network can be improved during runtime with the dynamic load distribution algorithm.
Future Work: – Investigate high availability– Techniques for optimizing stateful load migration between nodes
dynamically during runtime.– Investigate variations of stratification (currently in IBM HRL)
25
Backup Slides
26
Goal of Implementation
Explore benefits of event processing on stratified vs. centralized (single node) vs. partitioned network (single stratum in which load is distributed according to context) when system is under heavy load (when the number of incoming events that trigger the generation of derived events increases).
Compare stratification with partitioned approach when system is not heavily loaded.
Explore the effectiveness and scalability of the load distribution algorithm