Event Processing Course Event Patterns (relates to chapter 9)
-
date post
21-Dec-2015 -
Category
Documents
-
view
213 -
download
0
Transcript of Event Processing Course Event Patterns (relates to chapter 9)
Event Processing Course
Event Patterns(relates to chapter 9)
2Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Lecture outline
• About patterns
• The notion of pattern
• Pattern types
• Patterns in FFD
• Some code examples
3Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
The many faces of patterns in event processing
Software Engineering Design Patterns
Business User Patterns
Pattern in the “pattern detecting” meaning
We’ll concentrate on this one
Say a few words
about this one
And leave this one for another
opportunity
Event processing function patterns
We’ll start with this
one
4Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Event processing patterns – the idea
5Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
The EPA picture
Output
Not selected
Instanceselection
Context expression
Pattern detect EPA
Relevance filtering
Input terminal filter expression
Relevant event types
DerivationDerivation expression
MatchingPattern signature:
Pattern typePattern parametersRelevant event typesPattern policies
Pattern matching set
Participant events
6Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
EPA signature
The EPA signature consists of
<EPA identifier, EPA type, context, IET, OET, relevance assertion, pattern signature, derivation expression>
Where:
• EPA identifier is a unique name for the EPA instance
• EPA type: one of {filtering, transformation, pattern matching}
• Context: The set of criteria to partition the events of the input event types among the EPA instances.
• IET: a collection of input event types that the EPA receives from producers, channels or other EPAs
• OET: a collection of output event types derived by the EPA, and routed to channels, other EPAs or consumers.
• Relevance assertion: an assertion that is used to filter in the events instance in IET that are relevant for this specific EPA instance
• Pattern signature is defined in the next slide and defines the pattern logic.
• Derivation expression: Creation of the derived events.
7Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Pattern signatureEVENT PATTERN
An event pattern is a template specifying one or more combinations of events. Given any collection of events, you may be able to find one or more subsets of those events that match a particular pattern. We say that such a subset satisfies the pattern
The pattern signature is formally defined as a tuple
<pattern type, relevant event types, pattern parameters and assertions, policies>,
PATTERN TYPE
The pattern type is a label that determines the meaning and intention of the pattern and specifies the particular kind of matching function to be used.
RELEVANT EVENT TYPES LIST
The relevant event types list is a list of event types that forms part of the pattern matching function. The order of these event types has importance for some pattern functions.
Some pattern types require additional values to be provided in the form of pattern parameters.
PATTERN PARAMETERS
Pattern parameters provide additional values used in the definition of the pattern function. The parameters that may be specified, and their meanings, vary depending on the pattern type.
PATTERN POLICY
A pattern policy is a named parameter that disambiguates the semantics of the pattern and the pattern matching process.
8Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
An example
Pattern name: Manual Assignment Preparation
Pattern Type: Relative N highest
Context: Bid Interval
Participant event set: Delivery Bid
Pattern parameter: N = 5; value = Ranking
Cardinality: Single deferred
9Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Pattern types
Pattern
Logical
Operators Threshold
Subset
SelectionModal Temporal Spatial
Spatio
Temporal
10Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Logical operators
10:33 10:46 11:02
all: The all pattern is satisfied when the relevant event set contains at least one instance of each event type in the participant set
all: RET = {ET1…,ETn}; PS MS = {E1,…,En}. The all pattern type is a function, where (E1…En) MS iff i: ETi PET: Ei MS, such that: Type (Ei) = ETi
any: The any pattern is satisfied if the relevant event set contains an instance of any of the event types in the participant set
any: RET = {ET1… ETn}; PS MS = {E}. The any pattern type is a function, where E MS iff i: ETi RET: such that: Type (E) = ETi.
Absence: The absence pattern is satisfied when there are no relevant events
absence: (RET = {ET1…ETn}; RS MS = { }. The absence pattern type is a function, where PS = , i.e. there are no relevant events of any of the types in RET.
11Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
The not selected terminal example • Events are emitted on the not selected
terminal if they are not selected by any matching set.
book 1
book 3
book 2
Order 4
Order 3Order 1
Order 2
12Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Threshold patterns• The count pattern is satisfied when the number of
instances in the relevant event set satisfies the count threshold assertion.
• The value max pattern is satisfied when the maximal value of a specific attribute over all the relevant events satisfies the value max threshold assertion
• The value min pattern is satisfied when the minimal value of a specific attribute over all the relevant events satisfies the value min threshold assertion.
• The value average pattern is satisfied when the value of a specific attribute, averaged over all the relevant events, satisfies the value average threshold assertion.
10:39 12:17 14:10
10:00 11:02 13:35
Buy Amount: $2M
SellAmount: $7.8M
Buy Amount: $10.6M
13Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Threshold patterns definitions
Threshold pattern (PS, Threshold aggregate type pattern, binary operation, attribute name, threshold ) MS = PS iff the assertion: (aggregate type [attribute name] (PS), binary relation, threshold) is evaluated to true, where Aggregate types are: {count, value max, value min, value average, functor} and binary relations are {>, <, =, , , }.
14Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Subset selection patterns
• The relative N highest values pattern is satisfied by the events which have the N highest value of a specific attribute over all the relevant events, where N is an argument.
– relative N highest value pattern [attribute]: (PS = {E1…Em}) MS, where >> is a descending order of the events in PS by the value of attribute, and Ei MS iff its position in >> N.
• The relative N lowest values pattern is satisfied by the event which has the minimal value of a specific attribute over all the relevant events
– relative N lowest value pattern [attribute]: (PS = {E1…Em}) MS, where >> is an ascending order of the events in PS by the value of attribute, and Ei MS iff its position in >> N.
15Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Modal patterns
• The always pattern is satisfied when all the relevant events satisfy the always pattern assertion
– always pattern [assertion]: (PS = { E1…Em })
MS, returns the entire RS iff i: assertion (Ei) is
evaluates to true
• The sometimes pattern is satisfied when there is at least one relevant event that satisfies the sometimes pattern assertion
– sometimes pattern [assertion]: (PS = { E1…Em })
MS, returns the entire RS iff i: assertion (Ei) is
evaluates to true.
09:24 10:30 11:159:00 12:00 11:18
Printer problemSeverity – 3Customer type – goldReported by - Web
CommunicationproblemSeverity – 1Customer type – silverReported by - Web
Application software problem Severity – 2Customer type – platinumReported by - Web
Printer problem Severity – 3Customer type – goldReported by – Web
16Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Dimensional patterns – temporal • The sequence pattern is satisfied
when the relevant event set contains at least one event instance for each event type in the participant set, and the order of the event instances is identical to the order of the event types in the participant set.
– sequence pattern: RET = {ET1,
…,ETn} PS MS = {E1,…,En}.
The sequence pattern type is a function, where i: ETi RET: Ei
PS, such that: Type (Ei) = ETi,
and i, j: i > j IFF ETi >>ETj , where >> is a ascending order of the events according to the pattern order policy
48 hours
Sunday, 9:00
Monday, 11:30
17Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Trends patterns
• The increasing pattern is satisfied by an attribute A if for all the relevant events, e1 << e2 e1.A < e2.A
• The decreasing pattern is satisfied by an attribute A if for all the relevant events, e1 << e2 e1.A > e2.A
• The stable pattern is satisfied by an attribute A if for all the relevant events, e1 << e2 e1.A = e2.A
• The non increasing pattern is satisfied by an attribute A if for all relevant events e1 << e2 e1.A e2.A
• The non decreasing pattern is satisfied by an attribute A if for all relevant events e1 << e2 e1.A e2.A
• The mixed pattern is satisfied by an attribute A, if the relevant event set contains event instances e1, e2, e3, e4 such that: e1 << e2 and e1.A < e2.A and e3 << e4 and e3.A > e4.A
18Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Trends pattern examples
• Fever – decreasing
• Blood pressure:
– Systolic – increasing
– Diastolic – non increasing
• Pulse – non decreasing
• Respiratory rate: stable
• Pain scale: mixed
Fever: 39.4Blood pressure: 140 / 87Pulse: 110Respiratory rate:15Pain scale: 4
Fever: 39.0Blood pressure: 145 / 87Pulse: 112Respiratory rate:15Pain scale: 6
Fever: 38.4Blood pressure: 146 / 85Pulse: 112Respiratory rate:15Pain scale: 5
Fever: 37.9Blood pressure: 150 / 85Pulse: 115Respiratory rate:15Pain scale: 4
8:00 11:00 14:00 17:00
19Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Spatial patterns
• The min distance pattern is satisfied when the minimal distance of all the relevant events from a given point satisfies the min distance threshold assertion
• The max distance pattern is satisfied when the maximal distance of all the relevant events from a given point satisfies the max distance threshold assertion
• The average distance pattern is satisfied when the average distance of all the relevant events from a given point satisfies the average distance threshold assertion
3 KM
7 KM
12 KM 5 KM
4 KM
15 KM
Store A
Store B
Store C
20Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Relative distance patterns
• The relative min distance pattern is satisfied when the minimal distance between any two relevant events satisfies the min distance threshold assertion.
• The relative max distance pattern is satisfied when the maximal distance between any two relevant events satisfies the max threshold assertion
• The relative average distance pattern is satisfied when the average distance between any two relevant events satisfies the relative average threshold assertion 30 KM
21 KM
43 KM
1 KM 3 KM
5 KM
3 KM
3 KM
1 KM
2 KM
4 KM
4 KM
21Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Spatiotemporal patterns
• The moving in a constant direction pattern is satisfied if there exists a direction from the set {north, south, east, west, northeast, northwest, southeast, southwest} such that for any pair of relevant events e1, e2 we have e1 << e2 e2 lies in that direction relative to e1.
• The moving in a mixed direction pattern is satisfied if none of the eight moving in a consistent direction patterns is satisfied
• The stationary pattern is satisfied if the location of all relevant events is identical
• The moving toward pattern is satisfied when for any pair of relevant events e1, e2 we have e1 << e2 the location of e2 is closer to a certain object then the location of e1.
7:00 8:00 9:00
22Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
A simple example: heavy trading scenario
• Given:
– A stream of events of a single type, about the activity in the stock market for a certain stock.
– An event is produced every 10 minutes when there is trade in the stock.
– Each event consists of: quote (current stock-quote), volume (an accumulated volume of traded events within these 10 minutes).
– A selection specification: “trigger an automatic trade program if the volume exceeds 300,000 3 times within an hour; pass as an argument the last quote and the sum of the 3 volume values”.
Event-Id Time-Stamp Quote Volume E1 9:00 33.23 E2 9:10 33.04 320,000 E3 9:20 33.11 280,000 E4 9:30 33.01 400,000 E5 9:40 32.90 315,000 E6 9:50 33.04 320,000 E7 10:00 33.20 303,000 E8 10:10 33.33 219,000 E9 10:20 33.11 301,000 E10 10:40 33.00 210,000 E11 10:50 32.78 400,000 E12 11:00 32.70 176,000
How many times the trade programming is triggered ;Which arguments are used in each triggering?
Why defining patterns is not that easy? Because we need to tune up the semantics
23Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Pattern policies
• Evaluation policy—This determines when the matching sets are produced.
• Cardinality policy—This determines how many matching sets are produced within a single context partition.
• Repeated type policy—This determines what happens if the matching step encounters multiple events of the same type.
• Consumption policy—This specifies what happens to a participant event after it has been included in a matching set.
• Order policy—This specifies how temporal order is defined.
24Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Evaluation policy
An evaluation policy is a semantic abstraction that determines when the matching process is to be evaluated.
The evaluation policy lets you choose whether a Patter detect agent generates output incrementally, or only at the end of the temporal context. The two policies are:
• Immediate—The pattern is tested for each time a new event is added to the participant event set.
• Deferred—The pattern is only tested for when the agent's temporal context partition (window) closes.
25Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Cardinality policiesA cardinality policy is a semantic
abstraction that controls how many matching sets are created. The possible policies are: single, unrestricted and bounded.
The various policies are:
• Single—Only one matching set is generated. When this has been done no further action is performed within this context partition, so no more matching sets are generated.
• Unrestricted—Under this policy there are no restrictions on the quantity of matching sets that can be generated.
• Bounded—This policy specifies an upper bound on the number of matching sets that can be generated within a context partition. The Pattern detect agent continues generating matching sets until it reaches this bound.
9:20
14:3012:0611:23
9:50 10:30
15:56
Hertz
Avis
Hilton MarriottSheraton
United Continental
single
single deferred
Unrestricted
Hertz, Continental, Hilton
Avis, Continental, Sheraton
Hertz, Continental, Hilton Avis, United, Sheraton
26Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Repeated type policies A repeated type policy is a semantic abstraction
that defines the behavior of a Pattern detect agent when an excess type condition occurs. The possible policies are: override, every, first, last, with maximal value, with minimal value.
• Override The participant event set keeps no more instances of any event type than the number implied by the relevant event types list. If a new event instance is encountered and the participant set already contains the required number of instances of that type, then the new instance replaces the oldest previous instance of that type.
• Every: Every instance is kept in the participant event set, so that all possible matching sets can be produced.
• First Every instance is kept in the participant event set, but only the earliest instances of each type are used for matching.
• Last Every instance is kept, but only the latest instances of each type are used for matching.
• With maximal value <attribute name> Every instance is kept, but only the event or events with the maximal value of the specified attribute are used for matching.
• With minimal value <attribute name> Every instance is kept, but only the event or events with the minimal value of the specified attribute are used for matching.
9:20
14:3012:0611:23
9:50 10:30
15:56
Hertz
Avis
Hilton MarriottSheraton
United Continental
27Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Consumption policies
A consumption policy is a semantic abstraction that defines whether an event instance is consumed as soon as it is included in a matching set, or whether it can be included in subsequent matching sets. Possible consumption policies are: consume, reuse and bounded reuse.
• The consumption policies are quite straightforward:
• Consume—Under this policy each event instance is removed from the participant event set once it has been included in a matching set. This means that it cannot take part in any further matching for this particular pattern within the same context.
• Reuse—under this policy, an event instance can participate in an unrestricted number of matching sets.
• Bounded reuse—under this policy, you can specify the number of times that an event can be used in matching sets for this particular pattern within the same context.
28Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Order policies
An order policy is a semantic abstraction that defines the meaning of the << temporal order of the event instances in the participant event set. The possible policies are: by occurrence time, by detection time, by user-defined attribute, or by stream position.
The order policy is applicable to all temporal or spatiotemporal patterns. The possible policies are:
• By occurrence time—The order of events in the participant event set is determined by comparing their occurrence time attributes, so that the order reflects the order in which the events happened in reality (as accurately as the temporal granularity allows).
• By detection time—The order of events in the participant event set is determined by comparing their detection time attributes, that is the order in which events are detected by the event processing system. Note that this order may not be identical to the order in which events happened in reality.
• By user-defined attribute—Some event payloads contain a timestamp, sequence number or some other attribute that increases over time, and this can be used to determine the order. For example the Delivery Request events in the Fast Flower Delivery application could be ordered using their Delivery Time attribute.
• By stream position—In this case the order to be used is the order in which the events are delivered to the event processing agent from the channel that feeds it. Some channel implementations are designed so that this order is the same as the order in which events were delivered to the channel
29Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
FFD patterns
Pattern type Context Relevant types Parameters Policies
first n Request Delivery Bid N = 1 cardinality = single
evaluation = immediate
Pattern type Context Relevant types Parameters Policies
relative n
highest values
Bid interval Delivery Bid Attribute = ranking
N = 5
cardinality = single
evaluation = deferred
repeated Type = every
Pattern type Context Relevant types Parameters Policies
absence Response interval Manual Assignment
Pattern type Context Relevant types Parameters Policies
absence Pickup interval Pickup Confirmation
Pattern type Context Relevant types Parameters Policies
absence Delivery interval Delivery Confirmation
Pattern type Context Relevant types Parameters Policies
absence Driver evaluation Delivery Alert
Pattern type Context Relevant types Parameters Policies
count Driver evaluation Delivery Alert Relation:>
Threshold 5
repeated type = every
evaluation = deferred
Automatic assignment
Manual assignment preparation
Assignment not done
Pickup Alert
Delivery Alert
Ranking Increase
Ranking Decrease
30Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
FFD patterns (cont). Pattern type Context Relevant types Parameters Policies
sequence Driver Ranking Decrease,
Ranking Increase
repeated type = override
evaluation = immediate
cardinality= unrestricted
consumption = consume
Pattern type Context Relevant types Parameters Policies
always Monthly Driver Daily Assignments Assertion:
assignmentCount < 5
evaluation = deferred
Pattern type Context Relevant types Parameters Policies
sometimes Monthly Driver Daily Assignments Assertion:
assignmentCount = 0
evaluation = deferred
Pattern type Context Relevant types Parameters Policies
always Monthly Driver Relative Performance Assertion:
deviation< -2
evaluation = deferred
Pattern type Context Relevant types Parameters Policies
always Monthly Driver Relative Performance Assertion:
deviation > 2
evaluation = deferred
Pattern type Context Relevant types Parameters Policies
non-
decreasing
Monthly
Driver
Daily
Assignments
Attribute =
assignmentCount
evaluation = deferred
Improving note
Permanent weak driver
Idle driver
Consistent weak driver
Consistent strong driver
Improving driver
31Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Apama example
32Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Streambase example
33Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
Esper example
**/* No bid after 2 mins of a request*/insert into AlertW(requestId, message, driver)select d.requestId, "no bidder", ""from pattern[every d=DeliveryRequest -> (timer:interval(120 sec) and not DeliveryBid(requestId = d.requestId))];
34Copyright ©2009 Opher EtzionCopyright ©2009 Opher Etzion
What have we learned
• Pattern in general
• Event pattern notions
• Pattern types
• Patterns in FFD
• Some code examples