Wo stehen wir mit Open Government Data in der Schweiz? (Juni 2012)
Systemübergreifende Prozessüberwachung mit Event Stream Processing Thomas Dücker, SAP Schweiz...
-
Upload
elmer-garcia -
Category
Documents
-
view
220 -
download
1
Transcript of Systemübergreifende Prozessüberwachung mit Event Stream Processing Thomas Dücker, SAP Schweiz...
Systemübergreifende Prozessüberwachung mit Event Stream ProcessingThomas Dücker, SAP SchweizJuni, 2013
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 2
Agenda
Herausforderung und Trends
Übersicht Operational Process Intelligence Übersicht Beispiele Szenario
Mehrwert für Sie
Weitere Informationen
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 3
Event Stream Processing
Capture Disseminate & Decide Response & Action
ESP
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 4
Traditional Approaches to Event Stream Processing
• DB application (OLTP)— Pull not push (i.e. not event driven) or— Requires triggers – performance degradation / maintenance nightmare— latency in seconds
• Custom application (C/C++ / Java)— Specialized, high development / maintenance cost— Slow to change, unresponsive to the business
Applicationor
Dashboard
High-SpeedData
Queries /Results
Database Application
High-SpeedData
“Black-box”Application
Alerts /Actions
Custom Application
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 5
The SAP Approach – Sybase ESP
•Combines advantages of customized application with ease of visual data flow diagramming or SQL•Incoming data is processed as it arrives, according to the model rules and operations•Publish streaming results to other models, apps, message bus, dashboard, etc.
?
Input Streams PUBLISH
Market Data
Orders, Trades
ConsoleInput
Reporting Tools
Trading Systems
Visual / SQLAuthoring
Reference Data
SAP HANA / Sybase IQ
Dashboards
Message Bus
Output StreamsSUBSCRIBE
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 6
CEP Product Convergence
Aleri 3.2
Sybase ESP 5.0
Sybase ESP 5.1
Sybase CEP R4
Coral8 v5.6.5
Sept 2011
Sept 2012
Optional
(maintenance only)
Sybase ESP 5.1 SP1Dec 2012
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 7
Advantages of Sybase ESP Approach
Rapid Application Development•Reduce/eliminate dependence on specialist
programming skills for process•Cut implementation/deployment time•Broad out of the box connectivity
Non-intrusive Deployment•Event-driven integration with existing systems•Unify existing disparate data models
Analyze Events as they occur•Continuous Insight•Respond Immediately
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 8
Event Stream Processing Scenarios
Capital Markets
React to incoming ticks to determine trading strategies, monitor the success of the strategies by calculating PnL in real-time; monitor risk in real-time.
Retail
Provide clickstream analysis for online customer assistance / alternatives, use web scraping to react to competitor price changes
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 9
Event Stream Processing Scenarios
Energy
Monitor and optimize energy usage using Smart Metering to ensure QoS
Utilities / Manufacturing
Service Status Monitoring and Process Control using SCADA / OSISoft
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 10
Event Stream Processing Scenarios
Telecoms
Monitor events for QoS, use DPI to detect fraudulent use
Police / Homeland Security
ANPR event correlation for traffic monitoring, security
Click icon to add pictureClick icon to add picture
ESP Architektur
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 12
ESP Project Architecture
Publish Subscribe
Data Streams
Authoring Tools
Ad
apters
I/O G
ateway
Secu
rity
Secu
rity
Ad
apters
Stream Processor+ Multi-threaded+ 64 bit - Suse 11 / Redhat 6 / Solaris 10 (Sparc / i86) Win x64+ Low latency+ Optional persistence
C+C(XML-RPC)
Operational Console
Security (PAM, SSL, RSA, Kerberos)
Data Streams
MemStore
LogStore
I/O G
ateway
Data Streams Data Streams
LogStore Command & Control Studio
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 13
Out of the Box Connectivity
In Process
• JMS, MQ, TIBCO• ATOM (RSS)• TCP Sockets• Databases via JDBC /
ODBC• Files• MS Excel RT• SMTP (out)
External
• Market Data(Reuters, Bloomberg, IDC, ACTIV, NYSE Tech)
• FIX• OpenAdaptor Java
Framework• Replication Server• IQ / HANA
PubSub SDK
• C/C++• Java• .NET
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 14
ESP Cluster Architecture I
Controller
Manager
Manager
Manager
Manager
Project Server
Controller Controller
ManagerNodes Cache
Project Server Project Server
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 15
• Manager Node − Deploys project(s) to server(s) − Maintains heartbeat with project(s) to detect failure− Manages failover− In multi node, managers are run in a cache so can failover to manage other
controller nodes
• Controller Node− Basically acts as a launch daemon for Manager to start project(s) in a Container
ESP Cluster Architecture II
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 16
• Single node cluster – − Single manager/controller node runs multiple projects on a single machine− Detects when a project fails and tries to restart it− This is the default for project development
• Multi node cluster − Multiple equi-peer managers and controllers run multiple projects on multiple machines− Used to scale out projects across multiple machines− Allows for server failure – promotes failover recovery and data redundancy − Allows for Highly Available “active:active” deployment of projects within a Data Centre
ESP Cluster Architecture III
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 17
• Three clusters created by install script: -
• Studio Cluster (single node) on 9786 : – − Used for Studio Development − Can run multiple projects − Located in $ESP_HOME/studio/clustercfg
• User-defined Cluster (single node) on 19011− Located in $ESP_HOME/cluster/nodes/node1
• Example multi-node Cluster on 19011 - 19014− Located in $ESP_HOME/cluster/examples
ESP Installed Clusters
ESP Key Concepts
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 19
• Analyst-level skills (Excel, VBA)• Easy to understand complex models• No need to learn language syntax
• Rapid programming• Easy to use language (CCL – derived from
SQL)• Modular, project based approach
Visual Dataflow Authoring CCL Authoring
Authoring Options – Eclipse Plugin
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 20
Visual Dataflow Authoring
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 21
CCL Authoring
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 22
Procedural Logic - SPLASH
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 23
IMPORT 'CreateCompleteChannel.ccl';
----------CREATE MODULE CreateCompleteChannelIN MessageIn, CacheInOUT MessageOutBEGIN
// Module CodeEND;
LOAD MODULE CreateCompleteChannel AS Channel_AIN MessageIn = MessageA, CacheIn = CacheAOUT MessageOut = MessageOutA;
LOAD MODULE CreateCompleteChannel AS Channel_BIN MessageIn = MessageB, CacheIn = CacheBOUT MessageOut = MessageOutB;
ESP Modularity
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 24
Project Code, Compiled Code & Project Resources
• Project Source Code .CCL (Continuous Computation Language)− Used for schema, stream, window, method definitions compiled into
• Project Executable .CCX (Continuous Computation eXecutable)
• Project Resource File .CCR (Continuous Computation Resource)− An xml file used for in process adaptor, cluster, bindings, failover, HA definitions –
allows for easy migration between DEV, TEST and PROD
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 25
Constructing ESP Projects
•Build model
•Connect to data feeds using publish adaptor
•Connect to output devices via subscribe adaptor
ESP Project Container
Positions
Console
Trades
Tick data feed
DB
Replica
ted D
ata
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 26
• Events are retained as records in memory with a primary key so that further events may modify the state of the record - either update or delete it.
• A stateful event must have PK attribute(s) set.
• A stateful event MUST have an associated operation code (opcode), one of INSERT, UPDATE, UPSERT, DELETE. If no opcode is provided, ESP defaults it to UPSERT.
• When an event occurs it applies the opcode through the data flow model depending on the opcode. An insert in a source stream may become an update to an aggregate or join stream.
• The full set of unique (insert) events to be retained MUST fit in memory.
• Records in memory can be removed either by setting a retention window, or by publishing a DELETE event for that key.
• In ESP, stateful events are implemented using a WINDOW
Stateful Events
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 27
• An event is retained only while it is being processed. When an event moves from one stream to another it is removed from the first stream so it cannot be updated.
• All events are treated as insert.
• A stateless event ignores any PK attribute(s) and always outputs an insert
• In ESP stateless events are implemented with a STREAM
Stateless Events
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 28
ESP implements stateful and stateless events using 3 element types• WINDOW : used for stateful events. A WINDOW maintains state for a user defined period
•STREAM : used for stateless events. A STREAM does not maintain or react to state
•DELTA : used to process stateful events, but without maintaining state
• All can be INPUT, LOCAL, OUTPUT• INPUT subscribe to input adaptors i.e. events are published to an INPUT. They are the entry points
for events to enter an ESP project.
• LOCAL perform processing logic. They can NOT be published or subscribed to
• OUTPUT are derived elements which perform processing logic which CAN be subscribed to
ESP Elements
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 29
• WINDOWS must have a PK set
• Can have an optional retention period set
• Maintain state in memory or persisted to disk. i.e. State can be recovered in the event of system failure
• Can publish to STREAM, WINDOW or DELTA
• Only a WINDOW can be used for aggregation
• At lease one side of a join MUST be a WINDOW (unless using SPLASH). The side which maintains state depends on the type of the join.
• Use WINDOW when developing models as they help debugging
Windows (stateful)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 30
• Stream• Cannot be persisted so any data is lost in the event of system failure
• Can NOT publish to DELTA stream
• Can publish to a WINDOW, but WINDOW must aggregate the input
• Delta• Can only be LOCAL, OUTPUT
• A stateless element (a STREAM) that understands and processes state (a WINDOW)
• Can only use a WINDOW as a source
• Must have a PK set
• As it does not store events it can NOT be used for aggregation.
• Cannot be persisted so any data is lost in the event of system failure
Streams (stateless)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 31
Example: Simple Position Maintenance
AverageTrades Window
Symbol*Last PriceWeighted Avg PriceLast Time
Aggregate
Positions By Book Stream
BookId*Symbol*Current PositionAverage Position
Join
Trades Stream
Symbol*Time*PriceShares
Position Window
BookId*Symbol*Shares Held
Total Book Window
BookId*Current PositionAverage Position
Aggregate
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 32
MSFT 12:05:23 32.45 200
IBM 12:05:24 57.87 5000
CSCO 12:06:01 20.04 100
AveragePrices Window
Symbol*Last PriceWeighted Avg PriceLast Time
Aggregate
Individual Positions Stream
BookId*Symbol*Current PositionAverage Position
Join
Position Window
BookId*Symbol*Shares Held
Book Positions Window
BookId*Current PositionAverage Position
Aggregate
Trades Stream
Symbol*Time*PriceShares
MSFT 12:06:02 36.55 900
MSFT 32.45 33.01 12:05:23
IBM 57.87 56.90 12:05:24
CSCO 20.04 17.00 12:06:01
MSFT 36.55 34.00 12:06:02
IBM 57.87 56.90 12:05:24
CSCO 20.04 17.00 12:06:01
Book1 MSFT 3000
Book1 IBM 700
Book2 MSFT 1000
Book1 MSFT 97350 99030
Book1 IBM 40509 39830
Book2 MSFT 32450 33010
Book1 139539 138860
Book2 32450 33010
Book1 MSFT 109650 102000
Book1 IBM 40509 39830
Book2 MSFT 36550 34000
Book1 150159 141830
Book2 36550 34000
Example: Simple Position Maintenance
ESP Solution Architektur
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 34
Event Processing Solution Architecture
• Events don’t happen in isolation
• Most Event Stream Scenarios need access to historic data• Pattern Detection• Risk Calculation• Optimisation algorithm• To ensure sub millisecond response times use in memory e.g. HANA
• Many Event Stream Scenarios need access to master data• Data written directly to OLTP system• Updates to reference, semi-static data• For sub second updates without polling use log trawler e.g. Replication Server
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 35
In-database Algorithms
Analytical data
Historic data
Reference data
Real-time ANPR Alerting Framework
Event Stream Processor
CEP Engine
Inpu
t A
dapt
er O
utput A
dapte
r
In-Mem
ory
HANA
SQL
SQL
SQL
Monitoring
BusinessObjects DataService
ExternalDatabases
Business Inteligence
BO Predictive AnalysisPredictive models(Ad-hoc analysis)
OCR Radars
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 36
Real-time Retailing Framework
ES
PH
AN
A
Detect Event
Validate Event
Check Results Output
Run Optimization Algorithm
Manage Exception
Phase 1:
Event Detection & Validation
Phase 2:
Processing
Phase 3:
Validation & Exception
Phase 4:
Output
Straight Through Processing
Is competitor price<95% of our
price?
Is optimum price<95% of current our price?
GU
I
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 37
Integration with Hana I
• Generic input / output ODBC adapter• Data can enter via a source stream
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 38
Integration with Hana II
• Can be called from a FlexStream using SPLASH for event driven query: -
ON TRADE { if (TRADE.Price <> optimalPrice) { [ string s; | ] rec; vector (typeof(rec) v := new vector(typeof(rec)); getData(v,'HanaDSN','CALL DPTL.sp_price_opt (?, ?, ?, ?)', TRADE.Id, TRADE.Datetime, TRADE.Sector, TRADE.Price); if (not (isnull(v))) { newrec := v[0]; } // rest of method }}
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 39
Trading
SystemsTrading
Systems
Showing:• Concentration limits• Clustering• Outliers• Etc…
Trading
SystemsMarket Data
Trading
SystemsRisk
Sensitivities
Bu
s• Cleansing• Normalisation• Enrichment
Real-time Aggregation of Risk and P&L
CEP
Continual Mark t o Market
HANA
• Scenario Shifts• VAR calculations
Analytics:• Heatmaps• Drill-downs• Reports
By:• Instrument• Industry• Country• Debt Rating• Counterparty• VaR Time Horizon• Etc…
Transformation of granular data
Real-time Risk Framework
ASE
• Reference Data
Replication Server
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 40
Oil & Gas - Energy Efficiency Reporting Framework
ESP (Event Stream Processor)
ECC (IS-U)
OKONon SAP (top level MES)
Oracle
Oracle
BO 4.0
SOI2 server
Panopticon
RS(Replication
Server)
HANA
RA#1
RA#2
SOI2
Replication flowEvent flowSCADA SCADA SCADA
RS
HANA ODBC
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Besten Dank
Kontaktinformation:
Thomas DückerSAP [email protected]