DMM110_16014_WoottonJ_1

Public

Jeff Wootton / SAP HANA product management

run()

DMM110 SAP HANA smart data streaming Introduction to Continuous Queries

2014 SAP SE or an SAP affiliate company. All rights reserved. 2 Public

Disclaimer

This presentation outlines our general product direction and should not be relied on in making a

purchase decision. This presentation is not subject to your license agreement or any other agreement

with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to

develop or release any functionality mentioned in this presentation. This presentation and SAP's

strategy and possible future developments are subject to change and may be changed by SAP at any

time for any reason without notice. This document is provided without a warranty of any kind, either

express or implied, including but not limited to, the implied warranties of merchantability, fitness for a

particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this

document, except if such damages were caused by SAP intentionally or grossly negligent.


Agenda

Event stream processing

How its used

Introducing SAP HANA smart data streaming

Querying streams - continuously

Key Concepts

Building a streaming project

Define inputs

Connect to sources

Create derived streams & windows using continuous queries

Demo

Wrap-up and questions

Event Stream Processing

How its used

Introducing SAP HANA smart data streaming


Event streams are becoming ubiquitous

Market prices

Transactions

Social media

Click streams

Sensors


Continuous queries vs. on-demand queries

Continuous On-demand

Dataflow model:

()

Incremental

computation

Store and query model (real-time):

Load/capture

data

Query


How does smart data streaming extend the capabilities

of the SAP HANA Platform?

Capture data arriving as individual events at potentially high speeds Hundreds of thousands or millions of events per second

Micro-batching and parallel processing to optimize load speeds

Capture events that are published from

streaming sources e.g. message bus

Filter, transform or enrich the data on the way

in Capture only the data you want, in the form you need it

Prioritize data Capture high value data in HANA and direct other data into

Hadoop

Monitor incoming event streams Watch for trends or patterns

Monitor correlations

Detect missing events

Continuously update and monitor aggregate statistics

Generate alerts, notifications

Initiate immediate response

Stream capture Immediate Response


SAP HANA smart data streaming brings the event stream processing

capabilities of SAP ESP into the HANA platform

SAP ESP

HANA Streaming SAP ESP

Available as an independent

product

Include current HANA-

integration features

Does not require HANA

Sold and installed as an

optional HANA Platform

component

Runs as a separate engine

within the HANA landscape

SAP HANA smart data streaming incorporates the same stream

processing technology as SAP Event Stream Processor (ESP) SAP named a Leader in

Streaming Analytics in

Forrester WAVE

(June 2014)

Querying streams - continuously

Key concepts


Key Concepts

Input Streams (or windows) Events arrive on input streams

Derived Streams,

Windows Apply continuous query

operators to one or more input

streams to produce a new

stream

Database Analogy Streams are like tables Events are rows in the table Fields in an event message are columns in the table

Windows can Have State Retention rules define how many or how long events are kept

Opcodes in events can indicate insert/update/delete and can be automatically

applied to the window


The basic types of continuous queries

Filter Enrich Analyze Synthesize

Eliminate the noise

Only see interesting events

Complex filters can

evaluate events in

context

Transform, normalize

Calculate new fields or

values

Combine data from

different sources

Compute trends

Statistics

Maintain summary

information

Produce new

response events when conditions are

met

Detect patterns,

trends, conditions


SAP HANA Studio design time tools for streaming projects

Streaming

projects are

executable

Streaming

plug-in adds

streaming

authoring

and run/test

perspectives

Two editors:

visual and

text (CCL)


CCL: Continuous Computation Language SQL-based Event Processing Language

Leverage familiarity and simplicity of SQL

Instead of snapshot queries, continuous queries Filter (WHERE), Aggregate (GROUP BY), Join, Compute

Extensions for event streams

Streams, Windows, Patterns

Example CCL Query

Calculation of 10 minute

moving average

CREATE LOCAL WINDOW AVG10 PRIMARY KEY DEDUCED AS SELECT TICKER.SYMBOL AS SYMBOL , avg( TICKER.Price) AS AVG_PRICE FROM TICKER KEEP 10 MINUTES GROUP BY TICKER.SYMBOL ;

Building a streaming project

Define inputs

Connect to sources

Create derived streams & windows using continuous queries


Create a new Streaming project

Install streaming plug-in for HANA

studio

Choice of working in visual editor or

CCL editor

Can switch back and forth and edit in

either


Input streams and windows, event schema

Data arrives via input streams/windows

Events contain a set of typed fields

All events entering a stream or window

must conform to the schema of the

stream/window

Streams are stateless

Windows must have a primary key, and

will normally have a KEEP clause

Windows can accept events with

opcodes: insert, update, delete, upsert

CREATE INPUT STREAM MACHINEDATA SCHEMA ( MACHINEID string, EVENT_TIME timestamp, EVENT_NAME string, EVENT_DESCRIPTION string, EVENT_VALUE string ) ;

CREATE INPUT WINDOW EVENTS_IN SCHEMA ( ID string , EVENT_TIME timestamp , EVENT_VALUE string ) PRIMARY KEY (ID) KEEP 1000 ROWS;


Input Adapters connecting to data sources

Adapters can be attached to input streams/windows and/or output

streams/windows

Attached adapters are managed by the project: when the project starts, the

adapters are started, and when the project

is stopped, the attached adapters are

stopped

Attaching adapters in the project is

optional. Independent adapters or

applications can initiate connections

ATTACH INPUT ADAPTER REF_DATA_IN TYPE toolkit_file_csv_input TO ENRICH PROPERTIES dir = 'C:/datafiles' , file = 'refdata.csv' ;


Filter

uses a WHERE clause

events either pass through the filter or are discarded

filters can be complex and dynamic CREATE OUTPUT STREAM Filter1 AS SELECT * FROM Events WHERE Events.value = A';


Hello World streaming style

CREATE INPUT STREAM Events SCHEMA (word1 string , word2 string ); /**@SIMPLEQUERY=FILTER*/ CREATE OUTPUT STREAM Filter1 AS SELECT * FROM Events WHERE Events.word1 = 'Hello' AND Events.word2 = 'World' ; ATTACH INPUT ADAPTER JMS_CSV_Input1 TYPE toolkit_jms_csv_input TO Events PROPERTIES connectionFactory = 'ConnectionFactory' , jndiContextFactory = 'org.apache.activemq.jndi.ActiveMQInitialContextFactory' , jndiURL = 'tcp://localhost:61616' , destinationType = 'QUEUE' , destinationName = 'csvin.queue' ;

optional


Aggregate

GROUP BY

Compute values for the group

Typically used to compute trends

or statistics on groups

Moving averages

Sum, count, min, max

Generates an event when an

aggregate value changes or a new

key is added

Note the KEEP in the FROM

clause; KEEP at the Window level

will apply to output, not input

CREATE OUTPUT WINDOW AVG_TEMP PRIMARY KEY DEDUCED AS SELECT EVENTS.MACHINEID MACHINEID, last(EVENTS.E_TIME) E_TIME, avg(EVENTS.E_VALUE) AVG_TEMP FROM EVENTS KEEP 30 SEC GROUP BY EVENTS.MACHINEID;


REFERENCE element: including HANA tables/views in

streaming projects

Join event streams to tables in

HANA

Discovery tool imports the table

structure into the reference

schema

HANA is queried as needed

Optional caching with age property

allows repeat queries to be filled

from cache

CREATE REFERENCE REF1 SCHEMA ( MACHINEID string , MACHINETYPE string , MAX_TEMP integer , MIN_TEMP integer , LOCATION string , TEMP_UNIT string ) PRIMARY KEY ( MACHINEID ) PROPERTIES service = 'hdbdmm165', source = 'MACHINE_REF', sourceSchema = 'DMM165_0', service = 'hdbdmm165';


Join

Combine data from two or more

sources

Standard SQL-style joins. INNER

and OUTER

Join a stream to a window or join

windows. Cant join 2 streams.

One incoming event can produce

multiple events in the case of a

one-to-many join

Can join ESP streams/windows to

tables in SAP HANA

CREATE OUTPUT STREAM EVENTS AS SELECT INDATA.ID ID, INDATA.EVENT_TIME EVENT_TIME, INDATA.EVENT_VALUE EVENT_VALUE, REF1.MACHINETYPE MACHINETYPE, REF1.MAX_TEMP MAX_TEMP, REF1.LOCATION LOCATION , FROM INDATA INNER JOIN REF1 ON INDATA.ID = REF1.ID;


Pattern detection

MATCHING clause specifies a

combination or sequence of

events across an interval

Can include a missing event

Pattern can use AND, OR,

NOT and , (followed by)

ON clause further qualifies the

events

ON A.ID = 12 AND B.ID = 43

CREATE OUTPUT STREAM ALARM_POWER AS SELECT A.MACHINEID MACHINEID , A.EVENT_TIME EVENT_TIME , A.LOCATION LOCATION , 'POWER' ALARM_TYPE FROM EVENTS A, EVENTS B MATCHING [ 20 SEC : A , ! B ] ON A.MACHINEID = B.MACHINEID AND A.EVENT_VALUE = 'Power off' AND B.EVENT_VALUE = 'Power on' ;


Capturing output in SAP HANA

attach HANA output adapter to desired

output stream or window

designed for extreme scalability on

inserts

also supports updates and deletes ATTACH OUTPUT ADAPTER HANA_Output1 TYPE hana_out TO ACTIVITY_HIST PROPERTIES service = 'hdbdmm165' , table = 'ACTIVITY_HIST' ;


CCLScript adds extensibility to CCL

Formerly known as SPLASH - provides extensibility to CCL and overcome the limitations of SQL

It complements CCL, providing the ability to write custom functions and stream/window operators using a simple,

procedural language that is optimized for stream processing

Analogous to SQLScript but specifically designed for stream processing

if (temperature < 0) {

display := 'below zero';

} else if (temperature = 0) {

display := 'zero';

} else {

display := 'above zero';

}

Control Structures: if, while, switch

Data Structures:

- Variables - Vectors - Dictionaries - Event Cache

Flex Streams:

- Custom event handlers written in SPLASH


Custom Flex operator using CCLscript

CREATE INPUT WINDOW ORDERS SCHEMA (OrderID integer, Symbol string, Side string, Price money(2), Size integer) PRIMARY KEY (OrderID); CREATE FLEX Flex1 IN ORDERS // Output will be a sorted order book of up to top 5 orders on each side OUT OUTPUT WINDOW OrderBook SCHEMA (Position integer, Symbol string, Side string, Price money(2), Size integer) PRIMARY KEY (Position, Symbol, Side) BEGIN DECLARE //ecOrders is an event cache that will hold all open orders, sorted by price eventCache (ORDERS[Symbol, Side], coalesce, Price desc) ecOrders; integer depth := 5; END; ON ORDERS { integer i := 0; //on each incoming order, republished for the symbol/side affected while (i < count(ecOrders.OrderID) and i < depth) { OrderBook := [Position=i; Symbol=ORDERS.Symbol; Side=ORDERS.Side | Price=nth(i, ecOrders.Price); Size = nth(i, ecOrders.Size);]; output setOpcode (OrderBook, upsert); i++ ; } }; END;


Advanced CCL techniques

Window Types:

Union

Splitter - Multi-directional filter

Autogenerate - Add a sequence number column that automatically increments

Aging - Increment the value in a column when timer expires; reset timer

when the row updates

Modules - CCL modules can be used in multiple project

- Can be loaded multiple times in one project

- Can be parameterized

getdata() to run scheduled or event-driven queries against HANA, including analytics

Recoverable windows (log store) - Recover window contents after restart

Event Cache - Custom windows

Dictionaries - Key,value pairs

Vectors

Parameters, Variables

External function libraries

User Defined functions in CCLScript

- Time, Row count

- Sliding, jumping

- Count per value


Key takeaways

1. Define and run streaming projects from SAP HANA Studio using the Streaming plug-in

2. Define continuous queries in CCL and/or using the visual editor

3. Simple SQL-like queries for common relational operations

4. Custom operators via inline CCLScript

5. Continuous queries can be used to:

o Transform raw data streams into useful information

o Extract insight from streams, such as trends or patterns

o Synthesize action events to initiate immediate response to new information


SAP d-code Virtual Hands-on Workshops and SAP d-code Online Continue your SAP d-code education after the event!

SAP d-code Online

Access replays of keynotes, Demo Jam, SAP d-code live interviews, select lecture sessions, and more!

Hands-on replays

http://sapdcode.com/online

SAP d-code Virtual Hands-on Workshops

Access hands-on workshops post-event

Starting January 2015

Complementary with your SAP d-code registration

http://sapdcodehandson.sap.com


Further Information

SAP Education and Certification Opportunities

www.sap.com/education

Watch SAP d-code Online

www.sapcode.com/online

SAP Public Web

scn.sap.com/community/developer-center/esp

scn.sap.com/community/event-stream-processor

www.sap.com/esp

32 2014 SAP SE or an SAP affiliate company. All rights reserved.

Feedback Please complete your session evaluation for

DMM110

[email protected]

Thanks for attending this SAP TechEd && d-code session.



2014 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an

SAP affiliate company.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE

(or an SAP affiliate company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional trademark

information and notices.

Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors.

National product specifications may vary.

These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its

affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or

SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing

herein should be construed as constituting an additional warranty.

In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or

release any functionality mentioned therein. This document, or any related presentation, and SAP SEs or its affiliated companies strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for

any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-

looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place

undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.

DMM110_16014_WoottonJ_1

Documents

Transcript of DMM110_16014_WoottonJ_1