Making data-driven decisions with a HANA native agile datamart.sapevents.be/MDW/presentations/Data...

© 2016 SAP SE or an SAP affiliate company. All rights reserved. 1Customer

. . .nemeryphilippe@nemeryp

Making data-driven decisions with a HANA native agile datamart.

October 2016 [email protected]

https://twitter.com/nemeryp


https://be.linkedin.com/pub/philippe-nemery/0/400/392




mailto:[email protected]


Marketplace Early Signal Detection Systempowered by SAP HANA Data Platform & Predictive Analytics

Link

https://www.youtube.com/watch?v=hS-0ZadT6so

https://www.youtube.com/watch?v=hS-0ZadT6so


Predictive Maintenance


Predictive Quality Insurance


Forecasting - Predictions

?NOW

PAST FUTURE

Time-Window


NOWPAST FUTURE

Sensor-1

Sensor-2

?

Forecasting - Prediction


NOWPAST FUTURE

Time-Window

Sensor-1

Sensor-2

Action-Window

Action-Window: time

during which an

action can be taken

(e.g. 10 minutes)?


Classification: Will there be an incident in 10 minutes?

Regression: What is the value in 10 minutes?


NOWPAST FUTURE

Time-Window

Sensor-1

Sensor-2

Action-Window?


Classification: Will there be an incident in 10 minutes?

Regression: What is the value in 10 minutes?

- Values Now S1, S2, …

- Average Value S1 T-0




- Min Value S1 T-1

- Max Value S2 T-1

- Deltas between To, T-1, T-2

ToT-1T-2

Devices

BO Dashboards

External

DataFinancial,

Sales

Big Data Lake Training of Predictive Models

SDS

Triggering of alarms/actions

based on predictive models

SDS: Smart Data Streaming

SDI - SDQ

Hana Vora

Real Time

Gateway

System

Real-time Stream

Analysis

Visual

Exploration

SAP HANA

Platform

Hadoop

Devices SDS




Real Time

Gateway

System

Real-time Stream

Analysis

Smart Data Streaming

© 2016 SAP SE or an SAP affiliate company. All rights reserved. 13Public

Event stream processing uses continuous queries

Database Queries Continuous Queries

Step 1:

Store the data

Step 2:

Query the dataStep 1:

Define the

continuous

queries and the

dataflow

Step 2:

Wait for data to arrive.

As it arrives, it flows

through the continuous

queries to produce

immediate results


Complex Event Processing extracts insight from events

Sensor readings – 10’s of thousands per second

Virtually no useful

information in a

single isolated event history

e.g. Compare

variance of trends

across multiple

sensors against

historical norms

Event window – e.g. 30 min

Alert

© 2014 SAP AG or an SAP affiliate company. All rights reserved. 15

Stream Processing

Creating equidistant data and provisioning data stores

• Streams and windows, CCL script

• Create, enrich and aggregate equidistant data

• Publish raw data to hdfs and aggregates to HANA tables


Apply SAP Predictive Analytics Model within SAP Smart Data

Streaming

• Automated Analytics now supports smart

data streaming

• Generates CCL Code which can be

deployed to HANA SDS

HANA

Smart Data

Streaming

Predictive Analytics

Automated

Modeller


Overview of SDS integration options

SDS Project

Data sources

Alerts

Applications

Adapters

input output

Appplication

with SDS

connectivity

Application with

SDS

connectivity

SDS pub/sub library

Input adapters receive/fetch messages using a

protocol supported by the source, parse the

message, and publish it to SDS as an event.

Output adapters do the same – in reverse.

The SDS SDK provides a library that

can be embedded in adapters or

applications to connect to SDS and

publish/subscribe

Devices

BO Dashboards

External

DataFinancial,

Sales

Big Data Lake

SDS




SDI - SDQ

Real Time

Gateway

System

Real-time Stream

Analysis SAP HANA

Platform

Hadoop

Hana Vora


SAP HANA: The Real-time Platform of the Digital Economy

SAP HANA PLATFORMON-PREMISE | CLOUD | HYBRID

Web Server JavaScript

Fiori UX Graphic

Modeler

Data Virtualization ELT &

Replication

Application Services Integration & Quality Services

Columnar

OLTP+OLAP

Multi-Core &

Parallelization

Advanced

Compression

Multi-

tenancy

Multi-Tier

Storage

Spatial Graph Predictive Search

Text

Analytics

Data

Quality

Series

Data

Business

Functions

ALM

Processing Services

Database Services

Hadoop & Spark

Integration

Streaming

Analytics

Application Lifecycle

Management

High Availability &

Disaster Recovery

OpennessData

Modeling

Remote Data

Sync

Admin &

Security


SAP HANA: The Real-time Platform of the Digital Economy

SAP HANA PLATFORMON-PREMISE | CLOUD | HYBRID

Web Server JavaScript

Fiori UX Graphic

Modeler

ELT &

Replication

Application Services Integration & Quality Services

Columnar

OLTP+OLAP

Multi-Core &

Parallelization

Advanced

Compression

Multi-

tenancy

Multi-Tier

Storage

Spatial Graph Search

Text

Analytics

Data

QualityBusiness

Functions

ALM

Processing Services

Database Services

Streaming

Analytics

Application Lifecycle

Management

High Availability &

Disaster Recovery

OpennessData

Modeling

Remote Data

Sync

Admin &

Security

Data

VirtualizationPredictive

Series DataHadoop & Spark

Integration


Time Series Data in SAP HANA

Store

Support both equidistant and non-equidistant data

Support very high volumes of data using effective compression

techniques

Handle

Efficient grouping to different granularities (GROUP BY

SERIES_ROUND(…))

Built in SQL functions for efficient handling of Series Data

– SERIES_GENERATE;

– SERIES_DISAGGREGATE;

– SERIES_ROUND;

– SERIES_PERIOD_TO_ELEMENT;

– SERIES_ELEMENT_TO_PERIOD

Analyze

Analytic operations to be expressed naturally in SQL while

maintaining high performance

– AUTO_CORR, CROSS_CORR

– BINNING

– CUBIC_SPLINE_APPROX, LINEAR_APPROX

– DFT

– RANDOM_PARTITION

– SERIES_FILTER

– WEIGHTED_AVG

– Sliding window support

– {FIRST/NTH/LAST}_VALUE


IBM DB2, Netezza,

Oracle, MS SQL

Server, Teradata,

SAP HANA, SAP

ASE, SAP IQ

Modeling & SQL Script

S AP H AN A P L AT F O R M

Smart Data Access

Virtual Tables

Smart Data Integration

Built-In Adapters Custom Adapters

ODataDB2, Oracle,

MS SQL Server,

Teradata, SAP HANA,

SAP ASE

Adapter Framework

Metadata

S AP H AN A P L AT F O R M

Smart Data Access Smart Data Integration

Access any data from any source

Devices

BO Dashboards

External

DataFinancial,

Sales

Big Data Lake

SDS




SDI - SDQ

Real Time

Gateway

System

Real-time Stream

Analysis SAP HANA

Platform

Hadoop

Hana Vora


SAP HANA VoraWhat’s Inside and What Does It Do?

Democratize

Data

Access

Enable

Precision

Decisions

Making

Simplify

Big Data

Ownership

SAP HANA Vora is an in-memory query engine which leverages

and extends the Apache Spark execution framework to provide

enriched interactive analytics on Hadoop. Drill Downs on HDFS

Compiled Queries

HANA-Spark Adapter

Unified Landscape

Open Programming


Vora Time Series (in Vora 1.3 – GA End September)

Sequence of data points recorded over time

Can be equidistant or non-equidistant

Detect and correct errors / anomalies

Granularization

Standard aggregation

Analysis

Specify a series clause during table creation

Define the period column (timestamp)

Provide a compression definition (optional)

Define start/end of series (optional)

Define the series increment (optional)

Column Functions

Trend

Stddev

Median

Linear_Approx

Const_Approx

Cubic_Spline_Approx

Polynomial_Approx

Table Functions (coming soon!)

Auto_Corr

Cross_Corr

Histogram

DFT

Granulize

Devices

BO Dashboards

External

DataFinancial,

Sales

Big Data Lake Training of Predictive Models

SDS




SDI - SDQ

Hana Vora

Real Time

Gateway

System

Real-time Stream

Analysis

Visual

Exploration

SAP HANA

Platform

Hadoop


Visual Exploration

?

Demo

Any DataBase (Oracle, etc.)

Consume universes (.UNX) from SAP BI 4.x

Connect to SAP Business Warehouse (BW)

Support for Big Data sources (>15K columns) including Hadoop/Hive and Spark

Guided Timeseries ForecastingFeature Building

29

Choose the deployment.


Predictive Analytics Overview

Automated

Guided - Automated

Data Manager

Modeler

Expert

Workbench

Data Manip.

Workflow

Visualization

HANA

APL PAL

R

HANA SDSHadoop

Hana

Processing & DB ServicesHANA

APL PAL

R

Automated Predictive Library

(+6 BP)

Predictive Analysis Library (+100 M)

Conclusions


SAP SDS – HANA Advanced Analytical Platform

SAPAnalytical Platform

Future-Proof

Analytical Roadmap

Series Data

Capture and analyze a sequence of successive data points made

over a time interval

.

Enterprise E2E Platform

Centralized administration, management & auditing

Advanced & Real-Time Analytics

Experience

Project Approach and Partner network

Different Users

Role-based and maturity-based approach

Flexibility and Agility

Thank you

Contact information:

Philippe Nemery

[email protected]

mailto:[email protected]

Making data-driven decisions with a HANA native agile datamart.sapevents.be/MDW/presentations/Data...

Documents

Transcript of Making data-driven decisions with a HANA native agile datamart.sapevents.be/MDW/presentations/Data...