Post on 16-Apr-2017
Use this title slide only with an image
SAP & Big Data
@timoelliott
Timo ElliottInnovation EvangelistJune 2016
Technology Priorities for 2016 and beyond
Rank Technology Trend
1 BI/Analytics2 Cloud3 Mobile4 Digitalization / Digital Marketing5 Infrastructure & Data Center6 ERP7 Security8 Industry-Specific Applications9 Customer Relationships
10 Networking, Voice, and Data Comms
Nine out ofeleven years2006-2016
ANALYTICS
#1
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 3
Analytics Takes Over The World…
@timoelliott
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 4
By 2020, information will be used to
reinvent, digitalize, or
eliminate 80%of business processes and products
from a decade earlier.
From The Back Office To The Business Models of Future
”
“
@timoelliott
5
Digital Business and the Rise of the CDOs
@timoelliott
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 6
Bimodal IT…
“CIOs practicing bimodal IT are making a strategic mistake.”
—Forrester
“Customer experiences aren’t confined to a small subset of systems. Even simple purchases needs to reach back into fulfillment and billing systems.”
Live Business
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 9
Are you a BI-nosaur?
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 10
Culture Change
From Power to Empower
From Collection to Connection
From Control to Trust
“So we don’t need a centralized truth?!”
“Absolutely. Never worked, doesn’t work, will not work”
@frankbuytendijk
“BICCs are Dead”
Data-Driven Approach
Push:• From IT• Data-Driven• Data to Insight• Technology-Centric
Value-Driven Approach
Pull:• From LOB• Outcome-Driven• Insight to Data• Use-Case-Centric
Combination Approach
Push:• From IT• Data-Driven• Data to Insight• Technology-Centric
Pull:• From LOB• Outcome-Driven• Insight to Data• Use-Case-Centric
Use this title slide only with an image
Advanced Analytics@timoelliott
15© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Big Data Discovery =
Big DataData DiscoveryData Science
Gartner Strategic Planning Assumption: By 2017, Big Data Discovery Will Evolve Into a Distinct Market Category
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 16
Big Data Discovery
• Volume, velocity, or variety of data
• Potential business impact
• Difficult to implement• Potentially expensive• Lack of skills available
• Ease of use• Agility and flexibility• Time-to-results• Installed user base
• Complexity of analysis
• Potential impact• Range of tools• Smart algorithms• Difficult to implement• Slow and complex• Narrow focus of
analysis
• Limited depth of information exploration
• Low complexity of analysis
BIGDATA
DATASCIENCE
DATADISCOVERY
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 17
Big Data Discovery
• Simpler to use than data science
• Accessible to a wider range of users
• Broad range of data manipulation features
• Able to handle new types of data sources
• With adequate performance for big data
BIG DATA
DISCOVERY
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 18
Potential impact per user
Potential user base
The Rise of the Citizen Data Scientist?
Business analyst
Data scientist
Citizen data scientist
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 19
SAP’s Opportunity
Big Data
Discovery
SAP HANA SAP IQ
Vora / Spark / Hadoop
SAP Predictive Analytics 3.0
SAP Lumira
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 20
Bringing it All Together
ETL DW Q&R, Data
Discovery
Predictive Planning Other (spatial,
etc.)
Data Visualization
Operational Reporting
Enterprise Data Big Data
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 21
The New Multi-Polar World of Big Data Architectures
Data Warehouse
Hybrid Transaction/
Analytical Processing
Hadoop,MongoDB,Spark, etc Personal
Data / BI
Where does data arrive?When does it need to move?Where does modeling happen?What can users do themselves?What governance is required?
Big Data Architectures got complicated
What we want — consistent, seamless solution
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 22
Your BI Data ISN’T In Your Corporate System
“We found, on average, that 45% of the data business people use resides outside of the enterprise BI environments.
An astonishingly miniscule 2% of business decision-makers reported using solely enterprise BI applications.
This is undoubtedly connected to 76% of business respondents indicating they continue to resort to spreadsheets and other homegrown BI applications to analyze BI data. ”
Source: Forrester
55%
45%
In enterprise systemsNot in enterprise system
“Intricate calculations of sales by territories will appear as if by magic in the digital age ahead”
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 24
Decision Cockpits
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 25
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 26
SAP BusinessObjects Cloud
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 27
Cloud Analytics
@timoelliott
It’s not Pie in the Sky!
Data gravity: BI in the cloud when the data’s in the cloud
Securely Connect to Your Data In The Cloud AND On-Premise
SAP Cloud for AnalyticsWeb Client running in browser
S/4HANA
Public Cloud
BW
HANA
External data via HANA Smart Data Access
Public Cloud data sources
Data Connector
On-Premise System
Local Network
SAP Cloud for AnalyticsPlatform and Content Repository
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 29
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 30
SAP Lumira for Data Discovery
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 32
Health indicators across fleet
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 33
Lufthansa Systems LIDO
Lufthansa integrateddispatch operations
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 34
Mercy Health
Mercy — one of US Most Wired for 12th Year in a row!
• $4.48 billion revenue• 40K employees• > 8M patients/year
“It is mind-blowing how versatile and nimble our data warehouse is on SAP HANA.”
Agile self-service with SAP HANA and SAP Lumira. 9 years of data, structured & unstructured
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 38
What Pipes?
Type 1
Type 2?
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 42
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 43
Centerpoint Energy
New Business Models
Kaeser Compressor, a global leader in air compressors
≈€500 million, 4,800 employees, 50 countries, partners in additional 60 countries
@timoelliott
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 45
Modeling Example
E.g. Total energy consumption
• Aggregation of 10 sec values
• Calculation of typical consumption patterns
• Pattern associated with each compressor and day
Repeat for temperature, pressure, vibration, etc.
@timoelliott
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 46
Predictive Examples
Model combines sensor readings and ERP data (location, type of usage, last service, etc.)• Status alerts: “Oil change / oil analyze / no action”• Predict machine failure 24 hours in advance
@timoelliott
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 47
High-Level Technical View
Predictive Model(in-memory)
Long-term disk storage
User Interfaces
CRMERP
Event Stream Processing
all sampled
Customer Field Svs Sales R&D
DW
@timoelliott
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 48
Benefits
Customers• Less downtime• Decreased time to resolution• Optimal longevity and performance
Kaeser• More efficient use of spare parts, etc• New sales opportunities• Better product development
“We are seeing improved uptime of equipment, decreased time to resolution, reduced operational risks and accelerated innovation cycles.
Most importantly, we have been able to align our products and services more closely with our customers’ needs.” �
Kaeser CIOFalko Lameter
@timoelliott
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 4949
© 2014 SAP AG. All rights reserved. 50
Hadoop + Hana
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 51
Information Ecosystems
51
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 52
New Products & Services
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 53
Pret A Manger
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 54
55
HADOOP is key Part of SAP’s Open Source Development usage
1
10
100
1000
10000
Open source consumption Open source contribution SAP Contributes to over 100 Open Source Projects
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 56
SAP Hadoop Partnerships
SAP HANA Platform
The SAP focus: End-to-end value chain
SPATIAL PROCESSING
ANALYTICS, TEXT, GRAPH, PREDICTIVE
ENGINES
CONSUME
COMPUTE
STORAGE
SOURCE
INGEST
Application Development Environment
Transformations & Cleansing
Smart Data IntegrationSmart Data Quality
StreamProcessing
Smart Data Streaming
STREAM PROCESSING
LogsTextOLTP Social Machine GeoERP SensorStore & forward
Mobile applications and BI
Smart Data Access
Virtual Tables
User Defined Functions
101010010101101001110
Dynamic Tiering
Aged datain Disk
In-Memory
Data model& data
Calculation engine
Fastcomputing
Column Storage
High performance analytics
Series Data Storage
Store time-series data
Reporting &Dashboards
High Performance Applications
Data Exploration& Visualization
Adhoc & OLAP Analytics
PredictiveAnalysis
Business Planning & Forecasting Lumira / BI
But there is more work to do…
Hadoop / NoSQL
MapReduce
YARN
HDFS
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 58
Our Journey
SP06
SP07
SP09 HDFS
Yarn/MR
HBASEHive
SparkPig
Mahout
Ambari
Hive Added as a Remote Source
ODBC Based Communication
Query Optimization Like Remote Caching and Join Relocation
Reading HDFS Directly
Map Reduce Job Execution
SP10 Spark SQL added as a new Remote SourceAmbari launcher tile in HANA Cockpit
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 59
HANA & Hadoop Integration
HANA & Hadoop Integration SQL on Hadoop via SDA (virtual tables) – Hive
(SPS06) Remote caching with Hive (SPS07) Connectivity to Apache Spark using ODBC Execution of MR-Jobs via HANA (Virtual Functions)
and direct access to HDFS (SPS 09) Spark SQL adapter via SDA (SPS10) Join relocation to Hadoop thru SparkRDD Unified Admin thru Ambari integration for Hortonworks
Key Benefits Deep Integration for storage & processing Optimized data access between HANA & Hadoop Data tiering to Hadoop for cold storage
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 60
SAP HANA VoraWhat’s Inside and What Does It Do?
DemocratizeData Access
Make PrecisionDecisions
SimplifyBig DataOwnership
SAP HANA Vora is an in-memory query engine which leverages and extends the Apache Spark execution framework to provide enriched interactive analytics on Hadoop. Drill Downs on HDFS
Mashup API EnhancementsCompiled Queries
HANA-Spark AdapterUnified LandscapeOpen Programming
Any Hadoop Clusters
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 61
YARN
HDFS
Enable Precision DecisionsWith Contextual Insights In Enterprise Systems
Other Apps
Files Files Files
HANA-Spark Adapter for improved performance between distributed systems
Gain business coherence with business data and big data
Compiled queries enable applications & data analysis to work more efficiently across nodes
Familiar OLAP experience on Hadoop to derive business insights from big data such as drill-down into HDFS data
Compiled Queries
Spark Adapter
Drill Downs
SAP HANA in-memory platform
Vora
Spark
Vora
SparkIn-Memory
Store
Application Services
Database Services
Integration Services
Processing Services
SAP HANA Platform
Vora
SparkHANA-Spark
Adaptor
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 62
Democratize Data Access for Data Science Discovery
Extensive programming support for Scala, python, C, C++, R, and Java allow data scientists to use their tool of choice,
Pursue new inquiries without compromise on data and easily integrate these insights with all data
Enable data scientists and developers who prefer Spark R, Spark ML to mash up corporate data with Hadoop/Spark data easily
Optionally, leverage HANA’s multiple data processing engines for developing new insights from business and contextual data.
Mashup Enhancements
Open Programming
Optional Use of SAP HANA for Delegated, multi-engine pre-processing
Spark Data-source API enhancement
In-Memory Store
SAP HANA Platform
YARN
HDFSFiles Files Files
Vora
Spark
Vora
Spark
Vora
Spark
HANA Smart Data Access, UDFs, Others
Application Services
Database Services
Integration Services
Processing Services
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 63
SAP HANA VoraWhat’s Inside and What Does It Do?
DemocratizeData Access
Make PrecisionDecisions
SimplifyBig DataOwnership
SAP HANA Vora is an in-memory query engine which leverages and extends the Apache Spark execution framework to provide enriched interactive analytics on Hadoop. Drill Downs on HDFS
Mashup API EnhancementsCompiled Queries
HANA-Spark AdapterUnified LandscapeOpen Programming
Any Hadoop Clusters
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 64
Vora SQL Engine
#FEA433
Components
Written FromScratch
Multi Platform
Compressed Columns
Parallel QueryProcessing
In Memory Storage Fast Column Scans
Cache EfficientAlgorithms
Code Generation
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 66
SAP HANA Vora: Strategic Point of View
● Add functionality for enterprise applications● Hierarchies● OLAP modeling
● Boost SQL performance● Federate access across HANA and Hadoop● Integrate tooling
SAP HANA
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 67
SQL/OLAP on Big Data
• Hierarchical data storage of contextual data supports structured analysis
• Fast drill-down interaction aids in root-cause analysis
• Familiar OLAP tool enables experienced business analysts derive useful insights from contextual data
• Support for HDFS, Parquet and ORC formats
• LLVM/Clang – JIT compilation of query plans and execution Hadoop/NoSQL DATA
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 68
ALL IN-MEMORY
In-Memory Data Fabricfor Enterprise + Distributed Compute
Enterprise Compute Distributed Compute
CO
NSU
ME | C
OM
PUTE | STO
RE
HANAOLTP + OLAP
Scale Up
Scale Out+
Massive Scale Out
Appliance | TDI
Vora Vora Vora Vora
Vora Vora Vora Vora
Vora Vora Vora Vora
Distributed File System | Network Storage | Cloud Persistence | Any Hardware
Federated Queries &
Programming Model
+Tiering
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 69
SAP Predictive Analytics 3.0
Native Spark Modeling
Standalone or included in SAP HANA
Predictive Factory
Integration with cloud & other apps
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 70
Suite
Applications
S/4HANA
DigitalBoardroomIcon
Analytics
C4A
BOBJ
ExtensionsApplicationsIoT
HANA Cloud Platform
(Micro-) Services
IoTPlatform
Identity Management
Business Network
CEC
Platform
HANAEnterprise
Computing Platform
any DB Hadoop
VoraDistributed Computing
Platform
SAP Platform for Digital Transformation
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 71
Revenue Assurance
Detection
Investigation
Prevention
Correction
Revenue Assurance Lifecycle
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 72
The Big Idea
AR PUM
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 73
It’s Complicated
CostTransactional cost on monthly basis
for 3-6 month
SubscriberSubscriber master data
for the last 12 month
RevenueRevenue on monthly basis for 3-6 month
ProductProduct master data
Service ComponentsMaster data on service components that are the core element for building products and tariffs
Margin Margin table generated during data discovery service forms the basis for the analysis
=Billions of records
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 74
Seeing Clearly
Customer
Subscription
Cost and Revenue
Cost and Revenue
Classification
Unit Prices for Indirect and Direct Cost
ServicesService Specification
Billing Account
Carrier
Third Party
Product Offering
Inspired by TMForum’s Information Framework, follows ABDR (Analytics Big Data Repository) design principles.
Overview
Typical aggregated view of profitability only
SAP HANA & Lumira
SAP HANA & Infinite Insight
Find Opportunities
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 77
Visualization
Tree map: Profitability per level & drill down
Distribution: Lowest Margin Customer Analysis
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 78
Visualization
Identify SOC Combinations which make less margin per client
than all the SOC Combinations covering it
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 79
Visualization
green - acceptable margin
red - margin decrease
bubble size – number of customer
bubble – combination of service components
arrow - similar but slightly different service component combinations
thickness – similarity of service component combinations
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 80
Margin Assurance
Invoice ControlDelta-Calculation
Identification of Trends
&Trend-Alarming
Detection of Fraud and Misuse
Business Application Scientific Approach/ Method
SAP Solution
Identification of Margin Drivers
Analysis of Underperforming
Clients
Statistical Outlier Detection
Time-Series Analysis
Clustering
Network Analysis
Tarif
f & P
rodu
ct P
ortfo
lio M
anag
emen
t Too
ls Aut
omat
ed M
onito
ring
Tool
s
Da
ta-D
riv
en
To
ols
fo
r M
arg
in A
ss
ura
nc
eSAP Lumira
(Data Visualization)
HTML5 – based apps/dashboards
MNO Custom Built Web Applications
PAL(Data
Analytics)
HANA DB
HANA XSWeb
ContentXSJS REST
Services
Stored Procedures
(Automation Logic)
Views and SQL Scripts
R-ServerSAP Infinite
InsightApp-Server (Custom Built Application)
SA
P H
AN
A
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 81
Margin Assurance
€100m’s
€10m’s
Revenue LeakageInsight Value
Time
Learning, Prediction & Process Integration Decision
SAP Margin Assurance
Revenue Assurance Today
3-6 months
DataAnalysis
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 82
Margin Assurance
“The ability to understand granular customer profitability, specifically which customers are profitable or unprofitable and why, is a game-changer for our industry.”
-- Thomas Holtmanns, Vodafone Director Finance Operations Germany and Global Margin Assurance.
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 83
Use Cases
Underperforming Clients
Low margin customers mapped to Post-pay Tariffs & Products.
& Price simulation on static usage profiles.
Discount Control
Identification of low margin customers driven by discounts
e.g. Discount stacking. Or poor combinations.
Negative Margin Clusters
Combinations of Usage behaviour; Tariffs, Products and SoCs with negative margins.
e.g. M2M reverse charging and hotspots.
Profitability Time Analysis
Benchmarking, trend monitoring and alarms (against KPIs).
Identification of positive and negative trends month to month.
MarginDrivers
SoCs, Tariffs, Products, Customer Behaviour or Events, significantly driving positive or negative margin.
Mis-Use OR Fraud
Outlier detection and causal effect analysis
e.g. SMS boxes, breaching T&Cs.
Automated Invoice Control
Benchmarking, trend monitoring and alarms.
Identification of positive and negative trends month to month.
New Offer Performance
Tag & Track of new product OR tariff introductions.
Link with Profitability Time Analysis.
Learning & Prediction
WHAT IF analysis leveraging historical trends – price and usage elasticity.
e.g. Simulation of new propositions; competitor tariffs OR changing regulation
End to End Client Performance
Profitability of ALL Tariffs & Products for all market types & customer segments.
e.g. MMC, MMO / Corporate - VGE Top X….
ShowcasePhase 1 Phase 2
Monthly Data Source Near Real-time
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 84
Looking to The Future
Customer Value
(Profitability)
Customer Experience
Network Performance Geospatial ROI & Data
Monetisation
Granular Profitability – Customer, Group (Corporate, Family), Tariff and Product
Single View of Customer & Network (Learning & Prediction)
Customer Experience – Call, Data, Group Behaviour and NPS
Network Performance – Radio, Cell & Data Usage & Errors
Asset Value – ROI & Data Monetisation
Geospatial – Journey, Occupancy, Proximity, Qos, RTOM
Insight to Action Unique Profitable Propositions e.g. SLA Guarantees &
Dynamic Pricing and Policy Management
Investment in the right assets in the right locations (ROI)
Improvement to; or new business processes / workflows Learning & Prediction
Enablement environment for new ‘valuable’ business apps.
© 2014 SAP AG. All rights reserved. 85
Design Thinking
Understand
Ideate
Prototype
Test
Build
Deliver
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 86
Thank You!Timo ElliottVP, Global innovation Evangelist
Timo.Elliott@sap.com @timoelliott