IOT, Streaming Analytics and Machine Learning
-
Upload
dataworks-summithadoop-summit -
Category
Technology
-
view
1.844 -
download
0
Transcript of IOT, Streaming Analytics and Machine Learning
![Page 1: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/1.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
IOT, STREAMING ANALYTICS AND MACHINE LEARNINGDelivering Real-Time Intelligence With Apache NiFi
Paul Kent, VP of Big Data, Platform R&DDan Zaratsian, Sr. Solutions Architect
![Page 2: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/2.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
![Page 3: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/3.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
• Drop-and-Drag Interface
• Secure/Encrypted
• Bi-Directional Communication
• Data Provenance
![Page 4: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/4.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
![Page 5: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/5.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
SAS ESP + HDF WHY IS THIS IMPORTANT?
RAPID PROTOTYPING OF MACHINE LEARNING MODELS
ANALYTICS WITHIN AN OPEN FRAMEWORK
![Page 6: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/6.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
Events(Data In)
Events(Data Out)
FilteringAggregation
Pattern DetectionComputation
Merging / JoinsFunctions
Retention WindowText Analytics
Unsupervised LearningPredictive Modeling
…more…
Detect Events & Patterns of Interest
![Page 7: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/7.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
KEY CONCEPTSESP MODEL - PROCESS FLOW
SAS EVENT STREAM PROCESSING ENGINE
DATA IN (Events)
DATA OUT(Events)
Design of the rule model (called “Continuous Query”)using components (called “Windows”)
DATA IN (Events)
DATA IN (Events)
DATA OUT(Events)
SOURCE1
WINDOW
SOURCE2
WINDOW
SOURCE3
WINDOW
FILTERWINDOW
CALCULATIONSWINDOW
JOINWINDOW
JOINWINDOW
NOTIFICATIONWINDOW
PREDICTIVE MODEL
(SCORING)
WINDOW
![Page 8: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/8.jpg)
Copyr igh t © 2012, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
EVENT STREAM PROCESSING ESP STUDIO INTERFACE
![Page 9: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/9.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
DESIGNING ESP MODELS
DESIGN COMPONENTSDS2 PROCEDURAL WINDOW
In-Stream Analytics:
1. Build analytical model using EM,VA, etc.• Decision Tree• Neural Network• Regression• Rule Induction• And more
2. Use PROC DSTRANS to convert code to DS23. Deploy model to procedural window
Only when the existing model is additive in nature and can process one event at a time.
![Page 10: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/10.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
DESIGNING ESP MODELS
DESIGN COMPONENTSPATTERN WINDOW OVERVIEW
BUILD COMPLEX NETWORK OF EVENTS USING TEMPORAL CONDITIONSMULTIPLE EVENTS IN CAN PRODUCE ONE EVENT OUT
E1 E2And
FollowedBy
E4 E5AndNot
E6
E3Or
5 min
1 hourFollowed
By
“Detect when event A is followed by event B and not Event C in a 3min time frame”
![Page 11: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/11.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
DESIGNING ESP MODELS
DESIGN COMPONENTSTEXT ANALYTICS WINDOWS
• Process unstructured text fields• 3 dedicated Text Analytics windows
• Text Context (.liti files)• Text Category (.mco files)• Text Sentiment (.sam files)
• An appropriate Text Analytics license is required.
![Page 12: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/12.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
SAS EVENT STREAM PROCESSING HTML5 STREAMVIEWER
• HTML5 interface• Uses HTTP (RESTful) XML server• 2 Modes:
• Streaming mode: display all events• Update : events processed with opcode
• Google charts• Subscribe & Publish
![Page 13: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/13.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
INTEGRATION SAS EVENT STREAM PROCESSING & HORTONWORKS DATA FLOW (NIFI)
&SAS Event Stream Processing Hortonworks Data Flow (Nifi)
![Page 14: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/14.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
SAS® EVENT STREAM PROCESSING CONCEPTUAL OVERVIEW
SAS-generated Insights
Enrichment Data
Event Actions
SAS In-Memory
SAS® Event Stream Processing Model
Continuous QueryPu
blis
h
Subs
crib
e
Streaming Events
Analytic Models
Business Rules
Nifi
![Page 15: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/15.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
SAS EVENT STREAM PROCESSING CONNECTORS & ADAPTERS
PUB/SUB API Connect to any system with Java or CPublic, documented and easy to useAdapters are standalone processes and can be networkedPublish to ESP Source windows – Subscribe to any ESP windowAll Connectors & Adapters are built using the Pub/Sub API
•File/Socket•XML / JSON•Database (odbc)•SAS® LASR™•Hadoop•SAS® Dataset
OUT OF THE BOX
*Publish only**Subscribe only
•ESP Project•RabbitMQ•Solace•Tervela•Google Protobuff•Twitter*
•SAS® HDAT•JMS•IBM WebSphere MQ•Tibco RendezVous•Syslog *•Network Sniffer*
•HTTP RESTful•OSIsoft PI•Axeda•Teradata•SMTP **•ESP to ESP
![Page 16: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/16.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
DIRECTION: ADAPTERS & CONNECTORS
•Flume: Integrate ESP with streaming log data•Kafka: Integration with large scale message processing•MQTT: Support within IoT and Connected things•Cassandra (adaptor only): integration with large-scale, distributed data source•HortonWorks Data Flow (NiFi) Processor: support NiFI streams•MapR: MapR Streams support•Boardreader: Blogs, News, Boards, Reviews•Spryware: Market data through direct exchange feeds•IOT Gateways and devices
PUB/SUB APIConnect to any system with Java or CDocumented and easy to useAdapters are standalone processes and can be networkedPublish to ESP Source windows – Subscribe to any ESP windowAll Connectors & Adapters are built using the Pub/Sub API
SAS® EVENT STREAM PROCESSING 4.1
![Page 17: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/17.jpg)
Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.
SAS® EVENT STREAM PROCESSING 4.1 FLEXIBILITY AND INTEGRATION
• Python Pub/sub API: drive ESP using Python• Leverage Analytic Decisions within ESP
• Decision/Rules/Analytical Model Integration via SAS Micro Analytic Service• Lightweight, fast service for decision deployment
• Leverage Languages in ESP (In-process Event Stream Handlers)• DATAstep (native)• DS2 (current)• Python• Future: R language (post-16w48)
![Page 18: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/18.jpg)
Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.
Streaming Analytics Ecosystem
EdgeAnalytics
In-MotionAnalytics
At-RestAnalytics
Connected Systems, Devices
Monitor equipment on for failures and safety
issues, and take action.
Identify fraudulent transactions and be alerted in real-time.
Intelligently integrate customerinformation with real-time
streaming data
Strategic Data IntegrationTransactions, Logs, Clickstreams
![Page 19: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/19.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
STREAMING ANALYTICS Where are the Opportunities?
• Competitive Pressure (Technology, Sensors, Analytics)
• Risk
• Safety
• Security
• Personalization
Extend the existing analytical footprint!Capture value otherwise lost through information lag
![Page 20: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/20.jpg)
Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.
INTEGRATION SAS EVENT STREAM PROCESSING & HORTONWORKS DATA FLOW (NIFI)
&SAS Event Stream Processing Hortonworks Data Flow (Nifi)
![Page 21: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/21.jpg)
Copyright © 2012, SAS Institute Inc. All rights reserved.Copyright © 2012, SAS Institute Inc. All rights reserved.
Paul Kent, VP of Big Data, Platform R&DDan Zaratsian, Sr. Solutions Architect
Copyright © 2012, SAS Institute Inc. All rights reserved.Copyright © 2012, SAS Institute Inc. All rights reserved.
Demo
![Page 22: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/22.jpg)
Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.
![Page 23: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/23.jpg)
Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.
![Page 24: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/24.jpg)
Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.
![Page 25: IOT, Streaming Analytics and Machine Learning](https://reader034.fdocuments.in/reader034/viewer/2022052117/587dbab31a28ab1b498b4943/html5/thumbnails/25.jpg)
Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.