Post on 18-Jul-2015
Insights From Internet Of Things & Big Data
Insights From Internet Of Things & Big Data
Kostya Goldstein
Sr. Program Manager Microsoft Russia
Business insights through big data
Microsoft’s solution to big data
Intelligent systems service, HDInsightFeatures & capabilities
Demo – HDInsight
Office as Big Data visualization platformSelf service BI – Features & capabilities
Demo – Power BI
HackathonTessel
IoT Hands On Task
Agenda
World Today
2003 2010 2015 2020
50 млрд
GROWTH OF DEVICES
World tomorrow
1990
2020
“ ”
Internet of Things (IoT)
The network of physical
objects that contain
embedded technology to
communicate and interact
with their internal states or
the external environment.
Вопрос
COW IoT ?
Интернет вещей
Аудио /Видео
Журналы операций
Тексты/Изображения
Настроение
высказываний
Обновления витрин
данныхНовости электронного
правительства
Погода
Вики / БлогиПереходы по
ссылкамДатчики/ RFID / Устройства
Координаты GPS
WEB 2.0Мобильные
устро-ва
Реклама ВзаимодействиеЭлектроннная
коммерция
Цифровой
маркетинг
Поисковый
маркетинг
Протоколы веб-
серверов
Рекомендации
ERP / CRM
Конвейер
продаж
Кредитор
ы
Зарплата
Запасы
Контакты
Отслеживани
е торгов
терабайты
(1012)
гигабайты
(109)
экзабайты
(1018)
петабайты
(1015)
Скорость | разнообразие | изменчивость
Объ
ем
1980
190,000$
2010
0.07$
1990
9,000$2000
15$Стоимость хранения за гигабайт, долл
ERP / CRM WEB 2.0 Интернет вещей
What the big data is?
Two main IoT aspects
Start Justin
Customer
ServiceProvider
Microsoft
ConsistentPlatform
ONE
Cloud OS VisionMicrosoft’s vision of the unified platform
for modern business
Transform The Datacenter
Unlock Insights On Any Data
Empower People-centric IT
Enable Modern Business Apps
Create The Internet Of Your Things
Intelligent Systems Service
Microsoft Solution For Internet Of Things
Drive InsightsAnalytics ReadyCloud and infrastructure
Devices and
assets
10101010011000110101010111010011010101010100110111011110111001010100001101010101110100110101010111010011101010101011010011010101010101001101100010101111010011101010101011011110100111
10101010011000110101010111010011010101010100110111011110111001010100001101010101110100110101010111010011101010101011010011010101010101001101100010101111010011101010101011011110100111
Customer
portal Value
StreamInsights
Power BI
HDInsight
Windows Embedded
Connect new and existing
devices using open-source
agents or gateway
technologies
Azure, HDInsight
Store machine-generated
data with data from other
sources in the cloud
Office 365, Power BIView data, administer
devices, and configure
rules, alerts, and other
actions using out-of-box
or custom portals
Mine insights from your
data to find gaps and
opportunities to make
better decisions and realize
new business value
User
input
AlertsSensors Gateway
Agent
ADevices
IoT Services Architecture & Platform Components
ISS (Intelligent Systems Service)
Agent
Gateway
Event Hub & Azure Service
Bus
Event Processing&
Rules Engine
TablesBLOBS
SQL AzureHDFS
IF {condition}
THEN {action}
Azure Service Bus
Design & Engineering
Manufacturing & Supply Chain
Service & Maintenance
Customer Relationship
ISS (Intelligent Systems Service)
ID
Industrial
Equipment
Operational Data
Example of modern data storage
How To Generate Value From IoT Data
BIG DATA: Data powered by IoT & other business systems
BETTER Insights: Transform your business with better insights.
Unstructured
Structured
Streaming
PB
TB
GB Advanced analyticsData scientist
Interactivity +
ExplorationBusiness analyst
Self-service
analysisBI professional
Decision supportDevice operator
Big Data
BIG DATA: Data powered by IoT & other business systems
BETTER Insights: Transform your business with better insights.
Unstructured
Structured
Streaming
PB
TB
GB Advanced analyticsData scientist
Interactivity +
ExplorationBusiness analyst
Self-service
analysisBI professional
Decision supportDevice operator
Microsoft’s Big Data Solution Stack
Data Management and Enrichment
Insight
Familiar end user tool
Unstructured and structured data
Sensors Devices Bots Crawlers ERP CRM LOB APPs
Interactive Reports
With Power View
Excel With
PowerpivotPredictive Analytics
On MS Azure Cloud
HadoopHDInsight Machine
learning
Event
Hubs
Stream
AnalyticsData
Factory
Data Management And Enrichment
Data Management and Enrichment
Insight
Familiar end user tool
Unstructured and structured data
Sensors Devices Bots Crawlers ERP CRM LOB APPs
Interactive Reports
With Power View
Excel With
PowerpivotPredictive Analytics
On MS Azure Cloud
HadoopHDInsight Machine
learning
Event
Hubs
Stream
AnalyticsData
Factory
Hadoop And HDInsight Technology Stack
HDInsight Ecosystem
Metadata (Hcatalog)Graph
(Pegasus)
Scripting
(PIG)
Query
(Hive)
Machine
learning
(Mahout)
Distributed processing
(Man reduce)
Distributed storage (HDFS)
World’s data (Azure
data marketplace)
Windows Azure
storageAD, system center
Status
processing
(RHadoop
) Busin
ess in
tellig
ence
(Exce
l, ow
er
view
…)
Data
inte
gra
tion
OD
BC
\SQ
OO
P\R
EST
No
SQ
L D
ata
base
(Hb
ase
)
P
D
W
Pip
elin
e\w
ork
flo
w (O
ozie
)
Log
file
ag
gre
gatio
n
(Flu
me)
Top level
interfacesETL Tools BI Reporting RDBMS
Top level
abstractionsPIG HIVE Sqoop
Distributed
data
processingMap-Reduce
HBASEDatabase with
real time
access
At the base is a
self healing
clustered
storage system
Hadoop distributed file system
(HDFS)
Hadoop Ecosystem
HDInsight – Feature Set For Data Processing
Data Processing – Map Reduce Framework
Split (Combine) Partition
Read Map ReduceGroup Write
Data Processing – Map Reduce Framework
Костя Дима МишаАндрей Костя Юра
Сергей Андрей Миша
Костя Дима Миша
Андрей Костя Юра
Сергей Андрей Миша
Костя,1Дима,1Миша,1
Андрей,1Костя ,1Юра,1
Сергей,1Андрей ,1Миша,1
Костя,1Костя,1
Миша,1Миша,1
Андрей,1Андрей,1
Юра,1
Сергей,1
Дима,1
Костя,2
Дима,1
Миша,2
Андрей,2
Сергей,1
Юра,1
K1 ,V1 List(K2 ,V2) K2 ,List(V2)
Split (Combine) Partition
Read Map ReduceGroup Write
Data Preparation Using PIG Language
Data Storage Using HIVE Language
The prototypical MapReduce example counts the appearance of each word in a set of documents
function map(String name, String document):// name: document name// document: document contentsfor each word w in document:emit (w, 1)
function reduce(String word, Iterator partialCounts):// word: a word// partialCounts: a list of aggregated partial countssum = 0for each pc in partialCounts:sum += ParseInt(pc)
emit (word, sum)
en.wikipedia.org
PIG vs. HIVE
Sample of solving the same task by PIG &HIVE
PIG - Procedural
Users = load 'users' as (name, age, ipaddr);
Clicks = load 'clicks' as (user, url, value);
ValuableClicks = filter Clicks by value > 0;
UserClicks = join Users by name, ValuableClicks by user;
Geoinfo = load 'geoinfo' as (ipaddr, dma);
UserGeo = join UserClicks by ipaddr, Geoinfo by ipaddr;
ByDMA = group UserGeo by dma;
ValuableClicksPerDMA = foreach ByDMA generate group, COUNT(UserGeo);
store ValuableClicksPerDMA into 'ValuableClicksPerDMA';
HIVE-Declarative
insert into ValuableClicksPerDMA
select dma, count(*)
from geoinfo join (select name, ipaddr
from users join clicks on (users.name = clicks.user)
where value > 0;) using ipaddr
group by dma;
https://developer.yahoo.com/blogs/hadoop/comparing-pig-latin-sql-constructing-data-processing-pipelines-444.html
Demo
Event Hubs
Communication Patterns
TelemetryIngest
That‘s easy …
• Ingest rate
• Storage
• Security
• …
TelemetryIngest
6
machines
20
sensors / machine
X 120
sensors
/
productionline
=
Let‘s do the math …
Communication Patterns
TelemetryIngest
Communication Patterns
4
productionlines
/
plant
120
sensors /
productionline
X 480
sensors
/
plant
=
Let‘s do the math …
TelemetryIngest
Communication Patterns
480
sensors
/
plant
60
telemetryingests
/
minute
X 1,728,000
ingests
/
hour
=
Let‘s do the math …
TelemetryIngest
Communication Patterns
1,728,000
ingests
/
hour
50
e.g. customers
X 86,400,000
ingests
/
hour
=
Let‘s do the math …
On a 24/7 basis
Hyper Scale is needed
Services – Service Bus / Event HubOverview
Service Bus
Relay
Queue
Topic
Notification
Event Hub
Interactive Dashboard(s)Production Line(s)
Services – Service Bus / Event HubPartitions
Service
Bus
Interactive Dashboard(s)Production Line(s)
* 1 Mio Producers
* 1 MB/sec aggregate
per EventHub
Event Hub
Reader 1
Reader 2
Reader 3
….
Reader 1
Reader 2
Reader 3
….
Consumer
Group
Throughput Units
1 MB/s writes
2 MB/s reads
Stream AnalyticsReal-time stream processing in the cloud
Stream millions of events per second
Perform real-time analytics
Correlate across multiple streams of data
Reliable performance and predictable results
No hardware to deploy
Rapid development with familiar SQL-like language
Demo
BIG Data To Better Insights
BIG DATA: Data powered by IoT & other business systems
BETTER Insights: Transform your business with better insights.
Unstructured
Structured
Streaming
PB
TB
GB Advanced analyticsData scientist
Interactivity +
ExplorationBusiness analyst
Self-service
analysisBI professional
Decision supportDevice operator
Q&A
A Powerful New Way To Work With DataSelf-service business intelligence with familiar Excel and the power of the cloud
Discover And Access DataUsing power query to access data
From Internet From File From Database And More…
Easily Discover And Access Data
Analyzing Data With Excel
Easily discover and access public and
corporate data with Power Query
Model & analyze 100’s of millions of rows
lightning fast with Power Pivot
Explore and visualize data in new ways with
Power View and Power Map
Modules
▪ Accelerometer
▪ Ambient Light + Sound
▪ Audio
▪ Bluetooth Low Energy
▪ Camera
▪ Climate
▪ GPS
▪ GPRS
▪ Infrared
▪ MicroSD Card
▪ nRF24 Module
▪ Relay
▪ RFID
▪ Servo
What can you do with a Tessel?
▪ Ambient monitoring: monitor temperature, noise… Detect variations and take action / notify.– Is the light on at home? Turn on Hue lights automatically at dark.
▪ Accelerometer: game controllers, activity trackers…
▪ Camera: take pictures on event, motion detection…
▪ Infrared: control your TV– Clap your hands to turn it on
▪ Lots of projects ideas: https://projects.tessel.io/projects
Node.JS for the Tessel
▪ Node.JS is usually used on the server-side; here we are going to use it on the client side!
▪ Node.JS is well suited to real-time processing of events, thanks to its asynchronous nature; this is well adapted to a device whose main job is to monitor and process events (temperature / noise / light / etc.)
▪ Instead of listening to server-side events (GET, POST, etc.) you will be listening to module-specific events
▪ Events are handled using callbacks, functions that you pass when registering for the event
Hello World: tessel run blinky.js
// Import the interface to Tessel hardwarevar tessel = require('tessel');
// Set the led pins as outputs with initial states// Truthy initial state sets the pin high// Falsy sets it low.var led1 = tessel.led[0].output(1);var led2 = tessel.led[1].output(0);
setInterval(function () {console.log("I'm blinking! (Press CTRL + C to stop)");// Toggle the led statesled1.toggle();led2.toggle();
}, 100);
More getting started: Wi-Fi
▪ Connect to local WiFi – ExpoGeorgia– User:pav#3
– Pass:201567890
▪ OR
▪ Revert to using phone hotspot
▪ tessel wifi -n "iPhone 6" -p "Pass1234“
What can you do with Azure?
▪ In theory, anything you can do in Node.JS– In practice, some complex modules or projects will cause translation problems
because not all Node constructs are fully supported– Most notably, the Azure SDKs for Node.JS seem to be causing some problems– It might be easier to revert to plain old REST APIs when possible
▪ Upload stuff to Azure: Blob Storage
▪ Send monitoring/telemetry to Azure: Service Bus, Event Hubs– Experiment with different protocols: HTTPS, AMQP, MQTT…
▪ Interact with mobile devices through Mobile Services– Send notifications
▪ Samples on http://gist.github.com/tomconte and http://hypernephelist.com
Let’s hack!
▪ Grab your hardware
▪ Pair up– Might be best to have one person who knows JS/Node per pair
▪ Get something done in 4 hours– Install Node, Tessel module, plug in board, upgrade firmware– Use Notepad++ / Sublime Text / Visual Studio or whatever– Do the Hello World thing– Get connected to Wi-Fi– Plug in a module, test it– Do the lab https://github.com/Dx-ted-emea/iot-labs– For advanced
▪ HDInsight▪ Use in ML▪ Connect to mobile device
– Present your results/learnings/findings in the last 30 minutes
©2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or
trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this
presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee
the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN
THIS PRESENTATION.