Internet of Things in Tbilisi

Insights From Internet Of Things & Big Data

Insights From Internet Of Things & Big Data

Kostya Goldstein

Sr. Program Manager Microsoft Russia

Business insights through big data

Microsoft’s solution to big data

Intelligent systems service, HDInsightFeatures & capabilities

Demo – HDInsight

Office as Big Data visualization platformSelf service BI – Features & capabilities

Demo – Power BI

HackathonTessel

IoT Hands On Task

Agenda

World Today

2003 2010 2015 2020

50 млрд

GROWTH OF DEVICES

World tomorrow

1990

2020

“ ”

Internet of Things (IoT)

The network of physical

objects that contain

embedded technology to

communicate and interact

with their internal states or

the external environment.

Вопрос

COW IoT ?

Интернет вещей

Аудио /Видео

Журналы операций

Тексты/Изображения

Настроение

высказываний

Обновления витрин

данныхНовости электронного

правительства

Погода

Вики / БлогиПереходы по

ссылкамДатчики/ RFID / Устройства

Координаты GPS

WEB 2.0Мобильные

устро-ва

Реклама ВзаимодействиеЭлектроннная

коммерция

Цифровой

маркетинг

Поисковый

маркетинг

Протоколы веб-

серверов

Рекомендации

ERP / CRM

Конвейер

продаж

Кредитор

ы

Зарплата

Запасы

Контакты

Отслеживани

е торгов

терабайты

(1012)

гигабайты

(109)

экзабайты

(1018)

петабайты

(1015)

Скорость | разнообразие | изменчивость

Объ

ем

1980

190,000$

2010

0.07$

1990

9,000$2000

15$Стоимость хранения за гигабайт, долл

ERP / CRM WEB 2.0 Интернет вещей

What the big data is?

Two main IoT aspects

Start Justin

Customer

ServiceProvider

Microsoft

ConsistentPlatform

ONE

Cloud OS VisionMicrosoft’s vision of the unified platform

for modern business

Transform The Datacenter

Unlock Insights On Any Data

Empower People-centric IT

Enable Modern Business Apps

Create The Internet Of Your Things

Intelligent Systems Service

Microsoft Solution For Internet Of Things

Drive InsightsAnalytics ReadyCloud and infrastructure

Devices and

assets

10101010011000110101010111010011010101010100110111011110111001010100001101010101110100110101010111010011101010101011010011010101010101001101100010101111010011101010101011011110100111

10101010011000110101010111010011010101010100110111011110111001010100001101010101110100110101010111010011101010101011010011010101010101001101100010101111010011101010101011011110100111

Customer

portal Value

StreamInsights

Power BI

HDInsight

Windows Embedded

Connect new and existing

devices using open-source

agents or gateway

technologies

Azure, HDInsight

Store machine-generated

data with data from other

sources in the cloud

Office 365, Power BIView data, administer

devices, and configure

rules, alerts, and other

actions using out-of-box

or custom portals

Mine insights from your

data to find gaps and

opportunities to make

better decisions and realize

new business value

User

input

AlertsSensors Gateway

Agent

ADevices

IoT Services Architecture & Platform Components

ISS (Intelligent Systems Service)

Agent

Gateway

Event Hub & Azure Service

Bus

Event Processing&

Rules Engine

TablesBLOBS

SQL AzureHDFS

IF {condition}

THEN {action}

Azure Service Bus

Design & Engineering

Manufacturing & Supply Chain

Service & Maintenance

Customer Relationship

ISS (Intelligent Systems Service)

ID

Industrial

Equipment

Operational Data

Example of modern data storage

How To Generate Value From IoT Data

BIG DATA: Data powered by IoT & other business systems

BETTER Insights: Transform your business with better insights.

Unstructured

Structured

Streaming

PB

TB

GB Advanced analyticsData scientist

Interactivity +

ExplorationBusiness analyst

Self-service

analysisBI professional

Decision supportDevice operator

Big Data



Unstructured

Structured

Streaming

PB

TB


Interactivity +


Self-service



Microsoft’s Big Data Solution Stack

Data Management and Enrichment

Insight

Familiar end user tool

Unstructured and structured data

Sensors Devices Bots Crawlers ERP CRM LOB APPs

Interactive Reports

With Power View

Excel With

PowerpivotPredictive Analytics

On MS Azure Cloud

HadoopHDInsight Machine

learning

Event

Hubs

Stream

AnalyticsData

Factory

Data Management And Enrichment

Data Management and Enrichment

Insight

Familiar end user tool

Unstructured and structured data

Sensors Devices Bots Crawlers ERP CRM LOB APPs

Interactive Reports

With Power View

Excel With

PowerpivotPredictive Analytics

On MS Azure Cloud

HadoopHDInsight Machine

learning

Event

Hubs

Stream

AnalyticsData

Factory

Hadoop And HDInsight Technology Stack

HDInsight Ecosystem

Metadata (Hcatalog)Graph

(Pegasus)

Scripting

(PIG)

Query

(Hive)

Machine

learning

(Mahout)

Distributed processing

(Man reduce)

Distributed storage (HDFS)

World’s data (Azure

data marketplace)

Windows Azure

storageAD, system center

Status

processing

(RHadoop

) Busin

ess in

tellig

ence

(Exce

l, ow

er

view

…)

Data

inte

gra

tion

OD

BC

\SQ

OO

P\R

EST

No

SQ

L D

ata

base

(Hb

ase

)

P

D

W

Pip

elin

e\w

ork

flo

w (O

ozie

)

Log

file

ag

gre

gatio

n

(Flu

me)

Top level

interfacesETL Tools BI Reporting RDBMS

Top level

abstractionsPIG HIVE Sqoop

Distributed

data

processingMap-Reduce

HBASEDatabase with

real time

access

At the base is a

self healing

clustered

storage system

Hadoop distributed file system

(HDFS)

Hadoop Ecosystem

HDInsight – Feature Set For Data Processing

Data Processing – Map Reduce Framework

Split (Combine) Partition

Read Map ReduceGroup Write

Data Processing – Map Reduce Framework

Костя Дима МишаАндрей Костя Юра

Сергей Андрей Миша

Костя Дима Миша

Андрей Костя Юра

Сергей Андрей Миша

Костя,1Дима,1Миша,1

Андрей,1Костя ,1Юра,1

Сергей,1Андрей ,1Миша,1

Костя,1Костя,1

Миша,1Миша,1

Андрей,1Андрей,1

Юра,1

Сергей,1

Дима,1

Костя,2

Дима,1

Миша,2

Андрей,2

Сергей,1

Юра,1

K1 ,V1 List(K2 ,V2) K2 ,List(V2)

Split (Combine) Partition

Read Map ReduceGroup Write

Data Preparation Using PIG Language

Data Storage Using HIVE Language

The prototypical MapReduce example counts the appearance of each word in a set of documents

function map(String name, String document):// name: document name// document: document contentsfor each word w in document:emit (w, 1)

function reduce(String word, Iterator partialCounts):// word: a word// partialCounts: a list of aggregated partial countssum = 0for each pc in partialCounts:sum += ParseInt(pc)

emit (word, sum)

en.wikipedia.org

PIG vs. HIVE

Sample of solving the same task by PIG &HIVE

PIG - Procedural

Users = load 'users' as (name, age, ipaddr);

Clicks = load 'clicks' as (user, url, value);

ValuableClicks = filter Clicks by value > 0;

UserClicks = join Users by name, ValuableClicks by user;

Geoinfo = load 'geoinfo' as (ipaddr, dma);

UserGeo = join UserClicks by ipaddr, Geoinfo by ipaddr;

ByDMA = group UserGeo by dma;

ValuableClicksPerDMA = foreach ByDMA generate group, COUNT(UserGeo);

store ValuableClicksPerDMA into 'ValuableClicksPerDMA';

HIVE-Declarative

insert into ValuableClicksPerDMA

select dma, count(*)

from geoinfo join (select name, ipaddr

from users join clicks on (users.name = clicks.user)

where value > 0;) using ipaddr

group by dma;

https://developer.yahoo.com/blogs/hadoop/comparing-pig-latin-sql-constructing-data-processing-pipelines-444.html

Event Hubs

Communication Patterns

TelemetryIngest

That‘s easy …

• Ingest rate

• Storage

• Security

• …

TelemetryIngest

6

machines

20

sensors / machine

X 120

sensors

/

productionline

=

Let‘s do the math …


TelemetryIngest


4

productionlines

/

plant

120

sensors /

productionline

X 480

sensors

/

plant

=


TelemetryIngest


480

sensors

/

plant

60

telemetryingests

/

minute

X 1,728,000

ingests

/

hour

=


TelemetryIngest


1,728,000

ingests

/

hour

50

e.g. customers

X 86,400,000

ingests

/

hour

=


On a 24/7 basis

Hyper Scale is needed

Services – Service Bus / Event HubOverview

Service Bus

Relay

Queue

Topic

Notification

Event Hub

Interactive Dashboard(s)Production Line(s)

Services – Service Bus / Event HubPartitions

Service

Bus

Interactive Dashboard(s)Production Line(s)

* 1 Mio Producers

* 1 MB/sec aggregate

per EventHub

Event Hub

Reader 1

Reader 2

Reader 3

….

Reader 1

Reader 2

Reader 3

….

Consumer

Group

Throughput Units

1 MB/s writes

2 MB/s reads

Stream AnalyticsReal-time stream processing in the cloud

Stream millions of events per second

Perform real-time analytics

Correlate across multiple streams of data

Reliable performance and predictable results

No hardware to deploy

Rapid development with familiar SQL-like language

BIG Data To Better Insights



Unstructured

Structured

Streaming

PB

TB


Interactivity +


Self-service



Q&A

A Powerful New Way To Work With DataSelf-service business intelligence with familiar Excel and the power of the cloud

Discover And Access DataUsing power query to access data

From Internet From File From Database And More…

Easily Discover And Access Data

Analyzing Data With Excel

Easily discover and access public and

corporate data with Power Query

Model & analyze 100’s of millions of rows

lightning fast with Power Pivot

Explore and visualize data in new ways with

Power View and Power Map

Modules

▪ Accelerometer

▪ Ambient Light + Sound

▪ Audio

▪ Bluetooth Low Energy

▪ Camera

▪ Climate

▪ GPS

▪ GPRS

▪ Infrared

▪ MicroSD Card

▪ nRF24 Module

▪ Relay

▪ RFID

▪ Servo

What can you do with a Tessel?

▪ Ambient monitoring: monitor temperature, noise… Detect variations and take action / notify.– Is the light on at home? Turn on Hue lights automatically at dark.

▪ Accelerometer: game controllers, activity trackers…

▪ Camera: take pictures on event, motion detection…

▪ Infrared: control your TV– Clap your hands to turn it on

▪ Lots of projects ideas: https://projects.tessel.io/projects

https://projects.tessel.io/projects

Node.JS for the Tessel

▪ Node.JS is usually used on the server-side; here we are going to use it on the client side!

▪ Node.JS is well suited to real-time processing of events, thanks to its asynchronous nature; this is well adapted to a device whose main job is to monitor and process events (temperature / noise / light / etc.)

▪ Instead of listening to server-side events (GET, POST, etc.) you will be listening to module-specific events

▪ Events are handled using callbacks, functions that you pass when registering for the event

Hello World: tessel run blinky.js

// Import the interface to Tessel hardwarevar tessel = require('tessel');

// Set the led pins as outputs with initial states// Truthy initial state sets the pin high// Falsy sets it low.var led1 = tessel.led[0].output(1);var led2 = tessel.led[1].output(0);

setInterval(function () {console.log("I'm blinking! (Press CTRL + C to stop)");// Toggle the led statesled1.toggle();led2.toggle();

}, 100);

More getting started: Wi-Fi

▪ Connect to local WiFi – ExpoGeorgia– User:pav#3

– Pass:201567890

▪ OR

▪ Revert to using phone hotspot

▪ tessel wifi -n "iPhone 6" -p "Pass1234“

What can you do with Azure?

▪ In theory, anything you can do in Node.JS– In practice, some complex modules or projects will cause translation problems

because not all Node constructs are fully supported– Most notably, the Azure SDKs for Node.JS seem to be causing some problems– It might be easier to revert to plain old REST APIs when possible

▪ Upload stuff to Azure: Blob Storage

▪ Send monitoring/telemetry to Azure: Service Bus, Event Hubs– Experiment with different protocols: HTTPS, AMQP, MQTT…

▪ Interact with mobile devices through Mobile Services– Send notifications

▪ Samples on http://gist.github.com/tomconte and http://hypernephelist.com

http://gist.github.com/tomconte

http://hypernephelist.com/

Let’s hack!

▪ Grab your hardware

▪ Pair up– Might be best to have one person who knows JS/Node per pair

▪ Get something done in 4 hours– Install Node, Tessel module, plug in board, upgrade firmware– Use Notepad++ / Sublime Text / Visual Studio or whatever– Do the Hello World thing– Get connected to Wi-Fi– Plug in a module, test it– Do the lab https://github.com/Dx-ted-emea/iot-labs– For advanced

▪ HDInsight▪ Use in ML▪ Connect to mobile device

– Present your results/learnings/findings in the last 30 minutes

https://github.com/Dx-ted-emea/iot-labs

©2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or

trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this

presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee

the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN

THIS PRESENTATION.

Internet of Things in Tbilisi

Technology

Transcript of Internet of Things in Tbilisi