Internet of Things

Dilip Kumar Kari Dilip@hashmapinc.com

Agenda

•  Problem Statement •  IOT in Oil fields •  Describing the use case

•  Real-time operations monitoring •  Seismic survey in deep oceans

•  The Edge •  Protocols

•  Shipping the data •  Data processing frameworks •  Scalable Messaging Systems

•  Processing the data •  Real-time Distributed Processing Frameworks •  Storage •  Visualization

•  Demo •  Questions

Create New Requirements

• Businesses need trusted, real-time insights from full-fidelity data • Applications need access to both data in

motion and data at rest •  IT needs to capture multi-directional IoT data

flows from point to point • Challenge: the IoT perimeter is jagged and

outside the data center

Collecting and Analyzing IOT Data

IOT + Predictive Analytics affects all industries • BI vs Data Science approach

•  BI (traditional and doing for decades) •  only structured data •  Descriptive analysis •  Standard Reporting •  What happened?

•  Data Science •  Predictive analysis •  prescriptive analysis (guidance of what to do on predicted

outcome) •  Why? what will? what should id do?

Data Science Process

•  Define Hypothesis to test or prediction to be made •  Gather data DL (SQL + Hadoop) •  Build Schema (schema on read/query) •  Visualize the data •  Build Analytical Model (SAS, MAtLab, MADLib,

Mahout, R…) •  Evaluate the model results (probabilities, confidence

levels) - what is the output to different input params as they change

Today’s Architectural Limitations

• Data movement software was built for standardized data and one-way flows •  Tools for unstructured data are customized,

disjointed and difficult to manage •  Such tools cannot securely collect, conduct and

curate IoT data flows, which are multi-directional from point to point at the perimeter •  Today’s architectures limit potential insight by

making it difficult and expensive to manage IoT data both in motion and at rest

The Use cases

• A large Oil & Gas services company is monitoring deep sea and onshore drilling jobs

across the world from Houston.

• A seismic survey company is generating raw data for oceanic surveys on behalf of BP,

Shells, Exxon’s of the world.

OILFIELD OPERATIONS

1000s of sensors per site

Temperature, Pressure, Flow,

Pumps, Actuators, Valves, etc.

REMOTE COMMS DEVICES

RTUs, PLCs, Instrumentation

Systems

DATA LINK

Satellite, Cellular, Radio

SCADA NETWORK

OPC Servers 15-20 for a large basin

DATA LINK

Satellite, Cellular, Radio

HISTORIAN/s

Time series

logs and history

DATA LINK

Corporate WAN

CORP SYSTEMS & HADOOP Systems of

Record & Analytics

Platform

ANALYTICS Real Time

Data +

All Other Datasets

Oilfield Data Collection

Data from the field, factory floors and plants

Technologies that enables IoT

• Bluetooth •  zwave is the protocol used in most smart

home applications • LTE - communication specification for mobile

networks • Wifi-Direct - peer to peer communication

without need to have an access point • Open Standard Protocols like OPC UA/DA

Shipping the Data

AG.Imagesource"©SiemensAG2015,Allrightsreserved"

GATHER

DELIVER

PRIORITIZE

Trackfromtheedge(wellsite,equipment,etc.)

Throughthedatacenter(andback)

RECALIBRATE

Apache Nifi

It’s not just how quickly you move data… it’s about how quickly you can change behavior and seize new -  Guarantee high fidelity data delivery

and eliminate data loss even with intermittent communications - make decisions from credible and reliable data

-  Provide secure data collection and movement from data generation to data rest with 21 encryption algorithms

-  Enable comprehensive provenance and lineage with a complete digital chain of custody for real time multidirectional data streams, flow specific quality of service, and visibility into valuation of your field data capture infrastructure

-  Allow for real time prioritization of data and interactive changes to the data flow with visual command and control

RPC & OPC Specifications

•  COM, DCOM, OLE •  Classic OPC (Open Platform Communication) •  OPC XML DA (Data Access) •  OPC HDA (Historical Data Access) •  OPC AE (Alarm & Events) •  OPC Xi (Express Interface) •  OPC UA

OPC UA

•  OPC UA differs significantly from its predecessor ü International Standard IEC 62541 ü Multi-platform SOA based implementation, ü including portable ANSI C, Java and .NET implementations ü Scalability: from smart sensors and smart actuators to

mainframes (enterprise systems) ü Multi-threaded, as well as single-threaded/single-task

operation—necessary for porting the stack to embedded devices

ü Security, based on new standards ü Configurable time-outs for each service ü Chunking of big datagrams ü Model complex data structures for collaboration with other

standard organizations

Distributed Intelligence based on standards

•  OPC UA standard for enterprise to connect DCS, ERP, MES, SCADA, PLC and Sensors

SCADA

•  SCADA (Supervisory Control And Data Acquisition) is a system for remote monitoring and control that operates with coded signals over communication channels (using typically one communication channel per remote station). It is a type of industrial control system (ICS) •  SCADA systems historically distinguish

themselves from other ICS systems by being large-scale processes that can include multiple sites, and large distances •  These processes include industrial, infrastructure,

and facility-based processes

Choosing an Architecture

Do Lambda OR Kappa?

Lambda Architecture

Kappa Architecture

IOT Architecture

Producer Consumer+Spout RegionServer

HDFS+Hive

OPCServer

Broker

Push PushorPull

StormBolt(s)Bolt(s)

SCADA IoT Demo

Questions