The Critical Role of IoT Data Integration to develop Big Data Applications (for The Unmanned Systems...
-
Upload
rainer-sternfeld -
Category
Data & Analytics
-
view
2.686 -
download
0
Transcript of The Critical Role of IoT Data Integration to develop Big Data Applications (for The Unmanned Systems...
Rainer Sternfeld, CE September 2014
The Critical Role of IoT Data Integration to develop Big Data Applications
February 5, 2015
The Unmanned Systems Expo 2015, The Hague Rainer Sternfeld, CEO, Planet OS
@planet_osImage Credit: Northrop Grumman
2 February 2015
Sensor data will outgrow
social data in 2 yearsSensor data is
too big to moveData Management
will not scale as we know it
The bigger picture Swimming in sensors, drowning in data
3 February 2015
Hewlett Packard: “By 2020, 40% of all data
ever collected by human kind will be generated by sensors.”
5 February 2015
Case #1: NOAA Real-time weather and climate data + forecasts
They are working hard on a challenge presented by:
• Tens of thousands of devices deployed in the ocean, on land, and space
• Tens of terabytes coming in every day
• 700 scattered “web services” (FTPs, flat files, Threads servers, no APIs)
• Breaking connections (all.the.time.)
• 25,000 employees
• Critical data for the government, the industry, and citizens
What if all this data would be easily discoverable and machine-readable?
6 February 2015
Example to case #1: Marinexplore.org 40,000+ data streams from 33 organizations of open data
Advanced Data Discovery Raster data / heat map overlays Access with third party applications
Raster data / quiver plots Graph Monitor Build custom datasets
7 February 2015
Consolidate dataflows, organize and make sense of your data
Access your data with 3rd party tools and systems
One interface to search and discover your local and remote data
Securely exchange, acquire or sell datasets
Build advanced domain specific solutions without hassle
Industrial IoT Platform for Real-World Sensor Networks designed for ocean, land, air and space data
8 February 2015
Integrate, curate, control access and monitor dataflows
Analytics dashboard Integrate and configure new data channels
Roles, authorizations and logs Dataflow dashboard
9 February 2015
Case #2: Agriculture + data science Models, predictions, outliers, multi-source data
New companies are emerging:
• Predict a farmer’s profit statement based using adaptive models
(drone data + environmental data + in-situ data + soil/seed/pesticide choices)
• Open data is sparse and scattered
• Data dissemination is hard
• Farmer’s data input in hard
• “Do we really need to build a data platform to build our business?”
What if newco’s and farmers could focus on what they know best?
10 February 2015
Case #3: Surveillance and tactical in-flight data Having a 2-way data aggregation is critical
• End-users using a variety of drones need data versatility
• “How can I get a data-edge in a chase scenario?”
• Situational awareness based on the same surrounding data
for all unmanned vehicles in your fleet
• Drones need more accurate surrounding data than manned
crafts (don’t fly with your windows taped)
11 February 2015
Case #4: Planet OS data exchange in a swarm scenario Secure data exchange for tactical navigation & interoperability
• Connected cars / drones / UGVs / AUVs / fleets
• Multi-particle computation for ensuring privacy
• Collision avoidance using encrypted data
• Behavioral data collection using encrypted data
12 February 2015
Main Use Cases
Visualisation
Analytics
Event Processing
Models
Curation & Logistics
Marketplace
Search
ExplorationTECHNICAL DATA
• Spatio-Temporal • Sensors • Observations • Models • Predictions • Metadata
13 February 2015
Interfaces and IntegrationsDATA FORMATS
• CSV, Spreadsheet • GeoTiff, ESRI Raster • HDF, NetCDF (CF), GRIB • Shapefile, KML • OGC Geopackage • Metadata: Exif, SRID, ISO 19139 • Images, Audio, Video • Custom formats
DATA ACCESS / EXPORT
• CSV, Spreadsheet • GeoTIFF, HDF, NetCDF (CF) • OGC WFS / WCS / WMS • OpenDAP • ESRI ArcGIS • ISO-19139 • Custom formats and protocols
DATA SOURCES
• HTTP, FTP • NFS, Samba, Sharepoint • OpenDAP, THREDDS, ERDAP • OGC WFS / WCS / WMS • ESRI ArcGIS, PostGIS • Archives: ZIP, etc • Custom protocols