Big Data - ttja.ee · Big Data: Volume Fits into memory of one large server (up to 1TB) Tools:...
30
Rainer Sternfeld, CE September 2014 Big Data May, 2015 André Karpištšenko Planet OS Advisor, Co-founder
Transcript of Big Data - ttja.ee · Big Data: Volume Fits into memory of one large server (up to 1TB) Tools:...
Rainer Sternfeld, CE September 2014
Big Data
May, 2015
André Karpištšenko Planet OS Advisor, Co-founder
2 May 2015
Planet OS Presence
Tallinn
Rio de Janeiro
Washington DC
HoustonLos Angeles
Sunnyvale HQ
MontrealTartu
3 May 2015
Sensor Data Discovery EngineOcean Data Management
From a small buoy to Big Data
Data BuoysMarket: $2 billion
Competitors: 100+ producers Scalability: poor to limited
2008 2012 2014
Market: $5 billion Competitors: 25+
Scalability: good but slow
Market: $100+ billion Competitors: 15+
Scalability: very scalable and fast
4 May 2015
Big Data
VARIETY VELOCITYVOLUME
5 May 2015
Big Data: Volume
Fits into memory of one large server (up to 1TB)
Tools: Python (NumPy, SciPy, SciKit), R, Matlab, etc
10s of Terabytes, Petabytes
10+ computing nodes
Tools: Hadoop, Spark, etc
Small Data Big Data
6 May 2015
Variety of Devices & Data Types (Oceanic)
OBSERVER SIGHTINGS
ACOUSTIC RECORDINGS
AERIAL MONITORING
WAVE GLIDERS
IMAGERYANALYSIS
BUOYS & FLOATS
SATELLITETAGGING
ACOUSTIC MODELS
ACOUSTIC DETECTIONS
VESSELAIS DATA
7 May 2015
Variety of Formats & Locations
COMPRESSEDTAR / ZIP
ONLINEREPOSITORIES
NETWORK STORAGE
GIS GDB / SHP
FILE SHARING PLATFORMS
SCIENTIFIC HDF / NC
DOCUMENTS DOC / PDF
OFFLINEARCHIVES
TABULAR XLS / CSV
LOCALHARD DRIVES
8 May 2015
VARIETY VERACITY
The 5 V’s of Big Data
VELOCITYVOLUME VALUE
9 May 2015
Current Market
Connecting Devices Higher velocities Larger volumes Wider varieties
Future MarketAutomated Industries Real-time Decisions Data & Insight Markets
VERACITY
VELOCITY
Trends in industrial machine data
10 May 2015
11 May 2015
12 May 2015
Data
Time
Trends in Sensor Data
“By 2020, 40% of all data ever collected by human kind will be generated by sensors.”
Hewlett Packard:
13 May 2015
14 May 2015
An exabyte a day Compressed to 10 petabytes
15 May 2015
Map of all devices on the Internet
August 2, 2014
16 May 2015
Robotic ocean-borne sensor platforms
increase productivity
LIQUID ROBOTICS WAVE GLIDER
17 May 2015
Example: Improved Ocean Operations. Avoiding ship collisions with the North Atlantic Right Whales
72% 98%
Improved rate of whale detection model
18 March 2015
R/V Jean Charcot
Interactive reporting
Bravante Helping to deliver offshore data reports 80% faster
19 May 2015
Prediction models are applied to local sensors and are domain-specific
20 May 2015
Unmanned vehicles are estimated to grow 10x in 10 years
Image Credit: Northrop Grumman
21 May 2015
Satellites are getting smaller and cheaper.
150 launched since 2011 (3x of the market estimate)
SPIRE, A SAN FRANCISCO STARTUP BUILDING NON-IMAGING LOW-ORBIT NANOSATELLITES USING RF SENSORS
22 May 2015
Intelligent sensors in our homes and cities
23 May 2015
Sources http://www.wired.com/images_blogs/beyond_the_beyond/2012/11/ge-industrial.jpg
Image Credit: General Electric
24 May 2015
History of Big Data Technologies
2003 Google File System 2004 Map Reduce 2005 Big Table
2005 Open Source Started 2011 Stable Release
25 May 2015
Relational database Columnar storage
Array database Graph storage Key-value store Object storage
In-memory storage Hierarchical data format
…
Choose the Right Tool for the Right Job
26 May 2015
Crowd Sourcing, Data Labeling
fog computing next to cloud computing
27 May 2015
28 May 2015
29 May 2015
Google Trends, Interest Over TimeBig Data
![JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]](https://static.fdocuments.in/doc/165x107/58887b4a1a28ab34788b644d/javaone2016-microservices-terabytes-in-microseconds-con4516-58b31e1ae0dea.jpg)