Internet of Things: Connected Devices Enabling Energy Management
Enabling the Internet of Things with Real-time Hadoop
30
Real-time Hadoop & internet of things Presented by Pepperdata & MapR
-
Upload
becky-mendenhall -
Category
Technology
-
view
248 -
download
0
Transcript of Enabling the Internet of Things with Real-time Hadoop
Real-time Hadoop & internet of things Presented by Pepperdata & MapR
Real-time Hadoop & Internet of Things
Presented by Pepperdata and MapR
©2015 Pepperdata
Agenda
• How IoT is driving Hadoop adoption
• Key requirements for taking advantage of IoT
• Real-life use case examples
• How Pepperdata enables real-time Hadoop optimization
®© 2015 MapR Technologies 3
®
© 2014 MapR Technologies
Bill Peterson Director Product Marketing October 2015
®© 2015 MapR Technologies 4
What is the Internet of Things
Wearables
Connected Homes
Industrial Internet
Connected Cities
Connected Cars
®© 2015 MapR Technologies 5
Personal loT wearables smart phones clothes
Industrial loT smart factories agriculture retail manufacture
Group loT vehicles smart houses tourism education
Community loT smart cities smart roads parks
®© 2015 MapR Technologies 6
Internet of Things is Here Now and Growing
The New Essential Infrastructure
®© 2015 MapR Technologies 7
New Products
New Services
New Processes
REVENUE
PRODUCTIVITY & COST SAVINGS
Business Impact of Internet of Things
®© 2015 MapR Technologies 8
%
®© 2015 MapR Technologies 9 © 2015 MapR Technologies ®
IoT Use Cases
®© 2015 MapR Technologies 10
Increased Agility and Yield Motorcycle Assembly Line
New manufacturing facility with sensors continuously measuring and analyzing
• Time reduction from 21 days to 6 hours • Increased 25% in production • 30% staff reduction • Ability to ship a customized bike every 90 seconds
®© 2015 MapR Technologies 11
Operational Efficiency Waste Management
• 20K trucks, 30M GPS pings a day
• Integrate 17 different systems that have operational significance related to driver, vehicle, and route performance management
• Over 1.6M labor hours saved
®© 2015 MapR Technologies 12
IoE Creates a Smart City Barcelona
• Reduce traffic by in-ground sensors communicate with in-car devices to quickly direct cars to open parking
• Reduced costs by improving efficiency of waste collection, street lighting, other services
®© 2015 MapR Technologies 13 © 2015 MapR Technologies ®
IoT Ecosystem
®© 2015 MapR Technologies 14
At the center of IoT, with industry leading technology to support real-time, mission critical uses.
14
®© 2015 MapR Technologies 15
IoT Ecosystem
Mobile
Telecom Networks
Applications Sensors, devices
Networks
Data Processing & Analytics
®© 2015 MapR Technologies 15
®© 2015 MapR Technologies 16
IoT Ecosystem
Mobile
Telecom Networks
Applications Sensors, devices
Off the system Internal Processing
Networks
Data Processing & Analytics
®© 2015 MapR Technologies 16
®© 2015 MapR Technologies 17
IoT Ecosystem
Mobile
Telecom Networks
Applications Sensors, devices
Off the system Internal Processing
Data Processing & Analytics
Networks
®© 2015 MapR Technologies 17
®© 2015 MapR Technologies 18
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 0
1 0 0 1 1
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0
1 0 0 1
1 0 1 0 0 0
1 0 1 0 0 0
0 0 0 1 0 0 0
1 0 1 0 0 0
1 0 0 1
1 0 1 0 0 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 0 0 0
1 0 1 0 0 0
0 0 0 1 0 1 0
1 0 1 0 0 00
0 0 1 00
1 0 1 0 0 0
1 0 1 0 0 0
1 0 0 1 1 0
1 0 1 0 0 0
1 0 1 0 0 0
1 0 0 1 1 0
1 0 1 0 0 1 0
0 0 0 1 0 0 0
1 0 1 0 0 0
0 0 1 0
1 0 1 0 0 0
0 0 1 0
1 0 1 0 0 0
1 0 1 0 0 1
0 0 1 0
1 0 1 0 0 0
1 0 1 0 0 0 1 0
0 0 1 0
1 0 1 0 0 0 1 0
1 0 1 0 0 0
1 0 1 0 0 0 1 0
1 1 0 1 0 1
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 1 0 1 0 1
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 1 0 1 0 1
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
1 1 0 1 0 1
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0
0 1 0 1 0 1
1 0 1 0 0 0 1 0
0 0 0 1 0 0 0 1 0
Devices
Chip/Device
Edge
Cloud 1 1 0 1 0 1 0 1 1
0 0 0 1 0 0 0 1 0
1 0 0 1 1 0
1 0 0 1 1 0
1 1 0 1 0 1 0 1 1
1 1 0 1 0 1 0 1 1
1 1 0 1 0 1 0 1 1
1 1 0 1 0 1 0 1 1
1 1 0 1 0 1 0 1 1
0 0 1 0
1 1 0 1 0 1 0 1 1
Applications Processing Animation
Reporting & Alerting
Data Analysis & Visualization
Other Apps
®© 2015 MapR Technologies 19 © 2015 MapR Technologies ®
Considerations
®© 2015 MapR Technologies 20
MapR Real-time Advantages for IoT • High performance with consistent low latency
MapR-DB HBase
®© 2015 MapR Technologies 21
Global Real-time Synchronization
Multi-master (aka, active/active) replication
Active Read/Write
End Users
• Faster data access – minimize network
latency on global data with local clusters
• Reduced risk of data loss – real-time,
bi-directional replication for synchronized
data across active clusters
• Application failover – upon any cluster
failure, applications continue via
redirection to another cluster
®© 2015 MapR Technologies 21
©2015 Pepperdata
Pepperdata enables real-time Hadoop optimization….
©2015 Pepperdata
IoT requires real-time data ingestion, processing, & analysis
• Unable to support multi-tenant, multi-workload clusters
• Clusters typically overbuilt & underutilized
• Lack of real-time visibility into root causes
Hadoop has limitations :
©2015 Pepperdata
Customer:• Data analytics company serving multiple industries
– communications, Telco, and Industrial IoT• Ingesting over 100B records per day from sensors
& devices• 10K lookups per day on 800 node cluster
Challenges:• Real-time service critical internally and to
customers• One cluster for compute, lookup, and machine
learning across multiple groups (Engineering, Data Science, and non-tech)
• Customer SLAs critical – lookups need to be done in milliseconds
Case study: Customer faces challenges with IoT
800 nodes
MapReduce Machine learning
Compute layer
Lookup system
Records In
100B records / day
Customer users
Operational Alerts
SLA = seconds from record in
©2015 Pepperdata
Time and sweat won’t solve the problem
No human can make the thousands of decisions a second necessary for dynamic, real-time hardware resource management.
©2015 Pepperdata
Pepperdata fills the gaps
Node-level metrics
YARN
Node-level metrics
Pepperdata
Real-time metrics by queue, user, job, task
Allocate resources dynamically (maximize utilization)
Control hardware usage (priority jobs complete on time)
Schedule jobs; pre-allocate memory, CPU
Prevent rogue jobs from harming high-priority jobs
When jobs are scheduled
Once jobs are running
During & after job runtime
©2015 Pepperdata
Results:
• Pepperdata software installed in under 1 hour
• Stable, multi-tenant production cluster
• HBase, MapReduce, and Spark jobs run reliably
• SLA enforcement for the first time
• Increased cluster throughput
Case study: Pepperdata enables real-time Hadoop, customer improves IoT processing
800 nodes
MapReduce Machine learning
Compute layer
Lookup system
Records In
100B records / day
Customer users
Operational Alerts
SLA = seconds from record in
©2015 Pepperdata
• Pepperdata’s real-time, dynamic SLA enforcement means mission critical jobs complete on time
• Run multi-workload environments in a multi-
tenant system • Get hours of your life back – less time
troubleshooting bottlenecks and issues
Never Miss Another Deadline
©2015 Pepperdata
• Identify bottlenecks in real time or historically
• Quickly locate root causes of problems
• Visibility into 200+ metrics at job, user, and task level
Easy to troubleshoot cluster issues
©2015 Pepperdata
“ Thank you.
