Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge...
Transcript of Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge...
![Page 1: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/1.jpg)
Towards a Methodology for Benchmarking
Edge Processing Frameworks
Pedro Silva, Alexandru Costan, Gabriel Antoniu
Inria Kerdata, IRISA
![Page 2: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/2.jpg)
![Page 3: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/3.jpg)
![Page 4: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/4.jpg)
Edge processing / computing
Edge computing advantages:
- easier access to data
- bandwidth saving
- “privacy”
- potential high parallelism
EDGE
DATA
CLOUD / DC
DATA
FOG
![Page 5: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/5.jpg)
Edge processing tools
• Custom software
• Apache Edgent
• Amazon Greengrass
• Azure Stream Analytics
• IBM Watson IoT
• Intel IoT
• Oracle Edge Analytics
• …
EDGE
DATA
CLOUD / DC
DATA
FOG
![Page 6: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/6.jpg)
Edge processing tools
EDGE
DATA
CLOUD / DC
DATA
FOG
![Page 7: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/7.jpg)
Edge processing tools
EDGE
DATA
CLOUD / DC
DATA
FOGWhat’s their performances?
Under which conditions?
Do they integrate well with my app?
![Page 8: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/8.jpg)
Benchmarking Edge tools
• Understanding a tool's performance
through benchmarking
EDGE
DATA
CLOUD / DC
DATA
FOG
![Page 9: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/9.jpg)
Related work
• TPCx-IoT:
• Created for hardware benchmarking
• Fog oriented
• Academic benchmarks:
• Irreproducible
• Just a few commercial tools
• Lack a clear methodology (metrics, workloads,
parameters)
• Not focused on the tools
![Page 10: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/10.jpg)
Benchmarking Edge tools
FOGEDGE
DATA DATA
INGESTION
INGESTION
![Page 11: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/11.jpg)
General view
Workload
Ingestion
system
Deployed
Tools
Data- Latency
- Throughput
- Resource usage
![Page 12: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/12.jpg)
Benchmark objectives
• Processing performance
• Supported programming
languages
• Connectivity
• Development easiness
![Page 13: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/13.jpg)
Benchmark parameters
• Edge processing frameworks
• Edge infrastructure
• Scenarios / Workload
• Input data throughput
![Page 14: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/14.jpg)
Edge processing frameworks
• Apache Edgent
• Amazon Greengrass
• Azure Stream Analytics
• IBM Watson IoT
• Intel IoT
• Oracle Edge Analytics
• Baselines (C++, Java)
![Page 15: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/15.jpg)
Infrastructure
• Virtual machines and bare metal
• nano (1 core, 256MB)
• mini (1 core, 1GB)
• Raspberry PI2 (4 cores, 1GB)
• medium (4 cores, 4GB)
• large (8 cores, 8GB)
• Dell PowerEdge R630 (16 cores, 128GB)
![Page 16: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/16.jpg)
Scenarios / Workload
• New York City Taxi and
Limousine Commission
• Busiest driver in the last hour
minutes every 5 minutes
• CCTV footage from Univ. of
California San Diego
• Busiest places in the last hour
every 5 minutes
![Page 17: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/17.jpg)
Evaluation metrics
• Message processing throughput
• Processing latency
• Number of supported
programming languages
• Framework connections
• Lines of code
![Page 18: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/18.jpg)
Inflection: earthquake early warning
❑ Objective: process P-waves (time series) in order to characterize earthquakes before they start.
❑ DEEM: real time distributed hierarchical ML algorithm for earthquake magnitude measurement.
Image from http://ds.iris.edu
❑ Kevin Fauvel, Daniel Balouek-Thomert, Diego Melgar, Pedro Silva, Anthony Simonet, Gabriel Antoniu, Alexandru Costan, Manish Parashar, and Ivan Rodero. Towards a decentralized multi-sensor machine learning approach for Earthquake Early Warning. Soumission à ECML PKDD 2019
![Page 19: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/19.jpg)
Scientific Instruments
Intermediate machines with computing capabilities
…
Centralized data center Broadcasting users
… …
Data
Warning
❑ Deem: distributed
hierarchical ML algorithm
❑ Allows for heterogeneous
sensors
❑ Can be used on low quality
networks
❑ Allow for local decision
making
Deem: local decision
Deem: final decision
Inflection: earthquake early warning
![Page 20: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/20.jpg)
New requirements
• Benchmark a complete scenario
• Control network characteristics
• Control frameworks' configuration parameters
• Control Edge, Fog and Cloud infrastructures
![Page 21: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/21.jpg)
Updated workflow
…
…
…
Edge Fog Cloud
![Page 22: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/22.jpg)
Updated workflow
…
Workloads:
CCTV
Taxi
EEW
![Page 23: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/23.jpg)
…
Updated workflow
Edge:
Processing tools
![Page 24: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/24.jpg)
Updated workflow
…
Network connection:
Bandwidth
Loss
Latency
![Page 25: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/25.jpg)
…
Updated workflow
Fog:
Lightweight MQTT
server
+
processing tools
![Page 26: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/26.jpg)
…
Updated workflow
Network connection:
Bandwidth
Loss
Latency
![Page 27: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/27.jpg)
Updated workflow
…
Stream processing:
Kafka brokers
Zookeeper server
Flink Cluster
- There is a selection of Kafka, Zookeeper
and Flink parameters that can be set
![Page 28: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/28.jpg)
Updated workflow
…
- Latency
- Throughput
- Resource usage
![Page 29: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/29.jpg)
Glimpse on the implementation
• Experiment manager:
• Configures the infrastructure
• Deploys frameworks/tools
• Deploys applications and manages their executions
• Monitors resource usage
• Gathers metrics and logs
• Edge+Fog+Cloud processing
management:
• Wrappers / interfaces (metric generation, configuration, connection)
Exp
erim
en
t Man
ag
er
Infrastructure
VMs Bare Metal
Edge Fog Cloud
Python /
Execo
Grid5K
enoslib
app
stack
![Page 30: Towards a Methodology for Benchmarking Edge Processing … · 2019-05-31 · Benchmarking Edge tools EDGE FOG DATA DATA INGES TION INGES TION. General view Workload Ingestion system](https://reader036.fdocuments.in/reader036/viewer/2022081612/5f76a2402fadde22ca10ccd1/html5/thumbnails/30.jpg)
Future work
• Finish the benchmark prototype
• Finish paper with EEW use case
• Integrate a DL based use case