Towards a Methodology for Benchmarking Edge Processing Frameworks

Pedro Silva, Alexandru Costan, Gabriel AntoniuInria, IRISA

France

Invited Talk, BenchCouncil’19, Denver, November 2019

Data Shifts to the EdgeBy 2022 Gartner predicts that 75% of enterprise-generated data will be created and processed outside of the data center and cloud infrastructures compared with 10% today.

Source: Smarter with Gartner, What Edge Computing Means for Infrastructure and Operations, October 3, 2018Extract from: BullSequana Edge positioning paper (Atos) 2

3

4

Why Edge Processing?

Advantagesq Easier access to dataq Bandwidth savingq Privacyq High potential parallelism

EDGE

DATA

CLOUD / DC

DATA

FOG

5

Edge Processing Tools

qCustom softwareqGeneric frameworks

qApache EdgentqAmazon GreengrassqAzure Stream AnalyticsqIBM Watson IoTqIntel IoTqOracle Edge Analyticsq…

EDGE

DATA

CLOUD / DC

DATA

FOG

6

Edge Processing Tools Are Great! JEDGE

DATA

CLOUD / DC

DATA

FOG

7

How Great?EDGE

DATA

CLOUD / DC

DATA

FOGWhat is their performance?

Under which conditions?

Do they integrate well with my app?

8

We Need Benchmarking!

Goal: Understand performance

EDGE

DATA

CLOUD / DC

DATA

FOG

9

Benchmarking: Questionsq Are the cost models precise?

q What is the impact of networking on the performance?

q How do my algorithms react to real-time scenarios?

q How does my hybrid approach compare to a fullycentralized solution? FOG

EDGE

CLOUD

q SILVA, P., COSTAN A. and ANTONIU, G., Towards a Methodology for BenchmarkingEdge Processing Frameworks. 1st Workshop on Parallel AI and Systems for the Edge (PAISE workshop collocated with IPDPS 2019).

10

Benchmarking Platform: Objectives

q Benchmark complete scenarios

q Control network characteristics

q Control framework configuration parameters

q Control Edge, Fog and Cloud infrastructures

11

Benchmarking Edge Processing: Related Work

q TPCx-IoTq Created for hardware benchmarkingq Fog oriented

q Academic benchmarksq Difficult to reproduceq Lack of a clear methodology (metrics, workloads,

parameters)q Not focused on the tools

12

Benchmarking Edge Processing Tools

met

rics

q Edge/Fog data processing toolsq Processing performanceq Supported programming languagesq Connectivityq Development easiness

q Use casesq Overall application performanceq Viability on different infrastructure configurations

workload

datatransmission

processing

…13

Benchmarking Edge Processing Tools: Zoom

Edge Fog

Cloud

…

…

…

14

Benchmarking Edge Processing Tools : Parameters

Edge Fog

Cloud

…

…

…

Workloads:CCTV NYC TaxiEEW

Network:BandwidthLossLatency

Network:BandwidthLossLatency

Edge:Processingtools

Fog:MQTT server+ processingtools

Cloud:Kafka + Flink

15

Benchmarking Edge Processing Tools: Metrics

Edge Fog

Cloud

…

…

…

Throughput

Latency Edge to Fog Latency Fog to Cloud Processing Latency

Throughput

Each component has a resource utilization log.

16

Benchmarking Platform: Implementationq Experiment managerq Configures the infrastructureq Deploys frameworks/toolsq Deploys applications and manages their

executionsq Monitors resource usageq Gathers metrics and logs

q Edge+Fog+Cloud processing managementq Wrappers/interfaces

q Metric generation, configuration, connection

Experim

ent Manager

Infrastructure

VMs / Containers Bare Metal

Edge Fog Cloud

Python /Execo / EnosLib

Grid5K

enoslib

appstack

17

Earthquake Early Warning Systems (EEW)

Warning broadcaster

Seismometer

Data center

Data upload

P-wave

18

Earthquake Early Warning Systems (EEW)

ScientificInstruments

Intermediate machines with computing capabilities

…

Centralized data center Broadcasting users

… …

Data

Warning

q Deem: hierarchical and distributed ML algorithm

q Enables the usage of multiple types of sensors

q Enables the deployment on less powerful networks

q Enables local decisionmaking.

Deem: local decision

Deem: global decision

q FAUVEL, K. ; BALOUEK-THOMERT, D. ; MELGAR, D. ; SILVA, P., SIMONET, A. ; ANTONIU G. ; COSTAN, A ; MASSON, V ; PARASHAR, M. ; RODERO, I. ; TERMIER, A. A Distributed Multi-Sensor Machine Learning Approach to Earthquake Early Warning. Just accepted at AAAI 2020.

q SILVA, P., BALOUEK-THOMERT, D.; FAUVEL, K. ; MELGAR, D. ; SIMONET, A. ; ANTONIU G. ; COSTAN, A ; MASSON, V ; PARASHAR, M. ; RODERO, I. ; TERMIER, A A hybrid Fog and Cloud computing based approach for Earthquake Early Warning Systems. (In preparation.)

19

EEW: Fog-Based Infrastructure

q Thousands of producers

q High load on Fog and Cloud

q Objectivesq Reduction of network costsq Reduction of Cloud costsq Easier network reconfiguration (intelligent fog nodes)

20

Next Steps

q Improve the benchmark prototypeq Experiment with the EEW scenarioq Integrate extra scenarios and use cases (e.g., DL-based)

Thank you!