HOBBITA Brief Overview
Michael Röder, Axel-Cyrille Ngonga Ngomo
DICE research groupInstitute of applied Informatics, Leipzig, Germany
University Paderborn, Germany(Horizon 2020, GA No 688227)
BDV PPP SummitRiga, June 27th, 2019
Michael Röder (InfAI) Project Overview 17/01/2019 1 / 21
IntroductionA Lot of Data
11http://www.ibmbigdatahub.com/infographic/four-vs-big-data
Michael Röder (InfAI) Project Overview 17/01/2019 2 / 21
IntroductionA Lot of Tools
2
2https://cdn.datafloq.com/cms/os_big_data_open_source_tools-v2.pngMichael Röder (InfAI) Project Overview 17/01/2019 3 / 21
IntroductionA Lot of Tools
33https://hackernoon.com/great-power-great-responsibility-the-2018-big-data-ai-landscape-
6a35bcf34f7f?gi=3d49c063deb8
Michael Röder (InfAI) Project Overview 17/01/2019 4 / 21
IntroductionOne Question
Which tool(s) should I usefor my use case?
Michael Röder (InfAI) Project Overview 17/01/2019 5 / 21
IntroductionMany Questions
Where are the current bottlenecks?Which steps of the data lifecycle arecritical?Which solutions are available on themarket?Which key performance indicatorsare relevant?How well should tools perform?How do existing solutions performw.r.t. relevant indicators?
Michael Röder (InfAI) Project Overview 17/01/2019 6 / 21
IntroductionHOBBIT
Research project from 2015 – 2018(Horizon 2020, GA No 688227)Focus on Big Linked DataCover the business-critical steps ofthe Linked Data lifecycleUsed by a growing number ofcompaniesMature and maturing technologies
Michael Röder (InfAI) Project Overview 17/01/2019 7 / 21
IntroductionAims
1 Gathered real requirementsFocussed on industrial requirementsGathered relevant performance indicatorsGathered relevant performance thresholdsGathered real datasets
2 Developed benchmarks based on real data3 Provided universal benchmarking platform
Comparable resultsHosted as a free-to-use online instance
4 Periodic benchmarking challenges andreporting
5 Created an association (Special Group 7 ofTask Force 6)
Michael Röder (InfAI) Project Overview 17/01/2019 8 / 21
IntroductionAims
1 Gathered real requirementsFocussed on industrial requirementsGathered relevant performance indicatorsGathered relevant performance thresholdsGathered real datasets
2 Developed benchmarks based on real data
3 Provided universal benchmarking platformComparable resultsHosted as a free-to-use online instance
4 Periodic benchmarking challenges andreporting
5 Created an association (Special Group 7 ofTask Force 6)
Michael Röder (InfAI) Project Overview 17/01/2019 8 / 21
IntroductionAims
1 Gathered real requirementsFocussed on industrial requirementsGathered relevant performance indicatorsGathered relevant performance thresholdsGathered real datasets
2 Developed benchmarks based on real data3 Provided universal benchmarking platform
Comparable resultsHosted as a free-to-use online instance
4 Periodic benchmarking challenges andreporting
5 Created an association (Special Group 7 ofTask Force 6)
Michael Röder (InfAI) Project Overview 17/01/2019 8 / 21
IntroductionAims
1 Gathered real requirementsFocussed on industrial requirementsGathered relevant performance indicatorsGathered relevant performance thresholdsGathered real datasets
2 Developed benchmarks based on real data3 Provided universal benchmarking platform
Comparable resultsHosted as a free-to-use online instance
4 Periodic benchmarking challenges andreporting
5 Created an association (Special Group 7 ofTask Force 6)
Michael Röder (InfAI) Project Overview 17/01/2019 8 / 21
IntroductionAims
1 Gathered real requirementsFocussed on industrial requirementsGathered relevant performance indicatorsGathered relevant performance thresholdsGathered real datasets
2 Developed benchmarks based on real data3 Provided universal benchmarking platform
Comparable resultsHosted as a free-to-use online instance
4 Periodic benchmarking challenges andreporting
5 Created an association (Special Group 7 ofTask Force 6)
Michael Röder (InfAI) Project Overview 17/01/2019 8 / 21
Section 1
Project Highlights
Michael Röder (InfAI) Project Overview 17/01/2019 9 / 21
Project HighlightsHOBBIT platform
PlatformController
Data Generator
Task Generator
Data Generator
Data Generator
Task Generator
Task Generator
Front End
Benchmarked System
data flowcreates component
StorageAnalysis
BenchmarkController
Evaluation Module
Eval. Storage
User ManagementRepository
Scalable open-source benchmarking platformLocal, distributed and remote (AWS) deploymentFirst FAIR platform for benchmarking Big Linked Data in a holistic manner
Michael Röder (InfAI) Project Overview 17/01/2019 10 / 21
Project HighlightsProject Highlights
? 5 mimicking algorithms? 52 benchmarks? 200+ systems? 14 challenges
DEBS GC 2017 and 2018? 300+ users? 13K+ experiments
Michael Röder (InfAI) Project Overview 17/01/2019 11 / 21
Section 2
Benchmarking Machine Learning
Michael Röder (InfAI) Project Overview 17/01/2019 12 / 21
Benchmarking Machine LearningSML Benchmark v1 for DEBS GC 2017
The task: find anomalies in molding machine sensor data to predictmaintenance intervals (predictive maintenance).Mimicking algorithm based on real dataData was streamed as in the real worldParticipants had to use Markov Models to identify anomalies14 Participants, 7 made it into the last round
Michael Röder (InfAI) Project Overview 17/01/2019 13 / 21
Benchmarking Machine LearningSML Benchmark v1 for DEBS GC 2017
Michael Röder (InfAI) Project Overview 17/01/2019 14 / 21
Benchmarking Machine LearningSML Benchmark v2 for DEBS GC 2018
The task: predictions about ship routes based on AIS dataSpatio-temporal streaming dataPredictions for vessels’ destinations and arrival times
Michael Röder (InfAI) Project Overview 17/01/2019 15 / 21
Benchmarking Machine LearningSML Benchmark v2 for DEBS GC 2018
Michael Röder (InfAI) Project Overview 17/01/2019 16 / 21
Benchmarking Machine LearningSML Benchmark v2 for DEBS GC 2018
Michael Röder (InfAI) Project Overview 17/01/2019 17 / 21
Benchmarking Machine LearningSML Benchmark v2 for DEBS GC 2018
Michael Röder (InfAI) Project Overview 17/01/2019 18 / 21
Section 3
Future Directions
Michael Röder (InfAI) Project Overview 17/01/2019 19 / 21
Future Directions
KnowGraphs (Innovative Training Networks (ITN))4 years, starting in October 201915 Early-Stage Researchers (ESRs)HOBBIT will be used as central benchmarking platformFurther datasets will be integrated (e.g., UICML datasets)
RAKI (BMWi project)3 years, starting in September 2019HOBBIT will be used for evaluation
More projects pending
→ Further development of the HOBBIT platformHOBBIT is open for the community! Benchmarks, systems, datasets can beaddedNot limited to linked data
Michael Röder (InfAI) Project Overview 17/01/2019 20 / 21
Future Directions
KnowGraphs (Innovative Training Networks (ITN))4 years, starting in October 201915 Early-Stage Researchers (ESRs)HOBBIT will be used as central benchmarking platformFurther datasets will be integrated (e.g., UICML datasets)
RAKI (BMWi project)3 years, starting in September 2019HOBBIT will be used for evaluation
More projects pending→ Further development of the HOBBIT platform
HOBBIT is open for the community! Benchmarks, systems, datasets can beaddedNot limited to linked data
Michael Röder (InfAI) Project Overview 17/01/2019 20 / 21
Thank you
HOBBIT offersScalable benchmarkingBased on real world data in anExtendable,Open source platformFollowing the FAIR data principles
http://project-hobbit.eu/https://dice-research.org/about/
Michael Röder (InfAI) Project Overview 17/01/2019 21 / 21