marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION!...
Transcript of marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION!...
![Page 1: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/1.jpg)
1
Université Paris-Saclay / CNRSBALÁZS KÉGL
RAMP DATA CHALLENGES WITH
MODULARIZATION AND CODE SUBMISSION LESSONS LEARNED
![Page 2: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/2.jpg)
• Directeur de recherche CNRS
• machine learning (20 years)interfacing with particle physics (10 years)
• Director of the Paris-Saclay Center for Data Science
• interfacing with biology, economy, climatology, chemistry, etc. (4 years)
• Data science consulting and training (4 years)
2
WHO AM I?Balázs Kégl
![Page 3: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/3.jpg)
• A short history of RAMPs
• motivations, design principles, and the current tool
• Three data challenges
• anomaly detection in the LHC ATLAS detector
• classifying and quantifying drug preparations for cancer therapy
• time series forecasting of El Niño
• How can you use it?
• in a classroom: to teach ML
• as a domain science researcher: to crowdsource your predictive problem
• as a data science researcher: to benchmark your new techniques
3
OUTLINE
![Page 4: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/4.jpg)
Center for Data ScienceParis-Saclay
UNIVERSITÉ PARIS-SACLAY
4
+ horizontal multi-disciplinary and multi-partner initiatives to create cohesion
![Page 5: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/5.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
Biology & bioinformaticsIBISC/UEvry LRI/UPSudHepatinovCESP/UPSud-UVSQ-Inserm IGM-I2BC/UPSud MIA/AgroMIAj-MIG/INRALMAS/Centrale
ChemistryEA4041/UPSud
Earth sciencesLATMOS/UVSQ GEOPS/UPSudIPSL/UVSQLSCE/UVSQLMD/Polytechnique
EconomyLM/ENSAE RITM/UPSudLFA/ENSAE
NeuroscienceUNICOG/InsermU1000/InsermNeuroSpin/CEA
Particle physics astrophysics & cosmologyLPP/Polytechnique DMPH/ONERACosmoStat/CEAIAS/UPSudAIM/CEALAL/UPSud
The Paris-Saclay Center for Data ScienceData Science for scientific Data
250 researchers in 35 laboratories
Machine learningLRI/UPSud LTCI/TelecomCMLA/Cachan LS/ENSAELIX/PolytechniqueMIA/AgroCMA/PolytechniqueLSS/SupélecCVN/Centrale LMAS/CentraleDTIM/ONERAIBISC/UEvry
VisualizationINRIALIMSI
Signal processingLTCI/TelecomCMA/PolytechniqueCVN/CentraleLSS/SupélecCMLA/CachanLIMSIDTIM/ONERA
StatisticsLMO/UPSud LS/ENSAELSS/SupélecCMA/PolytechniqueLMAS/CentraleMIA/AgroParisTech
Data sciencestatistics
machine learninginformation retrieval
signal processingdata visualization
databases
Domain sciencehuman society
life brain earth
universe
Tool buildingsoftware engineering
clouds/gridshigh-performance
computingoptimization
Data scientist
Applied scientist
Domain scientist
Data engineer
Software engineer
Center for Data ScienceParis-Saclay
datascience-paris-saclay.fr
@SaclayCDS
LIST/CEA
5
Center for Data ScienceParis-Saclay
A multi-disciplinary initiative, building interfaces, matching people, helping them launching projects
345 affiliated researchers, 50 laboratories
![Page 6: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/6.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 6
Data scientist
Data trainer
Applied scientist
Domain expertSoftware engineer
Data engineer
Tool building Data domains
Data sciencestatistics
machine learning information retrieval
signal processing data visualization
databases
• interdisciplinary projects • data challenges • ultrawalls and interactive visualization
• coding sprints • Open Software Initiative • code consolidator and engineering projects
software engineeringclouds/grids
high-performancecomputing
optimization
energy and physical sciences health and life sciences Earth and environment
economy and society brain
!• data science RAMPs and TSs • IT platform for linked data
THE DATA SCIENCE ECOSYSTEMhttps://medium.com/@balazskegl
![Page 7: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/7.jpg)
Center for Data ScienceParis-Saclay
DATA CHALLENGES
7
• The HiggsML challenge on Kaggle
• https://www.kaggle.com/c/higgs-boson
![Page 8: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/8.jpg)
Center for Data ScienceParis-Saclay
HUGE PUBLICITY
8
B. Kégl / AppStat@LAL Learning to discover
CLASSIFICATION FOR DISCOVERY
14
![Page 9: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/9.jpg)
Center for Data ScienceParis-Saclay
SIGNIFICANT IMPROVEMENT OVER THE BASELINE
9
B. Kégl / AppStat@LAL Learning to discover
CLASSIFICATION FOR DISCOVERY
15
![Page 10: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/10.jpg)
10
HUGE PUBLICITY
SIGNIFICANT IMPROVEMENT OVER THE BASELINE
yet partially missing the objectives
![Page 11: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/11.jpg)
• Organizers have no direct access to solutions
• Emphasize competition: participants cannot build on each other’s solutions
• No modularization: ideas go unnoticed unless packaged into a top submission
11
LIMITATIONS OF DATA CHALLENGES
![Page 12: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/12.jpg)
12
We decided to design something better:
Data challenge with code submission
![Page 13: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/13.jpg)
• Design a crowdsourcing and teaching tool that
• hides heavy computational details and provides a simple interface to data scientists to experiment with algorithms
• promotes collaboration and the rapid propagation of ideas
• modularizes complex pipelines so the different expertise can be applied without having to understand all the details of the full workflow
• allows the challenge organizer to walk away with a working prototype
13
GOAL
![Page 15: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/15.jpg)
• Roughly two formats
• single day hackatons with 20-50 participants, open leaderboard, 15 minute timeout
• 1-3 week classroom challenges up 150 students (but no limit really): closed phase followed by an open phase
• 800+ users, 5000+ predictive models
15
RAMP RAPID ANALYTICS AND MODEL PROTOTYPING
![Page 16: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/16.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
CURRENT RAMPS
16
![Page 17: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/17.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
DATA SCIENCE THEMES
17
![Page 18: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/18.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
DATA DOMAINS
18
![Page 19: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/19.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
RAMP DATA CHALLENGE WITH CODE SUBMISSION
19
frontend
DB
backend
users submissions score problems workflow starting kit crossval
data workflow
train+test+blend
![Page 20: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/20.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
RAMP
20
![Page 21: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/21.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
RAMP
21
![Page 22: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/22.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
RAMP
22
![Page 23: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/23.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
ANOMALY DETECTION IN THE LHC ATLAS DETECTOR
23
reconstruction+simulated anomalies
classifier
anomaly (isSkewed = 1)
correct(isSkewed = 0)
?
![Page 24: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/24.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
CLASSIFYING AND REGRESSING ON MOLECULAR SPECTRA
24
chemotherapy drug in elastic pocket
laser spectrometer
molecular spectra
feature extractor 1
feature extractor 2
regressor
concentration
classifier
drug type
![Page 25: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/25.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
FORECASTING EL NINO SIX MONTHS AHEAD
25
![Page 26: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/26.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
FORECASTING EL NINO SIX MONTHS AHEAD
26
… 300.14 299.83 298.76 299.87 299.82 300.15 300.10 299.50… …
time series feature extractor
x (a fixed length feature vector)regressor
![Page 27: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/27.jpg)
27
Analyzing the process
![Page 28: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/28.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS)
OPEN PHASE LETS PARTICIPANTS CATCH UP
28
![Page 29: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/29.jpg)
29
THE DYNAMICS OF COLLABORATION
starting kit
the crowdearly influencers
inventors
![Page 30: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/30.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 30
THE DYNAMICS OF COLLABORATION
![Page 31: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/31.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 31
THE DYNAMICS OF COLLABORATION
![Page 32: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/32.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 32
THE DYNAMICS OF COLLABORATION
![Page 33: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/33.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 33
the single day hackaton ceiling
what you achieved with a well tuned deep net
the diversity gap
the human blender gap
![Page 34: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/34.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 34
the single day hackaton floor
![Page 35: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/35.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 35
the single day hackaton floor
![Page 36: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/36.jpg)
• Open phase helps novice participants to catch up: the goal of teaching!
• Sometimes also makes the best and blended score better
• Human blending often beats machine blending
• Human feature engineering easily beats deep learning on some data
• Course RAMPs beat single day hackatons significantly
• larger number of students?
• longer RAMPs?
• novice and master-level students are better than data science researchers?
• stronger incentives?
• closed phase preceding an open phase (vs pure open RAMP) helps to create diversity?
36
WHAT WE LEARNED
![Page 37: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/37.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 37
closed phase followed by open
the second diversity gap
![Page 38: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/38.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 38
![Page 39: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/39.jpg)
Center for Data ScienceParis-SaclayB. Kégl (CNRS) 39
![Page 40: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/40.jpg)
• More RAMPs
• galaxy morphology, detecting autism from brain fMRI, detecting Mars craters, forecasting space weather (solar storm early warning)
• More courses
• >1000 students next year
• Build your own RAMPs
40
WHAT’S NEXT
![Page 41: marseille ramp 1706 - madics.fr · RAMP! DATA CHALLENGES WITH! MODULARIZATION AND CODE SUBMISSION! ... • code consolidator and engineering projects software engineering energy and](https://reader036.fdocuments.in/reader036/viewer/2022071106/5fe0d99651dde349657bbc3f/html5/thumbnails/41.jpg)
• If you are a data science teacher
• we have been using classroom RAMPs in different formats: homework, final project, data camp
• students love it and work their butt off, they learn from each other and collaborate
• If you are a domain science researcher
• We can solve your predictive problems better than any single researcher in a classical project
• If you are a data science researcher
• You can benchmark your new techniques on a variety of problems
41
WHAT IS IN IT FOR YOU
sign up: www.ramp.studiocontact us: [email protected]