Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned...
Transcript of Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned...
![Page 1: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/1.jpg)
Zurich Universities of Applied Sciences and Arts
Lessons learned from 16 applied data science
(meta) case studies
on industrial applied data science, Lugano, Oct 18-19, 2018
Kurt Stockinger & Thilo Stadelmann
1
![Page 2: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/2.jpg)
Zurich Universities of Applied Sciences and Arts
Collecting lessons learned from half a decade of
data science
2
![Page 3: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/3.jpg)
Zurich Universities of Applied Sciences and Arts
Collecting lessons learned from half a decade of
data science
2
![Page 4: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/4.jpg)
Zurich Universities of Applied Sciences and Arts
Agenda
3
• The study
• Checklist: Eight commandments
• Inspiration: methodology, technology, innovation, education
![Page 5: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/5.jpg)
Zurich Universities of Applied Sciences and Arts
The study16 contributions, spanning much of data science
4
Taxonomy Discussed in chapters
Main focus 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23Fundamentals of Data science x x xMethodology or algorithm x x x x x x x x xTool x x xApplication x x x x x x x x xSurvey or tutorial x x x x
Stages in knowledge discovery process 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23Data recording x x x x x x xData wrangling x x x x x xData analysis x x x x x x x x x x x x xData visualization and/or interpretation x x x x x x xDecision making x x x x x x
Competence area 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23Technology x x x x xAnalytics x x x x x x x x x xData Management x x x x x x x xEntrepreneurship x x xCommunication x x
Taxonomy Discussed in chapters
Data modalities 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Numerical data x x x x x x x x x x x x
Text x x x x x
Images x x x
Audio x
Time series x x x x x
Transactional data x x x x
Open data x
Application domain 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Research x x x x x x x
Business x x x x x x
Biology x x
Health x x x x x x
eCommerce and retail x x x x
Finance x x
IT x
Industry and manufacturing x x
Services x x x x
![Page 6: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/6.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
![Page 7: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/7.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
1. DO: embrace interdisciplinarity, seek knowledge exchange
![Page 8: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/8.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
1. DO: embrace interdisciplinarity, seek knowledge exchange
2. DO: build trust by data usage transparency & security provisions
![Page 9: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/9.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
1. DO: embrace interdisciplinarity, seek knowledge exchange
2. DO: build trust by data usage transparency & security provisions
3. DO: cherish data wrangling, ideally automate it it’s the basis for analysis
![Page 10: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/10.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
1. DO: embrace interdisciplinarity, seek knowledge exchange
2. DO: build trust by data usage transparency & security provisions
3. DO: cherish data wrangling, ideally automate it it’s the basis for analysis
4. DO: leverage stream processing tools for real time big data analysis
![Page 11: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/11.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
1. DO: embrace interdisciplinarity, seek knowledge exchange
2. DO: build trust by data usage transparency & security provisions
3. DO: cherish data wrangling, ideally automate it it’s the basis for analysis
4. DO: leverage stream processing tools for real time big data analysis
5. DO: start machine learning from simple baselines
![Page 12: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/12.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
1. DO: embrace interdisciplinarity, seek knowledge exchange
2. DO: build trust by data usage transparency & security provisions
3. DO: cherish data wrangling, ideally automate it it’s the basis for analysis
4. DO: leverage stream processing tools for real time big data analysis
5. DO: start machine learning from simple baselines
6. DO: use visualization to gain insight (from debugging to result presentation)
![Page 13: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/13.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
1. DO: embrace interdisciplinarity, seek knowledge exchange
2. DO: build trust by data usage transparency & security provisions
3. DO: cherish data wrangling, ideally automate it it’s the basis for analysis
4. DO: leverage stream processing tools for real time big data analysis
5. DO: start machine learning from simple baselines
6. DO: use visualization to gain insight (from debugging to result presentation)
7. DO: make use of all of your data (no sampling necessary)
![Page 14: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/14.jpg)
Zurich Universities of Applied Sciences and Arts
✔ Eight commandments
5
1. DO: embrace interdisciplinarity, seek knowledge exchange
2. DO: build trust by data usage transparency & security provisions
3. DO: cherish data wrangling, ideally automate it it’s the basis for analysis
4. DO: leverage stream processing tools for real time big data analysis
5. DO: start machine learning from simple baselines
6. DO: use visualization to gain insight (from debugging to result presentation)
7. DO: make use of all of your data (no sampling necessary)
8. DO: take special care of small data (because of less redundancies)
![Page 15: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/15.jpg)
Zurich Universities of Applied Sciences and Arts
Inspiration #1: methodologyMake intuitive model inspection & data visualization “always on”
6
• Building trust with stakeholders
• Debugging capabilities for researchers & developers
negative X-ray positive X-ray
DNN training on the Information Plane a learning curve feature visualization
![Page 16: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/16.jpg)
Zurich Universities of Applied Sciences and Arts
Inspiration #2: technologyUnderstand influences on big data system performance
7
• Modern big data systems make parallel programming easy
• However, the complex distributed components need careful performance
analysis & tuning to arrive at state of the art results:
0
10'000
20'000
30'000
Kafka +Jackson
Kafka +Gson
MongoDB
Max producer throughput (alarms/s)
0
10'000
20'000
30'000
Kafka +Jackson
Kafka +Gson
MongoDB
Max consumer throughput (alarms/s)
0
10'000
20'000
30'000
Kafka +Jackson
Kafka +Gson
MongoDB
Max producer throughput (alarms/s)
0
10'000
20'000
30'000
Max consumer throughput (alarms/s)
Configuring the Kafka Direct Stream in
with proper settings…
(num partitions = num cores)
![Page 17: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/17.jpg)
Zurich Universities of Applied Sciences and Arts
Inspiration #3: innovationUse networks of experts to leverage different levels of innovation
Depth
of in
novatio
n
Apply the existing Recombine the existing Create tech. prerequisites
Business needs: purpose
Research: -
Business needs: consulting
Research: transfer
Business needs: development
Research: R&D
Products on the market,
know-how widely available
(e.g. process support by IT)
Product or technology on market,
know-how still novel
(e.g. business model innovation
through combination of
hardware & service)
Base technology exists,
case never implemented before,
transferability possible
(e.g. algorithms for automating
pattern recognition tasks)
![Page 18: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/18.jpg)
Zurich Universities of Applied Sciences and Arts
Inspiration #4: educationBuild interdisciplinary skills & experience on top of solid foundation
9
• Disciplinary bachelor establishes foundation in a constituting field
• Data science education imparts core methods, tools, and project experience
![Page 19: Lessons learned from 16 applied data science (meta) case studies · 2020-06-09 · Lessons learned from 16 applied data science (meta) case studies on industrial applied data science,](https://reader035.fdocuments.in/reader035/viewer/2022063001/5f1c0aa934f49c1a2735418f/html5/thumbnails/19.jpg)
Zurich Universities of Applied Sciences and Arts
Conclusions
10
• Crucial digital innovation needs to happen at the level of society:
how do we deal with the opportunities “making sense of data” is giving us?
On me:• Prof. AI/ML, head ZHAW Datalab, board Data+Service Alliance
• +41 58 934 72 08
• https://stdm.github.io/
On the topics:• Data science @ ZHAW: www.zhaw.ch/datalab
• Data science in CH: www.data-service-alliance.ch
• Applied data science book: https://stdm.github.io/data-science-book/
Happy to answer questions & requests.