DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science...
Transcript of DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science...
![Page 1: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/1.jpg)
Seagate Confidential 1
DATA IS IN OUR DNA
Guided Analytics at Seagate Seagate Technology Operations & Technology Advanced Analytics Group Allan Luk, Analytics Business Solutions Director Dr. Eric Lin, Sr Staff Analytics Business Solutions Engineer November 9, 2018
© Seagate Technology, LLC, 2018
![Page 2: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/2.jpg)
Seagate Confidential 2
You May Know Seagate as a Hard Drive Manufacturer…
• #1 OEM storage
• 1st to build and ship a collective zettabyte to the world
• Stores more than 40% of the world’s data
© Seagate Technology, LLC, 2018
![Page 3: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/3.jpg)
Seagate Confidential 3 3
But We’re also a Company that:
Serves many types of customers and
businesses
Delivers deep expertise and unique IP in storage & data management
Combines UX, software & design
capabilities to create new categories
of storage solutions
Ranks as one of the top 25 companies worldwide in supply chain operations
© Seagate Technology, LLC, 2018
![Page 4: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/4.jpg)
Seagate Confidential 4 4 Seagate Confidential
Presentation Overview
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
• Guided Analytics at Seagate Update
• Guided Analytics Software Development and Deployment Challenges
• KNIME Development and Deployment Examples • This Presentation (Allan Luk, Eric Lin) - Manufacturing, HR • Next Presentation (Debin Wang) - R&D, Engineering
• Summary
• Live Demo: • Parallel Execution of Data Source Query in KNIME
© Seagate Technology, LLC, 2018
![Page 5: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/5.jpg)
Seagate Confidential 5 5 Seagate Confidential
Background
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
Tons of Data everywhere Move up analytics capability curve Analytics INSIGHTS
![Page 6: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/6.jpg)
Seagate Confidential 6 6 Seagate Confidential
Background
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
Citizen Data Science (CDS) Initiative @ Seagate Turn Data into INSIGHTS
CDS & Guided Analytics Software Workshop
CDS Certification Guided Analytics Software Development & Deployment
CDS Community Building
Awareness Software & Tools Capability Ecosystem: Community Contributions
© Seagate Technology, LLC, 2018
Overview: Guided Analytics at Seagate
![Page 7: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/7.jpg)
Seagate Confidential 7 7 Seagate Confidential Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
Guided Analytics Software (KNIME) Workshop/Tech Talk
![Page 8: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/8.jpg)
Seagate Confidential 8 8 Seagate Confidential
Seagate Guided Analytics Journey
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
• 2nd year on this journey
• The numbers: • Exposed KNIME to over 700 employees within Seagate • About 70 KNIME users • What are we using KNIME for:
• 65% - ETL, getting data, data preparation and as an integration platform • 35% - Modeling and prediction
• Avid KNIME users: • 1st batch: Manufacturing, Engineering, R&D • 2nd batch: Other functions and businesses (e.g. HR)
• Using KNIME Desktop version
• KNIME Server version • Acquired license • To integrate with new IT infrastructure
© Seagate Technology, LLC, 2018
![Page 9: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/9.jpg)
Seagate Confidential 9 9 Seagate Confidential
Guided Analytics Software Development and Deployment Challenges
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
• Deploying to the whole company
• To get familiar with KNIME • Make it easier for employees to learn and apply at workplace
• Comparison to the existing software and solutions • Legacy system
• Ability to integrate with existing tools and software • Features. Ease of use.
• Customerization for Seagate use cases: • Why customerization?
• User experience • Connection to various data sources • Integration to the new IT infrastructure
• Analytics performed at the data source
• Our wish list about the software: • Visualization and dashboard
• Features. Ease of use. • Data exploration – more interactive
© Seagate Technology, LLC, 2018
![Page 10: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/10.jpg)
Seagate Confidential 10 10 Seagate Confidential
KNIME Development Examples Objectives: i) Improve Efficiency, ii) Enable Collaboration and iii) Enhance Adoption
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
![Page 11: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/11.jpg)
Seagate Confidential 11 11 Seagate Confidential
KNIME Development Example 1 – Manufacturing Data Query Node
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
Enable Data Wrangling and Automation. Essential to Product Performance Analysis.
M drive logs × N tables N files with M drives in each file
�����
![Page 12: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/12.jpg)
Seagate Confidential 12 12 Seagate Confidential
KNIME Development Example 2 – KNIME Containerization
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
Objectives: • End-to-end Task Automation Suite for
scheduling and automating various Data Science Workflows.
• Development of In-House Suite of Analytics Services.
• Containerization of KNIME workflows.
Deliverables: • A web application. Users upload data
science workflow files. • Users specify a schedule to run the
workflow automatically. • After the workflow runs, a report will be
mailed to the users.
![Page 13: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/13.jpg)
Seagate Confidential 13 13 Seagate Confidential
KNIME Development Example 3 – Custom Node Creation
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
• To augment ROC curve with additional all-in-one plot feature.
![Page 14: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/14.jpg)
Seagate Confidential 14 14 Seagate Confidential
• PRC: Precision vs Recall • ROC: TPR (true positive rate) vs FPR (false positive rate)
• TPR, FPR vs Threshold • F-Measure vs Threshold
KNIME Development Example 3 – Custom Node Creation
Seagate Technology Operations & Technology Advanced Analytics Group [email protected] © Seagate Technology, LLC, 2018
![Page 15: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/15.jpg)
Seagate Confidential 15 15 Seagate Confidential
KNIME Deployment Examples i) Product Quality Assessment ii) Equipment Health Prediction iii) Manufacturing Image Analytics iv) HR Analytics
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
![Page 16: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/16.jpg)
Seagate Confidential 16 16 Seagate Confidential
KNIME Deployment Example 1 – Component Manufacturing Quality Assessment
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
KNIME P. 14 KNIME KNIME
![Page 17: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/17.jpg)
Seagate Confidential 17 17 Seagate Confidential
KNIME Deployment Example 1 – Component Manufacturing Quality Assessment
Seagate Technology Operations & Technology Advanced Analytics Group [email protected] © Seagate Technology, LLC, 2018
![Page 18: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/18.jpg)
Seagate Confidential 18 18 Seagate Confidential
KNIME Deployment Example 1 – Component Manufacturing Quality Assessment
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
![Page 19: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/19.jpg)
Seagate Confidential 19 19 Seagate Confidential
KNIME Deployment Example 2 – Manufacturing Equipment Health Prediction
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
![Page 20: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/20.jpg)
Seagate Confidential 20 20 Seagate Confidential
KNIME Deployment Example 3 – Manufacturing Image Analytics via Tensorflow Integration
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
![Page 21: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/21.jpg)
Seagate Confidential 21 21 Seagate Confidential
KNIME Deployment Example 4 – HR Analytics
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
Integrated & Repeatable
Configurable
Pedagogic
On data exploration, data understanding & aggregation methods
On source data files, output chart format, & data filtering
From data cleaning to calculation roll-out & final report preparation
![Page 22: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/22.jpg)
Seagate Confidential 22 22 Seagate Confidential
Summary
Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
• Guided Analytics Software implementation at Seagate
• Significant progress made over the past year • Many activities underway
• KNIME software development and solutions deployment • Implementation areas: Manufacturing, Engineering, R&D, HR and other Business functions
• KNIME server integration and rollout will further accelerate adoption
• Application Areas • 1) ETL, Data Query, 2) Integration Platform, 3) Modeling, Prediction
• Benefits • Enable our citizen data scientists and data analysts to do more with their data • Automation (e.g. reduce task duration: from days to minutes, seconds)
• Our wish list to further enhance implementation • Better visualization and dashboard features • More interactive data exploration • Interactive learning materials for newcomers
© Seagate Technology, LLC, 2018
![Page 23: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/23.jpg)
Seagate Confidential 23 23 Seagate Confidential Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
KNIME Live Demo: Parallel Execution of Data Sources in KNIME
© Seagate Technology, LLC, 2018
![Page 24: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/24.jpg)
Seagate Confidential 24 24 Seagate Confidential Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
SQL Query
SQL Database
Parallel SQL Execution
Multi-threading, Not Multi-processing
Multitasking: Parallel Execution on Data Sources with KNIME/Java
KNIME Live Demo: Parallel Execution of Data Sources in KNIME
![Page 25: DATA IS IN OUR DNA Guided Analytics at Seagate...scheduling and automating various Data Science Workflows. • Development of In-House Suite of Analytics Services. • Containerization](https://reader030.fdocuments.in/reader030/viewer/2022013021/5ecd429b4289ce0dfd6e2f51/html5/thumbnails/25.jpg)
Seagate Confidential 25 25 Seagate Confidential Seagate Technology Operations & Technology Advanced Analytics Group [email protected]
© Seagate Technology, LLC, 2018
The Power of Multithreading: Parallel Execution on Data Sources
KNIME Live Demo: Parallel Execution of Data Sources in KNIME
Data collected with • Total In-List items = 2000 • Querying against 4
parametric tables