Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like...

30
1 Instrumentation, Observability, and Monitoring of Machine Learning Models

Transcript of Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like...

Page 1: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

1

Instrumentation, Observability, and Monitoring of Machine Learning Models

Page 2: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

About Me

● Google Engineer (2007-11)

● Cloudera’s Director of Data Science (2011-15)

● Slack’s Director of Data Engineering (2015-2017)

● Slack Engineer (now)

Page 3: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

“”

Page 4: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

“”

Page 5: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

The Genesis of This Talk

Page 6: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Machine Learning In the Wild

Page 7: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Data Science Meets DevOps

Page 8: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Some History

Page 9: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Logs via the ELK Stack

Page 10: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Metrics with Prometheus

Page 11: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Prometheus Architecture

Page 12: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Traces

Page 13: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

A Word About Cardinality

Page 14: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Make Good Decisions By Avoiding Bad Decisions

Page 15: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

The ML Test Score

Page 16: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

The Map Is Not The Territory

Page 17: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Monitor Model Decay

Page 18: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Build Lots of Models

Page 19: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Deploy Your Models Like They Are Code*

Page 20: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Stand On The Shoulders of Giants

● Ensembles

● Experiments

● Dark Tests

● Canary

● Sanity Checks

Page 21: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Tag All The Things

Page 22: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Circle of Competence

Page 23: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Garbage In...

Page 24: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Linking Online and Offline Metrics

Page 25: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Handling Cross-Language Feature Engineering

Page 26: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Know Your Dependencies

Page 27: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Monitoring For Critical Slices

Page 28: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

Second-Order Thinking

Page 29: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

On Razors

Page 30: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,

http://slack.com/careers

30

http://slack.com/careers

The classification problem in machine learning: An overview

The classification problem in machine learning: An overview

Problem-Based Learning: Data Analytics and Machine Learning … · Analytics and Machine Learning Techniques for Solving Real-World Challenges Dr Jasmina Lazić, ... Data Everywhere

Problem-Based Learning: Data Analytics and Machine Learning … · Analytics and Machine Learning Techniques for Solving Real-World Challenges Dr Jasmina Lazić, ... Data Everywhere

Speech Synthesis as a Statistical Machine Learning Problem Slides

Speech Synthesis as a Statistical Machine Learning Problem Slides

Finland - arxiv.org · EfficienthyperparametertuningforkernelridgeregressionwithBayesianoptimization 2 mization problem in machine learning of computational chemistry.

Finland - arxiv.org · EfficienthyperparametertuningforkernelridgeregressionwithBayesianoptimization 2 mization problem in machine learning of computational chemistry.

CS 195-5: Machine Learning Problem Set 4 - Brownmesh.brown.edu/dlanman/courses/cs195-5/PS4.pdf · CS 195-5: Machine Learning Problem Set 4 ... Matlab documentation, that alpha = quadprog

CS 195-5: Machine Learning Problem Set 4 - Brownmesh.brown.edu/dlanman/courses/cs195-5/PS4.pdf · CS 195-5: Machine Learning Problem Set 4 ... Matlab documentation, that alpha = quadprog

Accelerating Problem Solving by Combining Machine Learning ...ewh.ieee.org/cmte/cis/mtsc/ieeecis/David_B_Fogel.pdf · Accelerating Problem Solving by Combining Machine Learning and

Accelerating Problem Solving by Combining Machine Learning ...ewh.ieee.org/cmte/cis/mtsc/ieeecis/David_B_Fogel.pdf · Accelerating Problem Solving by Combining Machine Learning and

Object Correspondence as a Machine Learning Problem

Object Correspondence as a Machine Learning Problem

Resilient Machine Learning in Adversarial Environments · machine learning algorithms under various attacks) •Problem: ... Towards Evaluating the Robustness of Neural Networks •[Madry

Resilient Machine Learning in Adversarial Environments · machine learning algorithms under various attacks) •Problem: ... Towards Evaluating the Robustness of Neural Networks •[Madry

Machine Learning Problem - cs.uwaterloo.cadberry/ATRE/Slides/Bikram... · 1. Zhang, Du, and Jeffrey JP Tsai. "Machine learning and software engineering." Software Quality Journal

Machine Learning Problem - cs.uwaterloo.cadberry/ATRE/Slides/Bikram... · 1. Zhang, Du, and Jeffrey JP Tsai. "Machine learning and software engineering." Software Quality Journal

CS 195-5: Machine Learning Problem Set 4 1 Support Vector ...

CS 195-5: Machine Learning Problem Set 4 1 Support Vector ...

On the Diversity of the Accountability Problem. Machine Learning and Knowing Capitalism.

On the Diversity of the Accountability Problem. Machine Learning and Knowing Capitalism.

CSI 436/536 Introduction to Machine Learninglsw/MLmaterials/introduction.pdf• Machine learning algorithms • Design algorithm to solve a learning problem • Machine learning systems

CSI 436/536 Introduction to Machine Learninglsw/MLmaterials/introduction.pdf• Machine learning algorithms • Design algorithm to solve a learning problem • Machine learning systems

1 1)Introduction to Machine Learning 2)Learning problem 3)Learning system design IES 511 Machine Learning Dr. Türker İnce (Lecture notes by Prof. T. M.

1 1)Introduction to Machine Learning 2)Learning problem 3)Learning system design IES 511 Machine Learning Dr. Türker İnce (Lecture notes by Prof. T. M.

Machine*Learning,* Entropy*and*Fraud*in* Splunk* · Machine*Learning*Goal* Program*computers*to*use*example*dataor*past experience*to*solve*agiven*problem*

Machine*Learning,* Entropy*and*Fraud*in* Splunk* · Machine*Learning*Goal* Program*computers*to*use*example*dataor*past experience*to*solve*agiven*problem*

CSE 512 Machine Learning: Homework I · CSE 512 Machine Learning: Homework I Mari Wahl, marina.w4hl at gmail 1 Machine Learning - Problem Setup [10 points] In online debate forums,

CSE 512 Machine Learning: Homework I · CSE 512 Machine Learning: Homework I Mari Wahl, marina.w4hl at gmail 1 Machine Learning - Problem Setup [10 points] In online debate forums,

Android Malware Detection Based on System Calls · B. Machine Learning Our malware detection problem is an instance of a classi-cation problem in machine learning, and is solved using

Android Malware Detection Based on System Calls · B. Machine Learning Our malware detection problem is an instance of a classi-cation problem in machine learning, and is solved using

Machine Learning - B. Unsupervised Learning B.3 Frequent Pattern … · 2020-06-04 · Machine Learning Outline 1. The Frequent Itemset Problem 2. Breadth First Search: Apriori Algorithm

Machine Learning - B. Unsupervised Learning B.3 Frequent Pattern … · 2020-06-04 · Machine Learning Outline 1. The Frequent Itemset Problem 2. Breadth First Search: Apriori Algorithm

Machine Foundation Problem Machine Foundation Problem.

Machine Foundation Problem Machine Foundation Problem.

Machine Learning and Optimization - Andres Munoz, …munoz/files/ml_optimization.pdf · Machine Learning and Optimization Andres Munoz ... problem allowing people to use all of the

Machine Learning and Optimization - Andres Munoz, …munoz/files/ml_optimization.pdf · Machine Learning and Optimization Andres Munoz ... problem allowing people to use all of the

Applications of Machine Learning in Solving Vehicle Routing Problem

Applications of Machine Learning in Solving Vehicle Routing Problem