Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like...

Instrumentation, Observability, and Monitoring of Machine Learning Models

About Me

● Google Engineer (2007-11)

● Cloudera’s Director of Data Science (2011-15)

● Slack’s Director of Data Engineering (2015-2017)

● Slack Engineer (now)

“”

The Genesis of This Talk

Machine Learning In the Wild

Data Science Meets DevOps

Some History

Logs via the ELK Stack

Metrics with Prometheus

Prometheus Architecture

Traces

A Word About Cardinality

Make Good Decisions By Avoiding Bad Decisions

The ML Test Score

The Map Is Not The Territory

Monitor Model Decay

Build Lots of Models

Deploy Your Models Like They Are Code*

Stand On The Shoulders of Giants

● Ensembles

● Experiments

● Dark Tests

● Canary

● Sanity Checks

Tag All The Things

Circle of Competence

Garbage In...

Linking Online and Offline Metrics

Handling Cross-Language Feature Engineering

Know Your Dependencies

Monitoring For Critical Slices

Second-Order Thinking

On Razors

http://slack.com/careers

Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like...

Documents

Transcript of Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like...

Accelerating Problem Solving by Combining Machine Learning ... · Accelerating Problem Solving by Combining Machine Learning and Human Learning David B. Fogel, Ph.D. ... Turing (1950)

The classification problem in machine learning: An overview

More Machine Learning - Classification · Machine learning optimization With this notation, the “canonical” machine learning problem is written in the exact same way minimize!

Application example: Photo OCR Problem description and pipeline Machine Learning.

Machine Learning Lecture 18 - RWTH Aachen University · 2015-11-27 · 14.07.2015 Machine Learning ... • Expectation-Maximization (EM) Algorithm ... Learning problem: Try to find

Finland - arxiv.org · EﬃcienthyperparametertuningforkernelridgeregressionwithBayesianoptimization 2 mization problem in machine learning of computational chemistry.

Plagiarism Detection as a Problem of Machine Learning

CS 5751 Machine Learning Chapter 11 Explanation-Based Learning 1 Explanation-Based Learning (EBL) One definition: Learning general problem- solving techniques.

Lecture 1: Machine Learning Problem - University of Adelaidejaven/talk/L1 ML problem.pdf · 2014-08-04 · Lecture 1: Machine Learning Problem Qinfeng (Javen) Shi 28 July 2014 Intro.

Object Correspondence as a Machine Learning Problem

10-701 Machine Learning - Spring 2012 Problem Set 4: Solutions

Generalization of Machine Learning for Problem Reduction: A … · 2021. 6. 22. · Generalization of Machine Learning for Problem Reduction: A Case Study on Travelling Salesman Problems

Machine Learning and Optimization - Andres Munoz, …munoz/files/ml_optimization.pdf · Machine Learning and Optimization Andres Munoz ... problem allowing people to use all of the

Machine Learning Problem - cs.uwaterloo.cadberry/ATRE/Slides/Bikram... · 1. Zhang, Du, and Jeﬀrey JP Tsai. "Machine learning and software engineering." Software Quality Journal

I Why Machine Learning? - Computational Logic · 1 Machine Learning – Introduction IWhy Machine Learning? IWhat is a well-deﬁned learning problem? IAn example: learning to play

Regularization The problem of overfitting Machine Learning.

Applications of Machine Learning in Solving Vehicle Routing Problem

Mitigating the Compiler Optimization Phase-Ordering Problem using Machine Learning

The Machine Learning and Traveling Repairman Problemweb.mit.edu/jaillet/www/general/machine-learning-trp-short-2011.pdf · 2 The Machine Learning & Traveling Repairman Problem models,

Learning to Discover Complex Mappings from Web Forms to ... · the problem, we resort to machine learning methods that can learn knowledge from examples. Machine learning has been