Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big...
Transcript of Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big...
![Page 1: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/1.jpg)
Explaining Black-Box Machine Learning Predictions
Sameer SinghUniversity of California, Irvine
work with Marco T. Ribeiro and Carlos Guestrin
![Page 2: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/2.jpg)
Machine Learning is Everywhere…
![Page 3: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/3.jpg)
Classification: Wolf or a Husky?
Machine LearningModel
Wolf!
Adopt or not?
![Page 4: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/4.jpg)
Classification: Wolf or a Husky?
Machine Learning Model
Husky!
Adopt or not?
![Page 5: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/5.jpg)
Only 1 mistake!
Classification: Wolf or a Husky?
![Page 6: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/6.jpg)
Visual Question Answering
Is there a moustache in the picture?
> Yes
What is the moustache made of?
> Banana
![Page 7: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/7.jpg)
Essentially black-boxes!
How can we trust the predictions are correct?
Trust
How can we understand and predict the behavior?
Predict
How do we improve it to prevent potential mistakes?
Improve
![Page 8: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/8.jpg)
Only 1 mistake!
Classification: Wolf or a Husky?
We’ve built a snow detector…
![Page 9: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/9.jpg)
![Page 10: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/10.jpg)
Visual Question Answering
What is the moustache made of?
> Banana
What are the eyes made of?
> Bananas
What?
> Banana What is?
> Banana
![Page 11: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/11.jpg)
Text Classification
Why did this happen?
From: Keith Richards
Subject: Christianity is the answer
NTTP-Posting-Host: x.x.com
I think Christianity is the one true religion.
If you’d like to know more, send me a note
![Page 12: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/12.jpg)
Loan Applications (in a Blackbox-ML World)
Machine Learning
I would like to apply for a loan.vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Here is my information.vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Sorry, your request has been denied
Why? What were the reasons?
Currently
Cannot explain.. [0.25,-4.5,3.5,-10.4,…]
![Page 13: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/13.jpg)
How did we get here?
![Page 14: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/14.jpg)
What is Machine Learning?
HistoricalInput Data
HistoricalOutput Data
Machine Learning
NewInputs
PredictOutput
Classifier
ComplexMath
![Page 15: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/15.jpg)
Should I give out a loan?
X1
X2
Savings
Income
![Page 16: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/16.jpg)
Get Historical Data
X1
X2
Savings
Income
![Page 17: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/17.jpg)
Linear Classifiers
X1
X2You can interpret it…- Both have a positive effect- X1 > X2
10X1 + X2 - 5 > 0if:
otherwise
Savings
Income
![Page 18: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/18.jpg)
Decision trees
X1
X2
X1 > 0.5
X2 > 0.5
You can interpret it…- X2 is irrelevant if X1<0.5- Otherwise X2 is enough
![Page 19: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/19.jpg)
Looking at the structure
How can we trust the predictions are correct?
Trust
How can we understand and predict the behavior?
Predict
How do we improve it to prevent potential mistakes?
Improve
Test whether the structure agrees with our intuitions.
Structure tells us exactly what will happen on any data.
Structure tells you where the error is, thus how to fix it.
![Page 20: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/20.jpg)
Arrival of Big Data
![Page 21: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/21.jpg)
Big Data: Applications of ML
![Page 22: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/22.jpg)
Big Data: More Complexity
X1
X2
![Page 23: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/23.jpg)
Big Data: More Complexity
X1
X2
![Page 24: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/24.jpg)
Big Data: More Complexity
X1
X2
![Page 25: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/25.jpg)
http://www.mckinsey.com/industries/high-tech/our-insights/an-executives-guide-to-machine-learning
![Page 26: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/26.jpg)
Big Data: More Dimensions
SavingsIncome
Profession
Loan Amount
Age
Marital Status
Past defaults
Credit scores
Recent defaults
This easily goes to hundreds- Images: thousands- Text: tens of thousands- Video: millions- … and so on
![Page 27: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/27.jpg)
X1
X2
Complex Surfaces
SavingsIncome
Profession
Loan Amount
Age
Married
Past defaults
Credit scores
Recent defaults
…
Lots of dimensions+
Black-boxes!
![Page 28: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/28.jpg)
Accuracy vs Interpretability
Interpretability
Accuracy
10X1 + X2 - 5 > 0
X1 > 0.5
X2 > 0.5
millions of weights,complex features
Real-world use caseResearch on“interpretable models”
![Page 29: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/29.jpg)
Deep Learning
Interpretability
Accuracy
Real-world use caseResearch on“interpretable models”
Focus on accuracy!Human-level
![Page 30: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/30.jpg)
Looking at the structure
How can we trust the predictions are correct?
Trust
How can we understand and predict the behavior?
Predict
How do we improve it to prevent potential mistakes?
Improve
Test whether the structure agrees with our intuitions.
Structure tells us exactly what will happen on any data.
Structure tells you where the error is, thus how to fix it.
![Page 31: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/31.jpg)
Explaining PredictionsThe LIME Algorithm
![Page 32: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/32.jpg)
Applying for a Loan
Machine Learning
I would like to apply for a loan.vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Here is my information.vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Sorry, your request has been denied
Why? What were the reasons?
My Research
35% of my decision is based on your race,20% on your income, and
15% on your savings.
![Page 33: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/33.jpg)
Being Model-Agnostic…
No assumptions about the internal structure…
X1 > 0.5
X2 > 0.5f(x)
Explain any existing, or future, model
Data Decision
![Page 34: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/34.jpg)
LIME: Explain Any Classifier!
Interpretability
Accuracy
Real-world use case Make everything
interpretable!
![Page 35: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/35.jpg)
What is an “Explanation”?
Why did this happen?
From: Keith Richards
Subject: Christianity is the answer
NTTP-Posting-Host: x.x.com
I think Christianity is the one true religion.
If you’d like to know more, send me a note
![Page 36: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/36.jpg)
Being Model-Agnostic…
“Global” explanation is too complicated
![Page 37: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/37.jpg)
Being Model-Agnostic…
“Global” explanation is too complicated
![Page 38: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/38.jpg)
Being Model-Agnostic…
“Global” explanation is too complicated
Explanation is an interpretable model, that is locally accurate
![Page 39: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/39.jpg)
PerturbedInstances
P(Labrador)
Example – Image Classification
Original Image
0.92
0.001
0.34
P(labrador) = 0.21
Locally weightedregression
Explanation
![Page 40: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/40.jpg)
Google’s Object Detector
P( ) = 0.21 P( ) = 0.24 P( ) = 0.32
![Page 41: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/41.jpg)
Only 1 mistake!
Classification: Wolf or a Husky?
![Page 42: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/42.jpg)
Neural Network Explanations
We’ve built a great snow detector…
![Page 43: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/43.jpg)
Visual QA
![Page 44: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/44.jpg)
Neural Machine Translation
![Page 45: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/45.jpg)
Neural Machine Translation
![Page 46: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/46.jpg)
Neural Machine Translation
![Page 47: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/47.jpg)
Salary Prediction
IF Education < High SchoolThen Predict Salary < 50K
Salary
71%
29% >$50K
<$50K
![Page 48: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/48.jpg)
Salary Prediction
IF Married and Education = Doctorate
Then Predict Salary > 50K
Salary
91%
9%
>$50K
<$50K
Data from the US Census
![Page 49: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/49.jpg)
“Global” Behavior
What about explaining the rest of the model?
![Page 50: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/50.jpg)
Explaining Global behavior
LIME explains a single predictionlocal behavior for a single instance
Can’t examine all explanationsInstead pick k explanations to show to the user
DiverseShould not be redundant in their descriptions
RepresentativeShould summarize the model’s global behavior
Single explanation
![Page 51: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/51.jpg)
Are they useful?
![Page 52: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/52.jpg)
Quantitative Evaluation
Understand whatML is doing
Improve theexisting model
Compare differentML algorithms
Predict how ML will behave
![Page 53: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/53.jpg)
Quantitative Evaluation
Improve theexisting model
Compare differentML algorithms
Predict how ML will behave
Understand whatML is doing
![Page 54: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/54.jpg)
Understanding Behavior
We’ve built a great snow detector…
![Page 55: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/55.jpg)
Understanding Behavior
Question 1Would you trust this model?
Question 2What is the classifier is doing?
![Page 56: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/56.jpg)
Did they notice it?
0.
25.
50.
75.
100.
Didn't trust the model "Snow insight"
% o
f su
bje
cts
(ou
t o
f 2
7)
Before explanations After explanations
![Page 57: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/57.jpg)
Quantitative Evaluation
Understand whatML is doing
Improve theexisting model
Predict how ML will behave
Compare differentML algorithms
![Page 58: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/58.jpg)
Comparing Classifiers
Classifier 1
Classifier 2Explanations?
Look at Examples?
Deploy and Check?
“I have a gut feeling..”
Accuracy?
Change the modelDifferent dataDifferent parametersDifferent “features”…
![Page 59: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/59.jpg)
Comparing Classifiers
Original Image “Bad” Classifier “Good” Classifier
![Page 60: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/60.jpg)
Explanation for a bad classifier
From: Keith Richards
Subject: Christianity is the answer
NTTP-Posting-Host: x.x.com
I think Christianity is the one true religion.
If you’d like to know more, send me a note
After looking at the explanation,we shouldn’t trust the model!
![Page 61: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/61.jpg)
“Good” Explanation
It seems to be picking up on more reasonable things.. good!
![Page 62: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/62.jpg)
UI for Comparing Classifiers
![Page 63: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/63.jpg)
Comparing Models
If we picked basedon accuracy, we would get it wrong.
40
50
60
70
80
90
100
Guessing LIME-random LIME-global
% p
icke
d b
ette
r m
od
el
89% of users identifythe more trustworthy model
![Page 64: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/64.jpg)
Quantitative Evaluation
Understand whatML is doing
Compare differentML algorithms
Predict how ML will behave
Improve theexisting model
![Page 65: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/65.jpg)
Improving Classifiers
Classifier
Explanations
GenerateExplanations
SuggestChanges
Compute Accuracy
Lay peopleShowExplanations
![Page 66: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/66.jpg)
UI for fixing bad classifiers
![Page 67: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/67.jpg)
Fixing bad classifiers
0.5
0.575
0.65
0.725
0.8
w/o feedback 1 round 2 rounds 3 rounds
Acc
ura
cy o
n h
idd
en s
et
Train on original data
Train by us(without explanations)
Train by lay people
![Page 68: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/68.jpg)
Quantitative Evaluation
Understand whatML is doing
Improve theexisting model
Compare differentML algorithms
Predict how ML will behave
![Page 69: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/69.jpg)
Predicting Behavior
Classifier
Predictions & Explanations
Show to user
Data
Compare AccuracyPredictions
New Data
User guesses whatthe classifier would doon new data
![Page 70: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/70.jpg)
User Studies: Precision
0
10
20
30
40
50
60
70
80
90
100
No Explanation With Explanations
How good are users guesses on unseen instances?
VQA 1 VQA 2
![Page 71: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/71.jpg)
User Studies: Time
0
2
4
6
8
10
12
14
16
18
20
No Explanation With Explanations
How long do users take to make their guesses?
VQA 1 VQA 2
![Page 72: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/72.jpg)
Explanations are important!
How can we trust the predictions are correct?
Trust
How can we understand and predict the behavior?
Predict
How do we improve it to prevent potential mistakes?
Improve
Model Agnostic Explanations
![Page 73: Explaining Black-Box Machine Learning Predictionsoc.acm.org/docs/7-12-2017 - Sameer Singh... · Big Data: More Complexity X1 X2. Big Data: More Complexity X1 X2. Big Data: More Complexity](https://reader034.fdocuments.in/reader034/viewer/2022042409/5f25768268d2844fea161adc/html5/thumbnails/73.jpg)
Thanks! [email protected]
github.com/marcotcr/lime
sameersingh.org
Model Agnostic Explanations
“Why should I trust you?”Explaining the predictions of any classifier
Ribeiro, Singh, Guestrin, KDD 2016