ROC & AUC, LIFT
description
Transcript of ROC & AUC, LIFT
ROC & AUC, LIFT
רוזנפלד" אבי ר ד
Introduction to ROC curves
• ROC = Receiver Operating Characteristic
• Started in electronic signal detection theory (1940s - 1950s)
• Has become very popular in biomedical applications, particularly radiology and imaging
• מידע בכריית בשימוש גם
False Positives / Negatives
P N
P 20 10
N 30 90
Predicted
Actu
al
Confusion matrix 1
P N
P 10 20
N 15 105
Predicted
Actu
al
Confusion matrix 2
FN
FP
Precision (P) = 20 / 50 = 0.4Recall (P) = 20 / 30 = 0.666F-measure=2*.4*.666/1.0666=.5
4
Different Cost Measures• The confusion matrix (easily generalize to multi-class)
• Machine Learning methods usually minimize FP+FN • TPR (True Positive Rate): TP / (TP + FN) = Recall• FPR (False Positive Rate): FP / (TN + FP) = Precision
Predicted class
Yes No
Actual class
Yes TP: True positive
FN: False negative
No FP: False positive
TN: True negative
Specific Example
Test Result
People with disease
People without disease
Test Result
Call these patients “negative”
Call these patients “positive”
Threshold
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
True Positives
Some definitions ...
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
False Positives
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
True negatives
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
False negatives
Test Result
without the diseasewith the disease
‘‘-’’
‘‘+’’
Moving the Threshold: left
Tru
e P
osi
tive R
ate
(R
eca
ll)
0%
100%
False Positive Rate (1-specificity)
0%
100%
ROC curve
ה שינוי של הגרף THRESHOLDההשפעה על
Figure 5.2 A sample ROC curve.
של שונים גרפים ROCסוגים
Area under ROC curve (AUC)
כללי • מדד
לגרך • מתחת ROCהשטח
•0.50 , רנדומאלי מחירה .1.0הוא מושלם הוא
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
AUC = 50%
AUC = 90% AUC =
65%
AUC = 100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
AUC for ROC curves
Lift Charts
• X axis is sample size: (TP+FP) / N• Y axis is TP
40% of responses for 10% of costLift factor = 4
80% of responses for 40% of costLift factor = 2Model
Random
Lift factor
0
0.5
1
1.5
2
2.5
3
3.5
4
4.55 15 25 35 45 55 65 75 85 95
Lift
Sample Size
Lift
Val
ue
המדדים בין הקשר
התרגיל ...לקראת
ואז מודל על ימני לחצןCost / Benefit Analysis for Wood
ה את לראות וגם הסף את לשנות אפשרCONFUSION MATRIX
ה את גם לראות וגם Liftאפשרמחיר השפעת