Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Dynamic Classifier Systems forClassifier Aggregation
David Stefka
Seminar z umele inteligence11. 12. 2008
Martin Holena
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Outline
1 Classifier Combining
2 Classification Confidence
3 Classifier Systems
4 Experiments
5 Assessing Confidence Measures
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Classification
classification – process of assigning patterns into classes
X – feature space, ~x pattern
C1, . . . ,CN – classes
classifier – mapping φ : X → [0, 1]N
φ(~x) = (µ1(~x), . . . , µN(~x))
µi (~x) – degree of classification x ∈ Ci
interpretation of µi (~x) – depends on the classifier used(probability, fuzzy membership, . . . )
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Classifier Combining
method for improving the classification by using multipleclassifiers and combining their outputs
create a team T = (φ1, . . . , φr ) of classifiers (bagging,boosting, . . . )
aggregate the team using aggregator Amost of the aggregation methods are static
~x
φ1
φ2
...φr
T (~x) A Φ(~x)
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Classification example
C1 – bananas, C2 – apples
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Classification example ctnd.
φ(~x) = (0.1, 0.8)
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Classification example ctnd.
φ(~x) = (0.9, 0.1)
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Classification example ctnd.
φ(~x) = (0.1, 0.2)
Low confidence
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Classification Confidence
degree of trust we can give to the classifier φ
probability of correct classification of ~x by φ
ability to answer ”I don’t know”
confidence measure κφ : X → [0, 1]
static measures – constant of the classifier (e.g., accuracy)
dynamic measures – adapted to the currently classifiedpattern (e.g., local accuracy)
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Global Accuracy - GA
static confidence measure
validation (training) set Mproportion of patterns ~y ∈M correctly classified by φ
κ(GA)φ (~x) =
∑~y∈M I (φcr (~y)
?= c(~y))
|M|
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Euclidean Local Accuracy - ELA
dynamic confidence measure
validation (training) set MN(x) patterns from M neighboring with ~x (e.g., 20 nearestunder Euclidean metric)
proportion of patterns ~y ∈ N(~x) correctly classified by φ
κ(ELA)φ (~x) =
∑~y∈N(~x) I (φcr (~y)
?= c(~y))
|N(~x)|
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Euclidean Local Match - ELM
dynamic confidence measure
validation (training) set MN(x) patterns from M neighboring with ~x (e.g., 20 nearestunder Euclidean metric)
proportion of patterns ~y ∈ N(~x) from the same class as φ ispredicting for ~x
κ(ELM)φ (~x) =
∑~y∈N(~x) I (φcr (~x)
?= c(~y))
|N(~x)|
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Classifier Systems
S = (T ,K,A) – classifier system
T = (φ1, . . . , φr ) – classifiers
K = (κφ1 , . . . , κφr ) – confidence measures
A – aggregator
3 types of classifier systems
confidence-freestaticdynamic
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Types of classifier systems
~x
φ1
φ2
...φr
T (~x) A Φ(~x)
(a) Confidence-free
~x
φ1
φ2
...φr
T (~x)
Kconst
A Φ(~x)
(b) Static
~x
φ1
φ2
...φr
κφ1
κφ2
...κφr
T (~x)
K(~x)
A Φ(~x)
(c) Dynamic
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Mean value based aggregators
confidence-free – mean value of the classifier outputs
static – weighted mean; weights are static confidences
dynamic – weighted mean; weights are dynamic confidences
T (~x) =
0BBB@φ1(~x)φ2(~x)
...φr (~x)
1CCCA =
0BBB@µ1,1(~x) µ1,2(~x) . . . µ1,N(~x)µ2,1(~x) µ2,2(~x) . . . µ2,N(~x)
. . .
µr,1(~x) µr,2(~x) . . . µr,N(~x)
1CCCA , K(~x) =
0BBB@κφ1
(~x)κφ2
(~x)...
κφr (~x)
1CCCA
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Experiments
comparison: confidence-free vs. static vs. dynamic systems
systems of Quadratic Discriminant Classifiers
various confidence measures
aggregators based on mean value aggregator
4 artificial, 4 real-world datasets
10-fold crossvalidation
ELM confidence measure
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Results – QDC
Confidence−free Static Dynamic0
1
2
3
4
5
6
7
8
Combination method
Me
an
Err
± S
td E
rr
QDC − Breast
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Results – QDC
Confidence−free Static Dynamic19
20
21
22
23
24
25
26
27
28
Combination method
Me
an
Err
± S
td E
rrQDC − Clouds
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Results – QDC
Confidence−free Static Dynamic1
1.5
2
2.5
3
3.5
4
4.5
5
Combination method
Me
an
Err
± S
td E
rrQDC − Concentric
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Results – QDC
Confidence−free Static Dynamic19
19.5
20
20.5
21
21.5
22
22.5
23
23.5
24
Combination method
Me
an
Err
± S
td E
rrQDC − Gauss 3D
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Results – QDC
Confidence−free Static Dynamic14
16
18
20
22
24
26
Combination method
Me
an
Err
± S
td E
rrQDC − Phoneme
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Results – QDC
Confidence−free Static Dynamic16
18
20
22
24
26
28
30
32
34
Combination method
Me
an
Err
± S
td E
rrQDC − Pima
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Results – QDC
Confidence−free Static Dynamic13
13.5
14
14.5
15
15.5
16
16.5
17
17.5
Combination method
Me
an
Err
± S
td E
rrQDC − Satimage
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Results – QDC
Confidence−free Static Dynamic12
13
14
15
16
17
18
Combination method
Me
an
Err
± S
td E
rrQDC − Waveform
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Experiments - Results
experiments with QDC ensembles: dynamic classifier systemscan significantly improve classification quality
similar results with Random Forests on 17 datasets
improvement in classification quality vs. higher computationalcomplexity
ELM more successfull than ELA
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Assessing Confidence Measures
which confidence measure is better?
will a confidence measure bring improvement in theclassification quality?
are the benefits of a dynamic classifier system worth biggercompuational complexity?
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
OK/NOK Histograms
confidence measure = probability of correct classification?
distribution of correctly classified patterns (OK)
distribution of correctly classified patterns (NOK)
OK and NOK should not overlap and should be separated
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
OK/NOK Histograms – Waveform
0.0 0.2 0.4 0.6 0.8 1.00
2000
4000
6000
8000
10000ELA
(d) ELA – bad separation
0.0 0.2 0.4 0.6 0.8 1.00
5000
10000
15000
20000
25000
30000ELM
(e) ELM – relatively good separation
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
OK/NOK Histograms – Phoneme
0.0 0.2 0.4 0.6 0.8 1.00
5000
10000
15000
20000
25000
30000
35000
40000
45000ELA
(a) ELA
0.0 0.2 0.4 0.6 0.8 1.00
10000
20000
30000
40000
50000ELM
(b) ELM
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Restricted Histograms
sometimes the OK/NOK histograms do not correlate withimprovement in the prediction
for most of the patterns, the classifiers give similar outputs ⇒the OK/NOK separation is irrelevant
restrict the testing set to “unclear” patterns U(s)
0 ≤ s ≤ r . . . degree of consensus
for any class, at most s classifiers vote for the class
OK/NOK histograms restricted to U(s)
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Restricted Histograms – Waveform
0.0 0.2 0.4 0.6 0.8 1.001234567
ELA, s=7
0.0 0.2 0.4 0.6 0.8 1.002468
1012
ELA, s=8
0.0 0.2 0.4 0.6 0.8 1.00
102030405060
ELA, s=9
0.0 0.2 0.4 0.6 0.8 1.00
50100150200250300350400
ELA, s=10
0.0 0.2 0.4 0.6 0.8 1.00
200400600800
10001200
ELA, s=11
0.0 0.2 0.4 0.6 0.8 1.00
200400600800
10001200140016001800
ELA, s=12
0.0 0.2 0.4 0.6 0.8 1.00
5001000150020002500
ELA, s=13
0.0 0.2 0.4 0.6 0.8 1.00
500100015002000250030003500
ELA, s=14
0.0 0.2 0.4 0.6 0.8 1.00
5001000150020002500300035004000
ELA, s=15
0.0 0.2 0.4 0.6 0.8 1.00
10002000300040005000
ELA, s=16
0.0 0.2 0.4 0.6 0.8 1.00
100020003000400050006000
ELA, s=17
0.0 0.2 0.4 0.6 0.8 1.00
1000200030004000500060007000
ELA, s=18
0.0 0.2 0.4 0.6 0.8 1.00
10002000300040005000600070008000
ELA, s=19
0.0 0.2 0.4 0.6 0.8 1.00
2000400060008000
10000ELA, s=20
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Restricted Histograms – Waveform
0.0 0.2 0.4 0.6 0.8 1.002468
10121416
ELM, s=7
0.0 0.2 0.4 0.6 0.8 1.00
1020304050
ELM, s=8
0.0 0.2 0.4 0.6 0.8 1.00
20406080
100120140160
ELM, s=9
0.0 0.2 0.4 0.6 0.8 1.00
50100150200250300350
ELM, s=10
0.0 0.2 0.4 0.6 0.8 1.00
100200300400500600700800
ELM, s=11
0.0 0.2 0.4 0.6 0.8 1.00
200400600800
10001200
ELM, s=12
0.0 0.2 0.4 0.6 0.8 1.00
200400600800
1000120014001600
ELM, s=13
0.0 0.2 0.4 0.6 0.8 1.00
500
1000
1500
2000ELM, s=14
0.0 0.2 0.4 0.6 0.8 1.00
50010001500200025003000
ELM, s=15
0.0 0.2 0.4 0.6 0.8 1.00
50010001500200025003000350040004500
ELM, s=16
0.0 0.2 0.4 0.6 0.8 1.00
10002000300040005000600070008000
ELM, s=17
0.0 0.2 0.4 0.6 0.8 1.00
2000400060008000
100001200014000
ELM, s=18
0.0 0.2 0.4 0.6 0.8 1.00
5000
10000
15000
20000ELM, s=19
0.0 0.2 0.4 0.6 0.8 1.00
50001000015000200002500030000
ELM, s=20
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Restricted Histograms – Phoneme
0.0 0.2 0.4 0.6 0.8 1.00
50
100
150
200
250ELA, s=10
0.0 0.2 0.4 0.6 0.8 1.00
100200300400500600700
ELA, s=11
0.0 0.2 0.4 0.6 0.8 1.00
200400600800
10001200
ELA, s=12
0.0 0.2 0.4 0.6 0.8 1.00
200400600800
1000120014001600
ELA, s=13
0.0 0.2 0.4 0.6 0.8 1.00
500
1000
1500
2000ELA, s=14
0.0 0.2 0.4 0.6 0.8 1.00
50010001500200025003000
ELA, s=15
0.0 0.2 0.4 0.6 0.8 1.00
500100015002000250030003500
ELA, s=16
0.0 0.2 0.4 0.6 0.8 1.00
5001000150020002500300035004000
ELA, s=17
0.0 0.2 0.4 0.6 0.8 1.00
1000
2000
3000
4000
5000ELA, s=18
0.0 0.2 0.4 0.6 0.8 1.00
10002000300040005000600070008000
ELA, s=19
0.0 0.2 0.4 0.6 0.8 1.00
50001000015000200002500030000350004000045000
ELA, s=20
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Restricted Histograms – Phoneme
0.0 0.2 0.4 0.6 0.8 1.00
20406080
100120140160180
ELM, s=10
0.0 0.2 0.4 0.6 0.8 1.00
50100150200250300350400
ELM, s=11
0.0 0.2 0.4 0.6 0.8 1.00
100200300400500600700
ELM, s=12
0.0 0.2 0.4 0.6 0.8 1.00
200
400
600
800
1000ELM, s=13
0.0 0.2 0.4 0.6 0.8 1.00
200400600800
100012001400
ELM, s=14
0.0 0.2 0.4 0.6 0.8 1.00
200400600800
10001200140016001800
ELM, s=15
0.0 0.2 0.4 0.6 0.8 1.00
500
1000
1500
2000
2500ELM, s=16
0.0 0.2 0.4 0.6 0.8 1.00
50010001500200025003000
ELM, s=17
0.0 0.2 0.4 0.6 0.8 1.00
5001000150020002500300035004000
ELM, s=18
0.0 0.2 0.4 0.6 0.8 1.00
2000
4000
6000
8000
10000ELM, s=19
0.0 0.2 0.4 0.6 0.8 1.00
10000
20000
30000
40000
50000ELM, s=20
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
ROC analysis
convert the histograms to a ROC curve
summary – area under ROC (AUC)
AUC = 50% – random guessing
AUC = 100% – ideal
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
OK/NOK ROCs – Waveform
0.0 0.2 0.4 0.6 0.8 1.00.0
0.2
0.4
0.6
0.8
1.0ELA
s=7, AUC=0.50
s=8, AUC=0.49
s=9, AUC=0.51
s=10, AUC=0.50
s=11, AUC=0.51
s=12, AUC=0.51
s=13, AUC=0.52
s=14, AUC=0.53
s=15, AUC=0.55
s=16, AUC=0.56
s=17, AUC=0.58
s=18, AUC=0.61
s=19, AUC=0.64
s=20, AUC=0.67
(a) ELA
0.0 0.2 0.4 0.6 0.8 1.00.0
0.2
0.4
0.6
0.8
1.0ELM
s=7, AUC=1.00
s=8, AUC=0.94
s=9, AUC=0.97
s=10, AUC=0.78
s=11, AUC=0.76
s=12, AUC=0.77
s=13, AUC=0.79
s=14, AUC=0.80
s=15, AUC=0.82
s=16, AUC=0.84
s=17, AUC=0.87
s=18, AUC=0.89
s=19, AUC=0.90
s=20, AUC=0.91
(b) ELM
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
OK/NOK ROCs – Phoneme
0.0 0.2 0.4 0.6 0.8 1.00.0
0.2
0.4
0.6
0.8
1.0ELA
s=10, AUC=0.52
s=11, AUC=0.52
s=12, AUC=0.52
s=13, AUC=0.52
s=14, AUC=0.53
s=15, AUC=0.53
s=16, AUC=0.54
s=17, AUC=0.55
s=18, AUC=0.58
s=19, AUC=0.62
s=20, AUC=0.77
(a) ELA
0.0 0.2 0.4 0.6 0.8 1.00.0
0.2
0.4
0.6
0.8
1.0ELM
s=10, AUC=0.40
s=11, AUC=0.52
s=12, AUC=0.52
s=13, AUC=0.54
s=14, AUC=0.57
s=15, AUC=0.60
s=16, AUC=0.63
s=17, AUC=0.66
s=18, AUC=0.71
s=19, AUC=0.76
s=20, AUC=0.85
(b) ELM
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Improvement vs. AUC
AUC for U(s) covering 5% of the data
improvement of DWM over MV
better AUC?⇒ better improvement?
scatterplot for 17 datasets
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Improvement vs. AUC
50 60 70 80 90 100AUC [%]
20
0
20
40
60
80
Impro
vem
ent
[%]
ELAELA fitELMELM fit
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Classifier CombiningClassification Confidence
Classifier SystemsExperiments
Assessing Confidence Measures
Conclusions & Future Work
formalism of classifier systems with classification confidence
3 types of classifier systems
experiments with QDCs and RFs: dynamic systems canoutperform both static and confidence-free systems
methods for assessing confidence measures
future work: different classifier types (SVM)
future work: more advanced aggregation methods
better methods for assessing confidence measures
David Stefka Dynamic Classifier Systems for Classifier Aggregation
Top Related