L ++ An Ensemble of Classifiers Approach for the Missing Feature Problem Using learn ++ IEEE Region...

L++

An Ensemble of Classifiers Approach for the Missing Feature Problem Using learn++

IEEE Region 2 Student Paper ContestUniversity of Maryland Eastern Shore

April 5th, 2003Stefan Krause

Rowan University

This material is based upon work supported by the National Science Foundation under Grant No ECS-0239090. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Project Advisor: Dr. Robi Polikar

Branch Counselor: Dr. Shreekanth Mandayam

L++

Overview

• Background

• Problem Definition

• Motivation

• Approach and Theory

• Databases and Results

• Conclusions

• References

• Questions

L++

Background

Pattern recognition

– Recognizing and classifying a previously seen / familiar pattern

0 1 2 3 4 5 6 7 8 9

A classifier is necessary for automated machine recognition of patterns

Background

Problem Definition

Motivation

Approach and Theory

Databases and Results

Conclusion

References

Questions

2 2 2 2

2 2 2

2 2 2

L++

Background

Artificial neural network

– An artificial neural network (ANN) is an algorithmicmodel of the brain, albeit very crude, to allow a computer to emulate the brain’s decision making capability

2 ……

f1f2f3

f63f64

f1 f2 f3

f63 f64

…

……

C0

C1

C2

C8

C9

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++

Problem Definition

The missing feature problem

– The missing feature problem occurs when instances from a data set have features that are missing or corrupted

2 ……

f1f2f3

f63f64

f1 f2 f3

f63 f64

…

……

C0

C1

C2

C8

C9

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions ?

L++

Motivation

• Neural networks can only produce a valid classification when all features used for creating the network are available.

• Sensor failure / malfunction or corrupt data is very common in sensor based applications where multiple sensors are observing an event.

• Solving the missing feature problem adds considerable robustness to a data classification algorithm.

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

The missing feature problem is a significant issue in computational and machine learning because:

L++

Approach and Theory

• Learn++ automated classification algorithm

– Ensemble based incremental learning

– Modified for the missing feature problem

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++

Approach and Theory

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

classifier 1classifier 1



Complex decisionComplex decision boundary to be learnedboundary to be learned

OO

OO

OOOO

OOOO

OO

OO

OOOO


XXXX

XXXX

XXXX

XX

XXXX

XX

XXXX

XXXX

XX

XXXX

XX

XX

XX

XX

XXXX

XXXX

XXXX

XXXX

XX

XX

XXXX

XX

XX

XX

XXXXXX

OO

OO OO

OO

OOOO

OOOOOO

OO OOOO OO

OO

OOOO

OO

OOOO

OOOO

OO OOOO

OO

OO

OO

OO

OOOO

OO OOOO

OOOOOO OO

OO

OOOO OO

OOOOOO

OO

OOOOOO

OO

OO

OO

OO

OO

OO

OO

OO

OO

OO

OOOO

OO OOOO

OOOOOO

OO OOOO

OO

OO OOOO

OOOO

OOOO

OO OOOO

OO

OOOO

OO

OO

OO

OO

OO

OO

XX

XX

XX XX

XX

XXXX

XX

OO

OO

OOOO

OO

OO

OOOO

L++

Approach and Theory

Traditional ensemble of classifiers approach

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10f1 f1 f1 f1 f1 f1 f1 f1 f1 f1f2 f2 f2 f2 f2 f2 f2 f2 f2 f2f3 f3 f3 f3 f3 f3 f3 f3 f3 f3f4 f4 f4 f4 f4 f4 f4 f4 f4 f4f5 f5 f5 f5 f5 f5 f5 f5 f5 f5f6 f6 f6 f6 f6 f6 f6 f6 f6 f6

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10f1 X f1 X X f1 X f1 f1 XX f2 X X f2 X f2 f2 X XX X f3 X f3 X f3 X f3 f3X f4 X f4 f4 f4 X X f4 f4f5 X f5 f5 X f5 f5 X X f5f6 f6 X f6 X X X f6 X X

Approach and Theory

Creating networks in the ensemble with only some featuresBackground

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10f1 X f1 X X f1 X f1 f1 XX f2 X X f2 X f2 f2 X XX X f3 X f3 X f3 X f3 f3X f4 X f4 f4 f4 X X f4 f4f5 X f5 f5 X f5 f5 X X f5f6 f6 X f6 X X X f6 X X

Approach and Theory

Classifying an instance that is missing f2Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++


Gas Identification Database

Identification of 5 volatile organic compounds using 6 quartz crystal microbalance sensors.

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++


Gas Identification Database

Percentage of Missing Actual Used ClassificationFeatures in Test Data Networks Networks Performance

0.00% 50 50 81.4%2.50% 53 50 81.4%5.00% 56 50 81.4%7.50% 59 50 81.4%

10.00% 62 50 81.4%

33.3% (2 of 6) of features used for training

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++


Optical Character Recognition Database

Identification of handwritten characters of the numbers 0 through 9.

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++


Optical Character Recognition Database


0.00% 59 59 94.5%2.50% 80 59 94.5%5.00% 110 59 95.1%7.50% 149 59 92.3%

10.00% 210 59 93.7%


Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++


Ionosphere Radar Return Database

This system consists of a phased array of 16 high-frequency antennas with a total transmitted power on the order of 6.4

kilowatts. The targets were free electrons in the ionosphere.

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++


Ionosphere Radar Return Database


0.00% 34 34 94.7%


0.00% 53 53 94.7%2.50% 67 53 94.7%5.00% 85 53 94.7%7.50% 106 53 93.9%

10.00% 142 53 95.4%

100% (34 of 34) of features used for training


Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++

Conclusions

• Initial results indicate that the algorithm is capable of classifying data, even with up to 10% missing features, with virtually no drop off in performance.

• The mathematical equations for the algorithm as well as a flow chart describing the algorithm can be found in the paper.

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++

References

R. Polikar, L. Udpa, S. Udpa, and V. Honavar, “Learn++: an incremental learning algorithm for supervised neural networks,” IEEE Tran. Systems, Man and Cybernetics, C, vol. 31, no. 4, pp. 497-508, 2001.

R. Polikar, J. Byorick, S. Krause, A. Marino and M. Moreton, “Learn++: A Classifier Independent Incremental Learning Algorithm for Supervised Neural Networks,” Proc. Int. Joint Conf. Neural Networks (IJCNN2002), vol. 2 , pp. 1742-1747, Honolulu, HI, 2002.

L.K. Hansen and P. Salamon, “Neural network ensembles,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 10, pp. 993-1001, 1990.

Y. Freund and R. Schapire, “A decision theoretic generalization of on-line learning and an application to boosting,” Computer and System Sciences, vol. 57, no. 1, pp. 119-139, 1997

C.L. Blake and C.J. Merz, UCI Repository of machine learning databases at http://www.ics.uci.edu/~mlearn/ MLRepository.html. Irvine, CA: University of California, Dept. of In-formation and Computer Science, 1998.

R. Polikar, R. Shinar, L. Udpa, M. Porter, “Artificial intelligence Methods for Selection of an Optimized Sensor Array for Identification of Volatile Organic Compounds,” Sensors and Actuators B: Chemical, Volume 80, Issue 3, pp 243-254, December 2001.

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L++

Questions

This presentation and the paper are available online at:http://engineering.rowan.edu/~polikar/RESEARCH/PUBLICATIONS/publications.html

Background

Problem Definition

Motivation

Approach and Theory


Conclusions

References

Questions

L ++ An Ensemble of Classifiers Approach for the Missing Feature Problem Using learn ++ IEEE Region...

Documents

Transcript of L ++ An Ensemble of Classifiers Approach for the Missing Feature Problem Using learn ++ IEEE Region...