Pattern Recognition Artificial Neural Networks, and Machine...
Transcript of Pattern Recognition Artificial Neural Networks, and Machine...
![Page 1: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/1.jpg)
Pattern Recognition
Artificial Neural Networks,
and Machine LearningYuan-Fang Wang
Department of Computer Science
University of California
Santa Barbara, CA 93106, USA
![Page 2: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/2.jpg)
2PR , ANN, & ML
“Pattern Recognition”
What is a Pattern?
![Page 3: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/3.jpg)
3PR , ANN, & ML
![Page 4: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/4.jpg)
4PR , ANN, & ML
![Page 5: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/5.jpg)
5PR , ANN, & ML
![Page 6: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/6.jpg)
6PR , ANN, & ML
DNA patterns
AGCTCGAT
Protein Patterns
20 amino acids
![Page 7: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/7.jpg)
7PR , ANN, & ML
![Page 8: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/8.jpg)
8PR , ANN, & ML
![Page 9: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/9.jpg)
9PR , ANN, & ML
![Page 10: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/10.jpg)
10PR , ANN, & ML
Faces
Finger prints
![Page 11: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/11.jpg)
11PR , ANN, & ML
Other Patterns
Insurance, credit card applications
applicants are characterized by a pattern
# of accidents, make of car, year of model
income, # of dependents, credit worthiness,
mortgage amount
Dating services
Age, hobbies, income, etc. establish your
“desirability”
![Page 12: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/12.jpg)
12PR , ANN, & ML
Other Patterns
Web documents
Key words based description (e.g., documents containing War, Bagdad, Hussen are different from those containing football, NFL, AFL, draft, quarterbacks)
Intrusion detection
Usage and connection patterns
Cancer detection
Image features for tumors, patient age, treatment option, etc.
![Page 13: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/13.jpg)
13PR , ANN, & ML
Other Patterns
Housing market
Location, size, year, school district
University ranking
Student population, student-faculty ratio,
scholarship opportunities, location, faculty research
grants, etc.
Too many
E.g.,
http://www.ics.uci.edu/~mlearn/MLSummary.html
![Page 14: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/14.jpg)
14PR , ANN, & ML
What is a pattern?
A pattern is a set of objects, processes or
events which consist of both deterministic
and stochastic components
A pattern is a record of certain dynamic
processes influenced both by deterministic
and stochastic factors
![Page 15: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/15.jpg)
15PR , ANN, & ML
What is a Pattern? (cont.)
Completely regular,
deterministic
(e.g., crystal structure)
Completely
random
(e.g., white noise)
Constellation patterns,
texture patterns, EKG
patterns, etc.
![Page 16: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/16.jpg)
16PR , ANN, & ML
Classifies “patterns” into “classes”
Patterns (x)
have “measurements”, “traits”, or “features”
Classes ( )
likelihood (a prior probability )
class-conditional density
Classifier (f(x) -> )
An example
four coin classes: penny, nickel, dime, and quarter
measurements: weight, color, size, etc.
Assign a coin to a class based on its size, weight, etc.
P i( ) i
p x i( | )
What is Pattern Recognition?
i
We use P to denote probability mass function (discrete) and
p to denote probability density function (continuous)
![Page 17: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/17.jpg)
17PR , ANN, & ML
An Example
Many visual inspection systems are like this:
Circuit board, fruit, OCR, etc.
![Page 18: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/18.jpg)
18PR , ANN, & ML
Another Example
![Page 19: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/19.jpg)
19PR , ANN, & ML
Features
The intrinsic traits or characteristics that tell
one pattern (object) apart from another
Features extraction and representation allows
Focus on relevant, distinguishing parts of a pattern
Data reduction and abstraction
![Page 20: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/20.jpg)
20
Detection vs. Description
Detection: something
happened
Heard noise
Saw something
interesting
Non-flat signals
Description: what has
happened?
Gun shot, talking,
laughing, crying, etc.
Lines, corners,
textures
Mouse, cat, dog, bike,
etc.
PR , ANN, & ML
![Page 21: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/21.jpg)
21PR , ANN, & ML
Feature Selection
More an art than a science
Effectiveness criteria:
size
population
Size alone is not effective
![Page 22: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/22.jpg)
compactness
perimeter
Perimeter is not effective
Discrimination is accomplished by compactness alone
compactness
elongatedness
The two feature values are correlated, only one of them
is needed
![Page 23: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/23.jpg)
23PR , ANN, & ML
![Page 24: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/24.jpg)
24PR , ANN, & ML
![Page 25: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/25.jpg)
25PR , ANN, & ML
Too simple Too complicated
![Page 26: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/26.jpg)
26PR , ANN, & ML
Optimal tradeoff between performance and generalization
![Page 27: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/27.jpg)
27
Importance of Features
Cannot be over-stated
We usually don’t know which to select,
what they represent, and how to tune them
(face, gait recognition, tumor detection, etc.)
Classification and regression schemes are
mostly trying to make the best of whatever
features are available
PR , ANN, & ML
![Page 28: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/28.jpg)
28
Features
One is usually not descriptive (no silver
bullet)
Many (shotgun approach) can actually hurt
Many problems: Relevance
Dimensionality
Co-dpendency
Time and space varying characteristics.
Accuracy
Uncertainty and error
Missing values
PR , ANN, & ML
![Page 29: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/29.jpg)
29PR , ANN, & ML
Feature Selection (cont.)
Q: How to decide if a feature is effective?
A: Through a training phase
Training on typical samples and typical features
to discover
Whether features are effective
Whether there are any redundancy
The typical cluster shape (e.g., Gaussian)
Decision boundaries between samples
Cluster centers of particular samples
Etc.
![Page 30: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/30.jpg)
30PR , ANN, & ML
Classifiers
)(xng
i i j
i i
i i
i i
if g x g x for all j i
g x P
g x P x
g x R x
( ) ( )
( ) ( )
( ) ( | )
( ) ( | )
if no measurements are made
minimize misclassification rate
minimize associated risk
x max decision
g1
g2
gn
)(1 xg
)(2 xg
![Page 31: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/31.jpg)
31PR , ANN, & ML
Traditional Pattern Recognition
Parametric methods
Based on class sample exhibiting a certain
parametric distribution (e.g. Gaussian)
Learn the parameters through training
Density methods
Does not enforce a parametric form
Learn the density function directly
Decision boundary methods
Learn the separation in the feature space
![Page 32: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/32.jpg)
32PR , ANN, & ML
size
I. population
size
II. population
size
III. population
size
IV. population
1
2 2
1
2
2
2
( ) /
| |
n n
x x
e
Parametric Methods
![Page 33: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/33.jpg)
33PR , ANN, & ML
Density Methods
![Page 34: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/34.jpg)
34PR , ANN, & ML
Feature space
d dimensional (d the number of features)
populated with features from training samples
f1
f2
fd
![Page 35: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/35.jpg)
35PR , ANN, & ML
Decision
Boundary
Methods
f1
f2
fd
?
• Decision surfaces • Cluster centers
f1
f2
fd
?
f1
f2
fd
?
![Page 36: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/36.jpg)
36PR , ANN, & ML
![Page 37: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/37.jpg)
37
“Modern” vs “Traditional”
Pattern Recognition Hand-crafted features
Simple and low-level
concatenation of
numbers or traits
Syntactic
Feature detection and
description are
separate tasks from
classifier design
Automatically learned
features
Hierarchical and
complex
Semantic
Feature detection and
description are not
jointly optimized with
classifiers
PR , ANN, & ML
![Page 38: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/38.jpg)
38
Traditional Features
PR , ANN, & ML
![Page 39: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/39.jpg)
39
Modern Features
PR , ANN, & ML
![Page 40: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/40.jpg)
40
Modern Features
PR , ANN, & ML
![Page 41: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/41.jpg)
41
Modern Features
PR , ANN, & ML
![Page 42: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/42.jpg)
42
Modern Features
PR , ANN, & ML
![Page 43: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/43.jpg)
43
Modern Features
PR , ANN, & ML
![Page 44: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/44.jpg)
44
“Modern” vs “Traditional”
Pattern Recognition
PR , ANN, & ML
![Page 45: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/45.jpg)
45PR , ANN, & ML
Mathematical Foundation
Does not matter what methods or
techniques you use, the underlying
mathematical principle is quite simple
Bayesian theory is the foundation
![Page 46: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/46.jpg)
46PR , ANN, & ML
Review: Bayes Rule
Forward (synthesis) route:
From class to sample in a class
Grammar rules to sentences
Markov chain (or HMM) to pronunciation
Texture rules (primitive + repetition) to textures
Backward (analysis) route:
From sample to class ID
A sentence parsed by a grammar
A utterance is “congratulations” (not “constitution”)
Brickwall vs. plaid shirt
![Page 47: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/47.jpg)
47PR , ANN, & ML
Review: Bayes Rule
Backward is always harder
Because the interpretation is not unique
Presence of x has multiple possibilities
12
3
nx
![Page 48: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/48.jpg)
48PR , ANN, & ML
The simplest example
Two classes: pennies and dimes
No measurements
Classification:
based on the a prior probabilities
Error rate:
min( ( ), ( ))P P 1 2
1 1 2
2 1 2
1 2
if P P
if P P
or otherwise
( ) ( )
( ) ( )
12
![Page 49: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/49.jpg)
49PR , ANN, & ML
x
A slightly more complicated example
Two classes: pennies and dimes
A measurement x is made (e.g. weight)
Classification
based on the a posterior probabilities with
Bayes rule
1 1 2
2 1 2
1 2
if P x P x
if P x P x
or otherwise
P xp x
p x
p x P
p xi
i i i
( | ) ( | )
( | ) ( | )
( | )( , )
( )
( | ) ( )
( )
12
![Page 50: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/50.jpg)
50PR , ANN, & ML
weight
probability
p x( | )2p x( | )1
weight
probability
p x P( | ) ( ) 2 2p x P( | ) ( ) 1 1
P( )1 P( )2
p x( ) p x( )
weight
probability
p x( | )1 p x( | )2
![Page 51: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/51.jpg)
51PR , ANN, & ML
Why Both?
In the day time, some animal runs in front of
you on the bike path, you know exactly what it
is (p(x|w) is sufficient)
In the night time, some animal runs in front of
you on the bike path, you can hardly distinguish
the shape (p(x|w) is low for all cases, but you
know it is probably a squirrel, not a lion
because of p(w))
?)(&)|( ii Pxp
![Page 52: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/52.jpg)
52PR , ANN, & ML
Essence
Turn a backward (analysis) problem into
several forward (synthesis) problem
Or analysis-by-synthesis
Whichever model has a highly likelihood of
synthesizing the outcome wins
The formula is not mathematically provable
![Page 53: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/53.jpg)
53PR , ANN, & ML
Error rate
Determined by
The likelihood of a class
The likelihood of measuring x in a class
min( ( | ), ( | ))P x P x or 1 2
11 1 2 2
p xp x P p x P
( )min( ( | ) ( ), ( | ) ( ))
![Page 54: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/54.jpg)
54PR , ANN, & ML
Error Rate (cont.)
Bayes Decision Rule minimizes the average
error rate:
)|(maxarg
)|(1)|()|(
)()|(
*
)(
*
)(*
)(
xp
where
xpxpxerrorp
dxxpxerrorperror
ii
x
xi
xi
![Page 55: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/55.jpg)
55
Various types of errors
PR , ANN, & ML
![Page 56: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/56.jpg)
56PR , ANN, & ML
![Page 57: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/57.jpg)
57
Precision vs. Recall
A very common measure used in PR and
MI community
One goes up and the other HAS to go down
A range of options (Receiver operating
characteristic curves)
Area under the curve
as a goodness measure
PR , ANN, & ML
![Page 58: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/58.jpg)
58
Various ways to measure error rates
Training error
Test error
Empirical error
Some under your control (training and test)
Some not (empirical error)
How: n-fold validation
Why: Overfitting and underfitting problems
PR , ANN, & ML
![Page 59: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/59.jpg)
59PR , ANN, & ML
An even more complicated example
Two classes: pennies or dimes
A measurement x is made
Risk associated with making a wrong decision
Based on the a posterior probabilities with
Bayesian risk
R x P x P x
R x P x P x
thelossof action in state
R x theconditional risk of action with x
ij i j
i i
( | ) ( | ) ( | )
( | ) ( | ) ( | )
:
( | ):
1 11 1 12 2
2 21 1 22 2
![Page 60: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/60.jpg)
60PR , ANN, & ML
Observation
State1
Observation
State2
Decision1 Decision2
Mis-classification
Math
Mis-interpretation
Human factor
![Page 61: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/61.jpg)
61PR , ANN, & ML
Observation
State1 State2
Observation
Decision1 Decision2
Observation
State1 State2
Decision1 Decision2 Decision1 Decision2
Incorrect decisions
Incur domain-specific cost
![Page 62: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/62.jpg)
62PR , ANN, & ML
R(used as pennies | x) =
r(pennies used as pennies) * P(pennies | x) +
r(dimes used as pennies) * P(dimes | x)
R(used as dimes | x) =
r(pennies used as dimes) * P(pennies | x) +
r(dimes used as dimes) * P(dimes | x)
An even more complicated example
p(x|pennies)P(pennies)
p(x|dimes)P(dimes)
![Page 63: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/63.jpg)
63PR , ANN, & ML
A more credible example
R(call FD|smoke) =
r(call,fire)*P(fire|smoke) +
r(call, no fire)*P(no fire|smoke)
R(no call FD|smoke)=
r(no call, no fire)*P(no fire|smoke) +
r(no call, fire)*P(fire|smoke)
The risk associated with false negative is
much higher than that of false positive
False positive
False negative
![Page 64: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/64.jpg)
64PR , ANN, & ML
A more credible example
R(attack|battle field intelligence) =
r(attack,<50%)*P(<50%|intelligence) +
r(attack,>50%)*P(>50%|intelligence)
R(no attack|battle field intelligence)=
r(no attack, >50%)*P(>50%|intelligence) +
r(no attack, <50%)*P(<50%|intelligence)
False positive
False negative
![Page 65: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/65.jpg)
65PR , ANN, & ML
Baysian Risk Determined by
likelihood of a class
likelihood of measuring x in a class
the risk of making a wrong action
Classification
Baysian risk should be minimized
)|()()|()()|()|(
))|()|(),|()|(min())|(),|(min(
2221211121
121
222121212111
21
xPxPxRxR
orxPxPxPxPorxRxR
![Page 66: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/66.jpg)
66PR , ANN, & ML
Bayesian Risk (cont.)
Again, decisions depend on
likelihood of a class
likelihood of observation of x in a class
Modified by some positive risk factors
Why?
Because in the real world, it might not be the
misclassification rate that is important, it is the
action you assume
)|()()|()( 2221211121 xPxP
![Page 67: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/67.jpg)
67PR , ANN, & ML
Other generalizations
Multiple classes
n classes
Multiple measurements
X is a vector instead of a scalar
Non-numeric measurements
Actions vs. decisions
Correlated vs. independent events
speech signals and images
Training allowed or not
Time-varying behaviors
P ii
n
( )
11
![Page 68: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/68.jpg)
68PR , ANN, & ML
Difficulties
What features to use
How many features (the curse of
dimensionality)
The a prior probability
The class-conditional density
The a posterior probability
)( iP
)|( xP i
p x i( | )
![Page 69: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/69.jpg)
69PR , ANN, & ML
Typical Approaches
Supervised (with tagged samples x):
parameters of a probability function (e.g. Gaussian )
density functions (w/o assuming any parametric forms)
decision boundaries (classes are indeed separable)
Unsupervised (w/o tagged samples x):
minimum distance
hierarchical clustering
Reinforced (with hints)
Right or wrong, but not correct answer
Learning with a critic (not a teacher as in supervised)
p x Ni i i( | ) ( , )
![Page 70: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/70.jpg)
70PR , ANN, & ML
Parameter Density Decision Boundary
Supervised Learning
Minimum Distance Hierarchical Clustering
Un-supervised Clustering
Uncorrelated Events
Hidden Markov Models
Correlated Events
Pattern Recognition
![Page 71: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/71.jpg)
71PR , ANN, & ML
Applications
DNA sequence
Lie detectors
Handwritten digits recognition
Classification based on smell
Web document classification and search engine
Defect detection
Texture classification
Image database retrieval
Face recognition
etc.
![Page 72: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/72.jpg)
72
Other formulations
We talked about 1/3 of the scenarios – that
of classification (discrete)
Regression – continuous
Extrapolation and interpolation
Clustering
Similarity
Abnormality detection
Concept drift (discovery), etc.
PR , ANN, & ML
![Page 73: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/73.jpg)
73
Classification vs. Regression
Classification
Large vs. small hints
on category
Absolute values does
not matter as much
(can actually hurt)
Normalization is often
necessary
Fitting order stays low
Regression
Large means large,
small means small
Absolute values matter
Fitting orders matter
PR , ANN, & ML
![Page 74: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/74.jpg)
74
Recent Development
Data can be “massaged” Surprisingly,
massaging the data and use simple
classifiers is better than massaging the
classifiers and use simple data (for simple
problems & small data sets)
Hard-to-visualize concept
Transform data into higher dimensional space
(e.g., infinite dimensional) has a tendency to
separate data and increase error margin
Concept of SVM and later kernel methods
![Page 75: Pattern Recognition Artificial Neural Networks, and Machine Learningyfwang/courses/cs290i_prann/pdf/in… · · 2018-01-11Pattern Recognition Artificial Neural Networks, and Machine](https://reader034.fdocuments.in/reader034/viewer/2022052319/5acc1ce67f8b9a6a678bf2ad/html5/thumbnails/75.jpg)
75
More Recent Development
Think about fitting linear data with a model
Linear, quadratic, cubic, etc.
Higher the order, better the fit
n data points can be perfectly fit by an (n-1) order
polynomial
However
Overfitting is likely
No ability to extrapolate
“Massage” the classifiers (using deep networks)
Feature detection and description
Classification
Jointly optimization
PR , ANN, & ML