Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self...
-
Upload
gabriel-atkins -
Category
Documents
-
view
214 -
download
1
Transcript of Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self...
![Page 1: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/1.jpg)
CS344: INTRODUCTION TO ARTIFICIAL INTELLIGENCE
Pushpak BhattacharyyaCSE Dept., IIT Bombay
Lecture 38: PAC Learning, VC dimension; Self Organization
![Page 2: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/2.jpg)
VC-dimension
Gives a necessary and sufficient condition for PAC learnability.
![Page 3: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/3.jpg)
Def:-Let C be a concept class, i.e., it has
members c1,c2,c3,…… as concepts in it.
C1
C2
C3
C
![Page 4: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/4.jpg)
Let S be a subset of U (universe).
Now if all the subsets of S can be produced by intersecting with Ci
s, then we say C shatters S.
![Page 5: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/5.jpg)
The highest cardinality set S that can be shattered gives the VC-dimension of C.
VC-dim(C)= |S|
VC-dim: Vapnik-Cherronenkis dimension.
![Page 6: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/6.jpg)
IIT Bombay 6
2 – Dim surfaceC = { half planes}
x
y
![Page 7: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/7.jpg)
IIT Bombay 7
a
|s| = 1 can be shattered
S1= { a }
{a}, Ø
y
x
![Page 8: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/8.jpg)
IIT Bombay 8
a
|s| = 2 can be shattered
b S2= { a,b }
{a,b},
{a},
{b},
Ø
x
y
![Page 9: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/9.jpg)
IIT Bombay 9
a
|s| = 3 can be shattered
b
c
x
y S3= { a,b,c }
![Page 10: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/10.jpg)
IIT Bombay 10
![Page 11: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/11.jpg)
IIT Bombay 11
A B
D C
x
y
|s| = 4 cannot be shattered
S4= { a,b,c,d }
![Page 12: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/12.jpg)
Fundamental Theorem of PAC learning (Ehrenfeuct et. al, 1989)
A Concept Class C is learnable for all probability distributions and all concepts in C if and only if the VC dimension of C is finite
If the VC dimension of C is d, then…(next page)
IIT Bombay 12
![Page 13: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/13.jpg)
Fundamental theorem (contd)
(a) for 0<ε<1 and the sample size at least max[(4/ε)log(2/δ), (8d/ε)log(13/ε)]
any consistent function A:ScC is a learning function for C(b) for 0<ε<1/2 and sample size less than max[((1-ε)/ ε)ln(1/ δ), d(1-2(ε(1- δ)
+ δ))] No function A:ScH, for any hypothesis space is a learning function for C.
IIT Bombay 13
![Page 14: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/14.jpg)
Book1. Computational Learning Theory, M. H. G.
Anthony, N. Biggs, Cambridge Tracts in Theoretical Computer Science, 1997.
Paper’s 1. A theory of the learnable, Valiant, LG (1984),
Communications of the ACM 27(11):1134 -1142.
2. Learnability and the VC-dimension, A Blumer, A Ehrenfeucht, D Haussler, M Warmuth - Journal of the ACM, 1989.
![Page 15: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/15.jpg)
SELF ORGANIZATION
![Page 16: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/16.jpg)
Self Organization
Biological Motivation
Brain
![Page 17: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/17.jpg)
Higher brain
Brain
Cerebellum
Cerebrum3- Layers: Cerebrum
Cerebellum
Higher brain
![Page 18: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/18.jpg)
Maslow’s hierarchy
Search for
Meaning
Contributing to humanity
Achievement,recognition
Food,rest
survival
![Page 19: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/19.jpg)
Higher brain ( responsible for higher needs)
Cerebrum(crucial for survival)
3- Layers: Cerebrum
Cerebellum
Higher brain
![Page 20: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/20.jpg)
Mapping of Brain
Side areasFor auditory information processing
Back of brain( vision)
Lot of resilience:
Visual and auditoryareas can do eachother’s job
![Page 21: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/21.jpg)
Left Brain and Right Brain
Dichotomy
Left Brain
Right Brain
![Page 22: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/22.jpg)
Left Brain – Logic, Reasoning, Verbal abilityRight Brain – Emotion, Creativity
Music
Words – left Brain
Tune – Right Brain
Maps in the brain. Limbs are mapped to brain
![Page 23: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/23.jpg)
,O/p grid
. . . . I/p neuron
Character Reognition
![Page 24: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/24.jpg)
KOHONEN NET• Self Organization or Kohonen network fires a group of neurons instead of a single one. • The group “some how” produces a “picture” of the cluster.• Fundamentally SOM is competitive learning. • But weight changes are incorporated on a neighborhood. • Find the winner neuron, apply weight change for the winner and its “neighbors”.
![Page 25: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/25.jpg)
Neurons on the contour are the “neighborhood” neurons.
Winner
![Page 26: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/26.jpg)
Weight change rule for SOM
W(n+1) = W(n) + η(n) (I(n) – W(n)) P+δ(n) P+δ(n) P+δ(n)
Neighborhood: function of n
Learning rate: function of nδ(n) is a decreasing function of nη(n) learning rate is also a decreasing function of n0 < η(n) < η(n –1 ) <=1
![Page 27: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/27.jpg)
Pictorially
Winnerδ(n)
. . . .
Convergence for kohonen not proved except for uni-dimension
![Page 28: Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 38: PAC Learning, VC dimension; Self Organization.](https://reader036.fdocuments.in/reader036/viewer/2022070407/56649e425503460f94b341fd/html5/thumbnails/28.jpg)
…
… … .
P neurons o/p layer
n neurons
Clusters:
AA :
A
Wp
B :
C :
: :