Informed Search ECE457 Applied Artificial Intelligence Spring 2007 Lecture #3.
Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion...
Transcript of Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion...
![Page 1: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/1.jpg)
Applied Artificial Intelligence
Session 9: Homework Discussion
Linear Algebra for AI and Machine Learning II
Fall 2018
NC State University
Lecturer: Dr. Behnam Kia
Course Website: https://appliedai.wordpress.ncsu.edu/
1
Sep 25, 2018
![Page 2: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/2.jpg)
Homework 3: Document Classification with Naïve
Bayes Classifier
Discussion
2
![Page 3: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/3.jpg)
Probability Distribution Function
• Probability distribution function is a description of how likely a random variable or a set of variables is to take on each of its possible states.
33X=H X=T
0.5
P(X) Flipping a CoinP(X)
Cat’s weight
9lbs
![Page 4: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/4.jpg)
Homework 3
• You calculated conditional probability distribution function for each class:
4C=Positive C=Negative
p(C|Review) P(CN|Review)
P(CP|Review)
![Page 5: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/5.jpg)
Homework 3
• You calculated conditional probability distribution function for each class:
• And picked the class with the highest conditional probability.
5C=Positive C=Negative
p(C|Review) P(CN|Review)
P(CP|Review)
![Page 6: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/6.jpg)
Homework 3
• You calculated conditional probability distribution function for each class:
! " #$%&$' = ! #$%&$' " !(")!(#$%&$')
6C=Positive C=Negative
p(C|Review) P(CN|Review)
P(CP|Review)
posterior = likelihood × priorevidence
![Page 7: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/7.jpg)
Homework 3
• You calculated conditional probability distribution function for each class:
! " #$%&$' = ! #$%&$' " !(")!(#$%&$')
7C=Positive C=Negative
p(C|Review) P(CN|Review)
P(CP|Review)
posterior = likelihood × priorevidence
Naïve Bayes to Approximate
![Page 8: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/8.jpg)
What is the Error Rate?
8C=Positive C=Negative
p(C|Review) P(CN|Review)
P(CP|Review)
• You calculated conditional probability distribution function for each class:
• And picked the class with the highest conditional probability.
![Page 9: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/9.jpg)
What is the Error Rate?
9C=Positive C=Negative
p(C|Review) P(CN|Review)
P(CP|Review)
• You calculated conditional probability distribution function for each class:
• And picked the class with the highest conditional probability.
Error Rate
![Page 10: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/10.jpg)
Bayes Error Rate
10C=Positive C=Negative
p(C|Review) P(CN|Review)
P(CP|Review)
• You calculated conditional probability distribution function for each class:
• And picked the class with the highest conditional probability.
• Bayes error rate is the lowest possible error rate for ANY classifier.
Bayes Error Rate
![Page 11: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/11.jpg)
Performance of Classifiers
11
Error
Bayes Error Rate
Training Data Size
Learner 1
Learner 2
Learner 3
Optimal Bayes Classifier
![Page 12: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/12.jpg)
Let’s don’t confuse luck with statistical performance!
12
![Page 13: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/13.jpg)
Homework 2 vs Homework 3
13
![Page 14: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/14.jpg)
How Do you Compare Homework 2 with 3?
14
Homework 2Homework 3
cNB = argmaxc∈{Positive ,Negative}
p(c) p(xi | c)i∏
![Page 15: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/15.jpg)
How Do you Compare Homework 2 with 3?
15
Homework 2Homework 3
Discriminative ModelGenerative Model
cNB = argmaxc∈{Positive ,Negative}
p(c) p(xi | c)i∏
![Page 16: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/16.jpg)
Recommended Reading
16
Andrew Y. NG, and Michael I. Jordan. "On discriminative vs. generative classifiers: A
comparison of logistic regression and naive Bayes." Advances in neural information processing
systems. 2002.
![Page 17: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/17.jpg)
On discriminative vs. generative classifiers: A comparison of logistic
regression and naive Bayes
What is a Generative model, and What is a Discriminative Model.
17
![Page 18: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/18.jpg)
On discriminative vs. generative classifiers: A comparison of logistic
regression and naive Bayes
Starting Point of the Article: Common Wisdom: “One Should Solve the Classification Problem Directly and never [?] solve a more general problem as an intermediate step [such as modeling p(Review|C)]”
18
![Page 19: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/19.jpg)
On discriminative vs. generative classifiers: A comparison of logistic
regression and naive BayesResults:1: When m, the size of training data increases:Asymptotic error of Discriminative Models≤ Asymptotic error of Generative Models
19
Naïve Bayes (Generative)
Logistic Linear Regression (Discriminative)
![Page 20: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/20.jpg)
On discriminative vs. generative classifiers: A comparison of logistic regression and
naive BayesResults:2- Generative Models converge to their asymptotic errors fasters.For small datasets:
20
Error of Generative Models ≤ Error of Discriminative Models
20
Naïve Bayes (Generative)
Logistic Linear Regression (Discriminative)
![Page 21: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/21.jpg)
Recommended Reading
21
Ng, Andrew Y., and Michael I. Jordan. "On discriminative vs. generative classifiers: A
comparison of logistic regression and naive bayes." Advances in neural information processing
systems. 2002.
![Page 22: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/22.jpg)
Linear Algebra for AI and Machine Learning II
22
![Page 23: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/23.jpg)
Computational complexity for basic PCA
• Complexity of PCA: !(#$%&' #$())where #$%& = max(#/%$012, #42%5672/)
#$()= m8#(#/%$012, #42%5672/)
23
X=0.3 ⋯ −0.1⋮ ⋱ ⋮0.9 ⋯ 0.43
#42%5672/
#/%$012/
![Page 24: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/24.jpg)
Computational complexity for basic PCA
• Computational Complexity of PCA: !(#$%&' #$())where #$%& = max(#/%$012, #42%5672/)
#$()= m8#(#/%$012, #42%5672/)
• Computational Complexity of Randomized PCA:!(#$%&' #9:$0:)2)5/)
24
ipca = PCA(n_components=ncomponents, svd_solver=“randomized”)
![Page 25: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/25.jpg)
PCA’s limitations
• It is a linear transformationUse Kernel PCA
Kernel functions [implicitly] transform the data to a higher dimensional feature space, where linear PCA can be applied.
25
kpca = KernelPCA(kernel="rbf", fit_inverse_transform=True, gamma=10)
![Page 26: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/26.jpg)
PCA’s limitations
• It is a linear transformationUse Kernel PCA
Kernel functions [implicitly] transform the data to a higher dimensional feature space, where linear PCA can be applied.
26
kpca = KernelPCA(kernel="rbf", fit_inverse_transform=True, gamma=10)
![Page 27: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/27.jpg)
PCA’s limitations
• It is a linear transformation.
• [In scikit-learn implementation of PCA] the entire data matrix should fit into the memory. – Solution: Use Incremental PCA, called I
IncrementalPCA.
27
ipca = IncrementalPCA(n_components=n_components, batch_size=10)
![Page 28: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/28.jpg)
Homework
• Go to scikit-learn website, and follow any of PCA or decomposition examples
• http://scikit-learn.org/stable/auto_examples/index.html
28
![Page 29: Applied Artificial Intelligence · Applied Artificial Intelligence Session 9: Homework Discussion Linear Algebra for AI and Machine Learning II Fall 2018 NC State University Lecturer:](https://reader030.fdocuments.in/reader030/viewer/2022040608/5ec4f97e8b02eb017a32a20c/html5/thumbnails/29.jpg)
Final Project Proposal
• Teams of three people.• (Preferably) choose a problem from your own
field to solve.• 2-minute presentation in the class (4-5
slides).• An extended executive summary (1-2 page
word document.)
29