CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint -...
Transcript of CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint -...
![Page 1: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/1.jpg)
CS 559: Machine Learning Fundamentals and Applications
8th Set of Notes
Instructor: Philippos MordohaiWebpage: www.cs.stevens.edu/~mordohaiE-mail: [email protected]
Office: Lieb 215
1
![Page 2: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/2.jpg)
Project Proposal• Dataset
– How many instances• Classes (or what is being predicted)• Inputs
– Include feature extraction, if needed– If your inputs are images or financial data, this must
be addressed • Methods
– At least one simple classifier (MLE with Gaussian model, Naïve Bayes, kNN)
– At least one advanced classifier (SVM, Boosting, Random Forest, CNN)
2
![Page 3: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/3.jpg)
Project Proposal
• Typical experiments– Measure benefits due to advanced classifier
compared to simple classifier– Compare different classifier settings
• k in kNN• Different SVM kernels• AdaBoost vs. cascade• Different CNN architectures
– Measure effects of amount of training data available
– Evaluate accuracy as a function of the degree of dimensionality reduction using PCA
3
![Page 4: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/4.jpg)
Project Proposal
• Email me a pdf with all these• I must say “approved” in my response,
otherwise address my comments and resubmit
4
![Page 5: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/5.jpg)
Overview
• Linear Regression– Barber Ch. 17– HTF Ch. 3
5
![Page 6: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/6.jpg)
Simple Linear Regression
• How does a single variable of interest relate to another (single) variable?– Y = outcome variable (response,
dependent...)– X = explanatory variable (predictor, feature,
independent...)
• Data: n pairs of continuous observations (X1,Y1) … (Xn,Yn)
6
![Page 7: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/7.jpg)
Example• How does systolic blood pressure (SBP) relate to age?
• Graph suggests that Y relates to X in an approximately linear way
7
![Page 8: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/8.jpg)
Regression: Step by Step
1. Assume a linear model: Y = β0 + β1 X2. Find the line which “best” fits the data, i.e.
estimate parameters β0 and β1
3. Does variation in X help describe variation in Y ?
4. Check assumptions of model5. Draw inferences and make predictions
8
![Page 9: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/9.jpg)
Straight-line Plots
9
![Page 10: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/10.jpg)
Assumptions of Linear Regression
• Five basic assumptions1. Existence: for each fixed value of X, Y is
a random variable with finite mean and variance
2. Independence: the set of Yi are independent random variables given Xi
10
![Page 11: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/11.jpg)
Assumptions of Linear Regression
3. Linearity: the mean value of Y is a linear function of X
|
11
![Page 12: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/12.jpg)
Assumptions of Linear Regression
4. Homoscedasticity: the variance of Y is the same for any X
5. Normality: For each fixed value of X, Y has a normal distribution (by assumption 4, σ2 does not depend on X)
12
![Page 13: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/13.jpg)
Formulation
• Yi are linear function of Xi plus some random error
13
![Page 14: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/14.jpg)
Linear Regression
14
![Page 15: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/15.jpg)
Estimating β0 and β1
• Find “best” line• Criterion for “best”: estimate β0 and β1 to
minimize:
• This is the residual sum of squares, sum of squares due to error, or sum of squares about regression line
• Least Squares estimator15
![Page 16: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/16.jpg)
Rationale for LS Estimates
• 2 measures the “deviance” of Yi from the estimated model
• The “best” model is the one from which the data deviate the least
16
![Page 17: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/17.jpg)
Least Squares Estimators• Taking derivatives with respect to β, we obtain
• The residual variance is
17
![Page 18: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/18.jpg)
Example: SBP/age data
18
![Page 19: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/19.jpg)
Using the Model
• Using the parameter estimates, our best guess for any Y given X is
• Hence at
• Every regression line goes through ( , )• Also
19
![Page 20: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/20.jpg)
Correlation and Regression Coefficient
20
![Page 21: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/21.jpg)
Example
21
![Page 22: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/22.jpg)
Example
22
![Page 23: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/23.jpg)
Example
23
![Page 24: CS 559: Machine Learning Fundamentals and Applications 8th ... · Microsoft PowerPoint - cs559f15_Week8.pptx Author: mordohai Created Date: 10/19/2016 4:52:48 AM ...](https://reader035.fdocuments.in/reader035/viewer/2022071405/60fb020227a9c66db01e0b13/html5/thumbnails/24.jpg)
Example
24