Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave...
Transcript of Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave...
![Page 1: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/1.jpg)
Polynomial Regression and Regularization
![Page 2: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/2.jpg)
Administriviao If you still haven’t enrolled in Moodle yet
o Homework 1 posted later tonight. Good Milestones:
2
§ Enrollment key on Piazza
§ If joined course recently, email me to get added to Piazza
§ Problems 1-3 This Week § Problems 4 and 5 Next Week
![Page 3: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/3.jpg)
The RoadMapo Last Time:
o This Time:
o Next Time:
3
§ Regression Refresher (there was nothing fresh about it)
§ Polynomial Regression
§ Regularization (wiggles are bad, Man)
§ Bias-Variance Trade-Off (what does it all MEAN?)
![Page 4: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/4.jpg)
4
Previously on CSCI 4622Given training data for fit a regression of the form
where
Estimates of the parameters are found by minimizing
Construct the design matrix by prepending column of 1s to data matrix
For the moment, solve via normal equations:
✏i ⇠ N(0,�2)
(xi1, xi2, . . . , xip, yi) i = 1, 2, . . . , n
yi = �0 + �1xi1 + �2xi2 + · · ·+ �pxip + ✏i
X
� =⇣XTX
⌘�1XTy
RSS =nX
i=1
[(�0 + �1xi1 + · · ·+ �pxip)� yi]2 = kX� � yk2f
LOSS Function
![Page 5: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/5.jpg)
5
So, We Can Model Things Like This
![Page 6: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/6.jpg)
6
And Things Like This
X1
X2
Y
![Page 7: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/7.jpg)
But What About Things Like This?
7
![Page 8: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/8.jpg)
But What About Things Like This?
8
o So yeah, nonlinearity is a thing
o Clearly, the linear models of regression cannot handle this.
o Give me Neural Networks or Give me Death!
![Page 9: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/9.jpg)
But What About Things Like This?
9
o So yeah, nonlinearity is a thing
o Clearly, the linear models of regression cannot handle this.
o Give me Neural Networks or Give me Death!
o Nah. Regression can totally do this
![Page 10: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/10.jpg)
Start with the Obvious: Polynomials
10
o Suppose, as in the previous pictures, we have a single feature
o We want to go from this:
o To something like this:
o So what is the difference between these two?
Y = �0 + �1X + ✏
X
Y = �0 + �1X + �2X2 + · · ·+ �pXp + ✏
![Page 11: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/11.jpg)
Start with the Obvious: Polynomials
11
o Suppose, as in the previous pictures, we have a single feature
o We want to go from this:
o To something like this:
o So what is the difference between these two?
o Seems like it’s closer to this:
Y = �0 + �1X + ✏
X
Y = �0 + �1X + �2X2 + · · ·+ �pXp + ✏
Y = �0 + �1X1 + �2X2 + · · ·+ �pXp + ✏
![Page 12: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/12.jpg)
The One Becomes the Many
12
o Start with a single feature
o And derive new polynomial features:
o These two things:
o are exactly the same:
X
Y = �0 + �1X + �2X2 + · · ·+ �pXp + ✏
Y = �0 + �1X1 + �2X2 + · · ·+ �pXp + ✏
X1 = X, X2 = X2, · · · Xp = Xp
![Page 13: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/13.jpg)
The Polynomial Regression Pipeline
13
o Start with a single feature
o Derive new polynomial features:
o Solve the MLR in the usual way:
o Question: What does the Matrix Equation look like?
X
Y = �0 + �1X1 + �2X2 + · · ·+ �pXp + ✏
X1 = X, X2 = X2, · · · Xp = Xp
![Page 14: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/14.jpg)
The Polynomial Regression Pipeline
14
o Start with a single feature
o Derive new polynomial features:
o Solve the MLR in the usual way:
o Question: What does the Matrix Equation look like?
Before:
X
Y = �0 + �1X1 + �2X2 + · · ·+ �pXp + ✏
X1 = X, X2 = X2, · · · Xp = Xp
2
666664
1 x11 x21 · · · x1p
1 x21 x22 · · · x2p
1 x31 x32 · · · x3p...
......
...1 xn1 xn2 · · · xnp
3
777775
2
666664
�0
�1
�2...�p
3
777775=
2
666664
y1y2y3...yn
3
777775
![Page 15: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/15.jpg)
The Polynomial Regression Pipeline
15
o Start with a single feature
o Derive new polynomial features:
o Solve the MLR in the usual way:
o Question: What does the Matrix Equation look like?
After:
X
Y = �0 + �1X1 + �2X2 + · · ·+ �pXp + ✏
X1 = X, X2 = X2, · · · Xp = Xp
2
66666664
1 x1 x21 · · · xp
1
1 x2 x22 · · · xp
2
1 x3 x22 · · · xp
3
......
......
1 xn x2n · · · xp
n
3
77777775
2
666664
�0
�1
�2...�p
3
777775=
2
666664
y1y2y3...yn
3
777775-3
![Page 16: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/16.jpg)
The Polynomial Regression Pipeline
16
o Suppose I learn the parameters from the training set
o What if I want to make a prediction about data I haven’t seen yet?
HAVE TEST point Xo
NEED to PERFORM SAME Transformation
on Xo THAT WE PERFORMED On trainingdata . xo → ( 1
, × .
,X5
, ... ,XoP)
THEN PRE Pichon IS just §o= Xotp
![Page 17: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/17.jpg)
The Polynomial Regression Pipeline
17
o Suppose I learn the parameters from the training set
o What if I want to make a prediction about data I haven’t seen yet?
o Important Theme #1: If you transform your features for training, predicting is always as easy transforming the features of your new data in the same way.
Input space FEATURE SPACE-Training X → X
,Xn , ...
XP
Test Xo → Xo,Xp
, . .
,XOP
MAGIC Happens HERE !
![Page 18: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/18.jpg)
The Polynomial Regression Pipeline
18
o Suppose I learn the parameters from the training set
o What if I want to make a prediction about data I haven’t seen yet?
o Important Theme #1: If you transform your features for training, predicting is always as easy transforming the features of your new data in the same way.
o Important Theme #2: If your current features are not flexible enough, moving into a higher dimensional space will make your model more flexible (but BEWARE!)
![Page 19: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/19.jpg)
Example: Sinusoidal Data
19
True Model is f(x) = sin(⇡x) + ✏
o Question: What degree polynomial should I use?
![Page 20: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/20.jpg)
Example: Sinusoidal Data
20
True Model is f(x) = sin(⇡x) + ✏
o Question: What degree polynomial should I use?
o Degree = 1
![Page 21: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/21.jpg)
Example: Sinusoidal Data
21
True Model is f(x) = sin(⇡x) + ✏
o Question: What degree polynomial should I use?
o Degree = 4
![Page 22: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/22.jpg)
Example: Sinusoidal Data
22
True Model is f(x) = sin(⇡x) + ✏
o Question: What degree polynomial should I use?
o Degree = 7
![Page 23: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/23.jpg)
Example: Sinusoidal Data
23
True Model is f(x) = sin(⇡x) + ✏
o Question: What degree polynomial should I use?
o Degree = 11
![Page 24: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/24.jpg)
Example: Sinusoidal Data
24
True Model is f(x) = sin(⇡x) + ✏
o Question: What degree polynomial should I use?
o Degree = 11
o What Questions should we be asking?
![Page 25: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/25.jpg)
The Real ML Pipeline (Almost)
25
The more flexible (powerful) your model gets, the better it’ll do on training set
But that’s not useful. We want to know how model will do on new data
How well it’ll Generalize A better pipeline:
Train on training set. Validate (or evaluate) on validation set.
AllLabeledData TrainingSet ValidationSet
TRAIN TEST on
onTH 'S THIS
RANDOM
SPLIT ) )L L
HELD - OUT
DATA
![Page 26: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/26.jpg)
The Real ML Pipeline (Almost)
26
Need a performance measure:For Regression, we use the Mean-Squared Error:
AllLabeledData TrainingSet ValidationSet
MSE =1
n
nX
I=1
(yi � yi)2
yTRUE RESPONSE
← predictions-
![Page 27: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/27.jpg)
Example: Sinusoidal Data
27
What happens to the MSE on training set as p grows?
What happens to MSE on validation set as p grows?
MSE goesDOWNMSEgoes Down
At±T , then BACK-Up!
![Page 28: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/28.jpg)
Example: Sinusoidal Data
28
FLEXIBLEEnough
MODEL
/ OUERFITT '
ng←HAS STARTED
![Page 29: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/29.jpg)
Example: Sinusoidal Data
29
Revisit Polynomial Regression
What causes those wiggles that are hurting us so bad?
![Page 30: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/30.jpg)
Example: Sinusoidal Data
30
True Model is f(x) = sin(⇡x) + ✏
![Page 31: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/31.jpg)
Example: Sinusoidal Data
31
True Model is f(x) = sin(⇡x) + ✏
![Page 32: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/32.jpg)
Example: Sinusoidal Data
32
True Model is f(x) = sin(⇡x) + ✏
![Page 33: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/33.jpg)
Example: Sinusoidal Data
33
True Model is f(x) = sin(⇡x) + ✏
![Page 34: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/34.jpg)
Example: Sinusoidal Data
34
Coefficients grow very large. Need a way to control them. Regulate them even
![Page 35: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/35.jpg)
Regularization
35
Goal: Keep model coefficients from growing too large
Idea: Put something in the loss function to dissuade them growing too much
RSS� =nX
i=1
[(�0 + �1xi1 + · · ·+ �pxip)� yi]2 + �
pX
k=1
�2k
o Regularization weight is a tuning parameter
o Have to do regularization study to choose best value
�
minimize
ofTenafly
![Page 36: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/36.jpg)
Regularization
36
Goal: Keep model coefficients from growing too large
Idea: Put something in the loss function to dissuade them growing too much
RSS� =nX
i=1
[(�0 + �1xi1 + · · ·+ �pxip)� yi]2 + �
pX
k=1
�2k
o Why do we not regularize the bias parameter?
BIAS Tells Us About Average RESPONSE( HEIGHT OF DATA ) . Regularizing po pullsTHE whole model down FROM WHERE Should BE
![Page 37: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/37.jpg)
Ridge Regression
37
Goal: Keep model coefficients from growing too large
Idea: Put something in the loss function to dissuade them growing too much
RSS� =nX
i=1
[(�0 + �1xi1 + · · ·+ �pxip)� yi]2 + �
pX
k=1
�2k
o This form of penalty term is called Ridge Regression. But there are many more
![Page 38: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/38.jpg)
Sinusoidal Data with Ridge Regression
38
Watch the coefficients
![Page 39: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/39.jpg)
Sinusoidal Data with Ridge Regression
39
Watch the coefficients
![Page 40: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/40.jpg)
Sinusoidal Data with Ridge Regression
40
Watch the coefficients
![Page 41: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/41.jpg)
Sinusoidal Data with Ridge Regression
41
Watch the coefficients
![Page 42: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/42.jpg)
Sinusoidal Data with Ridge Regression
42
Watch the coefficients
![Page 43: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/43.jpg)
Sinusoidal Data with Ridge Regression
43
Watch the coefficients NOTE : COEFFICIENTS
STAYED Small !
![Page 44: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/44.jpg)
Example: Sinusoidal Data
44
No Regularization Ridge Regression
BLOW - UP HAPPENS
MUCH LATER !
![Page 45: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/45.jpg)
Example: Sinusoidal Data
45
Regularization Study:o Choose poly degreeo Vary over reg strengtho Look for best validation MSE
![Page 46: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/46.jpg)
Example: Sinusoidal Data
46
Regularization Study:o Choose poly degreeo Vary over reg strengtho Look at size of coefficients
COEFFS SHRINK
As × → oo
![Page 47: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/47.jpg)
Regularization Wrap-Up
47
You should always do regularization.
If you choose carefully, it will always help Generalization
Next Time: o Talk more about Ridge Regression Details
o Learning Curves and what they tell us about the Bias-Variance Trade-Off
�
![Page 48: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/48.jpg)
48
![Page 49: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/49.jpg)
49
![Page 50: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/50.jpg)
50
![Page 51: Polynomial Regression and Regularizationketelsen/files/... · p x ip + i X ˆ = ⇣ XT X ... oHave to do regularization study to choose best value minimize of Tenafly. Regularization](https://reader030.fdocuments.in/reader030/viewer/2022041013/5ec25b2fce6e8540ec28686e/html5/thumbnails/51.jpg)
51