Classification and Prediction:Regression Analysis

Bamshad MobasherDePaul University

Bamshad MobasherDePaul University

2

What Is Numerical Prediction(a.k.a. Estimation, Forecasting)

i (Numerical) prediction is similar to classification4 construct a model4 use model to predict continuous or ordered value for a given input

i Prediction is different from classification4 Classification refers to predicting categorical class label4 Prediction models continuous-valued functions

i Major method for prediction: regression4 model the relationship between one or more independent or predictor

variables and a dependent or response variablei Regression analysis

4 Linear and multiple regression4 Non-linear regression4 Other regression methods: generalized linear model, Poisson regression,

log-linear models, regression trees

3

Linear Regression

i Linear regression: involves a response variable y and a single predictor variable x y = w0 + w1 x

x y

i Goal: Using the data estimate weights (parameters) w0 and w1 for the

line such that the prediction error is minimized

Linear Regression

y

x

Observed Value of y for xi

Predicted Value of y for xi

xwwy 10

xi

Slope = β1

Intercept = w0

ei

Error for this x value

5

Linear Regression

i Linear regression: involves a response variable y and a single predictor variable x y = w0 + w1 x

4 The weights w0 (y-intercept) and w1 (slope) are regression coefficients

i Method of least squares: estimates the best-fitting straight line4 w0 and w1 are obtained by minimizing the sum of the squared errors (a.k.a.

residuals)

ii

iii

xx

yyxxw

21 )(

))((

xwyw 10

210

22

))((

)ˆ(

ii

iii

ii

xwwy

yye

w1 can be obtained by setting the partial derivative of the SSE to 0 and solving for w1, ultimately resulting in:

6

Multiple Linear Regression

i Multiple linear regression: involves more than one predictor variable

4 Training data is of the form (X1, y1), (X2, y2),…, (X|D|, y|D|)

4 Ex. For 2-D data, we may have: y = w0 + w1 x1+ w2 x2

4 Solvable by extension of least square method

4 Many nonlinear functions can be transformed into the above

x1 yx2

i Simple Least Squares:4 Determine linear coefficients , that minimize sum of

squared error (SSE).4 Use standard (multivariate) differential calculus:

h differentiate SSE with respect to , h find zeros of each partial differential equationh solve for ,

i One dimension:

Least Squares Generalization

ttt

N

jjj

xxy

x, yy,xxyx

yx

Nxy

samplefor test ˆ

trainingof means ]var[

],cov[

samples ofnumber ))((SSE1

2

i Multiple dimensions

4 To simplify notation and derivation, change to 0, and add a new feature x0 = 1 to feature vector x:

Least Squares Generalization

xβ

T

10 1ˆ

d

iii xy

yx2x1x0

11111

i Multiple dimensions

4 Calculate SSE and determine :

Least Squares Generalization

xβ

T

10 1ˆ

d

iii xy

ttt

j

j

N

j

d

ijiij

y

y

xy

xxβ

yXXXβ

xX

y

XβyXβy

samplefor test ˆ

)(

samples trainingall ofmatrix

responses trainingall ofvector

)()()(SSE

T1T

1

T2

0,

i The inputs X for linear regression can be:4 Original quantitative inputs4 Transformation of quantitative inputs, e.g. log, exp, square

root, square, etc.4 Polynomial transformation

h example: y = 0 + 1x + 2x2 + 3x3

4 Dummy coding of categorical inputs4 Interactions between variables

h example: x3 = x1 x2

i This allows use of linear regression techniques to fit much more complicated non-linear datasets.

Extending Application of Linear Regression

Example of fitting polynomial curve with linear model

i Complex models (lots of parameters) are often prone to overfittingi Overfitting can be reduced by imposing a constraint on the overall

magnitude of the parameters (i.e., by including coefficients as part of the optimization process)

i Two common types of regularization in linear regression:

4 L2 regularization (a.k.a. ridge regression). Find which minimizes:

h is the regularization parameter: bigger imposes more constraint

4 L1 regularization (a.k.a. lasso). Find which minimizes:

Regularization

N

j

d

ii

d

iiij xy

1 1

22

0

)(

N

j

d

ii

d

iiij xy

1 1

2

0

||)(

Example: Web Traffic Data

13

1D Poly Fit

14

Example of too much “bias” underfitting

Example: 1D and 2D Poly Fit

15

Example: 1D Ploy Fit

16

Example of too much “variance” overfitting

Bias-Variance Tradeoff

17

Bias-Variance Tradeoffi Possible ways of dealing with high bias

4 Get additional features

4 More complex model (e.g., adding polynomial terms such as x12, x2

2 , x1.x2, etc.)

4 Use smaller regularization coefficient l.4 Note: getting more training data won’t necessarily help in this case

i Possible ways dealing with high variance4 Use more training instances4 Reduce the number of features4 Use simpler models

4 Use a larger regularization coefficient l.

18

19

i Generalized linear models

4 Foundation on which linear regression can be applied to modeling categorical response variables

4 Variance of y is a function of the mean value of y, not a constant

4 Logistic regression: models the probability of some event occurring as a linear function of a set of predictor variables

4 Poisson regression: models the data that exhibit a Poisson distribution

i Log-linear models (for categorical data)

4 Approximate discrete multidimensional prob. distributions

4 Also useful for data compression and smoothing

i Regression trees and model trees

4 Trees to predict continuous values rather than class labels

Other Regression-Based Models

20

Regression Trees and Model Treesi Regression tree: proposed in CART system (Breiman et al. 1984)

4 CART: Classification And Regression Trees

4 Each leaf stores a continuous-valued prediction

4 It is the average value of the predicted attribute for the training instances

that reach the leaf

i Model tree: proposed by Quinlan (1992)

4 Each leaf holds a regression model—a multivariate linear equation for

the predicted attribute

4 A more general case than regression tree

i Regression and model trees tend to be more accurate than linear

regression when instances are not represented well by simple linear

models

21

Evaluating Numeric Predictioni Prediction Accuracy

4 Difference between predicted scores and the actual results (from evaluation set)4 Typically the accuracy of the model is measured in terms of variance (i.e., average

of the squared differences)

i Common Metrics (pi = predicted target value for test instance i, ai = actual target value for instance i)

4 Mean Absolute Error: Average loss over the test set

4 Root Mean Squared Error: compute the standard deviation (i.e., square root of the co-variance between predicted and actual ratings)

n

apapRMSE nn

2211 )(...)(

n

apapMAE nn )(...)( 11