1 12 Multiple Linear Regression 12-1 Multiple Linear Regression Model 12-1.1 Introduction 12-1.2...

12Multiple Linear Regression

CHAPTER OUTLINE

Learning Objectives for Chapter 12After careful study of this chapter, you should be able to do the

following:1. Use multiple regression techniques to build empirical models to

engineering and scientific data.2. Understand how the method of least squares extends to fitting multiple

regression models.3. Assess regression model adequacy.4. Test hypotheses and construct confidence intervals on the regression

coefficients.5. Use the regression model to estimate the mean response, and to make

predictions and to construct confidence intervals and prediction intervals.6. Build regression models with polynomial terms.7. Use indicator variables to model categorical regressors.8. Use stepwise regression and other model building techniques to select the

appropriate set of variables for a regression model.

12-1: Multiple Linear Regression Models

• Many applications of regression analysis involve situations in which there are more than one regressor variable. • A regression model that contains more than one regressor variable is called a multiple regression model.

12-1.1 Introduction

• For example, suppose that the effective life of a cutting tool depends on the cutting speed and the tool angle. A possible multiple regression model could be

whereY – tool lifex1 – cutting speedx2 – tool angle

12-1.1 Introduction

Figure 12-1 (a) The regression plane for the model E(Y) = 50 + 10x1 + 7x2. (b) The contour plot

12-1.1 Introduction

Figure 12-2 (a) Three-dimensional plot of the regression model E(Y) = 50 + 10x1 + 7x2 + 5x1x2. (b) The contour plot

12-1.1 Introduction

Figure 12-3 (a) Three-dimensional plot of the regression model E(Y) = 800 + 10x1 + 7x2 – 8.5x1

2 – 5x22 +

4x1x2. (b) The contour plot

12-1.1 Introduction

12-1.2 Least Squares Estimation of the Parameters

• The least squares function is given by

• The least squares estimates must satisfy

• The solution to the normal Equations are the least squares estimators of the regression coefficients.

• The least squares normal Equations are

Example 12-1

Figure 12-4 Matrix of scatter plots (from Minitab) for the wire bond pull strength data in Table 12-2.

Example 12-1

12-1.3 Matrix Approach to Multiple Linear Regression

Suppose the model relating the regressors to the response is

In matrix notation this model can be written as

We wish to find the vector of least squares estimators that minimizes:

The resulting least squares estimate is

Example 12-2

Estimating 2

An unbiased estimator of 2 is

12-1.4 Properties of the Least Squares Estimators

Unbiased estimators:

Covariance Matrix:

12-1.4 Properties of the Least Squares Estimators

Individual variances and covariances:

In general,

12-2: Hypothesis Tests in Multiple Linear Regression

12-2.1 Test for Significance of Regression

The appropriate hypotheses are

The test statistic is

12-2.1 Test for Significance of Regression

Example 12-3

R2 and Adjusted R2

The coefficient of multiple determination

• For the wire bond pull strength data, we find that R2 = SSR/SST = 5990.7712/6105.9447 = 0.9811.• Thus, the model accounts for about 98% of the variability in the pull strength response.

R2 and Adjusted R2

The adjusted R2 is

• The adjusted R2 statistic penalizes the analyst for adding terms to the model.• It can help guard against overfitting (including regressors that are not really useful)

12-2.2 Tests on Individual Regression Coefficients and Subsets of Coefficients

The hypotheses for testing the significance of any individual regression coefficient:

12-2.2 Tests on Individual Regression Coefficients and Subsets of Coefficients

The test statistic is

• Reject H0 if |t0| > t/2,n-p.• This is called a partial or marginal test

Example 12-4

The general regression significance test or the extra sum of squares method:

We wish to test the hypotheses:

A general form of the model can be written:

where X1 represents the columns of X associated with 1 and X2 represents the columns of X associated with 2

For the full model:

If H0 is true, the reduced model is

The test statistic is:

Reject H0 if f0 > f,r,n-p

The test in Equation (12-32) is often referred to as a partial F-test

Example 12-6

12-3: Confidence Intervals in Multiple Linear Regression

12-3.1 Confidence Intervals on Individual Regression Coefficients

Definition

Example 12-7

12-3.2 Confidence Interval on the Mean Response

The mean response at a point x0 is estimated by

The variance of the estimated mean response is

12-3.2 Confidence Interval on the Mean Response

Definition

Example 12-8

12-4: Prediction of New Observations

A point estimate of the future observation Y0 is

A 100(1-)% prediction interval for this future observation is

Figure 12-5 An example of extrapolation in multiple regression

Example 12-9

12-5: Model Adequacy Checking

12-5.1 Residual AnalysisExample 12-10

Figure 12-6 Normal probability plot of residuals

12-5.1 Residual Analysis

Example 12-10

Figure 12-7 Plot of residuals

Figure 12-8 Plot of residuals against x1.

Example 12-10

Figure 12-9 Plot of residuals against x2.

The variance of the ith residual is

12-5.2 Influential Observations

Figure 12-10 A point that is remote in x-space.

12-5.2 Influential Observations

Cook’s distance measure

Example 12-11

12-6: Aspects of Multiple Regression Modeling

12-6.1 Polynomial Regression Models

Example 12-12

Example 12-11

Figure 12-11 Data for Example 12-11.

Example 12-12

12-6.2 Categorical Regressors and Indicator Variables

• Many problems may involve qualitative or categorical variables.• The usual method for the different levels of a qualitative variable is to use indicator variables.• For example, to introduce the effect of two different operators into a regression model, we could define an indicator variable as follows:

Example 12-13

Example 12-12

Example 12-13

12-6.3 Selection of Variables and Model Building

12-6.3 Selection of Variables and Model BuildingAll Possible Regressions – Example 12-14

Figure 12-12 A matrix of Scatter plots from Minitab for the Wine Quality Data.

12-6.3: Selection of Variables and Model Building - Stepwise Regression

Example 12-14

12-6.3: Selection of Variables and Model Building - Backward Regression

Example 12-14

12-6.4 Multicollinearity

Variance Inflation Factor (VIF)

12-6.4 Multicollinearity

The presence of multicollinearity can be detected in several ways. Two of the more easily understood of these are:

Important Terms & Concepts of Chapter 12

1 12 Multiple Linear Regression 12-1 Multiple Linear Regression Model 12-1.1 Introduction 12-1.2...

Documents

Transcript of 1 12 Multiple Linear Regression 12-1 Multiple Linear Regression Model 12-1.1 Introduction 12-1.2...

Machine Learning - A. Supervised Learning A.1. Linear Regression · 2020. 12. 9. · Machine Learning 1. Linear Regression via Normal Equations Outline 1. Linear Regression via Normal

Chapter 8 Linear Regression. Objectives & Learning Goals Understand Linear Regression (linear modeling): Create and interpret a linear regression model.

Chapter 12: Correlation and Linear Regression 1.

Chapter 12. Simple Linear Regression and Correlationmathsci.kaist.ac.kr/~nipl/cc511/lectures/Chapter12.pdf · Chapter 12. Simple Linear Regression and Correlation 12.1 The Simple

The Algebra of Linear Regression - statpower.net Notes/RegressionAlgebra.pdf · The Algebra of Linear Regression 1 Introduction 2 Bivariate Linear Regression 3 Multiple Linear Regression

Chapter 13: SIMPLE LINEAR REGRESSION. 2 Simple Regression Linear Regression.

Regression Linear Regression Regression Trees

L21: Chapter 12: Linear regression...Multiple regression L21: Chapter 12: Linear regression Department of Statistics, University of South Carolina Stat 205: Elementary Statistics for

Regression Linear Regression

Chapter 12 Section 1 Inference for Linear Regression.

+ Chapter 12: More About Regression Section 12.1 Inference for Linear Regression.

Linear correlation and linear regression + summary of tests Week 12 Regression Theory.

REVIEW OF SIMPLE LINEAR REGRESSION SIMPLE LINEAR REGRESSION Determining the Regression ... of linear... · 2012. 6. 21. · REVIEW OF SIMPLE LINEAR REGRESSION SIMPLE LINEAR REGRESSION

Generalized linear regression - Little Dumb doctor to Multiple Linear Regression 3 tics tutorial at Multiple Linear RegressionMultiple Linear Regression • Regression analysis is

SIMPLE LINEAR REGRESSION. 2 Simple Regression Linear Regression.

Non-Linear Regression - Penn State College of Engineeringrtc12/CSE586/lectures/regrNonlinear... · 2013-04-01 · Non-linear regression Linear regression: Non-Linear regression: where

TOPIC 12 Simple Linear Regression and Correlation.

CHAPTER 12: LINEAR REGRESSION AND CORRELATION 12 New/Ch 12... · CHAPTER 12: LINEAR REGRESSION AND CORRELATION . Exercise 1. A vacation resort rents SCUBA equipment to certified divers.

Multiple Linear Regression - courses.physics.illinois.edu › bioe505 › fa...12-1: Multiple Linear Regression Model 12-1.3 Matrix Approach to Multiple Linear Regression Suppose the

Regression analysis Linear regression Logistic regression.