Multiple Regression(sec VIII)
Multiple Regression - Overview Multiple Regression in statistics is the science and art of
creating an equation that relates an outcome Y, to one or more predictors, X1, X2, X3, .. Xk.
Linear Regression
Y = a + b1X1 + b2 X2 + b3 X3 + ... + bk Xk + e = Ŷ + e
where "e" is the residual error between the observed Y and the prediction (Ŷ).
In linear regression, bi is the average change in Y for a single unit change in Xi.
“Performance” stats: R2, SDe
Logistic Regression
In multiple logistic regression, Y is 0 or 1 with mean P, the “risk”. The logit of P, not P, is assumed a linear function of the Xs
Logit(P) = ln(P/(1-P)) = a + b1 X1 + b2 X2 + b3 X3 + ... + bk Xk
(“Logit” = log of the odds since P/(1-P) is the odds) odds=exp(a + b1 X1 + b2 X2 + b3 X3 + ... + bk Xk) risk = P = odds/(odds+1)
“Performance” stats: Sensitivity, Specificity, Accuracy, Concordance (C), mean deviance
Poisson Regression
When Y is a positive integer (0,1,2,3…), we model the log of Y so Y can never be negative. This is the multiple Poisson regression model.
ln(mean Y) = a + b1 X1 + b2 X2 + b3 X3 + ... + bk Xk
mean Y = exp(a + b1 X1 + b2 X2 + b3 X3 + ... + bk Xk) mean Y cannot be negative.
“Performance” stats: R2, mean deviance
Logit function: P vs ln(P/(1-P))P vs logit
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
-4 -3 -2 -1 0 1 2 3 4
logit =log odds
P=ri
sk
Logistic regression Predictors of in hospital infection
Characteristic Odds Ratio (95% CI) p value
Incr APACHE score 1.15 (1.11-1.18) <.001
Transfusion (y/n) 4.15 (2.46-6.99) <.001
Increasing age (yr) 1.03 (1.02-1.05) <.001
Malignancy 2.60 (1.62-4.17) <.001
Max Temperature 0.70 (0.58-0.85) <.001
Adm to treat>7 d 1.66 (1.05-2.61) 0.03
Female (y/n) 1.32 (0.90-1.94) 0.16 *APACHE = Acute Physiology & Chronic Health Evaluation Score
Multiple Proportional Hazards Regr (Cox model)For time dependent outcomes (ie time to death), we model the hazard rate,
h , the event rate per unit time (for death, it is the mortality rate). Since h > 0, we model the log of the hazard as a linear function of the Xs so
h can never be zero (similar to Poisson regression) . ln(h) = a + b1 X1 + b2 X2 + b3 X3 + ... + bk Xk
so h = exp(a + b1 X1 + b2 X2 + b3 X3 + ... + bk Xk) > 0
If h0=exp(a) is the ‘baseline’ hazard, (that is, a=log(h0)) the hazard ratio is HR = h/h0 = exp(b1 X1 + b2 X2 + b3 X3 + ... + bk Xk) no intercept ‘a’.
If S0(t) is the ‘baseline’ survival curve corresponding to the baseline hazard, then the survival curve for a given combination of X1, X2, … Xk is given by
S(t) = S0(t)HR where HR is computed with the equation above.
exp(bi) is the hazard rate ratio for a one unit change in Xi.
“Performance” stats: Harrell’s Concordance (C) (0.5 < C < 1.0)
Cox regr-HR for patient mortality- Busuttil et al 2005
Cox HRs for donor age (Busuttil 2005)
Donor age HR 95% CI p value
1-18 1.00 (ref) -- --
18-32 1.23 0.88-1.72 0.20
32-48 1.40 1.02-1.92 0.03
48-55 1.51 1.02-2.24 0.04
55-60 2.29 1.48-3.55 < 0.001
60+ 1.61 1.10-2.37 0.01
Harrell C= 0.70
Regression coeff interpretationOutcome (Y) Regression interpretation
continuous Linear b is the average change in Y per one unit increase in X, the rate of change
Binary(P=proportion)
Logistic exp(b)=eb=odds ratio (OR) for a one unit increase in X
Low Positive integers
(0,1,2,3..)
Poisson exp(b)= mean ratio (MR) for a one unit increase in X
Hazard rate(h=events/time)
Cox exp(b)=hazard rate ratio (HR) for a one unit increase in X
S(t) = S0(t)HR
Multiple Linear Regression Example
Consider predictors of Y=Bilirubin (mg/dl) in liver transplant candidates. Two predictors are
X1=Prothombin time (PT) in seconds X2=ALT (alanine aminotransferase in U/L).
A multiple regression equation (on the log scale) is
Ŷ = (predicted) log Bilirubin = -3.96 + 3.47 log PT + 0.21 log ALT
Regression output - equation (Equation) Parameter estimates
term estimate SE t ratio p value
Intercept -3.96 0.257 -15.4 < 0.001
log PT 3.47 0.214 16.2 < 0.001
log ALT 0.21 0.055 3.8 0.0002
equation:
Log Bili=-3.96 + 3.47 log PT + 0.21 log ALT
Regression-analysis of variance table
Source df sum squares mean square F
Model 2 37.76 18.88 147.2
Error 363 46.56 0.1283=SDe2 --
Total 365 84.32
F = 18.88/0.1283 ->Screening F test that none of the predictors are related to Y.
R2 = 37.76/84.32 = 0.448 = 44.8%
R2 = model sum squares/total sum squares
Residual error plotResidual Log Bilirubin by Predicted
-1.0
-0.5
0.0
0.5
1.0
1.5
Log
Bili
rubi
n R
esid
ual
-1.0 -0.5 .0 .5 1.0 1.5 2.0
Log Bilirubin Predicted
When the model is valid, this plot should look like a circular cloud if the errors have constant variance.
The example above is a “good” result.
Example of a “good” residual error histogramerrors have a Gaussian distribution about zero
Moments - errors
Mean 0.000
Std Dev (SDe) 0.357
Std Err Mean 0.0187
n 366
-1 -0.5 0 .5 1 1.5
Quantiles - errors
100.0% maximum 1.711
97.5% 0.834
90.0% 0.454
75.0% quartile 0.177
50.0% median -0.025
25.0% quartile -0.213
10.0% -0.355
2.5% -0.630
0.0% minimum -1.191
Normal Quantile Plot
(Should be approximately a straight line if the residual error data are Gaussian)
.01
.05
.10
.25
.50
.75
.90
.95
.99
-3
-2
-1
0
1
2
3
Nor
mal
Qua
ntile
Plo
t
-1 -0.5 0 .5 1 1.5
residual error (e)
Interpreting multiple regression coefficients (cont.)The multiple regression coefficients will not in general be the same as the
individual regression coefficient for each variable one at a time, even though the same Y is being modeled.
Simple regression Multiple (simultaneous) variable (one Y, one X) (b1X1+b2X2)Log PT 3.560 3.470Log ALT 0.310 0.211 Log Bilirubin = - 3.70 + 3.56 log PT, R2 = 0.425
Log Bilirubin = -.105 + 0.310 log ALT, R2 = 0.049
Log Bilirubin = -3.96 + 3.47 log PT + 0.211 log ALT , R2 = 0.448
Log PT coefficients don’t match
Orthogonally (vs collinearity)In general, regression coefficients from
simple and multiple regression are not the same ↔ controlling for covariates does not give the same answer as ignoring covariates.
Only when all the X variables have correlation zero with each other will the simple and multiple regression coefficients be the same orthogonally
(Collinearity is when Xs are strongly correlated. It is the “opposite” of orthogonality).
Log PT (X1) vs log ALT (X2)since correlation is low, unadjusted and adjusted
regression results are similar
1.00
1.10
1.20
1.30
1.40
1.50
log
Pro
thro
m
1.0 1.5 2.0 2.5 3.0 3.5
log ALT
r12 = 0.111, R2 = 0.0123
Interaction Effects & subgroupsThe model Y = 0 + 1X1 + 2X2 + implies that change in Y due to X1 (=1) is the same
(constant) for all values of X2. An ADDITIVE model.
In the model Y = 0 + 1X1 + 2X2 + 3 X1X2 + the 3 term is an interaction term.
Change in Y for a unit change in X1 is (1+3X2) and is therefore not constant.
Positive 3 is often termed a “synergism”
Negative 3 is often termed an “antagonism”
Additive only if β3=0
How to implement? Make new variable W = X1X2.
Interaction example Response: Y= log HOMA IR (MESA study)
R2=0.280, Root Mean Square Error=SDe=0.623 Mean Response= 0.395, n= 6782
Parameter Estimates Term Estimate Std Error t Ratio p valueIntercept -1.388 0.049 -28.16 <.0001Gender -0.669 0.085 -7.83 <.0001BMI 0.061 0.00168 36.45 <.0001gender*BMI 0.028 0.00299 9.38 <.0001
Predicted log HOMA IR = -1.39 – 0.669 gender + 0.061 BMI + 0.028 gender * BMI (gender is coded 0 for female and 1 for male)
gender x BMI interaction- non additivity
In Females
Log HOMA IR = -1.39 +0.061 BMI
In Males
Log HOMA IR = -2.06 + 0.089 BMI
log e HOMA IR
-0.5
0.0
0.5
1.0
1.5
2.0
20 22 24 26 28 30 32 34 36 38 40 42 44
BMI
log
HO
MA
IR
F
M
Gender x BMI interactionrelation is different in males vs females
HOMA IR
0.0
1.0
2.0
3.0
4.0
5.0
6.0
20 22 24 26 28 30 32 34 36 38 40 42 44
BMI
HO
MA
IR
F
M
Hierarchically well formulated (WWF) regression models
HWF Rule – To correctly evaluate the X1*X2 interaction, must also have X1 and X2 in the model.
In general, one must include the lower order terms in order to correctly evaluate the higher order terms.
HWF example- NON HWF Model:chol = a0 + a1 smoke x age = SMOKEAGE
0, 1 (dummy) coding: smoke=0 or 1, smokeage = smoke x age
Variable DF Estimate std error t p value INTERCEPT 1 156.863 3.993 39.286 0.0001 SMOKEAGE 1 0.361 0.182 1.988 0.0567
----------------------------------------------------------------------------
-1, 1 (effect) coding: smoke=-1 or 1, smokeage =smoke x age Variable DF Estimate std error t p value INTERCEPT 1 162.278 3.103 52.320 0.0001 SMOKEAGE 1 0.055 0.100 0.548 0.5881
p value changes just because coding changes!
HWF example (cont)- HWF HWF:Model: chol = b0 + b1 smoke + b2 age + b3 smoke x age For HWF models, significance is the same regardless of coding
0, 1 (dummy) coding: smoke=0 or 1, smokeage = smoke x age
Variable DF Estimate std error t p value INTERCEPT 1 100.221 1.110 90.304 0.0001 SMOKE 1 3.812 1.570 2.429 0.0224 AGE 1 2.010 0.036 56.297 0.0001 SMOKEAGE 1 -0.009 0.050 -0.178 0.8599
-1, 1 (effect) coding:smoke=-1 or 1, smokeage=smoke x age
Variable DF Estimate std error t p value INTERCEPT 1 102.127 0.785 130.138 0.0001 SMOKE 1 1.906 0.785 2.429 0.0224 AGE 1 2.005 0.025 79.437 0.0001 SMOKEAGE 1 -0.004 0.025 -0.178 0.8599
Regression assumptionsRegression can simultaneously evaluate all
factors and thus reduce confounding. But must check two critical assumptions.
1. If X is continuous/interval, check if relation of X & Y is linear on some scale.
(otherwise polychotomize X)
2. Check if effects of X1, X2, X3 … are additive by adding interaction terms.
(Ex: X4 = X1 x X2). Not additive if interactions are significant.
3 Also, in linear regression, prefer residual errors (e) to have a Gaussian distribution with a constant variance that is independent of Y. But additivity and linearity are more important since lack of additivity and linearity lead to bias and is more misleading.
Nonlinear Regression
Log(Bilirubin)= -3.96 + 3.47 log(PT) + 0.211 log(ALT)
is a nonlinear model in terms of PT and ALT
but is a linear model in terms of log PT, log ALT and
the regression coefficients b0=-3.96, b1=3.47 and b2=0.211.
Consider model of the form: Ŷ= Drug concentration = b1 10 b2 X
This is nonlinear in b2 but can be made linear with a transformation: log10(concentration)=log10(b1) + b2 x
What about: Drug concentration = b0+ b1 10 b2 X
This model is nonlinear in b2 and cannot be transformed.
Nonlinear regression software is needed to estimate b0, b1, & b2.
Nonlinear ExampleCompartmental drug models
Model of how drug (or any chemical) is metabolized by an organism. Y1=concentration in serum, Y2=concentration in organ, x=time
d (Y1)/dx = -b1 Y1
d (Y2)/dx = b1Y1 - b2Y2
b1 > b2 > 0
solutions: Y1 = constant e -b1 x
Y2 = (b1/(b1-b2)) [e -b2x – e -b1x] < - model
Y2 takes on a maximum value when x = ln(b1/b2)/[b1-b2]Y2 is zero when x=0 or x is very largeThe constants b1 and b2 are rates. They are in units of 1/x (i.e 1/time).
Y1 serum
Y2
organ
Nonlinear equation
Y = (b1/(b1-b2)) [e -b2x – e -b1x]
t
Y
Y= [0.0967/(0.0967-0.0506)]*[exp(-0.0506*t)-exp(-0.0967*t)] at peak, t = 14, Ŷ= 0.49
Residual diagnostics & “model criticism” Assumptions of linear regression: 1. Linear relation between Y and each X except
for random “noise” (but can transform X). 2. Effect of each X is additive (but can make
interaction terms) 3. Errors (e) have constant variance and come
from a Gaussian distribution 4. All observations from the same population 5. All observations independent (usually ok)
A plot of Ŷ versus e, called a residual error (diagnostic) plot, can help verify if these assumptions are met.
“good” residual plot
Residual plots – diagnostics- “bad” residual plots
Regression diagnostics
Problem-outliersSolution – Find on residual plot & eliminate Problem-Curvilinear (non linear)Solution- try nonlinear trans formation (x2, 1/x, log(x), ex) Problem-Errors not GaussianSolution-Robust regression (future class)
Problem-Non constant SDe or Var(e)Solution-Weights (future class)
Adjusted means
-12.0
-10.0
-8.0
-6.0
-4.0
-2.0
0.0
2.0
4.0
20 25 30 35 40 45 50 55 60 65 70
dietary fat (g)
bo
dy
fat
chg
(%
)
mediationshamunadjusted meansadjusted means
unadjusted difference
adjusted difference
Ex: Meditation & change in pct body fat Overweight persons chose a meditation program or a “sham” (lectures) as
part of a weight loss effort. They were NOT randomized.
Change in percent body fat by treatment group (mediation or sham) over three months
Unadjusted Means Level n Mean pct body SEM Mean dietary fat (gm) fat change (before study start) 1-meditate 439 -7.51% 0.47% 32.7 g 2-sham 704 1.34% 0.35% 67.1 g
Unadjusted Mean difference (sham - meditation) = 8.85% SE of the difference = SEd = √0.47222 + 0.35322 = 0.59%
t = mean diff/SEd = 8.85% / 0.59% = 15.1, p < 0.0001
Overall unweighted dietary fat = 49.9 g
Result via “regression”
Y= change in body fat,
X = 1 if sham, 0 if meditation
Y = a + b X + error = a + b sham + error
term estimate SE t p value
a -7.51 0.46 -16.4 < 0.001
b 8.85 0.59 15.2 < 0.001
Ŷ = -7.51 + 8.85 sham
Regr-control for dietary fat pct body fat= Y = a + b1 X1 + b2 X2 + error
X1= 1 if sham, 0 if meditation
X2 = dietary fat in g
term estimate SE t p value
a -14.5 0.54 -26.6 < 0.001
b1 1.51 0.56 2.7 0.007
b2 0.21 0.011 18.9 <0.001
body fat chg=-14.5 +1.51 sham +0.21 diet fat
Adjusted meansPlug into equation for X1=1=sham or X1=0.
Hold X2 = diet fat= 49.9 g – overall mean X2
Med: -14.5 + 1.51(0)+0.213(49.9)=-3.84%
Sham:-14.5 + 1.51(1)+0.213(49.9)=-2.33%
Adj mean difference =2.33-(-3.84) =1.51%
(Unadjusted was 8.85%)
General procedure
Adjusted means for X1,controlling for (confounders) X2, X3, X4 …
1. Estimate model regression coefficients
2. Plug in different values for X1, and values for all other Xs held constant at their overall means.
Gives adjusted means and their SEs.Assumes ????
Top Related