Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance...
Transcript of Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance...
![Page 1: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/1.jpg)
Using Regression Residuals as Performance Measures:Pitfalls and Possibilities
Martin E. Sandbu
Center on Globalization
and Sustainable Development
Columbia University
![Page 2: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/2.jpg)
Two types of comparison
• “Descriptive” comparison of achievementsalong some dimension– “League tables” of GDP, GDP growth, etc.
• “Normative” comparison of performance along some dimension, controlling for inputs and external circumstances– Schools, hospitals
![Page 3: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/3.jpg)
Regression residuals as performance measures
![Page 4: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/4.jpg)
• Assume a causal relationship:Outcomei = a + b’Xi + ei
• Statistical fittingPredicted outcomei = α + β’Xi
Actual outcomei = α + β’Xi + εi
• Use difference as performance measure:Performancei = εi
Normalised performancei = εi/[α + β’Xi]
A simple way of constructing performance measures
![Page 5: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/5.jpg)
Absolute versus relative performance
• “Relative performance”: Compare countries on how they perform relative to predicted achievement– WHO 1999 study of country performance
• “Absolute performance”: Compare countries on their position between zero and maximum possible efficiency– WHO’s index of “health system performance”
![Page 6: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/6.jpg)
Conditioning on what?• Will depend on the purpose of the
comparison and the unit of analysis whose performance is being evaluated– Inputs: Condition on the resources “available”
to the unit– External circumstances: Condition on the
factors which “should” not be attributed to the unit’s performance
• Explaining performance: We may identify the controllable causes, but must include them in the performance measure
![Page 7: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/7.jpg)
A generic model
• Suppose we estimate the following model:
Xi = α + β’INPi + γ’EXTi + δ’INTi + εi
where
X is the outcome of interest (e.g. health)
INP are inputs (e.g. education, income)
EXT are external factors (e.g. geography)
INT are internal factors (e.g. number of doctors)
• δ’INTi + εi measures how well i performs relative to how it “should”
![Page 8: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/8.jpg)
Examples
• Hospital performance
• School performance– Chicago public schools, MBA programs
• Health performance of countries and country health systems
• Economic performance: Total Factor Productivity Growth
![Page 9: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/9.jpg)
Pitfalls: The case of WHR 2000• World Health Report 1999 had investigated
country performance in health• World Health Report 2000 set out to
measure “health system performance”• Absolute performance concept: Produced an
index from 0 to 1 where 0 equivalent to no health system, 1 the best possible system
• Used a residual from regression of disability-adjusted life expectancy on education and health expenditure per capita
![Page 10: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/10.jpg)
WHR 2000 methodology
![Page 11: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/11.jpg)
WHR 2000 methodology
• Efficiency index defined by:
[ODi + (PHOi – LBi)]/[ODmax + (PHOi –LBi)]
or
[HOi – LBi]/[ODmax + (PHOi – LBi)]
• Note an equivalent relative performance index would be:
ODi/PHOi
![Page 12: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/12.jpg)
Problems with WHR 2000 index• Using absolute performance requires more
guesswork and is unnecessarily obscure when the goal is cross-comparison
• Inadequate partitioning of variables:– Controls for inputs like education and health
spending, but not for circumstances external to the health system, like economic policy
• Jamison and Sandbu (2001) test the robustness of the WHR 2000 ranking:– Repeat exercise but control for geography
![Page 13: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/13.jpg)
Including geography controls
![Page 14: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/14.jpg)
Including geography controls• When two geography variables are included
(tropical location and access to sea), ranks change dramatically
• Only 17 out of 96 countries remain within “uncertainty interval”
• The amended ranking not necessarily better: Still many conditioning variables left out
![Page 15: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/15.jpg)
Three measures of performance• Note that achievements can be compared in
three ways:– Achievement at point in time
– Growth rate of achievement levels
– Inputs to the achievement of outcomes
• Similar distinction for performance:– Actual relative to predicted outcome
– Actual relative to predicted change over time
– Actual relative to predicted effect of inputs
![Page 16: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/16.jpg)
Country performance in healthat a point in time and over time
Jamison, Sandbu and Wang’s (2004) model of infant mortality rates (IMR), estimated for 94 countries over 25-year period:
LIMRit = β0i + β1i TIMEt + β2 LY5it + β3 FEDUCit + εit
LIMR = 1n(IMR)
LY5 = 1n (per capita income)
FEDUC = Female education level
(all in country i at time t, 5-year intervals)
![Page 17: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/17.jpg)
Residuals• Model of the country-specific coefficients:
– Intercept: β0i = γ00 + γ01TROPICSi + γ02COASTALi + µ0i
– Time trend:β1i = γ10 + γ11TROPICSi + γ12COASTALi + µ1i
(where µ0i and µ1i are normally distributed with mean zero)
– Total residual:
µ0i + TIMEt* µ1i
![Page 18: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/18.jpg)
Performance
• How much lower than predicted was IMR in 1962? Beginning-of-period performance:
BPi ≡ 1 – exp(µ0i)
• How much faster than predicted did IMR fall? Within-period performance:
WPi ≡ 1 – exp(25*µ1i)
• We can combine the measures.End-of-period performance:
WPi ≡ 1 – exp(µ0i + 25*µ1i)
![Page 19: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/19.jpg)
End-of-period performance:Low-/middle-income countries
0
50
100
150
200
250
Costa R
ica
Sudan, Z
imbab
weVen
ezuela
Sri Lan
ka, M
alays
iaBulg
aria
Kenya
Bolivia
Sierra
Leo
ne
Lesoth
o, Guin
ea B
issau
Indones
ia, T
urkey
The Gam
biaBan
glades
h
Country
% o
f Pre
dic
ted
IMR
![Page 20: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/20.jpg)
End-of-period performance:High-income countries
0
50
100
150
Singap
ore
Hong K
ong
Finlan
dSpa
inSwed
en
Canad
aUK
Israe
lUS
New Z
ealan
d
Country
% o
f Pre
dict
ed IM
R
![Page 21: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/21.jpg)
Input efficiency performance• Work by Or, Wang and Jamison estimates
the following model for OECD countries:
HOit = β0i + β1iDOCit + γ’Xit + εitwith
HO = various health outcomesDOC = number of doctors per capitaX = GDP/capita, education, tobacco, alcohol,
private/public financing mix
• Country-specific health productivity:
β1i = β1 + µ4iand µ4i normally distributed with mean zero
![Page 22: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/22.jpg)
Findings for IMRInfant Mortality
a . Lines represent the 95 percent confidence intervals for country estimates.
Rank coefficients are multiplied by (-1) for visual convenience.
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
![Page 23: Using Regression Residuals as Performance Measures ... · Using Regression Residuals as Performance Measures: Pitfalls and Possibilities Martin E. Sandbu Center on Globalization and](https://reader035.fdocuments.in/reader035/viewer/2022062908/5aefb7f87f8b9ac57a8db829/html5/thumbnails/23.jpg)
Conclusions• Regression residuals can be used as
performance measures for normative comparison
• Caution required in identifying appropriate conditioning variables– Partial out inputs and external factors
– Don’t partial out factors the institution controls
• Methods allow rich performance analysis:– Decompose performance into various types
– Decompose performance into various causes