1
Helsinki University of Technology Systems Analysis Laboratory
INFORMS 2007 Seattle
Efficiency and Sensitivity Analyses in the Efficiency and Sensitivity Analyses in the Evaluation of University DepartmentsEvaluation of University Departments
Ahti Salo and Antti PunkkaHelsinki University of Technology
Systems Analysis Laboratory02015 TKK, Finland
Helsinki University of TechnologySystems Analysis Laboratory
2INFORMS 2007 Seattle
ContextContext National efforts to increase the efficiency of universities
– ”Productivity programme” ~1500 positions to be slashed in 2007-10– Efficiency studies commissioned by the Ministry of Finance
» ”Measurable Productivity in Universities” by Gov’t Econ. Research Cntr in 09/06
Developments at Helsinki University of Technology (TKK)– Rector asked us to produce comments to the above report – TKK has been using various resource allocations models over the years – Considerable dissatisfaction with many of these models– Resources Committee requested to develop different principles
Tasks Develop value efficiency models in support of resource allocation Explore methodological extensions in view of decision making needs
Helsinki University of TechnologySystems Analysis Laboratory
4INFORMS 2007 Seattle
Efficiency of University DepartmentsEfficiency of University Departments Departments consume inputs in order to produce outputs
Valuation of inputs and outputs involves subjective preferences
University / Department
x1 (Budget funding) y1 (Master’s Theses)
y2 (Doctor’s Theses)
y3 (Int’l publications)
x2 (Project funding)
VALUE OF OUTPUTSEFFICIENCY
VALUE OF INPUTS
Helsinki University of TechnologySystems Analysis Laboratory
5INFORMS 2007 Seattle
Data Envelopment Analysis Data Envelopment Analysis (Charnes et al., 1978)(Charnes et al., 1978)
Approach – Multiple inputs xi and outputs yi of decision making units (DMU) aggregated
by non-negative multipliers (’weights’) – Efficiency ratio of each DMU is maximed, subject to the condition that
this ratio does not exceed one for any DMUs
Observations– Extending the set of inputs cannot worsen the efficiency of any DMU – In Value Efficiency Analysis, the DMs’ preferences are explicitly modelled
(VEA, Halme et al., 1999; Korhonen and Syrjänen, 1998)
,max ( , )
1,
o i io i ioi i
i is i isi i
E y x
y x s S
Helsinki University of TechnologySystems Analysis Laboratory
6INFORMS 2007 Seattle
Valuation of Inputs and Outputs Valuation of Inputs and Outputs Preferences elicited from the Resources Committee
How valuable are the different outputs in relative terms? – What is the value of an MSc degree relative to a PhD degree etc?– 44 outputs from the reporting system using 3-year annual averages
» Degrees granted – publications activity – international activities » Mitigation of impacts due to large annual fluctuations
How important are budget funding and project funding in terms of producing this output?
Helsinki University of TechnologySystems Analysis Laboratory
9INFORMS 2007 Seattle
Feasible valuations – Responses by individual respondents plus convex combinations thereof
Efficient departments (efficiency = 1) – For some feasible valuation of inputs and outputs, the efficiency ratio of a
this Dept is either greater than or equal to that of all other Depts
Inefficient deparments (efficiency < 1) – For all feasible valuations, the efficiency ratio of some other Dept is strictly
greater
If the aim is to maximize overall efficiency and Depts increase their outputs in proportion to the use of inputs, resources should be shifted from inefficient to efficient Depts
Feasible Valuations and EfficienciesFeasible Valuations and Efficiencies
Helsinki University of TechnologySystems Analysis Laboratory
10INFORMS 2007 Seattle
Efficiencies of departmentsEfficiencies of departments Very significant differences in departmental efficiencies
Results still in alignment with resource allocation models
0,00
0,10
0,20
0,30
0,40
0,50
0,60
0,70
0,80
0,90
1,00
A B C D E F G H I J K L TKK
Department
Eff
icie
nc
y
Helsinki University of TechnologySystems Analysis Laboratory
11INFORMS 2007 Seattle
Motivations for Methodological Extensions Motivations for Methodological Extensions Results of Value Efficiency Analysis may not be robust
– Introduction of an outlier may produce radical changes in efficiency results – Hence the results may appear counterintuitive to DMs
Pairwise dominances among DMUs– It may be of interest to enable comparisons among all DMUs – Efficient DMUs need not be greatest relevance for very inefficient DMUs
Rank-based information about relative efficiencies – Ranking lists (e.g., Shanghai Jian Tao University) have been influential– Yet this list (and many others) do not account for the value of inputs – Hence the interest to examine efficiencies in terms of rankings, too
Helsinki University of TechnologySystems Analysis Laboratory
12INFORMS 2007 Seattle
Pairwise Efficiency Dominance of DMUsPairwise Efficiency Dominance of DMUs For any feasible input and output valuations ,
the efficiency of DMU s is defined as
Definition: If DMU s and DMU t are such that
for all feasible input and output valuations (with strict inequality for some feasible valuations), then DMU s dominates DMU t.
( , ) i isi Ms
i isi N
yE
x
,
( , )min 1 0
( , )s
t
E
E
,S S
Helsinki University of TechnologySystems Analysis Laboratory
13INFORMS 2007 Seattle
Pairwise Efficiency Dominance of DMUsPairwise Efficiency Dominance of DMUs
Definition: If the efficiency ratio of DMU s is greater than or equal to that of DMU t for all ,
(with strict inequality for some feasible valuations), then DMU s dominates DMU t.
This dominance holds if the minimum
is positive
,S S
( , ) ( , ) ,s tE E S S
,
( , )min 1
( , )s
t
E
E
Helsinki University of TechnologySystems Analysis Laboratory
14INFORMS 2007 Seattle
Pairwise Dominance Pairwise Dominance This minimization problem gives a lower bound on how
much more efficient DMU s is incomparison with DMU t
DMU t
DM
U s
Helsinki University of TechnologySystems Analysis Laboratory
15INFORMS 2007 Seattle
Ranking of DMUs’ Efficiencies Ranking of DMUs’ Efficiencies Definition: Let , be a feasible valuation. The ranking of
DMU t among DMUs S is
The ranking of the most efficient DMU is 1 If several DMUs have the same efficiency ratio, they have a tie with the same ranking
Different feasible valuations assign different rankings to DMUs
Best and worst possible of rankings computed with an MILP model
( , ) 1 |{ | ( , ) ( , )} |t s tr s S E E
Helsinki University of TechnologySystems Analysis Laboratory
16INFORMS 2007 Seattle
Ran
king
Ranges of Rankings for TKK DepartmentsRanges of Rankings for TKK Departments
0,00
0,10
0,20
0,30
0,40
0,50
0,60
0,70
0,80
0,90
1,00
A B C D E F G H I J K L TKK
Department
Eff
icie
nc
y
Helsinki University of TechnologySystems Analysis Laboratory
17INFORMS 2007 Seattle
Ranking List of Shanghai Jia Tong-University Ranking List of Shanghai Jia Tong-University
Helsinki University of TechnologySystems Analysis Laboratory
18INFORMS 2007 Seattle
Weight Sensitivity of Rankings Weight Sensitivity of Rankings
Helsinki University of TechnologySystems Analysis Laboratory
19INFORMS 2007 Seattle
ConclusionsConclusions Lessons learned
– Different models complement each other – Thinking about the value of intangibles is useful - does our data matter?– Efficiency analysis alone does not suggest strategic changes
Useful methodological extensions– Inter-departmental comparisons supported by pairwise comparisons – Ranges of rankings show sensitivities in the relative efficiency of Depts
Possible extensions – Development of analyses to account for intermediate inputs/outputs – Explicit linkages to resource allocation through goal-setting – Interactive decision support tools with Internet-based user interfaces
Top Related