1 Helsinki University of Technology Systems Analysis Laboratory INFORMS 2007 Seattle Efficiency and...

1

Helsinki University of Technology Systems Analysis Laboratory

INFORMS 2007 Seattle

Efficiency and Sensitivity Analyses in the Efficiency and Sensitivity Analyses in the Evaluation of University DepartmentsEvaluation of University Departments

Ahti Salo and Antti PunkkaHelsinki University of Technology

Systems Analysis Laboratory02015 TKK, Finland

[email protected]

Helsinki University of TechnologySystems Analysis Laboratory

2INFORMS 2007 Seattle

ContextContext National efforts to increase the efficiency of universities

– ”Productivity programme” ~1500 positions to be slashed in 2007-10– Efficiency studies commissioned by the Ministry of Finance

» ”Measurable Productivity in Universities” by Gov’t Econ. Research Cntr in 09/06

Developments at Helsinki University of Technology (TKK)– Rector asked us to produce comments to the above report – TKK has been using various resource allocations models over the years – Considerable dissatisfaction with many of these models– Resources Committee requested to develop different principles

Tasks Develop value efficiency models in support of resource allocation Explore methodological extensions in view of decision making needs



Efficiency of University DepartmentsEfficiency of University Departments Departments consume inputs in order to produce outputs

Valuation of inputs and outputs involves subjective preferences

University / Department

x1 (Budget funding) y1 (Master’s Theses)

y2 (Doctor’s Theses)

y3 (Int’l publications)

x2 (Project funding)

VALUE OF OUTPUTSEFFICIENCY

VALUE OF INPUTS



Data Envelopment Analysis Data Envelopment Analysis (Charnes et al., 1978)(Charnes et al., 1978)

Approach – Multiple inputs xi and outputs yi of decision making units (DMU) aggregated

by non-negative multipliers (’weights’) – Efficiency ratio of each DMU is maximed, subject to the condition that

this ratio does not exceed one for any DMUs

Observations– Extending the set of inputs cannot worsen the efficiency of any DMU – In Value Efficiency Analysis, the DMs’ preferences are explicitly modelled

(VEA, Halme et al., 1999; Korhonen and Syrjänen, 1998)

,max ( , )

1,

o i io i ioi i

i is i isi i

E y x

y x s S



Valuation of Inputs and Outputs Valuation of Inputs and Outputs Preferences elicited from the Resources Committee

How valuable are the different outputs in relative terms? – What is the value of an MSc degree relative to a PhD degree etc?– 44 outputs from the reporting system using 3-year annual averages

» Degrees granted – publications activity – international activities » Mitigation of impacts due to large annual fluctuations

How important are budget funding and project funding in terms of producing this output?



Feasible valuations – Responses by individual respondents plus convex combinations thereof

Efficient departments (efficiency = 1) – For some feasible valuation of inputs and outputs, the efficiency ratio of a

this Dept is either greater than or equal to that of all other Depts

Inefficient deparments (efficiency < 1) – For all feasible valuations, the efficiency ratio of some other Dept is strictly

greater

If the aim is to maximize overall efficiency and Depts increase their outputs in proportion to the use of inputs, resources should be shifted from inefficient to efficient Depts

Feasible Valuations and EfficienciesFeasible Valuations and Efficiencies



Efficiencies of departmentsEfficiencies of departments Very significant differences in departmental efficiencies

Results still in alignment with resource allocation models

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90

1,00

A B C D E F G H I J K L TKK

Department

Eff

icie

nc

y



Motivations for Methodological Extensions Motivations for Methodological Extensions Results of Value Efficiency Analysis may not be robust

– Introduction of an outlier may produce radical changes in efficiency results – Hence the results may appear counterintuitive to DMs

Pairwise dominances among DMUs– It may be of interest to enable comparisons among all DMUs – Efficient DMUs need not be greatest relevance for very inefficient DMUs

Rank-based information about relative efficiencies – Ranking lists (e.g., Shanghai Jian Tao University) have been influential– Yet this list (and many others) do not account for the value of inputs – Hence the interest to examine efficiencies in terms of rankings, too



Pairwise Efficiency Dominance of DMUsPairwise Efficiency Dominance of DMUs For any feasible input and output valuations ,

the efficiency of DMU s is defined as

Definition: If DMU s and DMU t are such that

for all feasible input and output valuations (with strict inequality for some feasible valuations), then DMU s dominates DMU t.

( , ) i isi Ms

i isi N

yE

x

,

( , )min 1 0

( , )s

t

E

E

,S S



Pairwise Efficiency Dominance of DMUsPairwise Efficiency Dominance of DMUs

Definition: If the efficiency ratio of DMU s is greater than or equal to that of DMU t for all ,

(with strict inequality for some feasible valuations), then DMU s dominates DMU t.

This dominance holds if the minimum

is positive

,S S

( , ) ( , ) ,s tE E S S

,

( , )min 1

( , )s

t

E

E



Pairwise Dominance Pairwise Dominance This minimization problem gives a lower bound on how

much more efficient DMU s is incomparison with DMU t

DMU t

DM

U s



Ranking of DMUs’ Efficiencies Ranking of DMUs’ Efficiencies Definition: Let , be a feasible valuation. The ranking of

DMU t among DMUs S is

The ranking of the most efficient DMU is 1 If several DMUs have the same efficiency ratio, they have a tie with the same ranking

Different feasible valuations assign different rankings to DMUs

Best and worst possible of rankings computed with an MILP model

( , ) 1 |{ | ( , ) ( , )} |t s tr s S E E



Ran

king

Ranges of Rankings for TKK DepartmentsRanges of Rankings for TKK Departments

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90

1,00

A B C D E F G H I J K L TKK

Department

Eff

icie

nc

y



Ranking List of Shanghai Jia Tong-University Ranking List of Shanghai Jia Tong-University



Weight Sensitivity of Rankings Weight Sensitivity of Rankings



ConclusionsConclusions Lessons learned

– Different models complement each other – Thinking about the value of intangibles is useful - does our data matter?– Efficiency analysis alone does not suggest strategic changes

Useful methodological extensions– Inter-departmental comparisons supported by pairwise comparisons – Ranges of rankings show sensitivities in the relative efficiency of Depts

Possible extensions – Development of analyses to account for intermediate inputs/outputs – Explicit linkages to resource allocation through goal-setting – Interactive decision support tools with Internet-based user interfaces

1 Helsinki University of Technology Systems Analysis Laboratory INFORMS 2007 Seattle Efficiency and...

Documents

Transcript of 1 Helsinki University of Technology Systems Analysis Laboratory INFORMS 2007 Seattle Efficiency and...