NTTS Conference, Brussels, 18-20 February 20091 OECD/JRC Handbook on constructing composite...

NTTS Conference, Brussels, 18-20 February 2009 1

OECD/JRC Handbook on

constructing composite indicators-

Putting theory into practice

Michela Nardo & Michaela Saisana

[email protected] & [email protected]

JRC-IPSC- G09

Econometric and Applied Statistics Unit

NTTS (New Techniques and Technologies for Statistics) Seminar

Brussels, 18-20 February 2009


Step 1. Developing a theoretical framework

Step 2. Selecting indicators

Step 3. Imputation of missing data

Step 4. Multivariate analysis

Step 5. Normalisation of data

Step 6. Weighting and aggregation

Step 7. Robustness and sensitivity

Step 8. Back to the details (indicators)

Step 9. Association with other variables

Step 10. Presentation and dissemination

The ten recommended steps

JRC/OECD Handbook on composite indicators [… and ranking systems]

http://composite-indicators.jrc.ec.europa.eu/

2 rounds of consultation with OECD high level statistical committee

Finally endorsed in March 2008


What is badly defined is likely to be badly measured….

This means :

• To give a definition of the phenomenon to describe

• To define the number and identity of indicators

• To describe the nested structure of these indicators

Step 1. Developing a Theoretical/conceptual framework


a. SAFETY & SECURITY

b. RULE OF LAW, TRASPARENCY & CORRUPTION

c. PARTICIPATION & HUMAN RIGHTS

d. SUSTAINABLE ECONOMIC OPPORTUNITY

e. HUMAN DEVELOPMENT

PIL

LA

RS

SU

B -

PIL

LA

RS

a.1 National security – weight 2/3

a.2 Public safety – weight 1/3

5 quantitative indicators1 qualitative indicator

1 qualitative indicator

b.1 Ratification of norms – weight 1/3

b.2 Judicial independence – weight 1/3

b.3 Corruption – weight 1/3

2 qualitative indicators1 quantitative indicator

3 quantitative indicators

1 quantitative indicator

c.1 Participation in elections – weight 1/2

c.2 Respect for rights – weight 1/2

4 qualitative indicators

3 qualitative indicators1 quantitative indicator

d.1 Wealth creation – weight 1/4

d.2 Macroeconomic stability&financialintegrity – weight 1/4

d.3 Arteries of commerce – weight 1/4

d.4 Environmental sensitivity – weight 1/4




1 quantitative indicator -EPI

e.1 Poverty – weight 1/3

e.2 Health & sanitation – weight 1/3

e.3 Educational opportunity – weight 1/3






Framework: a way of encoding a natural/social system into a formal system

No clear way

Definition of the phenomenon

Systems are complex

A framework is valid only within a giveninformation space

Formal coherence but also reflexivity



Problems found in practice…• Go from an ideal to a workable framework

• Recognize that a framework is only a possible way of representing the reality

• Recognize that a framework is shaped by the objective of the analysis – implies the definition of selection criteria (e.g. input, process, output for measuring innovation activity)

• Define the direction of an indicator (i.e. define the objective of the CI). E.g. social cohesion – OECD lists 35 indicator among which number of marriages and divorces, the fertility rate, number of foreigners,.. BUT Does a large group of foreign-born population decrease social cohesion?

• Select sound variables (merge different source of information, soft data, different years, …)



A composite indicator is above all the sum of its parts…

Step 2. Selecting variables


Dempster and Rubin (1983), “The idea of imputation is both seductive and dangerous”



Rule of thumb: (Little Rubin ‘87) If a variable has more than 5% missing values, cases are not deleted, and many researchers are much more stringent than this.

Case deletion

Pros: no artificial assumptions

Cons: higher standard error (less info is used)

Risk: think that the dataset is complete



Imputation with MAR and MCAR assumptions

Single imputation: one value for each missing data

implicit modelingexplicit modeling

Multiple imputation: more than 1 value for each missing data (Markov Chain Monte Carlo algorithm)



Analysing the underlying structure of the data is still an art …

Step 4. Multivariate Analysis


Grouping information on individual indicators

•Is the nested structure of the composite indicator well-defined?•Is the set of available individual indicators sufficient/ appropriate to describe the phenomenon?

Expert opinion and Multivariate statistics

Revision of the indicators or the structure



Correlation one may see a high correlation among sub-indicators as something to correct for, e.g. by making the weights inversely proportional to the strength of the overall correlation for a given sub-indicator,

e.g. Index of Relative Intensity of Regional Problems in the EU (Commission of the European Communities, 1984).

schools of thought

Practitioners of multicriteria decision analysis would instead tend to consider the existence of correlations as a feature of the problem, not to be corrected for, as correlated sub-indicators may indeed reflect non-compensable different features of the problem.

e.g. A car’s speed and beauty are likely correlated with one another, but this does not imply that we are willing to trade speed for design.



a1 1 a2 0.85 1 a3 0.83 0.87 1 a4 0.77 0.81 0.89 1 a5 0.56 0.55 0.72 0.72 1 a6 0.64 0.61 0.66 0.73 0.56 1 b1 0.50 0.72 0.69 0.63 0.40 0.54 1 b2 0.61 0.80 0.75 0.74 0.62 0.54 0.86 1 b3 0.43 0.39 0.54 0.54 0.39 0.48 0.10 0.29 1 b4 0.35 0.26 0.41 0.30 0.31 0.37 0.16 0.24 0.75 1 b5 0.70 0.49 0.48 0.45 0.60 0.31 0.09 0.38 0.21 0.13 1 b6 0.55 0.75 0.78 0.77 0.59 0.57 0.77 0.83 0.44 0.24 0.29 a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5

Component Initial Eigenvalues Total % of variance Cumulative %

1 7.242 60.35 60.35 2 1.523 12.69 73.04 3 1.178 9.82 82.86 4 0.554 4.61 87.47 5 0.512 4.26 91.73 6 0.385 3.20 94.94 7 0.242 2.01 96.95 8 0.131 1.09 98.04 9 0.098 0.82 98.04

10 0.064 0.54 99.40 11 0.043 0.36 99.76 12 0.029 0.24 100.00

0

1

2

3

4

5

6

7

8

0 2 4 6 8 10 12 14

Component number

Eig

enva

lues


Principal Component Analysis


Grouping information on countries/indicators

Cluster analysis serves as: • a purely statistical method of aggregation of the indicators, • a diagnostic tool for exploring the impact of the methodological choices made during the construction phase of the composite indicator, • a method of disseminating information on the composite indicator without losing that on the dimensions of the individual indicators, and • a method for selecting groups of countries to impute missing data with a view to decreasing the variance of the imputed values.

Cluster analysis



Tree Diagram for 27 Cases

Single Linkage

Euclidean distances

2 3 4 5 6 7 8

Linkage Distance

MEXTURPOL

ISLKORJPNCZECHESWEUKMNZL

DEUUSACANNORNLDESP

HUNBELFRALUXFIN

PRTITA

AUTDNKAUS

Cluster analysis: group information on countries based on their similarity on different individual indicators.



Avoid adding up apples and oranges …



Convert variables expressed in different measurement units (e.g. no. of people, money per capita, % of total exports) into a single (dimensionless) unit

Issue: which is the most suitable normalization

robustness



Method Equation

1. Ranking )x(RankI tqc

tqc

2. Standardization (or z-scores)

tcqc

tcqc

tqct

qc

xxI

3. Re-scaling

)x(min)x(max

)x(minxI

00

0

tqc

tqc

tqc

tqct

qc

.

4. Distance to reference country

0tcqc

tqct

qcx

xI

or 0

0

tcqc

tcqc

tqct

qcx

xxI

5. Logarithmic transformation )xln(I tqc

tqc

6. Categorical scales

...

60yx

80yx

100yx

tqc

tqc

tqc

tqc

tqc

tqc

else then percentile th- 35upperthe in if

else then percentile th-15 upperthe in if

else then percentile th-5 upperthe in if

Pro: simple, indep. to outliersCons: levels lost

Pro: no distortion from meanCons: outliers

Pro: simpleCons: outliers

Pro: benchmarkingCons: max be outliers

Pro: reduce skewnessCons: values near to 0

Pro: indep on slight changes in def. Cons: no track slight improvements



7. Indicators above or below the mean

0Ip)1(xx)p1(

1Ip)1(xx

1Ip)1(xx

tqc

tcqc

tqc

tqc

tcqc

tqc

tqc

tcqc

tqc

0

0

0

then if

then if

then if

8. Cyclical indicators (OECD) t

qcttqct

tqct

tqct

qcxExE

xExI

9. Balance of opinions (EC) eN

e

1tqc

tqce

e

tqc xxsgn

N

100I

10. Percentage of annual differences over consecutive years

tqc

1tqc

tqct

qc x

xxI

Pro: simple, no outliers Cons: p arbitrary, no absolute levels

Pro: minimize cyclesCons: penalize irregular indic.

Pro: based on opinionsCons: no absolute levels

Pro: no levels but changesCons: low levels are equal to high levels



The relative importance of the indicators is a source of contention …

6. Weighting and aggregation

should be done along the lines of the underlying theoretical framework.

To select appropriate weighting and aggregation procedure(s) that respect both the theoretical framework and the data properties.

To discuss whether correlation issues among indicators should be accounted for.

To discuss whether compensability among indicators should be allowed.



No general consensus

A composite cannot be “objective”

Weighting implies a subjective choice

Soundness and transparency



Weights based on statistical models

Factor analysisBenefit of the doubtRegressionUnobserved components model

Weights based on opinions

Budget allocationPublic opinionAnalytic hierarchy processConjoint analysis


Pros: deals with correlation

Cons:• correlation does not mean redundancy• only used when correlation• sensitive to factor extraction and rotation

Factor analysis

Table 17. Factor loadings of TAI based on maximum likelihood

M L PCA Patents 0.19 0.17 Royalties 0.20 0.15 Internet 0.07 0.11 Tech exports 0.07 0.07 Telephones 0.15 0.08



Indicator 1

Indicator 2

a

b

c

d

d’

0

Source: Rearranged from Mahlberg and Obersteiner (2001).

The performance indicator is the ratio of the distance between the origin and the actual observed point and that of the projected point in the frontier:

'd0/d0

.


Benefit of the doubt

Data Envelopment Analysis (DEA) employs linear programming tools to estimate an efficiency frontier that would be used as a benchmark to measure the relative performance of countries


Benefit of the doubt

Pros: * the composite will be sensible to national priorities * real benchmark * any other weighting scheme gives a lower index * reveal policy priorities

Cons: * weights are country specifics (no comparability) * a priori constraints on weights * weights depend on the benchmark


But the cross-efficient DEA allows ranking (and comparability)


Budget allocationThe budget allocation process (BAP) has four different phases:

• Selection of experts for the valuation;• Allocation of budget to the individual indicators;• Calculation of the weights;

Pros: weights not based on statistical manipulation experts tends to increase legitimacy and acceptanceCons: reliability (weights could reflect local conditions) not suited for more than 10 indicators at the time



Analytic hierarchy processThe core of AHP (Saaty 70s) is an ordinal pair-wise comparison of attributes. The comparisons are made per pairs of individual indicators: Which of the two is the more important? And, by how much? The preference is expressed on a semantic scale of 1 to 9. A preference of 1 indicates equality between two individual indicators, while a preference of 9 indicates that the individual indicator is 9 times more important than the other one. The results are represented in a comparison matrix (Table 20), where Aii = 1 and Aij = 1 / Aji.

Table 20. Comparison matrix of eight TAI individual indicators

Objective Patents Royalties Internet Tech exports Telephone Electricity Schooling University

Patents 1 2 3 2 5 5 1 3 Royalties 1/2 1 2 1/2 4 4 ½ 3 Internet 1/3 ½ 1 1/4 2 2 1/5 1/2 Tech. exports 1/2 2 4 1 4 4 1/2 3 Telephones 1/5 1/4 1/2 1/4 1 1 1/5 1/2 Electricity 1/5 ¼ 1/2 1/4 1 1 1/5 1/2 Schooling 1 2 5 2 5 5 1 4 University 1/3 1/3 2 1/3 2 2 1/4 1



Analytic hierarchy process

Pros: suitable for qualitative and quantitative data Cons: high number of pairwise comparisons depends on the setting of the experiment



Aggregation rules:

Linear aggregation implies full (and constant) compensability rewards sub-indicators proportionally to the weights

Geometric mean entails partial (non constant) compensability rewards more those countries with higher scores

The absence of synergy or conflict effects among the indicators is a necessary condition to admit either linear or geometric aggregation

&weights express trade-offs between indicators

Multi-criteria analysisdifferent goals are equally legitimate and important (e.g. social, economic dimensions), thus

a non-compensatory logic



When different goals are equally legitimate and important, then a non compensatory logic may be necessary.

Example: physical, social and economic figures must be aggregated. If the analyst decides that an increase in economic performance can not compensate a loss in social cohesion or a worsening in environmental sustainability, then neither the linear nor the geometric aggregation are suitable.

Instead, a non-compensatory multicriteria approach will assure non compensability by formalizing the idea of finding a compromise between two or more legitimate goals.

+ does not reward outliers + different goals are equally legitimate and important + no normalisation is required

BUT- computational cost when the number of countries is high

Multi-criteria type of aggregation




“Cynics say that models can be built to conclude anything provided that suitable assumptions are fed into them.”

[The Economist, 1998]

“many indices rarely have adequate scientific foundations to support precise rankings: […] typical practice is to acknowledge uncertainty in the text of the report and then to present a table with unambiguous rankings”

[Andrews et al. (2004: 1323)]

Step 7. Robustness and Sensitivity


Uncertainty + Sensitivity Analysis


stakeholder involvement

Steps in the Development of an Index

Step 1. Developing a theoretical framework

Step 2. Selecting indicators


Step 4. Multivariate analysis




Step 8. Links to other variables

Step 9. Back to the details

Step 10. Presentation and dissemination

Sources of uncertainty


As a result, an uncertainty analysis should naturally include a careful mapping of all these uncertainties (/assumptions) onto the space of the output (/inferences). Two things can happen:

The latter outcome calls in turn for a revision of the ranking system, or a further collection of indicators.

The space of the inference is still narrow enough, so as to be meaningful.

The space of the inference is too wide to be meaningful.



Why the ten steps?

The three pillars of a well-designed CI:1. Solid conceptual framework, 2. good quality data,3. sound methodology (+ robustness

assessment)

Composite indicators between analysis and advocacy, Saltelli, 2007, Social Indicators Research, 81:65-77

If the 3 conditions are met then CIs can be used for policy-making


Some recent CIs developed in collaboration with the JRC using the philosophy of the Handbook

2008 European Lifelong Learning Index (Bertelsmann Foundation, CCL)2008/2006 Environmental Performance Index (Yale & Columbia University)2007 Alcohol Policy Index (New York Medical College)2007 Composite Learning Index (Canadian Council on Learning)2002/2005 Environmental Sustainability Index (Yale & Columbia University)


http://composite-indicators.jrc.ec.europa.eu/


Figure 1. Relationship between the Composite Learning Index and the Economic and Social Well-Being Index in Canada.

y = 0.7691x + 20.249

R2 = 0.6979

40

50

60

70

80

90

100

40 50 60 70 80 90 100

Composite Learning Index

Ec

on

om

ic a

nd

So

cia

l W

ell

-be

ing

The Composite Learning Index (Canadian Council on Learning)

Results

Policy message

Sensitivity analysis

Composite Learning Index (Canadian Council on

Learning)

Framework

Scenario

Pillar Structure Normalisation Weighting Aggregation

CLI Preserved z-scores FA within pillar, Regression weights to Factors, FA pillars, Regression weights to pillars

Linear

S1 Preserved z-scores FA within pillar, FA pillars Linear S2 Preserved Min-max FA within pillar, FA pillars Linear S3 Not preserved z-scores FA all indicators Linear S4 Not preserved Min-max FA all indicators Linear S5 Preserved z-scores FA within pillar, EW pillars Linear S6 Preserved Min-max FA within pillar, EW pillars Linear S7 Not preserved z-scores EW all indicators Linear S8 Not preserved Min-max EW all indicators Linear S9 Preserved z-scores EW within pillar, EW pillars Linear S10 Preserved Min-max EW within pillar, EW pillars Linear S11 Preserved z-scores FA within pillar, FA pillars Geometric S12 Preserved Min-max FA within pillar, FA pillars Geometric S13 Not preserved z-scores FA all indicators Geometric S14 Not preserved Min-max FA all indicators Geometric S15 Preserved z-scores FA within pillar, EW pillars Geometric S16 Preserved Min-max FA within pillar, EW pillars Geometric S17 Not preserved z-scores EW all indicators Geometric S18 Not preserved Min-max EW all indicators Geometric S19 Preserved z-scores EW within pillar, EW pillars Geometric S20 Preserved Min-max EW within pillar, EW pillars Geometric S21 Preserved Raw data FA within pillar, FA pillars Multi-criteria S22 Not preserved Raw data FA all indicators Multi-criteria S23 Preserved Raw data FA within pillar, EW pillars Multi-criteria S24 Not preserved Raw data EW all indicators Multi-criteria S25 Preserved Raw data EW within pillar, EW pillars Multi-criteria


Framework (WHO report)

Results

Policy message

Sensitivity analysis

The Alcohol Policy Index (New York Medical College)

Comparative Analysis of Alcohol Control Policies in 30 Countries

Brand, Saisana, Rynn, Pennoni, Lowenfels, 2007, PLoS Medicine, 4(4):752-759


The Alcohol Policy Index (New York Medical

College)

Maybe an example of a country where strong laws are poorly enforced.

high estimated amount of unrecorded consumption (>50% of the total).

predominantly Islamic population


Our Tools for Sensitivity Analysis

free software

Two fractional variance measures First order effect – one factor

influence by itself

Total effect - influence inclusive of all interactions with other

factors

Y

iXTi V

YVES ii

XX

Y

iXi V

XYEVS ii X


Saltelli A., M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, S. Tarantola, 2008, Global sensitivity analysis. The Primer, John Wiley & Sons, England.

OECD/JRC, 2008, Handbook on Constructing Composite Indicators. Methodology and user Guide, OECD Publishing, ISBN 978-92-64-04345-9. Authored by Nardo M., Saisana M., Saltelli A., Tarantola S., Hoffman A., Giovannini E.

Selected References

Books


Brand, D. A., Saisana, M., Rynn, L. A., Pennoni, F., Lowenfels, A. B., 2007, Comparative Analysis of Alcohol Control Policies in 30 Countries, PLoS Medicine 4(4):752-759.

Cherchye, L., Moesen, W., Rogge, N., van Puyenbroeck, T., Saisana, M., Saltelli, A., Liska, R., Tarantola, S., 2008, Creating Composite Indicators with DEA and Robustness Analysis: the case of the Technology Achievement Index, Journal of Operational Research Society 59:239-251.

Saisana, M., Saltelli, A., Tarantola, S., 2005, Uncertainty and sensitivity analysis techniques as tools for the analysis and validation of composite indicators, Journal of the Royal Statistical Society A 168(2):307-323.

Saltelli, A., M. Ratto, S. Tarantola and F. Campolongo (2005) Sensitivity Analysis for Chemical Models, Chemical Reviews, 105(7) pp 2811 – 2828.

Munda G., Nardo M., Saisana M., 2009, Measuring uncertainties in composite indicators of sustainability. Int. J. Environmental Technology and Management (forthcoming).

Selected References

Journal articles


Munda G., Saisana M., 2009, Methodological Considerations on Regional Sustainability Assessment based on Multicriteria and Sensitivity Analysis. Regional Studies (forthcoming).

Tarantola S., Nardo M., Saisana M., Gatelli D., 2006, A new estimator for sensitivity analysis of model output: An application to the e-business readiness composite indicator. Reliability Engineering & System Safety 91(10-11), 1135-1141.

Selected References

Journal articles


Saisana, M., 2008, The 2007 Composite Learning Index: Robustness Issues and Critical Assessment, EUR Report 23274 EN, European Commission, JRC-IPSC, Italy.

Saisana M., D’Hombres B., 2008, Higher Education Rankings: Robustness Issues and Critical Assessment, EUR Report 23487 EN, European Commission, JRC-IPSC, Italy.

Saisana M., Munda G., 2008, Knowledge Economy: measures and drivers, EUR Report 23486 EN, European Commission, JRC-IPSC.

Saisana M., Saltelli, A., 2008, Sensitivity Analysis for the 2008 Environmental Performance Index, EUR Report 23485 EN, European Commission, JRC-IPSC.

Selected References

EUR reports


Munda G. and Nardo M. (2005), Constructing Consistent Composite Indicators: the Issue of Weights, EUR 21834 EN, Joint Research Centre, Ispra.

Nardo M., Tarantola S., Saltelli A., Andropoulos C., Buescher R. , Karageorgos G. , Latvala A. and Noel F. (2004), The e-business readiness composite indicator for 2003: a pilot study, EUR 21294.

Saisana M., Tarantola S., 2002, State-of-the-art Report on Current Methodologies and Practices for Composite Indicator Development, EUR Report 20408 EN, European Commission, JRC-IPSC, Italy.

Selected References

EUR reports

NTTS Conference, Brussels, 18-20 February 20091 OECD/JRC Handbook on constructing composite...

Documents

Transcript of NTTS Conference, Brussels, 18-20 February 20091 OECD/JRC Handbook on constructing composite...