1
Composite indicators
Andrea Saltelli [email protected]
OECD WORKSHOP ON INDICATORS OF REGULATORY MANAGEMENT SYSTEMS
EXPERT MEETING
2-3 April 2009
BERR CONFERENCE CENTRE LONDON, UNITED KINGDOM
2
Course based on:
Joint OECD-JRC
handbook.
•5 years of preparation,
•2 rounds of consultation
with OECD high level
statistical committee,
•finally endorsed March
2008 with one abstention
3
How many composites are around? Searching “composite indicators” on Scholar Google:
Scholar Google
October 2005 992
June 2006 1,440
May 2007
1,900
October 2008 3,030
February 2009 3,300
4
• Analytic work based on CI‟s
From Jean Philippe Cotis and Romain Duval, Competitiveness, Innovation and economic growth Istanbul 2007, Conference ‘Measuring and Fostering Progress’
See http://www.oecd.org/dataoecd/20/52/38859413.pdf
5
See www.education-economics.org
From Ludger Wößmann Contribution of Education and Training to Innovation and Growth Symposium on the Future Perspectives of European Education and Training for Growth, Jobs and Social Cohesion, Brussels, 2007
6
7
• Composite indicators and the media
8
Use: Rankings from the Economist web
edition, March 30, 2009
Our rankings Business …
Economics Big Mac index: A light-hearted guide to whether
currencies are at their “correct” level
All Economics rankings »
Education …
Living…
Politics… Global peace index: The world's most and least
peaceful countries
All Politics rankings »
Risk…
Technology…
9
Economically literate press does have appetite for statistic based narratives.
10
Jean Pisani-Ferry and Andre‟ Sapir argument on league tables
11
“[…] civil societies learn from the experience of others. Such policy learning can be enhanced by initiatives that facilitate cross country comparison and benchmarking. A telling example in this respect is […] PISA.”
12
“Transparency benefits the democratic process as it empowers national electorates to review the performance of their own governments and it helps focus the debate on key areas of underperformance. The use of league tables facilitates this process.”
13
Tito Boeri and the World
Bank’s Doing Business
report, speaking at the
Italian Rai3, March 16
2009
14
A ten-pillars snapshot of
doing business in
different countries
181 economies, from
Afghanistan
to Zimbabwe
15
16
65
53 83
75
58
84
53
128
60
156
27
“Italy lost six position
in a year”, says Boeri,
then he goes into the
details. Paying taxes
and enforcing contracts
…
Ranks
for Italy
17
International university rankings: How reliable are these rankings?
THES World University Rankings
Jiao Tong ranking of World Universities
What can we do
to improve our
position on the
international
scene?
18
• Can good practices help ? Ten steps to build Composite Indicators
19
Ideally, a composite indicator should be based on:
-a solid theoretical framework,
-underlying data of good quality,
-a defensible construct.
20
A good technical preparation for a CI can make it more robust (to uncertainties in data, weights,…) more resilient (remain relevant over time), more defensible (in dialogue with stakeholders…)
21
Step 1. Developing a solid theoretical framework
What is badly defined is likely to be badly measured …
Examples …
The challenges are:
• To integrate a broad set of (probably conflicting) points of view while keeping within a manageable construct
• To have a community of peers (individuals, regions, countries) willing to accept the theoretical framework.
22
‘’Doing Business provides a quantitative
measure of regulations for starting a business,
dealing with construction permits, employing
workers, registering property, getting credit,
protecting investors, paying taxes, trading
across borders, enforcing contracts and closing
a business […]’’.
Examples …
23
‘’Doing Business does not measure all aspects of the
business environment […]. It does not, for example,
measure security, macroeconomic stability,
corruption, the labor skills of the population, the
underlying strength of institutions or the quality of
infrastructure […]’’.
Examples …
24
The Canadian Council on Learning has developed the Composite Learning Index
(http://www.ccl-cca.ca/). The framework:
Learning to Know Learning to Do Learning to Live Together Learning to Be[*] **+ Pillars from Jacques Delors’ Task Force: UNESCO's International Commission on Education for the Twenty-first Century.
Jacques Delors
25
I. Learning to Know involves developing the foundation
of skills and knowledge needed to function in the world. This includes literacy, numeracy, general knowledge and critical thinking.
II. Learning to Do refers to the acquisition of applied
skills. It can encompass technical and hands-on skills and knowledge, and is closely tied to occupational success.
26
III. Learning to Live Together involves developing
values of respect and concern for others, fostering social and inter-personal skills, and an appreciation of the diversity of Canadians. This area of learning contributes to a cohesive society.
IV. Learning to Be refers to the learning that helps develop
the whole person ― mind, body and spirit. This aspect concerns personal discovery, self-knowledge, creativity and achieving a healthy balance in life (~Maslow’s top).
27
Step 1. Developing a solid theoretical framework
After Step 1. the developer should have… • A clear understanding and definition of the multidimensional phenomenon to be measured. • A nested structure of the various sub-groups of the phenomenon. • A list of selection criteria for the underlying indicators, e.g., input, process, output.
28
Step 2. Selecting indicators
A composite indicator is above all the sum of its parts…
Excerpt: The strength of a composite indicator can largely depend on the quality of the underlying data. *…+. The theoretical framework should guide the choice of the underlying indicators. The selection process can be quite subjective and therefore should involve stakeholders. Moreover, depending on availability of data, certain indicators cannot be used and proxies need to be considered.
29
Step 3. Multivariate analysis
Analysing the underlying structure of the data is an art …
a1 1
a2 0.85 1
a3 0.83 0.87 1
a4 0.77 0.81 0.89 1
a5 0.56 0.55 0.72 0.72 1
a6 0.64 0.61 0.66 0.73 0.56 1
b1 0.50 0.72 0.69 0.63 0.40 0.54 1
b2 0.61 0.80 0.75 0.74 0.62 0.54 0.86 1
b3 0.43 0.39 0.54 0.54 0.39 0.48 0.10 0.29 1
b4 0.35 0.26 0.41 0.30 0.31 0.37 0.16 0.24 0.75 1
b5 0.70 0.49 0.48 0.45 0.60 0.31 0.09 0.38 0.21 0.13 1
b6 0.55 0.75 0.78 0.77 0.59 0.57 0.77 0.83 0.44 0.24 0.29
a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5
Component Initial Eigenvalues
Total % of variance Cumulative %
1 7.242 60.35 60.35
2 1.523 12.69 73.04
3 1.178 9.82 82.86
4 0.554 4.61 87.47
5 0.512 4.26 91.73
6 0.385 3.20 94.94
7 0.242 2.01 96.95
8 0.131 1.09 98.04
9 0.098 0.82 98.04
10 0.064 0.54 99.40
11 0.043 0.36 99.76
12 0.029 0.24 100.00
0
1
2
3
4
5
6
7
8
0 2 4 6 8 10 12 14
Component number
Eig
enva
lues
30
Step 4. Imputation of missing data.
The idea of imputation could be both seductive and dangerous … Almost all datasets contain missing data.
Country a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 b6
AT 0.97 0.78 0.84 0.53 0.72 0.34 0.42 0.18 0.49 0.19 0.85 0.02
BE 0.97 0.72 0.82 0.59 0.86 0.45 0.43 0.18 0.49 0.18 0.87 0.02
BG 0.75 0.31 0.54 0.20 0.61 0.16 0.03 0.01 0.17 0.05 0.44 0.01
CY 0.88 0.47 0.67 0.42 0.69 0.20 0.12 0.07 0.44 0.06 0.54 0.01
CZ 0.95 0.71 0.77 0.40 0.77 0.31 0.22 0.09 0.31 0.08 0.87 0.01
DE 0.95 0.78 0.90 0.61 0.80 0.47 0.52 0.24 0.52 0.19 0.76 0.05
DK 0.97 0.84 0.93 0.80 0.39 0.36 0.33 0.62 0.21 0.93 0.04
EE 0.94 0.62 0.74 0.39 0.78 0.26 0.13 0.07 0.24 0.11 0.93 0.03
EL 0.93 0.60 0.52 0.37 0.72 0.35 0.08 0.06 0.71 0.01
ES 0.94 0.49 0.76 0.49 0.90 0.27 0.16 0.08 0.30 0.11 0.81 0.02
EU27 0.93 0.63 0.80 0.49 0.77 0.34 0.29 0.15 0.41 0.14 0.77 0.03
FI 0.99 0.81 0.97 0.70 0.91 0.47 0.19 0.15 0.53 0.15 0.91 0.03
FR 0.96 0.57 0.82 0.52 0.89 0.35 0.43 0.11 0.76 0.02
31
Three common approaches to deal with missing data: •case deletion (removes either country or indicator from the analysis) •single imputation (e.g. Mean/Median substitution, Regression, etc.)
•multiple imputation (e.g. Markov Chain Monte Carlo algorithms).
Trying to minimize bias and to keep „expensive to collect‟ data that would otherwise be discarded.
32
Step 5. Normalisation of data
Avoid adding up apples and oranges …
Ranking
Standardization
Re-scaling
Distance to reference country
Categorical scales
Cyclical indicators
Balance of opinions
33
Step 6. Weighting and aggregation
The relative importance of the indicators can become the substance of a negotiation …
Weights based on statistical models
Principal component/Factor analysis
Data envelopment analysis
Regression approach
Unobserved components models
34
Step 6. Weighting and aggregation
The relative importance of the indicators can become the substance of a negotiation …
Weights based on opinions: participatory methods
Budget allocation
Public opinion
Analytic hierarchy process
Conjoint analysis
35
Space of alternatives
Including/ excluding variables
Normalisation
...
Imputation Weights
Aggregation
Performance index
Italy Greece Spain
10
20
30
40
50
60
Step 7. Robustness and sensitivity
Uncertainty analysis can be used to assess the robustness of composite indicators …
36
Step 8. Links to other variables
Composite indicators can be linked to other variables and measures
• Comparing effectively complex dimensions:
Figure 1. Relationship between the Composite Learning Index and the Economic
and Social Well-Being Index in Canada.
y = 0.7691x + 20.249
R2 = 0.6979
40
50
60
70
80
90
100
40 50 60 70 80 90 100
Composite Learning Index
Ec
on
om
ic a
nd
So
cia
l W
ell
-be
ing
37
Step 9. Back to the details De-constructing composite indicators can help extend the analysis …
Ad hoc clustering
38
The four-quadrant model of the Composite Learning Index (Canadian Council on Learning)
Step 10. Presentation and dissemination
A well-designed graph can speak louder than words …
39
• In house CI development
40
The Alcohol Policy Index (New York Medical College)
Concept: (WHO report)
Results
Policy message Sensitivity analysis
Published in
PLoSMedicine
41
Figure 1. Relationship between the Composite Learning Index and the Economic
and Social Well-Being Index in Canada.
y = 0.7691x + 20.249
R2 = 0.6979
40
50
60
70
80
90
100
40 50 60 70 80 90 100
Composite Learning Index
Ec
on
om
ic a
nd
So
cia
l W
ell
-be
ing
The Composite Learning Index (Canadian Council on Learning)
Results
Policy message Sensitivity analysis
CCL Report
Framework
Scenario
Pillar Structure Normalisation Weighting Aggregation
CLI Preserved z-scores FA within pillar, Regression
weights to Factors, FA pillars,
Regression weights to pillars
Linear
S1 Preserved z-scores FA within pillar, FA pillars Linear
S2 Preserved Min-max FA within pillar, FA pillars Linear S3 Not preserved z-scores FA all indicators Linear S4 Not preserved Min-max FA all indicators Linear S5 Preserved z-scores FA within pillar, EW pillars Linear S6 Preserved Min-max FA within pillar, EW pillars Linear S7 Not preserved z-scores EW all indicators Linear S8 Not preserved Min-max EW all indicators Linear S9 Preserved z-scores EW within pillar, EW pillars Linear S10 Preserved Min-max EW within pillar, EW pillars Linear S11 Preserved z-scores FA within pillar, FA pillars Geometric
S12 Preserved Min-max FA within pillar, FA pillars Geometric
S13 Not preserved z-scores FA all indicators Geometric
S14 Not preserved Min-max FA all indicators Geometric
S15 Preserved z-scores FA within pillar, EW pillars Geometric
S16 Preserved Min-max FA within pillar, EW pillars Geometric
S17 Not preserved z-scores EW all indicators Geometric
S18 Not preserved Min-max EW all indicators Geometric
S19 Preserved z-scores EW within pillar, EW pillars Geometric
S20 Preserved Min-max EW within pillar, EW pillars Geometric
S21 Preserved Raw data FA within pillar, FA pillars Multi-criteria
S22 Not preserved Raw data FA all indicators Multi-criteria
S23 Preserved Raw data FA within pillar, EW pillars Multi-criteria
S24 Not preserved Raw data EW all indicators Multi-criteria
S25 Preserved Raw data EW within pillar, EW pillars Multi-criteria
42
Environmental Sustainability Index ESI
and Environmental Performance Index EPI.
http://epi.yale.edu/Home
43
Uncertainty & sensitivity analysis
http://epi.yale.edu/
Home
Ran
k 1-1
0
Ran
k 11-2
0
Ran
k 21-3
0
Ran
k 31-4
0
Ran
k 41-5
0
Ran
k 51-6
0
Ran
k 61-7
0
Ran
k 71-8
0
Ran
k 81-9
0
Ran
k 91-1
00
Ran
k 101-1
10
Ran
k 111-1
20
Ran
k 121-1
30
Ran
k 131-1
40
Ran
k 141-1
49
Ran
k 1-1
0
Ran
k 11-2
0
Ran
k 21-3
0
Ran
k 31-4
0
Ran
k 41-5
0
Ran
k 51-6
0
Ran
k 61-7
0
Ran
k 71-8
0
Ran
k 81-9
0
Ran
k 91-1
00
Ran
k 101-1
10
Ran
k 111-1
20
Ran
k 121-1
30
Ran
k 131-1
40
Ran
k 141-1
49
Switzerland 31 30 20 11 6 Viet Nam 5 10 18 29 20 8Sweden 63 25 10 Nicaragua 8 21 28 14 10 14Norway 55 31 6 Saudi Arabia 11 13 23 16 10 11Finland 81 16 Tajikistan 9 6 14 19 18 19 11Costa Rica 81 16 Azerbaijan 10 23 9 24 26 5Austria 15 19 16 21 18 Nepal 5 9 8 15 28 14 13 6New Zealand 98 Morocco 14 20 15 13 15 16Latvia 25 39 26 6 Romania 14 10 33 16 11 5Colombia 74 18 5 Belize 9 29 16 13 14 6France 15 26 30 14 13 Turkmenistan 5 6 14 8 19 25 11 9Iceland 11 15 5 14 15 9 10 13 Ghana 11 14 16 10 10 13 9 8 5Canada 58 33 9 Moldova 8 10 13 30 31Germany 14 40 21 20 Namibia 16 16 25 18 16United Kingdom 11 44 29 11 Trinidad & Tobago 6 20 11 23 8 13 8 5Slovenia 9 18 25 23 8 10 Lebanon 5 15 13 13 5 5 6 20 8 11Lithuania 16 20 14 9 8 9 6 9 Oman 10 25 18 24 5 10Slovakia 15 21 14 25 5 6 8 Fiji 6 15 13 19 14 20 5Portugal 20 46 16 11 5 Congo 9 23 18 13 13 13 9Estonia 56 34 Kyrgyzstan 5 6 15 13 30 15 9Croatia 16 19 23 10 6 9 5 5 5 Zimbabwe 21 11 15 10 15 13 6Japan 6 38 35 14 5 Kenya 18 15 11 11 11 11 6 8 6Ecuador 63 26 5 South Africa 14 16 20 11 16 11Hungary 6 6 13 16 20 6 10 9 Botswana 5 13 18 18 19 16 6Italy 6 28 24 16 13 5 Syria 16 11 21 13 30 6Denmark 8 9 6 15 13 14 8 6 11 Mongolia 5 13 9 25 18 15 10Malaysia 31 48 15 5 Laos 10 8 9 10 6 10 9 19 11Albania 9 11 6 13 10 16 5 6 9 5 Indonesia 9 10 11 23 19 14 5 6Russia 9 33 43 9 Côte d'Ivoire 10 13 15 20 8 11 9 6Chile 16 46 25 8 Myanmar 6 5 16 24 26 15 5Spain 5 30 18 19 14 11 China 9 13 9 25 19 13 5Luxembourg 9 15 16 20 26 5 5 Uzbekistan 14 16 29 29 11Panama 73 20 Kazakhstan 5 15 16 36 24Dominican Rep. 18 54 21 6 Guyana 6 10 19 15 20 20 5Ireland 5 16 13 15 13 13 9 5 Papua New Guinea 6 10 14 9 11 20 20 8Brazil 5 20 29 24 11 Bolivia 8 10 23 13 20 11Uruguay 11 15 9 8 9 10 9 14 Kuwait 9 5 15 28 41Georgia 8 8 19 15 16 10 13 5 United Arab Em. 10 9 36 19 11 11Argentina 10 23 28 24 11 Tanzania 11 13 16 9 11 11 8 6 6United States 5 23 19 24 13 8 Cameroon 6 10 6 13 23 23 15Taiwan 20 13 19 16 10 13 Senegal 5 6 9 16 30 24 6Cuba 5 24 29 19 13 5 Togo 8 6 18 18 18 18 10Poland 5 11 20 35 15 Uganda 8 5 5 6 11 20 21 11 10Belarus 11 10 10 18 16 16 13 Swaziland 6 16 31 24 15Greece 8 18 14 19 15 5 10 6 Haiti 10 21 23 30 10Venezuela 5 11 36 25 18 5 India 11 15 31 25 13Australia 30 30 14 10 9 Malawi 9 13 13 14 11 15 9 6 8Mexico 11 15 34 28 6 Eritrea 6 13 16 16 25 18Bosnia & Herzegovina 5 10 11 24 9 6 8 14 6 Ethiopia 6 8 9 8 9 25 26 5Israel 5 31 19 19 13 5 6 Pakistan 23 9 26 18 18Sri Lanka 19 36 16 16 10 Bangladesh 9 18 24 48South Korea 6 14 14 19 9 8 13 8 Nigeria 6 5 13 15 24 23 6 5Cyprus 10 9 25 14 28 6 Benin 10 11 10 14 13 9 11 13Thailand 8 30 35 11 11 Central Afr. Rep. 13 14 16 38 13Jamaica 8 15 24 11 11 9 10 5 Sudan 10 34 46 6Netherlands 9 11 14 10 21 9 11 9 Zambia 10 10 14 9 21 21 11Bulgaria 5 19 25 15 8 10 6 Rwanda 6 11 18 11 18 5 13 6 9Belgium 13 6 11 6 6 16 10 13 9 Burundi 9 8 15 9 18 29 11Mauritius 6 9 19 18 8 16 15 Madagascar 8 13 16 20 21 15Tunisia 5 10 10 10 14 19 18 9 Mozambique 6 6 9 11 14 18 21 9Peru 15 30 18 30 Iraq 11 26 60Philippines 6 13 26 21 16 9 5 Cambodia 8 15 11 31 28Armenia 6 13 19 8 16 18 8 6 Solomon Islands 16 81Paraguay 11 18 20 18 9 8 5 6 Guinea 13 14 23 36 6Gabon 6 35 28 16 5 6 Djibouti 8 18 35 39El Salvador 5 6 13 16 9 10 9 8 9 8 5 Guinea-Bissau 15 14 28 19 15 5Algeria 5 5 15 26 24 11 6 5 Yemen 6 29 63Iran 11 23 26 18 16 Dem. Rep. Congo 13 29 26 23Czech Rep. 9 8 15 11 13 19 15 10 Chad 8 16 33 40Guatemala 10 16 23 26 14 8 Burkina Faso 9 6 18 43 25Jordan 8 14 24 20 6 14 8 Mali 5 18 36 41Egypt 19 21 24 13 10 6 Mauritania 9 25 40 24Turkey 18 15 18 16 9 6 6 Sierra Leone 11 18 70Honduras 9 28 20 15 13 8 5 Angola 19 79Fyrom 5 5 15 10 18 21 13 6 5 Niger 6 19 73Ukraine 8 15 6 23 11 10 13 10
Legend:Probability between 5 and 15% Source: JRC calculationsProbability between 15 and 30% Notes:Probability between 30 and 50% 1. Numbers express probabilities for the country rankProbability greater than 50% 2. Countries listed based on the 2008 EPI scores from highest to lowest (left to right)Probability lower than 5% is not shown
2008 EPI: Uncertainty analysis for 149 countries
44
Results
Sensitivity analysis
The Knowledge Economy Index
(FP6 - DG RTD)
0
5
10
15
20
25
30
Sw
eden
Denm
ar
Luxem
b
Fin
land
US
A
Japan
Unite
d
Neth
erl
Irela
nd
Austr
ia
Belg
ium
Fra
nce
EU
15
EU
27
Germ
an
Slo
veni
Esto
nia
Malta
Cypru
s
Spain
Czech
Latv
ia
Italy
Gre
ece
Lith
oua
Hungar
Port
uga
Slo
vaki
Pola
nd
20
04
Kn
ow
led
ge
Ec
on
om
y In
de
x (
ran
k)
45
What are the levels of Civic Competence of young people in Europe?
84 indicator index
IEA data on:
youth knowledge,
skills,
attitudes,
values and beliefs
towards
citizenship
Results: Newer democracies perform better on Attitudes towards participation and
Citizenship values.
Older and more stable democracies perform better on Social Justice and Cognitive
tasks on civic knowledge and skills.
46
COMPARATIVE ANALYSIS OF
INNOVATION PERFORMANCE of THE
EUROPEAN COUNTRIES
THE 2007 SUMMARY INNOVATION INDEX (SII)
Data from Eurostat Science and
Technology Indicators and Community
Innovation Survey (CIS)
Country scores, analysis of trends, and
variability across countries and possible
underlying reasons
Nordic countries, Germany and UK are
ahead of the US.
0.0
8
0.2
3
0.2
3
0.2
4
0.2
5
0.2
5
0.2
6
0.2
6
0.2
7
0.2
9
0.3
1
0.3
3
0.3
3
0.3
5
0.3
6
0.3
6
0.3
6
0.3
7
0.4
4
0.4
7
0.4
7
0.4
8
0.4
8
0.4
9
0.5
0 0.5
3
0.5
5
0.5
7
0.5
9
0.6
0
0.6
1
0.6
2
0.6
4
0.6
7
0.7
3
0.1
8
0.1
9
0.4
5
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
TR RO LV BGHR PL SK PT EL HU LT M TES CY IT SI CZ NOAU EECA EU BE FR NL AT IE IS LU US UK DEJP DK IL FI CH SE
AU
IL
SKEL LT
PT
LU
ATIE
FR
NL
BE
UKUS
DE
JP
CH
DK
HU
CYES
MT
EE
ITCZSI
IS
CA
HR
PL
LV
RO
BG
SE
FI
NO
TR
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0
Average grow th rate of SII (2003-2007)
2007 S
um
mary
Innovatio
n Index
Sw eden Innovation leaders Innovation follow ersModerate innovators Catching-up countries Turkey
Dotted lines show EU performance.
47
Other sources:
Saisana M., Saltelli A., Tarantola S. (2005) Uncertainty and Sensitivity analysis techniques as
tools for the quality assessment of composite indicators, Journal of the Royal Statistical Society,
A 168(2), 307-323.
Saltelli, A., 2007, Composite
indicators between analysis and advocacy Social Indicators Research, 81 , 65-77.
Brand DA, Saisana M, Rynn LA, Pennoni F, Lowenfels AB, 2007, Comparative Analysis of Alcohol Control Policies in 30 Countries, PLoS Medicine, 4(4), 752-
759.
Top Related