Measuring Sustainable Development and its trade-offs · of Mauritius love to prove famous people...

FEEM International Conference

“Methodologies and Indicators for

Green Growth Measurement”

12 November 2013, Milan

1





Measuring Sustainable

Development and its

trade-offs

Michaela Saisana

[email protected]

European Commission

Joint Research Centre

Econometrics and Applied Statistics Unit





2





These two reports, among many others, urge statisticians to capture

what people do live by:

• Measuring what matters

• Man does not live by GDP alone





3





(p.7): “the role of statistical indicators

has increased over the last two decades”

(i) more literacy,

(ii) more complexity,

(iii) more information society





4





16/10/2008

THE 1.3m people of Mauritius love to prove famous people wrong. On independence from

Britain in 1968, pundits such as a Nobel prize-winning economist, James Meade, and a novelist, V.S. Naipaul, did not give much of a chance to this tiny, isolated Indian Ocean island 1,800km (1,100 miles) off the coast of east Africa. Its people depended on a sugar economy and enjoyed a GDP per person of only $200. Yet the

island now boasts a GDP per person of $7,000, and very few of its people live in absolute

poverty. It once again ranks first in the latest annual Mo Ibrahim index, which measures governance in Africa. And it bagged 24th spot in the World Bank’s global ranking for ease of doing business—the only

African country in the top 30, ahead of countries such as Germany and France. How does it pull it off?

Statistics' best known face (to general public & media)





5





Definition: A composite indicator is formed when individual indicators are

compiled into a single index, on the basis of an underlying model of the

multi-dimensional concept that is being measured.

(OECD Glossary of statistics)

http://www.oecd.org/home/0,3305,en_2649_201185_1_1_1_1_1,00.html





6





29

11800

7780

3420

1550

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

1940 1960 1980 2000 2020

Scholar Google hits on

"composite indicators"

Almost ten-fold increase since 2000

Guess how many

contain the word

“sustainability”?

36%

(it was 28% in 2012)





7





Plenty of indicator lists, databases & composite indicators on sustainability





8





Indices of

Societal Progress Developer(s) Main Dimensions Number of indicators

Number of

units

assessed

Human

Development

Index (HDI, 2011)

United Nations

Development

Programme

1) Life expectancy index, 2) Education index, 3) Income index 4 187 countries

Human

Sustainable

Development

Index (HSDI,

2010)

Chuluun Togtokh

& Owen Gaffney

and; UNEP

Same as HDI but with GHG emissions per capita 5 163 countries

Environmental

Sustainability

Index (ESI, 2005)

Yale University,

Columbia

University

1) Environmental Systems, 2) Reducing Stresses, 3) Reducing

Human Vulnerability, 4) Social and Institutional Capacity and 5)

Global Stewardship.

21 146 countries

Environmental

Performance

Index (EPI, 2012)

Yale University,

Columbia

University

1) Environmental burden of disease, 2) water (effects on human

health), 3) air pollution (effects on human health), 4) air pollution

(ecosystem effects), 5) water resources (ecosystem effects), 6)

biodiversity and habitat, 7) forestry, 8)fisheries, 9) agriculture, 10)

climate change.

22 132 countries

…

FEEM

Sustainability

Index (FEEM SI

methodology

report 2011)

Fondo Eni

Enricco Mattei

(2009)

1) Economic, 2) Environment, 3) Social 18 40 countries

Source: Saisana & Philippas, 2012





9





2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

First JRC report:

State of the art on

CIs (EUR 20408)

2002

Launch -JRC site:

Server, 10 steps

(>1000 links, 1st hit)

2003

First JRC Seminar:

Ispra, 12 May 2003

2005

First External request:

Env. Sustainability

Index (EUR 21807)

First Paper:

UASA for CIs

(JRSSA, 168:307-323)

2012-2013

1. Human Capital (WEF)

2. Country Resilience Capacity Rating (WEF)

3. Environmental Performance Index (Yale &

Columbia)

4. Global Innovation Index (WIPO, INSEAD)

5. Drug Consequences Index (ONDCP)

6. Sustainable Society Index (SSF, NL)

7. Corruption Perceptions Index (Trans. Int.)

8. Rule of Law Index (World Justice Project)

9. Global Focus Model (UN OCHA)

10. Global Talent Competitiveness Index (INSEAD)

1. Environmental Pressures Index (ENV)

2. Small Business Act Principles (ENTR)

3. Regional Human Development Index (REGIO)

4. Regional Human Poverty Index (REGIO)

5. Research Excellence (RTD)

6. Youth Civic Competencies Index (EaC)

7. Europe 2020 Innovation Indicator (Commissioner

M.Geoghegan-Quinn)





10





Step 10. Presentation & dissemination

Step 9. Association with other variables

Step 8. Back to the indicators

Step 7. Robustness & sensitivity

Step 6. Weighting & aggregation

Step 5. Normalisation of data

Step 4. Multivariate analysis

Step 3. Data treatment (missing, outliers)

Step 2. Selection of indicators

Step 1. Development of a conceptual framework

Decalogue Consecutive

steps but

with an

iterative

nature

2 rounds of consultation with OECD high level statistical committee

Finally endorsed in March 2008





11









12





• Suggestions for composite indicators on

sustainability (and not only…)





13





Score

Economy 5

Environment 9

Social 2

How do we combine these

dimensions into a single number?

Arithmetic Average

n

i

ii wxy1

Geometric Average

n

i

w

iixy

1

1/Carefully select

aggregation formula





14





ECO ENV SOC Arithmetic

Average Geometric

Average Comparison

with B

A 7 7 7 7.0 7.0

B 5 9 2 5.3 4.5

B1 10 9 2 7.0 5.6 26.0%

B2 5 9 7 7.0 6.8 51.8%

Advantages of geometric versus arithmetic mean for a sustainability index

1) implies only partial compensability, i.e. poor performance in one dimension cannot be

fully compensated by good performance in another,

2) rewards balance by penalizing uneven performance between dimensions,

3) encourages improvements in the weak dimensions, i.e. the lower the performance in a

particular dimension, the more urgent it becomes to improve in that dimension.

1/Carefully select

aggregation formula





15





43

32

25

29

38

33

38

34

28

39

26

36

39

32

2931

33

36 35

30

34

38

33

29

33 33 33 3335

32

0

10

20

30

40

50

60

70

Co

nsu

mp

tio

n

Acce

ss S

tab

ility

Nu

tritio

n Q

ua

lity

Qu

alit

y

Ava

ilab

ility

Acce

ss

Sta

tus

Acce

ss

Qu

alit

y

To

ilet

Fa

cili

tie

s

Wa

ste

Ma

na

ge

me

nt

Pra

ctice

s

Qu

alit

y

Fa

cili

tie

s

En

erg

y

Qu

alit

y

Ava

ilab

ility

Acce

ss

Te

nu

re

Qu

alit

y

Inp

uts

Skill

s

Se

rvic

es

Asse

ts

Exp

osu

re

Co

pin

g a

bili

ty

Re

co

ve

ry a

bili

ty

Fo

od

Ed

uca

tio

n

He

alth

ca

re

Education Farm

Assets

Exposure &

Resilience to

Shocks

Gender

Equality

In 4 dimensions of poverty,

the average expert weights are

similar to equal weighting

Weighting based on budget

allocation - 42 experts

2/Carefully apply

expert-based weighting





16





USING PAIRWISE COMPARISONS, THE

RELATIVE IMPORTANCE

OF ONE CRITERION OVER ANOTHER

CAN BE EXPRESSED

1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME

1 2 3 4 5 6 7 8 9

Patents vs. x Royalties x

x Patents vs. Internet x

x Patents vs. Technology exports x

x Patents vs. Telephones x

x Patents vs. Electricity x

Patents vs. x Schooling years x

Patents vs. x University Students x

x Royalties vs. Internet x

Royalties vs. x Technology exports x

x Royalties vs. Telephones x

x Royalties vs. Electricity x

Royalties vs. x Schooling years x

Royalties vs. x University Students x

Internet vs. x Technology exports x

x Internet vs. Telephones x

x Internet vs. Electricity x

Internet vs. x Schooling years x

Internet vs. x University Students x

x Technology exports vs. Telephones x

x Technology exports vs. Electricity x

Technology exports vs. x Schooling years x

Technology exports vs. x University Students x

x Telephones vs. Electricity x

Telephones vs. x Schooling years x

Telephones vs. x University Students x

Electricity vs. x Schooling years x

Electricity vs. x University Students x

x Schooling years vs. University Students x

Which Indicator Do You Feel Is More Important? To What Degree?

Questionnaire

Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St.

Patents 1 1/3 5 4 3 9 1/6 1/8

Royalties 3 1 3 1/4 5 9 1/3 1/4

Internet 1/5 1/3 1 1/6 2 2 1/7 1/6

Tech.Exports 1/4 4 6 1 5 9 1/4 1/5

Telephones 1/3 1/5 1/2 1/5 1 7 1/9 1/9

Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9

Schooling 6 3 7 4 9 9 1 2

University St. 8 4 6 5 9 9 1/2 1

solve for the Eigenvector

Patents 0.109

Royalties 0.103

Internet hosts 0.029

Tech exports 0.117

Telephones 0.030

Electricity 0.014

Schooling 0.301

University st. 0.297

Weights

Inconsistency

17.4 %

Weights based on Analytic Hierarchy Process

3/Check for inconsistency

in expert-based weighting





17





4/Check for statistical

coherence

Within a dimension:

• Avoid negative

correlations

• Avoid random

correlations

•Match effective

weights (e.g., R2 Pearson

correlation ratio) to

nominal weights





18





EPI relatively balanced in the two

objectives

But there are several silent indicators


coherence





19






coherence

Do not take the average

of negatively correlated

dimensions.

Why?

…

Next





20





Country X1 X2 I1 I2 Y Rank

H 2 000 500 100 100 100 1

A 160 435 8 87 47.5 2

B 400 370 20 74 47.0 3

C 640 305 32 61 46.5 4

D 880 240 44 48 46.0 5

E 1 120 175 56 35 45.5 6

F 1 360 110 68 22 45.0 7

G 1 600 45 80 9 44.5 8

21 5.5. IIY


coherence





21





Country X1 X2 I1 I2 Y Rank

H 2 000 700 100 100 100 1

A 160 435 8 62.14 35.07 8

B 400 370 20 52.86 36.43 7

C 640 305 32 43.57 37.79 6

D 880 240 44 34.29 39.14 5

E 1 120 175 56 25 40.5 4

F 1 360 110 68 15.71 41.86 3

G 1 600 45 80 6.43 43.21 2

21 5.5. IIY

Only the best performer (H) improves BUT the

ranking gets completely reversed and country A

is last as opposed to 2nd !

Rank

1

2

3

4

5

6

7

8


coherence





22





0

100

200

300

400

500

600

0 500 1000 1500 2000 2500

X1

X2

r (X1,X2) = -0.26

How did that happen?


coherence





23





How to shake coupled

stairs

How coupled stairs are shaken

in most of available literature

5/Do uncertainty analysis: which countries are most

volatile?





24





UA results

Look for countries:

• outside the

confidence interval

• with wide intervals

5/Do uncertainty analysis: which countries are most

volatile?





25





6/Do sensitivity analysis: which assumptions have the

highest impact?

• Use uncertainty and

sensitivity analysis during

the process of building an

index, not only to criticize

an existing one!

• Focus discussions on

assumptions that matter

the most.





26





7/Deconstruct an index

At country level

At indicator level





27





Framework

(WHO report)

Policy message

Sensitivity analysis

The Alcohol Policy Index

(2007, PLoS Medicine, 4(4):752-759)

8/Link to policy





28





• Powerful evidence based narratives supported

by good statistical measures and good analytic

work are a possibility which should not be left

untried. We need relevant and sound …





29





(Composite) Indicators

More reading at

http://composite-indicators.jrc.ec.europa.eu

Measuring Sustainable Development and its trade-offs · of Mauritius love to prove famous people...

Documents

Transcript of Measuring Sustainable Development and its trade-offs · of Mauritius love to prove famous people...